This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Tool that detects data types

Hi guys

I am looking for a tool that is able to automatically detect and validate type of a file.

More specifically I am interested for detecting between SAM, BAM, FASTA, FASTQ (if possible discriminating between these types), BED, BED12, BED15, GFF, GFF2, GFF3.

The detection should be performed without just looking at the extension of the file....

Is there an easy way to accomplish something like this?

Thank you in advance

datatype rna-seq
• 4,483 views • link
updated by Ram Ram • written by fitziano • fitziano

1 answer

OK from what I got from SE; you can create and compile a magic file the following way.

Create a file 'bioinfo' containing the pattern for your format. Example: from the specification a BAM file starts at 1st byte (position=0) with 4 bytes: BAM\1:

0	string	@HD\tVN:1.0\ SO:coordinate	SAM file v1.0 sorted on coordinates
0	string	@HD\tVN:1.0\ SO:coordinate	SAM file v1.0 sorted on coordinates
0	string	BAM\1	BAM file v1.0
0	string	CRAM\2\1	CRAM 2.1 file
0	string	BAI\1	BAM index file v1.0
0	regex	[\r\n]^>.*\n[ATGCatgc]*\n	Fasta DNA sequence
0	string	\#\#fileformat=VCFv4\.1 VCF format 4.1
0	string	BCF\4	BCF file v1.0
0	string	BCF\2\1	BCF file v2.0
0	string	TBI\1	Tabix index file v1.0

compile the magic file 'bioinfo.mgc':

file -C -m bioinfo

use this magic file :

file -z -m bioinfo.mgc ex1.sam.gz
ex1.sam.gz: SAM file v1.0 sorted on coordinates (data)
file -z -m bioinfo.mgc file.bam file.bam: BAM file v1.0 (data)
file -z -m bioinfo.mgc file.fasta file.fasta: Fasta DNA sequence
file -z -m bioinfo.mgc file.vcf
file.vcf: VCF format 4.1
file -z -m bioinfo.mgc file.gz.tbi
file.gz.tbi: Tabix index file v1.0 (data)

UPDATE: I started a git repo to store the bioinfo formats: https://github.com/lindenb/magic

• 1 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

That is neat. To make more people benefit from this, could you consider submitting the magic strings directly to http://www.darwinsys.com/file/ and http://freedesktop.org/wiki/Software/shared-mime-info/? This would make these patterns recognised by default on many Linux distributions. One could even consider to submit a media type for SAM, BAM, etc. files to the IANA (Internet Assigned Numbers Authority). In that case, the best would be to do it through the current maintainers of the SAM specification.

• 0 views • link
updated by Ram Ram • written by Charles Plessy Charles Plessy

Log in to answer this question.

More posts like this