Hi, I have a suspicion one or more of my fastq files is corrupted: some reads have sequence and quality of different lengths, for example. Would this python script work to detect a problem?
import sys
from Bio import SeqIO
for record in SeqIO.parse(sys.stdin, "fastq"):
pass
And I use it like this:
pigz -dc -p 3 serum-3_2.fq.gz | python check.py
Will it raise an error if one of the reads is non-valid? Or is there a better quick way to check this?
•
written
by
flyamer •
6 answers
If you only care about read length differing from quality-score length, you could just run this:
zcat fastq.gz | paste - - - - | awk -F"\t" '{ if (length($2) != length($4)) print $0 }' | wc -l
That will give you a count of aberrant records.
You can use fqlint, a Rust program that identifies a broad range of issues Illumina-based FASTQ files. To install it, you can do the following after installing Rust.
cargo install --git https://github.com/stjude/fqlib.git
It's very difficult to say what is and isn't an invalid FASTQ, as there is no definitive specification.
A better approach is to write some tests that you think should be true, for example, that the SEQ and QUAL values are the same length, that there is always 4 lines per entry, etc. Do not assume that if a bioinformatics tool/parser does not give errors, it means there are no errors. This is so very often not the case, particularly surrounding FASTA/FASTQ as the specs are so open to interpretation.
One you have determined that the data is not "nonsense" but rather "missense" to borrow a term from genetics, then you should perhaps analyse distributions of the data to see if it makes sense. You can use uQ to do this very quickly with the --peek parameter, for example the command:
python /Users/John/Desktop/uq.py -i /Users/John/Downloads/ENCFF000ZZU.fastq.txt --peek
outputs this:
http://pastebin.com/raw/6qTMwNTp
This also checks that SEQ/QUAL is the same length, and some other very very basic things. It's just a proof-of-concept for compressing FASTQ files to be as small as possible, but it's not intended to be used.
With this tool : FastQValidator
Provide a link, when you are referring to a specific program. This is important since software programs may have similar names and searching the web may sometimes lead one down an undesired path (e.g. malware etc).
I will include a link for FastQValidator this time.
I will know it. Thanks.
Try FASTQC
•
written
by
BioinfGuru
Log in to answer this question.
-
Append assembly accession to nucleotide accession number in RefSeq Genbank file
written by genomes_and_MGEs •
-
SeqIO object get cleared away after being accessed
written by zincjiang •
-
How to remove features from GBK files with biopython ?
written by hugo.avila
-
comlement fastq file using Biopython
written by Andrei Makhon •
-
Import the quality scores from a FASTQ file in Python 3 Biopython
written by biomagician
-
Splitting fastq from stdin into multiple files ?
written by flyamer •
-
Biopython script to change file formats and headers
written by skbrimer
-
SeqIO.write() in biopython is extremely slow
written by jincheng.wang1986 •
-
How To Retrieve Multiple Sequence In One Python Script
written by ahmedakhokhar
-
Biopython Embl Parser Only Reads One Entry
written by sinanugur •
why not try with test file?
Ha, I don't know why not, good idea, thanks! I checked, seems like it at least picks up differences in sequence and quality lengths.
A: Fastq Quality Read And Score Length Check