This is a test version of Biostars. For the public version, visit https://www.biostars.org.
fastq file error

Hi,

After removing rRNAs using sortmerna, 2/42 files failed fastqc with error:

uk.ac.babraham.FastQC.Sequence.SequenceFormatException: ID line didn't start with '@' at uk.ac.babraham.FastQC.Sequence.FastQFile.readNext(FastQFile.java:158) at uk.ac.babraham.FastQC.Sequence.FastQFile.next(FastQFile.java:125) at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:76) at java.lang.Thread.run(Thread.java:745)

I checked the fastq file using fastq_stats which suggested the fastq file is malformed

Malformed fastq record (length mismatch) at line 69042283 Segmentation fault (core dumped)

can anybody please tell me what's the problem in this file?

zcat 4590_END_sortmerna_R2.fastq.gz | sed -n '69042275,69042295p'

+
BBBFFFFFFFFFFIIIIIIIIIIIIIIIIFIIFIIIIIIFIIIIIFIIIIIIIIIIIIIFFFFFFFFFFFFFFFFFFFFFFFFFFBFFFFFFFFBFFFFFF
@HWI-ST1072:210:C5LGGACXX:4:1309:19821:97176 2:N:0:AGTTCC
CTGGAATCCCTCGATCACTGAACAGGAAGGAAACCTGATGCAGAGACTCAGGACGCAGGCTCCAGAAGTCCCAGACCATGTCCGGATCCTTCAGGTGTGTT
+
BBBFFFFFFFFFFIIIIIIIIIIIIIIIIIIIIIIFFIIIIIIIBFIIIIFFFIIIIIFFFFFFFFFBBBBBFFBFBFFFFFFBFFFFFFFFBBBBBBBBB
@HWI-ST1072:210:C5LGGACXX:4:1309:19994:97188 2:N:0:AGTTCC
CTCATGTACCGGGGGTTACTGGATGCTCTGCTGCAGACGGCTCGGACAGAGGGCATTTTTGGCATGTACAAGGGAATAGGTGCCTCCTACTTCCGCCTTGG
+
BBBFFFFFFFFFFIIFFIIIIIIIIIIIIIIIIIIIIIFFFFFFFBBBFB<BB7<BFFFFFFBBBBBFFFFFFFBBFBFBBBBFFFFFFFFFFFFFFFFBB
@HWI-ST1072:210:C5LGGACXX:4:1309:19837:97199 2:N:0:AGTTCC
CCTACTTCTCTTTTAATTAAAATGTAGCCACTATATAACAGATATGGTTTACGATCTGTTTCAAAAACATTTACCTAAGTGACTCTGTAAAAACCTCAGCA
+
BBBFFFFFFFFFFIIIIIIIIIIIIIIIIIIIIIIIIIIIIIFIIIIFIIBFFIIIIIIIIIIIIIIIIIIIIIIIIIIFFFFFFFFFFFFFFFFFFFFFB
@HWI-ST1072:210:C5LGGACXX:4:1309:19815:97203 2:N:0:AGTTCC
GGGCCATTCGGGGTTGGCTGGGGGTGAGTGGCAGAGCAAGCTTTCAGCACAGGGCGGCATGTGCCAAAAGCACGGTGGCCTGAGGCTGCCCCTCGTGAGCA
+
BBBFFFFFFFFFFFIIIIIIIIFIFFFIFIFIIIIIIFFFFFBFFFFFFF<BBBBBFFFFBBBBFFFFFFBFFFFFFFBFFFFFFF0BBFFFBFFBF<BB0
@HWI-ST1072:210:C5LGGACXX:4:1309:19975:97229 2:N:0:AGTTCC
AGAGACTGCATGGCTTTCATGACGTGAAGATTGGGCACATTCTTGTCTGCCAGCTCCGGGTGCTTGGGCATGTGGACATCCTTCTTGGCCACCATCACCCC
+

Thanks a lot!

fastq fastqc sortmerna rna-seq
• 9,526 views • link
written by Pals Pals

I would just like to also report the same problem. I also had corrupted fastq file output after SortMeRNA run and it is also missing lines with quality scores. If someone has sorted out this issue would be grateful if you could share your solution. Thanks

• 0 views • link
written by bdadonaite • bdadonaite

1 answer

Purely looking at the line in question nothing seems to be wrong but obviously the error is there with FastQC. This appears to be a paired end dataset so you could use repair.sh from BBMap suite to see if you can identify the malformed read pairs and remove them.

To verify proper pairing: reformat.sh in1=r1.fq.gz in2=r2.fq.gz vpair
To repair: repair.sh in1=r1.fq.gz in2=r2.fq.gz out1=fixed1.fq.gz out2=fixed2.fq.gz outsingle=singletons.fq.gz

• 0 views • link
written by GenoMax GenoMax

After the following error, the output file size is 3.1 compared to 3.3 (original). It seems sortmerna steps created the problem in reads.

Started output stream.
java.lang.AssertionError: 
Error in 4563_END_sortmerna_R2.fastq.gz, line 194876135, with these 4 lines:
BBBFFFFFFFFFFIIIIIIIIIFIIIIIIIIIIIIIFIIFIIIIIIIIIIIIIIIIIIIIIFFFFFFFFF7<BFF<FFFFFFFFFFFFFFFFFFFFFFFFF
@HWI-ST1072:210:C5LGGACXX:2:2305:2793:35345 2:N:0:CAGATC
CGGGGATTCTCCATTCCGGATGTGTTTCGTGGAGTGCATCGATACCTGCGCAATGCCTATGCTAGGGAAGAGTTTGCTTCCACCTGTCCAGATGATGAGGA
+

    at stream.FASTQ.quadToRead(FASTQ.java:779)
    at stream.FASTQ.toReadList(FASTQ.java:710)
    at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:111)
    at stream.FastqReadInputStream.nextList(FastqReadInputStream.java:96)
    at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:656)
    at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:635)

Set cris2Active=false
java.lang.AssertionError: 
Error in 4563_END_sortmerna_R1.fastq.gz, line 194876135, with these 4 lines:

@HWI-ST1072:210:C5LGGACXX:2:2305:2793:35345 1:N:0:CAGATC
ATTTTGGAGTGTGTCCGTTGGGTAGTATGTGGAAACCACCCAGGGCCTTTGTGGAGAAAATGGAGGGGGGTGCCGGGGGGCCCTAGGAAGGGCCTTATTTG
+

    at stream.FASTQ.quadToRead(FASTQ.java:779)
    at stream.FASTQ.toReadList(FASTQ.java:710)
    at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:111)
    at stream.FastqReadInputStream.nextList(FastqReadInputStream.java:96)
    at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:656)
    at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:635)

Set cris1Active=false
Exception in thread "main" java.lang.NullPointerException
    at jgi.SplitPairsAndSingles.process3_repair(SplitPairsAndSingles.java:562)
    at jgi.SplitPairsAndSingles.process2(SplitPairsAndSingles.java:310)
    at jgi.SplitPairsAndSingles.process(SplitPairsAndSingles.java:236)
    at jgi.SplitPairsAndSingles.main(SplitPairsAndSingles.java:45)
• 0 views • link
written by Pals Pals

What step is this error from? Before running repair.sh or while running repair.sh?

I thought you were able to repair the files before.

• 0 views • link
written by GenoMax GenoMax

While running the repair.sh

• 0 views • link
written by Pals Pals

Did you check the files with vpair first? Is the error from that step or actual repair?

I am going to tag Brian Bushnell . He should be along in 2-3 hours. It seems to me that some fastq records themselves may be broken (in addition to pairing).

• 0 views • link
written by GenoMax GenoMax

Here is the error from vpair step:

/homeappl/home/kipokh/appl_taito/mytools/bbmap/reformat.sh in1=4563_END_sortmerna_R1.fastq.gz in2=4563_END_sortmerna_R2.fastq.gz vpair

java -ea -Xmx200m -cp /homeappl/home/kipokh/appl_taito/mytools/bbmap/current/ jgi.ReformatReads in1=4563_END_sortmerna_R1.fastq.gz in2=4563_END_sortmerna_R2.fastq.gz vpair
Executing jgi.ReformatReads [in1=4563_END_sortmerna_R1.fastq.gz, in2=4563_END_sortmerna_R2.fastq.gz, vpair]

No output stream specified.  To write to stdout, please specify 'out=stdout.fq' or similar.
Set INTERLEAVED to false
Input is being processed as paired
java.lang.AssertionError: 
Error in 4563_END_sortmerna_R1.fastq.gz, line 194876135, with these 4 lines:

@HWI-ST1072:210:C5LGGACXX:2:2305:2793:35345 1:N:0:CAGATC
ATTTTGGAGTGTGTCCGTTGGGTAGTATGTGGAAACCACCCAGGGCCTTTGTGGAGAAAATGGAGGGGGGTGCCGGGGGGCCCTAGGAAGGGCCTTATTTG
+

    at stream.FASTQ.quadToRead(FASTQ.java:779)
    at stream.FASTQ.toReadList(FASTQ.java:710)
    at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:111)
    at stream.FastqReadInputStream.nextList(FastqReadInputStream.java:96)
    at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:656)
    at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:635)
java.lang.AssertionError: 
Error in 4563_END_sortmerna_R2.fastq.gz, line 194876135, with these 4 lines:
BBBFFFFFFFFFFIIIIIIIIIFIIIIIIIIIIIIIFIIFIIIIIIIIIIIIIIIIIIIIIFFFFFFFFF7<BFF<FFFFFFFFFFFFFFFFFFFFFFFFF
@HWI-ST1072:210:C5LGGACXX:2:2305:2793:35345 2:N:0:CAGATC
CGGGGATTCTCCATTCCGGATGTGTTTCGTGGAGTGCATCGATACCTGCGCAATGCCTATGCTAGGGAAGAGTTTGCTTCCACCTGTCCAGATGATGAGGA
+

    at stream.FASTQ.quadToRead(FASTQ.java:779)
    at stream.FASTQ.toReadList(FASTQ.java:710)
    at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:111)
    at stream.FastqReadInputStream.nextList(FastqReadInputStream.java:96)
    at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:656)
    at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:635)
Exception in thread "Thread-2" java.lang.AssertionError: 
There appear to be different numbers of reads in the paired input files.
The pairing may have been corrupted by an upstream process.  It may be fixable by running repair.sh.
    at stream.ConcurrentGenericReadInputStream.pair(ConcurrentGenericReadInputStream.java:480)
    at stream.ConcurrentGenericReadInputStream.readLists(ConcurrentGenericReadInputStream.java:345)
    at stream.ConcurrentGenericReadInputStream.run(ConcurrentGenericReadInputStream.java:189)
    at java.lang.Thread.run(Thread.java:745)
• 0 views • link
written by Pals Pals

Are you able to actually fix the broken pairing by running reformat.sh (second command)?

One additional thing to test would be to validate the individual files to see if the fastq records themselves are broken: http://genome.sph.umich.edu/wiki/FastQValidator

• 0 views • link
written by GenoMax GenoMax

It appears that you have records with bases but no quality scores. Those are not valid fastq records. It looks like maybe, some of the records have 2 newlines after the "+" instead of 1.

Probably, sortmerna is doing something inappropriate that results in corrupted output.

• 0 views • link
written by Brian Bushnell Brian Bushnell

@Brian: Is there something in BBMap to check and fix broken fastq records in files?

• 0 views • link
written by GenoMax GenoMax

Well... I tried to add some capability to do so in some limited ways, but it doesn't work very well. The problem is that there are an infinite number of ways a file could be broken, so the software will always end up guessing. So it's impossible to write a program that can correctly fix any broken fastq file, or even that can ensure that the corrections it did were correct. You can only write a program that will fix very specific problems, like a correctly-formatted fastq file with interleaved reads out of order.

• 0 views • link
written by Brian Bushnell Brian Bushnell

Thanks a lot Brian and genomax2. I had also asked this problem to Jenya Kopylov (author of sortmerna) and I decided to skip sortmerna step for now. By the way, I found BBTools really useful :)

• 0 views • link
written by Pals Pals

Log in to answer this question.

More posts like this

No similar posts found.