How to set fq.gz as input for bbduk (BBmap)?

I have a pair of WGS files (phiM_x.fq.gz) from Illumina sequencing placed in the folders phiM/Raw. I previously set up the pipeline with BBmap using bbduk.sh for preprocessing. However, I now get an error for the file type: Unspecified format for input phiM/Raw/phiM_1.fq.gz; defaulting to fastq.

Is there a flag for giving bbduk fq.gz as input? Or might this be a problem with the files themselves (maybe they are corrupted)?

Thank you

$ bbduk.sh in=phiM/Raw/phiM_1.fq.gz in2=phiM/Raw/phiM_2.fq.gz out=phiM/Trimmed/phiM_1_Trm.fq.gz out2=phiM/Trimmed/phiM_2_Trm.fq.gz \
    ktrim=r k=23 mink=11 hdist=1 tbo tpe minlen=70 ref=adapters ftm=5 -Xmx200m

java -ea -Xmx200m -Xms200m -cp /home/gigiux/src/bbmap/current/ jgi.BBDuk in=phiM/Raw/phiM_1.fq.gz in2=phiM/Raw/phiM_2.fq.gz out=phiM/Trimmed/phiM_1_Trm.fq.gz out2=phiM/Trimmed/phiM_2_Trm.fq.gz ktrim=r k=23 mink=11 hdist=1 tbo tpe minlen=70 ref=adapters ftm=5 -Xmx200m
Executing jgi.BBDuk [in=phiM/Raw/phiM_1.fq.gz, in2=phiM/Raw/phiM_2.fq.gz, out=phiM/Trimmed/phiM_1_Trm.fq.gz, out2=phiM/Trimmed/phiM_2_Trm.fq.gz, ktrim=r, k=23, mink=11, hdist=1, tbo, tpe, minlen=70, ref=adapters, ftm=5, -Xmx200m]
Version 39.33

Input is being processed as paired
java.lang.AssertionError: 
Error in phiM/Raw/phiM_2.fq.gz, line 1030, with these 4 lines:

>>> RANDOM ASCII CHARACTERS <<<

at stream.FASTQ.quadToRead_slow(FASTQ.java:742)
at stream.FASTQ.toReadList(FASTQ.java:678)
at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:110)
at stream.FastqReadInputStream.hasMore(FastqReadInputStream.java:76)
at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:677)
at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:666)

preprocessing bbmap sequencing trimming

• 607 views • link

updated 4 months ago by fernando •

• written 4 months ago by marongiu.luigi

1 answer

I now get an error for the file type

That says the file is not actually compressed (and additionally may be corrupt).

What do you get from the command file phiM_1.fq.gz? Are you able to cat the file and see its contents. If you can then the file may be plain text.

Based on the error message posted, there is also a possibility that file phiM_2.fq.gz is corrupt. Check around line 1030 as the error message above says.

• 0 views • link

written 4 months ago by GenoMax

Thank you, the output is:

$ file phiM_1.fq.gz # same for the second pair 
phiM_1.fq.gz: gzip compressed data, max speed, from Unix, original size modulo 2^32 461033219 gzip compressed data, unknown method, has CRC, extra field, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 461033219
$ gunzip -t phiM_1.fq.gz # same for the second pair 
$

I can't cat so it looks like zipped; how can I check the sanity of the files?

However, I got an error here:

$ seqkit stats --all phiM_1.fq.gz phiM_2.fq.gz 
[ERRO] phiM_1.fq.gz: fastx: invalid FASTA/Q format
processed files:  2 / 2 [======================================] ETA: 0s. done

• 0 views • link

written 4 months ago by marongiu.luigi

Looks like the data files are corrupt. Find other copies. You could try to repair the files, but only as a last resort, if you can't find another copy of the data.

• 0 views • link

written 4 months ago by GenoMax