This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How To Merge Two Fastq.Gz Files?

I need to merge two fastq.gz files. I could decompress the files and do "cat" on them, but is there any faster way? can I use

gzcat file1.fastq.gz file2.fastq.gz | gzip > merged.fastq.gz

?

merge fastq

How to find and merge all fastq files into one log file

1 answer

See SO : Fast Concatenation of Multiple GZip Files

A gzip file consists of a series of "members" (compressed data sets). [...] The members simply appear one after another in the file, with no additional information before, between, or after them.

cat file1.gz file2.gz file3.gz > allfiles.gz

Note that this will someitmes lead to trouble, if tools do not implement gzip compression correctly. I spend an hour or so to find out that FastQC was only reading the first of 10 fastq.gz files I had combined in this way (they seem to have fixed this in the newest release).

This worked well for me. Note that you can confirm that this works in most cases by doing something like the following:

cat file1.gz file2.gz file3.gz > allfiles-cat.gz
zcat file1.gz file2.gz file3.gz | gzip -c > allfiles-zcat.gz
zcat allfiles-cat.gz | md5sum
zcat allfiles-zcat.gz | md5sum

The resulting hash/message digests should be identical.

My experience is that the zcat method is around 40x slower, but the cat method's resulting file is a few percent bigger depending on your the gzip parameters used in the methods.

Can I know why after the cat my two 1.5G fastq.gz files become one 700M file? Why it got much smaller?

Probably the compression found more patterns when more data was added, thus it could reduce more information.

Log in to answer this question.