This is a test version of Biostars. For the public version, visit https://www.biostars.org.
how to unzip the files in batch?

Hi guys, I know this question is dumb. How to convert the .fastq.gz into fastq in batches? For example, if we have 100 .fastq.gz, how can we use the command to let them to be .fastq then? Thank you

next-gen rna-seq snp alignment

Look at a few "xargs" usage on internet or simply try gunzip *.fastq.gz

Why do you need to unzip the fastq files? In most cases it is better to keep them compressed. Most NGS tools can handle compressed files directly, and it is generally faster to read a compressed file than an uncompressed one.

If you have LSF (replace bsub with qsub -cwd for SGE):

ls *.fastq.gz | xargs -i echo bsub gzip -d {} | sh

7 answers

GNU Parallel, FTW!

If you have access to a HPC cluster, open a interactive job session. In my case I have a 64 core node, so I do it as:

qsub -I -l nodes=1:ppn=64 -l walltime=1:00:00

then run gunzip on all 64 processors

parallel -j64 "gunzip {}" ::: *.fastq.gz

I'll be done in no time!

$ gunzip *.gz

or

$ for f in *.gz; do gunzip $f; done

For parallel processing (assuming more cores than jobs)

for f in *.gz; do gunzip $f & done

This would start as many jobs as the number of files, which is not an elegant way. Best way to do it is to use GNU Parallel as told by arnstm.

parallel --jobs <int cores> gunzip {} ::: *.fastq.gz

Isn't this the easiest way? =)

Or don't.

pass them compressed if the tool handles it ,or decompress on the fly i.e.

zcat file.gz | whatever_tool

There are a bunch of utilities for processing gzip without explicitly writing out the unzipped file

http://www.nongnu.org/zutils/manual/zutils_manual.html

Hi,

PIGZ could be potential solution, its parallel version of GZip and uses the multi threading out of the box with so many other parameters to tweak the performance. You can find one here: http://zlib.net/pigz/

You can simply run pigz -d *.gz to extract the .gz files

Thanks

pigz is indeed a very simple way to achieve such a task.

http://stackoverflow.com/questions/16038087/extract-all-gz-in-a-directory-linux

It might do the job

gunzip '*.gz'

gunzip *.fastq.gz will do the job!

Log in to answer this question.