For parallel processing (assuming more cores than jobs)
for f in *.gz; do gunzip $f & done
Hi guys, I know this question is dumb. How to convert the .fastq.gz into fastq in batches? For example, if we have 100 .fastq.gz, how can we use the command to let them to be .fastq then? Thank you
GNU Parallel, FTW!
If you have access to a HPC cluster, open a interactive job session. In my case I have a 64 core node, so I do it as:
qsub -I -l nodes=1:ppn=64 -l walltime=1:00:00
then run gunzip on all 64 processors
parallel -j64 "gunzip {}" ::: *.fastq.gz
I'll be done in no time!
$ gunzip *.gz
or
$ for f in *.gz; do gunzip $f; done
For parallel processing (assuming more cores than jobs)
for f in *.gz; do gunzip $f & done
This would start as many jobs as the number of files, which is not an elegant way. Best way to do it is to use GNU Parallel as told by arnstm.
parallel --jobs <int cores> gunzip {} ::: *.fastq.gz
Awesome! Worked great on my 480 files
Isn't this the easiest way? =)
Or don't.
pass them compressed if the tool handles it ,or decompress on the fly i.e.
zcat file.gz | whatever_tool
There are a bunch of utilities for processing gzip without explicitly writing out the unzipped file
Hi,
PIGZ could be potential solution, its parallel version of GZip and uses the multi threading out of the box with so many other parameters to tweak the performance. You can find one here: http://zlib.net/pigz/
You can simply run pigz -d *.gz to extract the .gz files
Thanks
pigz is indeed a very simple way to achieve such a task.
It might do the job
gunzip '*.gz'
gunzip *.fastq.gz will do the job!
Log in to answer this question.
Look at a few "xargs" usage on internet or simply try gunzip *.fastq.gz
Why do you need to unzip the fastq files? In most cases it is better to keep them compressed. Most NGS tools can handle compressed files directly, and it is generally faster to read a compressed file than an uncompressed one.
If you have LSF (replace
bsubwithqsub -cwdfor SGE):