I am creating sorted bam files from multiple paired-end fastq files using an array along with the parallel command. I am not specifying job numbers for parallel. I thought parallel should make the job faster but it takes longer to finish the job with parallel as compared to array without parallel. Any help is appreciated to understand why it is so?
1 answer
GNU parallel, by default, will use all cores, to avoid this behaviour, one has to use the option -j. As genomax said, you likely consumed all RAM and started to use swap, or hit disk IO limits, thus slowing down the overall run time compared to serial execution.
As lieven.sterck said, you don't need to use a for loop with GNU parallel, it has plenty of resources to deal with multiple inputs and simplify the command line. You can use the -max-lines option to control how many arguments will be passed to parallel:
ls *.fastq.gz | parallel --max-lines=2 echo "{1} {2} - Are a pair"
Log in to answer this question.
-
Is a SLURM job array submission suitable for parallelizing multiple FASTQ file downloads?
written by kalavattam •
-
In the NGS pipeline, why read are sorted before marking duplicates?
written by ManuelDB
-
salmon light weight alignment mode
written by evelyn
-
rseqc bed file for library type paired end RNA fastq files
written by evelyn
-
FastQC on multiple paired end files
written by evelyn
-
split sorted bam file chromosome wise
written by evelyn
-
samtools sort error
written by evelyn
-
vcf from sorted bam
written by evelyn
-
sorted bam file
written by evelyn
-
How to run simultaneous jobs in parallel using Job arrays in SLURM?
written by Biologist
I think this has been covered here already. I'm no parallel expert myself but from what I understood the thing is that you should not do parallel in combination with a for loop. You either use the for loop and do then serial or you stream your input files to parallel and do them all in parallel.
At some point you simply run out of RAM or I/O capacity on your system. Parallel will start multiple jobs but it can't overcome limitation of hardware you have available.