This is a test version of Biostars. For the public version, visit https://www.biostars.org.
bgzip all VCFs in a directory

Hi guys,

So I've been trying to bgzip around 100 VCF files in parallel, but although the jobs are submitted and files get created there's definitely something wrong.

So far I've been trying:

$ for file in *.vcf
> do
> bsub /foo/bar/bgzip $file
>> $file.gz

What is the correct way to do this?

Thanks in advance!

sequencing seq
• 13,342 views • link
updated by Ram Ram • written by stevenlang123 stevenlang123

1 answer

Use GNU parallel

parallel bgzip {} ::: *.vcf

or xargs:

ls *.vcf | xargs -P 10 bgzip
• 0 views • link
updated by Ram Ram • written by Pierre Lindenbaum Pierre Lindenbaum

Is there any way to do this using an LSF manager to split the jobs up instead ? The problem I'm having is that bgzip requires to redirect the file output so

bsub < bgzip $file > $file.gz

does not work

• 0 views • link
updated by Ram Ram • written by stevenlang123 stevenlang123

Never mind, working great Pierre! Thanks!

• 0 views • link
written by stevenlang123 stevenlang123

When I use ls *.vcf | xargs -P 10 bgzip it only compresses the first file in the folder. Using -n1 (=use at most 1 argument per command line) instead of -P 10 worked for me:

ls *.vcf | xargs -n1 bgzip
• 1 views • link
written by Scleroz • Scleroz

You can use both (xargs -n1 -P0) to get a computer to use max_procs and dispatch one file (line) per process.

• 0 views • link
written by Ram Ram

Log in to answer this question.

More posts like this