This is a test version of Biostars. For the public version, visit https://www.biostars.org.
HPC settings for concatenating large VCF.GZ files

Dear all,

First-time poster here. :)

I was hoping to ask for your recommendations on what to request on a HPC to concatenate ~22 chromosomes (each vcf.gz files is ~15GB, so ~330GB in total).

e.g. with bcftools concat -Oz chr1.vcf.gz chr2.vcf.gz ... chrX.vcf.gz > allchr.vcf.gz

Could I request 32 CPUs each with 16GB of memory (512GB)? Would that work?

Any suggestions at all would be appreciated!!

Thanks in advance

concatenate vcf hpc
• 1,704 views • link
written by Katherine • Katherine

Katherine - There are many ways to resolve that question, none of which should require that much dedicated memory. Irrespective, I really encourage you to tabix index your vcf files. Once this index file is created (which only needs to be done once), you will be able to perform operations that would have taken minutes in seconds, including operations similar to what you propose. VAL

• 0 views • link
written by LauferVA LauferVA

1 answer

bcftools concat stores only a small stream of data in memory. Therefore, you would need, at most, 1Gb of memory to perform this operation and one cpu core. You can request multiple threads to perform multithreaded gzipping of the output though.

• 0 views • link
written by 4galaxy77 4galaxy77

Thanks for this information @4galaxy77 ! How many hours would you expect a job to run using bcftools to concatenate just TWO vcf.gz files (~15GB each)? At the moment, this job has been running for 20hrs. Thanks in advance, Katherine.

• 0 views • link
written by Katherine • Katherine

Log in to answer this question.

More posts like this