This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Optimizing VCF File Merging Using bcftools

Hello,

I am trying to merge 1200 WGS VCF files (around 9 GB each) using bcftools. I want to merge them by chromosome to make the output files easier to work with.

I am using an HPC cluster with 156 threads and 1to RAM, but I’m not sure how to optimize the resources to make the merging process faster. I used parallel but it didnt work as i get this error.

"Could not load local index file 'path/to/file.tbi' : Too many open files"

Even when I ran bcftools for a single chromosome, I got the same error message.

Any advice on how to resolve this issue and optimize resource usage would be greatly appreciated!

Thank you,

bcftools merge parallelize wgs
• 2,092 views • link
written by j.k3096 • j.k3096

3 answers

This may be related to ulimit option set for you account. See: https://askubuntu.com/questions/1182021/too-many-open-files

• 0 views • link
written by GenoMax GenoMax

just so that there is also an answer here try something like as GenoMax points out in the link

ulimit -n 100000

though the sys administrator may have some limits what you can set the limit to,

in that case merge in batches, merge 100 at the time per process.

now you have 12 files now merge those

• 0 views • link
written by Istvan Albert Istvan Albert

see my answer using nextflow: How to merge 20K single-sample VCFs *without* using plink or plink2?

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

Thank you all! I adjusted the ulimit, and it worked perfectly.

• 0 views • link
written by j.k3096 • j.k3096

Log in to answer this question.

More posts like this