This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to efficiently/quickly merge ~500k vcfs?

I have a frankly ludicrous number of single-sample vcf.gz files (with their tabix) that I want to merge into on big file. I've previously used bcftools merge on 48 threads to merge 1000 and it took 15+ minutes. I'm pretty sure that time to complete won't scale linearly once I increase the number of samples to 500k+. Any suggestions? Should I merge groups of samples at a time like going up a tree? Should I use a different toos?

bcftools vcf variant-calling
• 2,709 views • link
updated by Jeremy Leipzig Jeremy Leipzig • written by Lei • Lei

2 answers

I would suggest TileDB-VCF, which enables downstream analysis (and export) without the need to fire up a Spark cluster. (Disclaimer: I work for TileDB)

• 0 views • link
written by Jeremy Leipzig Jeremy Leipzig

Interesting! I looked into TileDB-VCF a couple of months back and it looks like the tutorial has much improved! I'll give it a try as well.

• 0 views • link
written by Lei • Lei

Feel free to reach out to me directly. I can walk you through some notebooks and/or provide some free credits to get you started.

• 0 views • link
written by Jeremy Leipzig Jeremy Leipzig

Do you have VCFs or gVCFs? For gVCFs you could also use HAIL (https://hail.is/) or GLNexus (https://github.com/dnanexus-rnd/GLnexus).

• 0 views • link
written by DBScan • DBScan

Log in to answer this question.

More posts like this