Hello,
In order to perform population gnomic analysis, I am trying to merge many and huge variants data (gVCF), such as several dozens Gb, over 20 files.
Bcftools merge and vcf-merge were used so far but very slow to merge those files into just one file.
Do you have any ides to merge huge gVCF files? I want to use variants as many as possible so I used gVCF not for VCF.
Thank you!
•
written
by
ymat •
4 answers
merge per region in parallel and then concatenate each chunk.
how would you go about doing this?
I have a similar issue, and came across GLnexus: https://github.com/dnanexus-rnd/GLnexus I haven't used this myself yet, but according to the authors (their paper) seems to be faster than GATK.
Use GATK4 GenomicsDBImport and GenotypeGVCFs in parallel for many callable regions.
The callable regions you can calculate with picard based on locations were there are 99+N nucleotide in the reference genome.
picard ScatterIntervalsByNs R=$fasta O=callalbe_regions.bed N=99 -Xmx1G OT=ACGT
Then loop over the regions in that file and call GenomicsDBImport and GenotypeGVCF for each region(line) in the bed file.
Works best of course if you can submit many GenomicsDBImport and GenotypeGVCFs commands to a somewhat large cluster.
Finally use bcftools concat (naive) tot get a single VCF/BCF file. For BCF first convert to the per callable region output VCF file to BCF.
Log in to answer this question.
-
Is this the correct order of steps in GATK germline cohort variant calling (VQSR workflow)?
written by Ramnaresh •
-
Tool that can merge 2 VCF files while taking "representational ambiguity" of (multi-allelic) varian…
written by William
-
Fast genotyping for PRS calculation
written by German.M.Demidov
-
Efficient way to merge multiple single sample vcf files with same variants / load into PLINK
written by dmitry.s.kolobkov •
-
Error while merging multiple vcf files using Picard MergeVcfs
written by nitinra •
-
how to merge gvcf using bcftools
written by evelyn
-
Merging population vcf files without gvcf
written by Myo Naung •
-
How to update the INFO field of VCF after merging, subsetting?
written by kirannbishwa01
-
CombineGVCFs + GenotypeGVCFs for Freebayes gVCF files?
written by William
-
gVCF files from 1000 Genomes samples
written by donfreed
Merging files that size as VCFs is always going to be slow. If you don't mind losing some metadata and have a lot of memory at your disposal then converting to plink binary and merging in that format will speed things up a lot.
I also think converting to plink binary and merging is a good solution! I will try it. Thank you.