speeding up bcftools view
Hi all - I have a very large multi sample vcf file which I am trying to subset by a list of sample IDs, however, my current approach is working very slowly (>2hr per chromosome) and I am wondering if there are any tricks to making it run faster with large files? Here is my current approach:
for file in /vcffiles/*.vcf.gz; do
bcftools view -Oz -S sample_list.txt $file > /output/subset_"${i##*/}"
done
Thanks in advance for any suggestions!
• 4,619 views
•
link
2 answers
Another solution with tsp and background processes.
# this sets number of max jobs. Here we use the number of processes. You might want to change this to another number.
tsp -S $(nproc)
# rest is similar. we just add tsp to start of the command and & at the end.
# & at the end calls all the processes at once but tsp queues them and calls them in batches.
for file in /vcffiles/*.vcf.gz; do
tsp bcftools view -Oz -S sample_list.txt $file > /output/subset_"${i##*/}" &
done
• 0 views
•
link
let's do it using nextflow, I won't test it so there will be some small bugs, but you get the idea.
• 0 views
•
link
Log in to answer this question.
Maybe this link is useful: How to parallelize bcftools mpileup with GNU parallel?
thank you for your helpful responses!