This is a test version of Biostars. For the public version, visit https://www.biostars.org.
speeding up bcftools view

Hi all - I have a very large multi sample vcf file which I am trying to subset by a list of sample IDs, however, my current approach is working very slowly (>2hr per chromosome) and I am wondering if there are any tricks to making it run faster with large files? Here is my current approach:

for file in /vcffiles/*.vcf.gz; do
    bcftools view -Oz -S sample_list.txt $file > /output/subset_"${i##*/}" 
done

Thanks in advance for any suggestions!

vcf bcftools

thank you for your helpful responses!

2 answers

Another solution with tsp and background processes.

# this sets number of max jobs. Here we use the number of processes. You might want to change this to another number.
tsp -S $(nproc)

# rest is similar. we just add tsp to start of the command and & at the end.
# & at the end calls all the processes at once but tsp queues them and calls them in batches.
for file in /vcffiles/*.vcf.gz; do
    tsp bcftools view -Oz -S sample_list.txt $file > /output/subset_"${i##*/}" &
done

let's do it using nextflow, I won't test it so there will be some small bugs, but you get the idea.

Log in to answer this question.