Should Making Bgzip Files Take A Long Time?
I'm bgzipping some huge vcf files on a server and wondering whether I'm doing something wrong as it is taking a long time (file size increasing with a 100kb every few seconds.)
The command I'm running is
tabix -h vcfs/ALL.chrX.phase1_release_v3.20101123.snps_indels_svs.genotypes.vcf.gz X | vcf-subset -c population_files/CEU_panel | bgzip > population_files/CEU_X.vcf.gz
head /proc/meminfo gives:
MemTotal: 140567216 kB
MemFree: 2109700 kB
Buffers: 295408 kB
Cached: 106374448 kB
SwapCached: 991620 kB
Active: 62698940 kB
Inactive: 71434832 kB
Active(anon): 55538428 kB
Inactive(anon): 35691912 kB
Active(file): 7160512 kB
(Dunno if this is relevant)
Thanks.
• 4,114 views
•
link
updated
by
Matt Shirley
•
written
by
Click downvote
•
written
by
Click downvote
1 answer
You're not just bgzip-ing the file, but extracting, then subsetting, and then compressing them back to a file. You could be CPU-limited. If you run top and the vcf-subset command is running at 100% utilization, then that's your bottleneck. Depending on how little of the data you are subsetting, your output file might not grow very quickly either.
• 141 views
•
link
written
by
Matt Shirley
Log in to answer this question.
You're not just bgzip-ing the file, but extracting, then subsetting, and then compressing them back to a file. You could be CPU-limited. If you run
topand the vcf-subset command is running at 100% utilization, then that's your bottleneck. Depending on how little of the data you are subsetting, your output file might not grow very quickly either.this should be an answer, rather than a comment ;)
Good point. Moved.