Hello,
I'm trying to run bedops vcf2bed on a huge (27G) VCF file. I'd like to extract a BED of the deletions in the VCF.
My usage:
vcf2bed --max-mem=4G --deletions <file.vcf >file.bed
I'm running it on a cluster and I tried giving the job 8G and 16G RAM, but it maxes out all available RAM in a matter of seconds (100-150 seconds) and the job quits. I know splitting the VCF into per-chromosome chunks might solve this, but is there any other option I could use?
I am open to using other tools to extract the BED as well, given that they ensure end-start equals length(REF).
EDIT:
I combined a couple of ideas from the recommendations below to get to my solution - YMMV and not all of these mods might be necessary. I piped in the file using GNU parallel's --pipe and also used --do-not-sort to get unsorted bed files.
cat huge_file.vcf | parallel --pipe vcf2bed --do-not-sort >huge_file.unsorted.bed
--
Ram
3 answers
This helped. I used
cat huge_file.vcf | parallel --pipe vcf2bed --do-not-sort --insertions >huge_file.insertions.bed
and it worked like a charm, consuming only ~600M of memory and taking ~25 minutes. Amazing!
Maybe Alex Reynolds can add this as a tip in the manual?
Good that this thing worked.Parallel default block size is 1MB (as I understand) and this block or vcf lines (some where between million to 10 million) could have been bigger.
$ parallel --pipepart -a huge.vcf --block -10 convert2bed -i vcf > huge.bed or $ parallel --pipepart -a huge.vcf --block 10M convert2bed -i vcf > huge.bed
I would suggest using --do-not-sort instead of --max-mem. This skips sorting, which otherwise uses system memory (though it shouldn't here, because of the use of --max-mem).
If skipping sorting works, please let me know as I'd like to investigate further, if there's a problem.
You should definitely update your BEDOPS installation to v2.4.29.
There have been fixes to convert2bed between what you're running and the current version. There were VCF-specific fixes in v2.4.21 and v2.4.24, for example: http://bedops.readthedocs.io/en/latest/content/revision-history.html#v2-4-21
OK, there's some inconsistency. I downloaded using homebrew on my MBP, and it's 2.4.29 (typical). However, the github dowload link gives me a 2.4.20 version on unpacking.
Also, the version flag mentioned in the docs is -w, not -v, although the latter does show usage with version when it errors out. Plus, the -w only works as expected with convert2bed, not with bedops or vcf2bed. -v just errors out and shows usage (that also has version number in it). -v always exits with a non-0 exit code, while convert2bed -w exits with a 0.
I'll investigate the github download. That should provide 2.4.29 binaries, not 2.4.20.
I wasn't able to reproduce what you saw:
$ cd /tmp
$ wget https://github.com/bedops/bedops/releases/download/v2.4.29/bedops_linux_x86_64-v2.4.29.tar.bz2
...
$ tar xvf bedops_linux_x86_64-v2.4.29.tar.bz2
...
$ ./convert2bed --version
convert2bed
version: 2.4.29 (typical)
author: Alex Reynolds
The idea is to do things in the "Unix way" as much as possible, and relying on the PATH is one reliable and consistent way to find binaries. I can't guarantee I'll use it, but I'll look at your approach and see if it can work for us.
Vcf2bed is a wrapper to call convert2bed. So you could use the absolute path to vcf2bed, and it would call the old convert2bed found in your environment PATH. Just install BEDOPS wherever you have 2.4.20 installed.
See if this helps: vcf2bed argument in bedops partially process vcf file
This is also a good idea. If --max-mem is used, the /tmp directory can still fill up with temporary files used for a merge sort.
Using --sort-tmpdir with an alternative directory that can hold ~27G avoids this problem.
As VCF files can be much larger than /tmp, I may change defaults in vcf2bed to require specifying an alternative sort directory, or the use of --do-not-sort to skip sorting.
Log in to answer this question.
Sounds like giving it 72G of RAM might work, Ram.
...Sorry, couldn't pass it up.
Seriously though, the
vcf2bedandconvert2bedcommands don't have a ton of options. Alex could probably give you a solution though, dude's a wizard. Alternatively, you could filter based on the variant type in the VCF to create a file of only the INDELs, which I'm sure would cut down your file to probably only a few gigs.vcf2bedshould be able to easily handle that file then.