This is a test version of Biostars. For the public version, visit https://www.biostars.org.
vg index memory allocation

I'm attempting to call variants using a reference graph I've generated first with minigraph ./minigraph -xggs -t16 f1.fa f2.fa f3.fa f4.fa f5.fa > PG_1.gfa and then vg./vg convert -g -a PG_1.gfa > PG_2.vg.

When I try to index, I'm prompted to modify the length of the nodes to 256 ./vg mod -X 256 PG_2.vg > PG_3.vg but it stalls out when I use this to index ./vg index -x -p PG_3.xg -g PG_3.gcsa PG_3.vg

exiting at

Building XG index Saving XG index to Ldec_vg.xg Generating kmer files... Building the GCSA2 index... InputGraph::InputGraph(): 2193921420 kmers in 1 file(s)

`

The PG_3.vg input is 1.3G in size, how much memory and CPUs should I be using to generate an index and call variants using fastq reads approximately 1.5-2GB in size?

vg reference graph

2 answers

Assuming that the graph is not too complex locally (in a 256 bp window), ~2 billion initial kmers in a single graph file should require 100-200 GB memory and 200-300 GB disk space in $TMPDIR.

GCSA construction uses a semi-external algorithm that works best when the graph is partitioned (e.g. by chromosome) into multiple .vg files. It can then reduce the memory usage significantly by loading kmers from one graph file at a time.

Excellent, thank you! The first graph I generated, using a minigraph gfa, was incompatible due to missing fields (P). I'm roughly following the pggb pipeline (minimap2 -> seqwish -> vg) now so hopeful this will work.

For the first attempt I used vg prune according to your index construction on the vg wiki. So, with this new, and probably larger graph, should I construct using the -r argument, and then prune?

I'm also unsure how to break up the large graph by chromosome as 1) I'm not working with vcf's rather gfa and 2) they're of varying contiguity/quality with different scaffold names.

Will the vg construct find the reference scaffolds with the -r fasta file or do I have to input them as a variable like you did in your example (e.g seq 1 22) and will this work on a gfa? Thank you!

This should have been a comment on Jouni's answer. Please do not add answers unless you're actually answering the top level post.

Follow these steps to add this post as a comment:

  1. Copy the contents of your reply from this answer (you can edit this answer (Ctrl/Cmd + click the link to open it in a new tab) and do a Select All -> Copy there).
  2. Click on Add Comment on Jouni's post here: A: vg index memory allocation
  3. Paste the copied text
  4. Click on the green Add Comment button
  5. Click on moderate back in your answer here: A: vg index memory allocation
  6. Choose Delete Post
  7. Click on the blue Submit button.

Log in to answer this question.