This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Runtime expectations and alternatives for vsearch --usearch_global on large nucleotide FASTA datasets

I'm using vsearch --usearch_global to identify high-identity overlaps between two large nucleotide FASTA files and would like to understand whether my observed runtimes are reasonable and whether more efficient approaches exist.

Dataset

  • Query FASTA: ~2 to 3 million nucleotide sequences
  • Database FASTA: ~2 to 3 million nucleotide sequences
  • Sequence length: >=300 bp (after length filtering)
  • Dereplication: performed upstream
  • Identity threshold: 99%

Command used

vsearch \
  --usearch_global query.fasta \
  --db db.fasta \
  --id 0.99 \
  --maxaccepts 1 \
  --maxrejects 0 \
  --threads 32 \
  --blast6out hits.tsv

I'm running on a Linux-based high-performance computing system managed by SLURM, using a single compute node with up to 48 CPU cores and 64 GB of RAM available. The job has been running for >24 hours, with a continuously growing BLAST6 output file.

My questions are as follows:

  1. Is this runtime expected for vsearch --usearch_global given: millions of nucleotide sequences; >=300 bp length; 99% identity threshold?
  2. Are there more efficient alternatives for this task?
  3. What are the main performance bottlenecks at this scale (CPU, memory)?
  4. My understanding is that to find the shared sequences between two FASTA files, I have to run it in both directions (i.e., use each sample as query and db, then find overlaps). Is that necessary?
ngs performance vsearch sequence-alignment immune-repertoire
• 1,040 views • link
updated by dthorbur dthorbur • written by emmzee • emmzee

1 answer

Is this runtime expected for vsearch --usearch_global given: millions of nucleotide sequences; >=300 bp length; 99% identity threshold?

Absolutely. You are performing a 2-3m by 2-3m sequence search. This is going to take a long time using this approach. To my knowledge it's not exhaustive, and you can probably slightly decrease runtime by adding some more parameters.

Are there more efficient alternatives for this task?

Yes, a few that come to mind.

  1. Use a more efficient tool, especially if you can get one that is GPU accelerated. MMseqs2 is the gold standard in my opinion, and there is even a GPU accelerated version available.
  2. Clustering your data first, then performing local searches only within clusters. You could even do this first with vsearch, but again, I would probably switch to MMseqs2 for this task since it is considerably quicker in my experience (including the CPU version)
  3. You have access to a HPC with a job scheduler. Chunk the input into more manageable sizes (e.g., 10k-20k sequences) and run them across an array. This then decreases the likelihood of failure if any single job fails, and is easier to identify problematic sequences if they exist. Also quicker queue times since each job can request less resources.
  4. Read the vsearch documentation since there are many paramaters you could add that would make minor improvements to runtime.

What are the main performance bottlenecks at this scale (CPU, memory)?

Almost certainly CPU. Depending on HPC setup, you should be able to monitor usage - though most only offer diagnostics after the job has finished without including your own loggers.

My understanding is that to find the shared sequences between two FASTA files, I have to run it in both directions (i.e., use each sample as query and db, then find overlaps). Is that necessary?

No, this command finds sequences with a 99% similarity between the two files. It shouldn't matter which file is query and which is the database. The output should have query_id and target_id columns.

• 0 views • link
written by dthorbur dthorbur

Log in to answer this question.

More posts like this