This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Tool: Benchmarked nf-core/rnaseq across AWS HealthOmics, AWS Batch (on-demand + spot), and our own Nextflow executor — methodology and numbers

Disclosure up front: I work on Carolina Cloud, one of the four platforms below. Methodology and raw data are public and I would genuinely rather you find the holes than not.

Setup

  • nf-core/rnaseq 3.22, test_ful` profile
  • 8 human RNA-seq samples (123.3 GiB gzipped FASTQ)
  • GRCh37, pre-built STAR index
  • 128 concurrent vCPU cap on every platform

Measured wall-clock time and total cost for one complete run Storage costs excluded everywhere (too retention-dependent to compare fairly) Results

  • Batch (spot) 7–24 h under $10
  • nf-ccloud 4.5 h $19.26
  • HealthOmics 5.5 h $45
  • Batch (on-demand) 7–8 h ~$70

Spot wins on cost and I'm not going to pretend otherwise. If cost is the only thing you're optimizing, use Batch on spot. The catch is variance — our slowest spot run took 24h because the instance types we requested weren't available. Whether that matters depends entirely on whether anything downstream is waiting on the run.

The Sentieon caveat — read this before trusting our number. Our run used Sentieon-accelerated STAR. The AWS runs used open-source STAR. Our reasoning was "best available with no extra configuration on each platform," since Sentieon is integrated on our side and sits behind a Marketplace listing on AWS's. But it is not an apples-to-apples aligner comparison and you should weight it accordingly.

Hidden line items we hit

HealthOmics requires pre-populating ECR with container images: $0.04/day, never surfaced during setup VPC: $0.13/day Inter-region egress at $0.02/GiB makes chasing spot capacity across regions impractical at 123 GiB — you pay the transfer to reach the cheap instances Where the wall-clock gap comes from: mostly task orchestration, not compute. nf-core/rnaseq is 300+ tasks and many are short. AWS bills a 60-second minimum per instance, so short tasks pay a floor they never use. Our spin-up/spin-down measured ~47x faster, which compounds across that many tasks.

Attributing cost was its own project. Cost Explorer wasn't granular enough to isolate a single pipeline run, so we limited ourselves to one run per day and avoided every other AWS service in the account just to get a clean number.

Full methodology and raw data: https://github.com/Ddfulton/nextflow-benchmark-appendix

nextflow

Thanks for sharing these numbers. Interesting. Now, when using a quantifier like salmon, the same job could probably be done in a few hours on an iMac. I always wondered whether the results would differ substantially.

0 answers

No answers yet.

Log in to answer this question.