This is a test version of Biostars. For the public version, visit https://www.biostars.org.
ENCODE pipelines vs Nextflow/Snakemake: design choices and use cases

I recently completed a course in which students built Nextflow pipelines from scratch, and there was a strong emphasis on reproducibility. While preparing for the class, I looked into the ENCODE pipelines and their DAGs to see how similar problems are addressed there. From what I can tell, ENCODE pipelines are largely assembled using shell scripts, with dependencies defined explicitly rather than through a workflow DSL. Since shell-based steps can be wrapped within Nextflow or Snakemake, I am curious about the motivation for this design.

In particular, I would like to understand the intended use cases for running ENCODE pipelines as provided rather than reimplementing or wrapping them in a workflow manager, and whether these choices are driven by historical reasons, infrastructure constraints, or ENCODE-specific requirements such as standardization and large-scale production. More generally, how does ENCODE handle reproducibility and portability, and how does this compare to modern workflow systems?

pipelines encode nextflow snakemake
• 1,896 views • link
written by DdogBoss • DdogBoss

when people shoot from the hip their questions have no premise in reality

lmgt

• 0 views • link
written by Jeremy Leipzig Jeremy Leipzig

I was not familiar with that. Thanks for sharing.

• 0 views • link
written by DdogBoss • DdogBoss

The ATAC-seq pipeline was the one I was looking at.

• 0 views • link
written by DdogBoss • DdogBoss

2 answers

Since shell-based steps can be wrapped within Nextflow or Snakemake, I am curious about the motivation for this design.

It's difficult to answer this without asking someone senior in ENCODE. I suspect the main consideration was timing. Workflow managers like snakemake and nextflow weren't around or mainstream during the initial phases of the project.

More generally, how does ENCODE handle reproducibility and portability, and how does this compare to modern workflow systems?

I don't know what you mean by portability? They are designed only for UNIX, so these aren't portable to other operating systems. They also didn't generally use environment managers like conda or containerisation through docker/singularity, but rather enforced strict version requirements on stable pipelines. Again, this was likely a decision because environment managers weren't mainstream during the core ENCODE development.

What ENCODE means by reproducibility is dataset-level reproducibility. You feed the pipeline the same dataset on different machines, and it should produce the same results. This is preserved so long as the tool versions are still available and maintained. Some of the ENCODE github repos see contemporary updates, but with github's version control you retain that reproducibility. Just use the same version hash as reported in a methods and you should be able to reproduce results.

• 0 views • link
written by dthorbur dthorbur

From what I can tell, ENCODE pipelines are largely assembled using shell scripts, with dependencies defined explicitly rather than through a workflow DSL

that's not true - ENCODE Data Coordination Center (DCC) is/was a WDL shop and also pioneered a number of JSON-LD based methods for metadata management. WDL is a configuration-based description language, rather than an inline DSL, but definitely not a shell script.

https://github.com/ENCODE-DCC

Fun fact: more genomes have been processed with WDL pipelines (between Broad, ENCODE, and others) than Snakemake and Nextflow combined.

disclaimer: I was the bfx person at a portal that served a number of ENCODE-built pipelines

• 0 views • link
written by Jeremy Leipzig Jeremy Leipzig

that's not true - ENCODE Data Coordination Center (DCC) is/was a WDL shop and also pioneered a number of JSON-LD based methods for metadata management. WDL is a configuration-based description language, rather than an inline DSL, but definitely not a shell script.

I didn't know that.

Fun fact: more genomes have been processed with WDL pipelines (between Broad, ENCODE, and others) than Snakemake and Nextflow combined.

Neat.

• 0 views • link
written by DdogBoss • DdogBoss

Log in to answer this question.

More posts like this