Bottleneck for scattered subworkflows

Hi I have a question regarding when steps are able to run in parallel. We noticed that sometimes all of the jobs from a scatter must complete before moving to the next

I was wondering whether a scattered subworkflow can start before all the inputs to that step are available, or does a subworkflow have to wait for all of the previous inputs to be available regardless of whether they are available for just the first set of scattered inputs?

cwl toil

• 1,765 views • link

updated 7.9 years ago by Carlos Borroto

• written 7.9 years ago by ionox0

1 answer

All inputs for the step have to be available. See this, look for bullet point "Instead of scattering separate steps, prefer to scatter over a subworkflow."

• 1 views • link

written 7.9 years ago by Carlos Borroto

More posts like this

Conventions for designing flowcharts for bioinformatics workflows
written 1.4 years ago by kalavattam •

What are widely followed conventions for designing flowcharts in bioinformatics workflows or pipelines? For example, I often see inputs and outputs represented as parallelograms, processes …
downloading the VCF files of CCLE project (or cancer genomics WGS)
written 3.6 years ago by Bogdan

Dear all, has anyone any suggestions regarding the resource (database, webserver) that I can use in order to download the VCF files produced by CCLE …
The memory is not enough when run cellranger to align the fq using the two parameters --localcores…
written 5.1 years ago by whaiyu06 •

Hello, I met a problem when I run the cellranger (cellranger-6.0.2) to align the fq files to ref genome on the supercomputer. I used the …
possible to show job names in toil stats command using cwl workflow
written 7.8 years ago by ionox0

I'm using toil 3.15.0 with a cwl workflow, and I was wondering if there was a way to have the `toil stats` command list the …
Possible to combine inputs using valueFrom for scatter
written 7.9 years ago by ionox0

Is it possible to combine multiple inputs from another step using `valueFrom` so that they can be scattered over? For example something like: inputs: input_one: …
How do I specify the elements of an array of output files?
written 8.2 years ago by biokcb

Hi - I would like to have a workflow with the following set up: **Step 1:** creates an array of N files with specific naming …
Can I run the EBI metagenomics pipeline locally?
written 8.3 years ago by alesssia

Hello everyone, I was wondering whether the EBI metagenomics pipeline ( https://www.ebi.ac.uk/metagenomics/ ) could be run locally. Here https://emg-docs.readthedocs.io/en/latest/_images/pipeline_v4.1b_overview.png there is an overview of the …
More concise way to pass many parameters through to subworkflows
written 8.4 years ago by ionox0

I have a lot of parameters in a pipeline, and would like to pass them down through several layers of subworkflows so the system is …
Conflicting errors with "baseCommand" vs "arguments" usage
written 8.4 years ago by ionox0

When I include all of my static arguments in the `baseCommand` section I get an IOError that the filename for the log file is too …
Split up record input when using scatter/gather
written 9.6 years ago by psaffrey

I am building a CWL workflow that takes an array of record types as an input, in my case a list of pairs where each …