Hi, I'm running a nextflow pipeline and I have a limit of jobs i can submit to the cluster, however nextflow submits jobs for different sections of a pipeline as soon as the required files are available. Is there a way to tell it to fits submit all the tasks for a section (e.g. aligning) without trying to run the next step right away?
Thanks
Please read the question and answer what is asked. I don't need any notes on what i'm doing.
1 answer
workflow {
mulitple_ch = ....
step_ch = MANY(mulitple_ch)
WAIT_FOR_MANY(step_ch.output.collect())
}
@Raygozak Please be nice to the people who have provided you with answers and have some humility. I understand exactly what they mean, and have learned something here and or already use this pattern in my pipelines. If you do not, then it is not their fault, but yours.
Log in to answer this question.
-
Forum: Nextflow and self-made pipelines opinion
written by karlaarz •
-
Anyone know any clever snakemake/SLURM tricks to run a big analysis with limited storage?
written by steel1990 •
-
Choosing the right sbatch parameters
written by drabiza1 •
-
Resources for learning how to submit jobs in unix-based cluster
written by dk0319 •
-
How to set AWS_BATCH_JOB_ATTEMPT value when using curl to submit cromwell jobs to batch?
written by alexandriapinto1 •
-
News: Nextflow Training Course
written by Physalia-courses
-
CWL limit number of job instances created through scatter
written by silverspanch •
-
PASTEC repeat classification software (part of REPET) help needed
written by mcsimenc •
-
Errors attempting to start OMA job on SLURM-managed cluster
written by Adamc
-
Generating shell scripts
written by ebrown1955
You can make each alignment process emit a value, then gather/concat this value from all alignment jobs into a new value/variable and let the downstream process read that new one as input. It's not actually used by the downstream process for any operation other than just serving as a serialization instance. The new processes won't start until all alignmrnt jobs are finished. Not aware of an in-built method.
this doesn't answer the question asked, nor is related in the least.
hum... yes it is. My answer below is an illustration of what ATpoint said.
It answers the question exactly, and I would appreciate a different tone. Consider paying a consultant in case a free online community is not serving you appropriately.
Why did you delete this post, Raygozak?