Hi everybody,
If I have a workflow which has a step that performs a scatter over n samples, what would be the best way to spawn n EC2 instances so that each instance is responsible for the computation of a single sample?
I realize that for local execution one can specify the --parallel flag to launch n threads and each thread will be responsible for the computation of one sample. However, how do I do this on a per-instance level in AWS?
I was looking into Toil and while I couldn't get much information about their parallelization capabilities, my understanding is that Toil, when used with the autoscaling feature, will distribute the workload of a scatter step over n instances. Is my understanding correct? If not, then what options do I have?
Thanks.
•
written
by
uray10 •
One way to do this is to set one of the ResourceRequirements in CWL (e.g. coresMin) to be equal to the instance size.
For instance, you can run your workflow using: toil-cwl-runner ... --nodeTypes m4.xlarge ... and use coresMin: 4 in your CWL definition. You may optionally include --maxNodes to limit the number of instances Toil launches. If your workflow is heterogeneous where you have multiple instance types then limiting based on disk by setting outdirMin: (in MiB) to be slightly less than --nodeStorage (in GiB) [*] is also very robust.
[*] Note that Toil/Mesos use up some disk on the instance, which is why outdirMin should be slightly less than --nodeStorage.
This is a pretty good hack. Be sure to tell the underlying tool that it has access to all the cores via arguments: [ --threads=$(runtime.cores) ] or similar based on the command line options that tool has.
This is a pretty good hack. Be sure to tell the underlying tool that it has access to all the cores via arguments: [ --threads=$(runtime.cores) ] or similar based on the command line options that tool has.
Log in to answer this question.
Hello uray10,
As a reminder, we moved CWL community support from this forum to https://cwl.discourse.group/ which is why I didn't see this sooner.