I need to download hundreds of large files, and run each of them through my snakemake pipeline. The file download is fast compared to my downstream pipeline. I'd like to limit the number of parallel downloads to 5, but allow the downsteam processing to use 100 cores. In snakemake, is there a way to limit the number of cores used by a certain rule? I picture 5 cores constantly grabbing data, while my other cores are working on the data I've already downloaded. If I run snakemake as usual with 100 cores, it will try to download all files at once, and overload the server.
•
written
by
samesense •
Is this what you're looking for: https://bitbucket.org/johanneskoester/snakemake/wiki/Documentation#markdown-header-threads?
Otherwise, you can try posting in the official Snakemake Google group: https://groups.google.com/forum/#!forum/Snakemake The author is pretty responsive.
•
written
by
maciek.kandula •
I saw that too. I think it would multithread my download, but wait on the rule to finish before moving onto the downstream pipeline. I'll ask the user group.
•
written
by
samesense •
Log in to answer this question.
I tried setting a download resource, according to this answer
This did not limit the downloads as expected.