This is a test version of Biostars. For the public version, visit https://www.biostars.org.
snakemake - Dynamic memory allocation using a workflow profile

Hey everyone,

I am looking for a way to set dynamic resources without having access to the rules itself. This workflow uses snakedeploy. So you only have the main and pretty simple Snakefile locally, but the other rules get downloaded on the fly from the GitHub repo.

I often set my resources dynamically with funtions like this:

def bwa_mem_mem(wildcards, attempt):
    possibilities = [16384, 32768, 65536, 131072, 262144, 524288]
    return possibilities[attempt - 1]

However, with the snakedeploy routine, I don't have direct access to the rules, but I assign the resources using a workflow-profile, which is a YAML file.

...
set-resources:
  bwa_mem:
    mem: 16384
...

Does anyone know, how I can define a function within a workflow profile, so I don't have to clone or fork the whole repo to change the rules itself?

snakemake
• 499 views • link
written by gernophil • gernophil

Seems like I can just to

set-resources:
  bwa_mem:
    mem: [16384, 32768, 65536, 131072, 262144, 524288][attempt - 1] # 2 ** (attempt + 3) * 1024

Sorry, haven't thought of that (or didn't know).

• 0 views • link
written by gernophil • gernophil

The following will prevent index errors after 6 retries, limiting max memory to 524288 for further attempts.

mem: min(2 ** (attempt + 3) * 1024, 524288)
• 0 views • link
written by Arup Ghosh Arup Ghosh

No, it was a formatting issue.

• 0 views • link
written by Arup Ghosh Arup Ghosh

I tried a lambda function but this didn't work:

mem_mb: (lambda x: x[min(attempt - 1, len(x) - 1)])([44000, 66000])

This failed, because it couldn't be parsed. I guess due to the ":". Any ideas about that?

snakemake: error: Couldn't parse config file: mapping values are not allowed here
  in "workflow_profile/profile.yaml", line 48, column 22
Note that certain characters like colons have a special meaning in YAML and hence keys or values containing them have to be quoted.

EDIT: Also this didn't work:

mem_mb: [44000, 66000][min(attempt - 1, len([44000, 66000]) - 1)]

due to this

snakemake: error: Couldn't parse config file: while parsing a block mapping
  in "workflow_profile/profile.yaml", line 48, column 5
expected <block end>, but found '['
  in "workflow_profile/profile.yaml", line 48, column 27

EDIT2: Even this didn't work:

mem_mb: [44000, 66000][attempt - 1]

due to the same error:

snakemake: error: Couldn't parse config file: while parsing a block mapping
  in "workflow_profile/profile.yaml", line 48, column 5
expected <block end>, but found '['
  in "workflow_profile/profile.yaml", line 48, column 27
• 0 views • link
written by gernophil • gernophil

Yes, but that's just an example. This time I want 44000MB in the first and 66000MB in the second attempt. That's just easier to accomplish with list indexing. I also tried escaping the : and [] with backslashes, but that didn't help, right now I try quotes, but I fear these will be fowarded as string and not as function to snakemake, but let's see.

• 0 views • link
written by gernophil • gernophil

Can you try this in the workflow profile?

set-resources:
  map_reads_bwa:
    mem_mb: "44000 if attempt == 1 else 66000"
• 0 views • link
written by Arup Ghosh Arup Ghosh

1 answer

That might also work, but I prefer it less "hard coded". However, it seems you can simply quote the functions and they are still evaluated by Snakemake. This did work and also set the correct resources:

mem_mb: "(lambda x: x[min(attempt - 1, len(x) - 1)])([44000, 66000])"
• 0 views • link
written by gernophil • gernophil

Log in to answer this question.

More posts like this