This is a test version of Biostars. For the public version, visit https://www.biostars.org.
snakemake job status monitoring

Hi everyone, I'm writing the simple linear snakemake pipeline on multiple samples.

pipe

I'm runnign snakemake with -k parameters, so failing in one branch does not cause to fail whole pipeline. But I'd like also to have a meaningfull info (possibly via emails) on the pipeline status (which joabs are done, and which are failed). There is a --log-handler-script parameter which supposetly should give me necessary functioonality. BUT, the msg's produced by sankemake rather uninformative:

log

as one might see the one of the job failed but msg (i just printed it to terminal from the def log_handler(msg):) does not tell me which step went wrong. And I do not see how I can get the jobID of failed job before I the pipeline is over. Any suggestions are appreciated

Best, Eugene

snakemake pipelines
• 4,006 views • link
written by Eugene A Eugene A

if something crashed it is usually pretty easy to figure out what happened but if a task exited with a normal status code but a file is missing then yes it won't be super helpful

• 0 views • link
written by Jeremy Leipzig Jeremy Leipzig

It is pritty easy to figure out what happend in manual examination. My task was to automatically detect failed rules and report them (via emails or something like that), while the rest of a pipeline goes on (-k option). And _automatic_ detection of failed jobs is surprisengly inconvenientin snakemake

• 0 views • link
written by Eugene A Eugene A

I assume this something you may have already added, but to be sure, do you have the log directives as part of your rules? https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html?highlight=log%20directive#step-5-logging

This allows you to see errors more specifically linked to the jobs submitted.

As for emails I find the following helpful: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?highlight=onstart#onstart-onsuccess-and-onerror-handlers

• 0 views • link
written by lianov • lianov

2 answers

after some more search I've figured that there is other option for monitoring appart from --log-handler-script, namely panoptes server (https://github.com/panoptes-organization/panoptes, https://snakemake.readthedocs.io/en/stable/executing/monitoring.html). It also has some problems (failed jobs currently displayed as running, so does not solve initial issue), but will suite my purpuses for now.

• 0 views • link
written by Eugene A Eugene A

Some additions, which might be helpfull for someone - it seems that the only way to detect failed jobs in snakemake and automatically keep track of them is to intercept exit codes of the used commands and processe it separetly

• 0 views • link
written by Eugene A Eugene A

Log in to answer this question.

More posts like this