Snakemake rule to copy files
As part of a pipeline, I am trying to create a rule which takes a file containing file paths to gVCF files and copies them to a compute node. I would like to have all output files defined in the output of the rule, so they will get delete when I mark them with temp(). Currently I am doing the following:
rule CopyToNode:
input:
gVCF_list
output:
gVCFs_copy_node
threads:
2
shell:
"""
cat {input} | xargs -n1 -P{threads} -I% rsync -avh % {SCRATCH_DIR}
awk -F/ '{{print $NF}}' {input} | awk '{{printf "{SCRATCH_DIR}%s\\n", $0}}' > {output}
"""
This however doesn't actually have the file names in the rule output (the output is just a file containing the file path).
• 1,316 views
•
link
2 answers
Turns out there is a solution to this problem with Snakemake 8.
- Install newest version of Snakemake (8.16.0 at this date)
- Install snakemake-storage-plugin-fs
- Then you can define remote and local storage prefixes like this:
snakemake --cores 16 --default-storage-provider fs --shared-fs-usage persistence software-deployment sources source-cache --local-storage-prefix /scratch/ --remote-job-local-storage-prefix /scratch/
I still need to figure out what all of these options mean, but it seems to work.
• 0 views
•
link
I'm not sure of what you're trying to do , may be you want:
awk -F/ '{{print $NF}}' {input} | while read F; do cp -v "$F" {output} ; done
• 0 views
•
link
Log in to answer this question.
If I understand well, you wish to copy files on a node and flag the copied files as temp? Not sure you can do this, the desired output files would be remote there. Snakemake does not support this I think.