Try to find a tool for creating a pipeline with PBS scripts?

Question

[longsong]

I have been doing some next-gen data analysis. I often need to do trimming, mapping, calling vcf etc. for genome sequencing data with PBS script on an HPC.

However, it is tedious to submit several jobs manually one by one.

As far as I know, make can not be used with PBS scripts.

What I need is a kind of tool like make, and the tool could be used with PBS scripts. If I have this kind of tool, I could type one command to do the whole data analysis.

Anyone has good suggestions?

Thanks!

Answer 1

[h.mon]

It is possible to use make with PBS, one just needs some creativity - see gnu-make-job-scheduler. But you probably want a workflow manager such as CWL, Nextflow or Snakemake - there are many others, but I think those are the "most famous".

[Robert W. Link]

I agree with h.mon. I've been using Snakemake, and it's been easy to scale input quantity

[longsong]

Thanks!

I'll check these tools.