So far, I can describe my question only in general words, but hopefully someone can help me further narrow it down. I am trying to run modified version of GATK pipeline through cromwell in a server mode on a slurm cluster.

So far I setup a slurm cluster with one node, as a test environment and everything works reasonably well: I'm able to push several instances of my pipeline in parallel to the cromwell server and slurm creates a nice queue of tasks. The problem is, that currently I am hitting the limit of IO operations on the disk where cromwell server is running, which slows everything down:

Here is the result of 18 pipelines submission with 1h interval. As can be seen I/O operations are at it's limit restricting the overall performance.

Now I'd like to add some more nodes to Slum to deal with it. _BUT_ from what I learn so far - all slurm nodes will use the same shared disk, meaning that I will not gain any performance increase from additional computational resources. Is it correct? Or can I force slurm to use local hard drives for each node? If so, how I can configure cromwell server and nodes to correctly copy files back and force?

The question is quite vague but I thank the main idea is clear - ho to increase performance over what I have by addition more nodes to the cluster (if it is possible at all)

Best wishes, Eugene