Hello everyone,
I am trying to run a program called HPVDetector inside the Compute Canada cluster which uses SLURM to schedule jobs. For some reason, when I simply execute the program inside the head node it works perfectly, but when I submit it as a job with sbatch or srun, it takes MUCH longer (not counting queue waiting times), and the output files are all empty (even though the program shows no errors throughout its run). I have looked absolutely everywhere for a solution or explanation but I can find none. I am new to HPC and would really appreciate any help. Thank you so much in advance.
For time duration :
Could you please check if memory and partition of your script look similar like this
SBATCH --mem=16G
Above 16G means 16 Gb memory have been allocated but if your program demand more memory replace 16G with MaxMemPerNode
SBATCH --partition=standard
Here generally we use standard but you may replace standard with fast
Note: As other people must be using that cluster at the same time, use these two parameters wisely
Regarding output file
I think there might be problem with some variable or file name. Take special care of $ symbol
Hope this helps :)
You sir are an absolute genius. I just did what you said about the memory:
SBATCH --mem=MaxMemPerNode
And it ran perfectly, actually even BETTER because it was done in under a minute. I cannot thank you enough. Thank you so much!!!
Log in to answer this question.
Might want to familiarize yourself with this: https://slurm.schedmd.com/sbatch.html
Thank you very much! This looks extremely useful and complete. Will definitely take a look.