This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Optimizing GROMACS MD performance and reducing queue time on HPC (Slurm cluster)

Hello everyone,

I am running molecular dynamics simulations using GROMACS (2021.x) on a Slurm-based HPC cluster with shared CPU nodes. (it's my 1st time using an HPC and I couldn't find the most optimal strategy- somethings are just learned from the get go but more through experience, that's why I am requesting an answer from you the experts and experienced, thank you soo much for your time & help)

My system is a protein–protein complex and I am running production MD (~100 ns target).

Current setup Partition: mediumq (shared CPU nodes) Node type: 48 cores total (2 × 24 cores, no hyperthreading) Current allocation: 8 CPUs per task Walltime: 24 hours

GROMACS run command:

gmx mdrun -deffnm md -ntmpi 1 -ntomp 8

Observed performance

At 8 cores on a shared node, I observe relatively slow throughput:

~0.04–0.06 ns/hour (fluctuating depending on node load) Node is often heavily shared (90–100% CPU allocation) Performance varies between runs due to node contention

So effectively, MD speed is inconsistent and slower than expected for a 48-core architecture.

Main questions

  1. Core scaling (8 vs 16 vs 24 cores) Is 8 cores too low for a 48-core node in GROMACS production MD? Would increasing to 16 or 24 cores significantly improve performance, or is scaling limited by shared-node contention? At what point does adding more cores stop being efficient for this kind of system?
  2. Slurm strategy: queue time vs performance In practice on shared clusters, does requesting more cores (e.g. 24–48) significantly increase queue time compared to 8–16 cores? Is it generally better to stay at 16 cores for faster scheduling, even if scaling is suboptimal?
  3. Exclusive nodes (--exclusive)

Would requesting:

--exclusive

meaning full node allocation, improve MD performance significantly?

Is the performance gain typically worth the increased queue time on busy clusters?

  1. Optimal configuration for my case

Given:

48-core nodes (2 sockets × 24 cores) heavy node sharing protein–protein system GROMACS 2021.x

What would be the optimal trade-off between:

core count walltime (24h vs longer) MPI/OpenMP settings (ntmpi / ntomp) scheduling strategy

Additional context

I am trying to balance:

faster MD throughput (ns/day) reasonable queue time stable production runs (~100 ns trajectories)

Right now I suspect I am underutilizing the node (8 cores), but I am unsure whether increasing cores or requesting exclusive nodes is worth the scheduling cost.

Any practical HPC/GROMACS tuning advice would be highly appreciated.

Thank you.

md hpc queue
• 511 views • link
written by DAUNTLESS • DAUNTLESS

Please include the SLURM part of the command line since that would be relevant here. Your program command line resource requests need to match the SLURM allocations.

• 0 views • link
written by GenoMax GenoMax

About the SLURM I wrote it this way (I was told not to use more than 1 node)

#!/bin/bash
#SBATCH --job-name=pp_md_100ns
#SBATCH --partition=mediumq
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --time=48:00:00
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err

set -e

module purge
module load GROMACS/2021.3-foss-2021a

echo "Loaded GROMACS:"
gmx --version

WORKDIR=/scratch/users/
• 0 views • link
updated by GenoMax GenoMax • written by DAUNTLESS • DAUNTLESS

You are not specifying memory in this request so you are likely getting the minimum value set by cluster admins. That may be one of the issues here. I would expect this kind of simulation to require tens of GB of RAM. So add an explicit --mem=NNg option.

Node is often heavily shared (90–100% CPU allocation)

While that may be so once your job starts running it should hold on to the cores it got assigned.

Right now I suspect I am underutilizing the node (8 cores)

Have you checked into that? Just throwing more cores at a task may not always result in an expected speed up since the code itself may not run efficiently when you exceed a certain number of cores. You can use commands like seff <jobid> and sacct to check on actual utilization stats.

• 0 views • link
written by GenoMax GenoMax

Thank you for the suggestions that’s very helpful (I didn't think of that).

I checked the node characteristics and runtime behavior more closely. The compute nodes have ~48 cores (2 × 24 cores), but they are heavily shared. For example, one of my jobs was running on a node with ~47/48 CPUs already allocated, which suggests significant contention.

Regarding memory: I haven’t explicitly requested --mem, but based on system size (protein–protein complex), I would expect usage in the ~5–20 GB range. I will confirm this using seff/sacct once the job finishes, but I suspect memory is not the main bottleneck here.

On the CPU side, I agree that scaling is not linear. My concern is that with only 8 cores I may be underutilizing the node, but at the same time the shared-node environment seems to limit performance regardless of core count.

At the moment I observe ~0.04–0.06 ns/hour on 8 cores, which seems lower than expected for this hardware. This might be due to node contention rather than pure scaling inefficiency.

Given this, I’m trying to decide between:

increasing to 16–24 cores (while still sharing the node), or requesting --exclusive access to a full node (longer queue time but better isolation)

Do you think, in practice, that node contention can dominate performance to this extent, and that requesting exclusive access would be worth it for GROMACS MD runs?

Also, when evaluating scaling with seff, what CPU efficiency range would you consider acceptable for this type of workload?

Thanks again for your help.

• 0 views • link
written by DAUNTLESS • DAUNTLESS

but I suspect memory is not the main bottleneck here.

Best to check on that before deciding since your job could be doing a lot of disk swapping (while it manages to run).

For example, one of my jobs was running on a node with ~47/48 CPUs already allocated, which suggests significant contention.

Not essentially. SLURM should have reserved 8 of those cores for your job as you requested. It does not matter what other cores are doing as far as your job is concerned.

increasing to 16–24 cores (while still sharing the node), or requesting --exclusive access to a full node (longer queue time but better isolation)

Since we don't know how your cluster is administered this is something you will need to try. You are already asking for the cores to remain on the same node. I guess you will need to ask for a lot more cores to get the exclusive access but if your system allows for it then no harm in trying it out.

what CPU efficiency range would you consider acceptable for this type of workload?

Not having used GROMACS I can't specifically comment. Multicore jobs may not use all cores all the time since steps in an analysis may not always be parallizable. Not having all cores pegged at all times can be a relatively normal observation.

NOTE: If your GROMACS install is compiled to use MPI/OpenMPI then you should actually be using mpirun or srun --mpi to start the run. https://docs.open-mpi.org/en/v5.0.x/launching-apps/slurm.html

• 0 views • link
written by GenoMax GenoMax

Thank you for the detailed explanation (it really helps clarify several aspects).

Regarding memory, I agree it’s important to rule out swapping. Based on the size of my system (a protein–protein complex), I would expect memory usage to be on the order of ~5–20 GB, while the nodes have ~190 GB RAM. So I suspect memory is not the limiting factor, but I will confirm this using sacct/MaxRSS once the job completes.

On the CPU side, I understand that Slurm guarantees the requested cores, so in principle my 8 cores should be reserved and not used by other jobs. However, in practice I am consistently seeing jobs placed on nodes that are almost fully allocated (e.g. ~47/48 CPUs in use with high load averages). This made me wonder whether, even with exclusive core allocation, shared resources such as last-level (L3) cache, memory bandwidth, and NUMA interconnects could still impact performance.

Since GROMACS is known to be sensitive to memory bandwidth and data locality, my hypothesis is that heavy node occupancy may reduce throughput even when cores themselves are properly allocated. This seems consistent with what I observe empirically: performance varies depending on how busy the node is, and I currently obtain ~0.08–0.10 ns/hour on 8 cores, which appears somewhat lower than expected for this type of hardware (I am losing roughly 30 to 50% in performance) .

Given this, I am trying to determine whether the main limitation is:

scaling (i.e. 8 cores being too few), or the shared-node environment (resource contention beyond CPU cores)

I am considering- but I really don't know if it's sane:

increasing to 16–24 cores while remaining on shared nodes, and testing --exclusive node allocation to see whether isolation improves throughput

Regarding MPI, I am currently running with -ntmpi 1 -ntomp N, so this should be a thread-MPI setup rather than a multi-rank MPI job. If I switch to multi-rank runs, I will test launching via srun as suggested (but from what other users of the HPC told me I could be flagged for doing it).

I will also run a small scaling test (8 vs 16 cores) and check efficiency metrics after completion to better quantify the behavior.

Thanks again for your insights, I really appreciate the guidance and time you invested in your help.

• 0 views • link
written by DAUNTLESS • DAUNTLESS

Does your cluster have GPU's? If so, you should use more recent gromacs package compiled with CUDA support if it is provided (it's hard to compile gromacs with GPU support).

• 0 views • link
written by Michael Dondrup Michael Dondrup

we don't have access to the GPUs since December, I've been emailing the IT for 3 months about it no replies :(

• 0 views • link
written by DAUNTLESS • DAUNTLESS

0 answers

No answers yet.

Log in to answer this question.

More posts like this