This is a test version of Biostars. For the public version, visit https://www.biostars.org.
"Memory Used" output for various assemblers is greater than total memory that physically exists on compute node

Greetings BioStars,

Our SLURM log output for a successful co-assembly with MegaHit includes the following:

2026-08-06 17:53:53 - b'INFO  utils/utils.h:  152 - Real: 577.4736\tuser: 316.8872\tsys: 67.0934\tmaxrss: 257052'
2026-08-06 17:53:53 - k-max reset to: 127 
2026-08-06 17:53:53 - Start assembly. Number of CPU threads 8 
2026-08-06 17:53:53 - k list: 27,37,47,57,67,77,87,97,107,117,127 
2026-08-06 17:53:53 - Memory used: 486634136371

This SLURM job ran on node 29, which has a maximum of only 453 GB of RAM total (according to free_res).

MegaHit cannot be using 486 GB, like it says in the log output above, because that amount of RAM does not physically exist on this node. I don't think it could represent an estimate of RAM to be used either, because it hasn't counted kmers for our reads yet, according to the log output. The SLURM sacct outputs for this completed job show MaxRSS was 106 GB, MaxVM size 107 GB. It seems unlikely that 486 GB is cumulative RAM either (e.g. 100 + 100 + 100 + 186, at different times). The RequestedMem was 256 GB.

I noticed the same with other assemblers (MetaHipMer2): the software is claiming more memory than exists on the node, at the beginning of the job. "Initial free memory across all 1 nodes: 472.06GB (472.06GB avg, 472.06GB min, 472.06GB max)" per node. Again, not physically possible on that node.

What is the correct interpretation of these memory statements in these logs?

/////////////////////

The reason I ask is that, in trying to get various assemblers to work on our (larger and larger) datasets, it frequently comes down to some memory issue. Things get murky for our team between the Linux OS, SLURM, and the assembler software itself, in how memory is detected and allocated. For example, Slurm controls memory through the Linux cgroup functionality. I verified that our ConstrainRAMSpace setting is active in our cgroup.conf, which should enable SLURM to limit the amount of RAM provided during a job. But, it seems to make no difference: we still get these statements like above where the software is claiming more RAM than exists on the node.

We are at a disadvantage in troubleshooting memory issues, if we do not understand log outputs regarding memory for successful runs such as this.

Thanks very much for any additional information you could provide.

memory cgroup slurm linux megahit
• 350 views • link
written by GallopingAvocado 3

Thanks v much Arup!

def detect_available_mem():
    try:
        psize = os.sysconf('SC_PAGE_SIZE')
        pcount = os.sysconf('SC_PHYS_PAGES')
        if psize < 0 or pcount < 0:
            raise SystemError
        return psize * pcount
    except ValueError:
        if sys.platform.find("darwin") != -1:
            return int(float(os.popen("sysctl hw.memsize").readlines()[0].split()[1]))
        elif sys.platform.find("linux") != -1:
            return int(float(os.popen("free").readlines()[1].split()[1]) * 1024)
        else:
            raise

So, now I can learn more about these OS vars, and why they are more than the physical memory on the node, as reported by free_res.

It still seems like the OS and then SLURM should take precedence in allocating compute resources such as RAM. We're still working on the mystery of why the ConstrainRAMSpace=yes in our cgroup.conf is not successfully enabling our OS to limit the amount of RAM claimed by the assembler script in a SLURM job (should have been 256 GB RAM above), regardless of whether or not memory flags are set in the assembler scripts themselves.

Or a better way to visualize / understand when and how the assembler scripts are interacting with the SLURM vars such as mem-per-cpu. I will try to dig through the source code some more as well.

• 0 views • link
written by GallopingAvocado 3

SLURM sacct outputs for this completed job show MaxRSS was 106 GB, MaxVM size 107 GB.

Is your SLURM config setup to allow a main job to spawn subjobs? It is possible that assemblers are spawning sub-jobs and then accounting for the memory they are using that to come up with the larger number.

Since you have a finite amount of physical memory on node, there is no way to use more than what OS/SLURM allows. So the output of sacct is what I would trust. If you are asking for 256G RAM then there is no way SLURM is going to allow that main process to use more (as long as it is properly configured). You also don't say how much swap is configured on your node. That could also be counting towards the number reported.

We are at a disadvantage in troubleshooting memory issues

What kind of issues? Are some jobs failing compared to others? Different assemblers are probably managing memory differently and some programs may be better then others at cleaning up after an operation completes.

• 0 views • link
written by GenoMax 16K

1 answer

Looks like I'm learning about memory pages today:

"When combined with virtual memory, it is known as paged virtual memory. In this scheme, the operating system retrieves data from secondary storage in blocks of the same size (pages). Paging is an important part of virtual memory implementations in modern operating systems, using secondary storage to let programs exceed the size of available physical memory"

• 0 views • link
written by GallopingAvocado 3

MegaHit cannot be using 486 GB, like it says in the log output above, because that amount of RAM does not physically exist on this node.

Not to be pedantic, but 486634136371 bytes is not 486 GB. It is ~453 GB because a GB is more than 1e-9 bytes (1024^3, or almost 1.1e-9).

• 0 views • link
written by Mensur Dlakic 3.0K

Please do be pedantic, I welcome it!

That's a good point, I should make sure I know whether the units being used are Gigabytes (GB, 1E9 bytes) or Gibibytes (GiB, 1024^3 bytes).

• 0 views • link
written by GallopingAvocado 3

Log in to answer this question.

More posts like this