HISAT index incomplete even with HUGE RAM

Question

[emcc]

Hello,

I'm hoping someone can help. I'm attempting to create index files for a species not available on the pre-made list and continually getting errors associated with memory. I have allocated 480GB RAM (for a 1.3GB genome) on a HPC cluster and it produces 6 of the 8 ht2 files (file 5 and 6 missing) and all of the .rf files, which I realise are temporary and I should not see. I have successfully used this command with sample data. I can successfully create the splice sites and exons files (I do need this to be annotated). And the really frustrating thing is, a colleague can successfully finish this job using a lower memory allocation (and I have tried their exact script with my file names). I'm hoping there is something simple I am missing. Any help would be greatly appreciated.

The script I submit looks like this:

#!/bin/bash
#$ -N HS_Index
#$ -o /mnt/scratch/users/*******/file/jobs/create_HISAT2_index.$JOB_ID
#$ -M [e-mail] -m bea

#request 3 nodes with 160GB RAM each
#$ -pe mpinodes-verbose 3

#load hisat2 module environment
module load hisat2

# set working directory to scratch space project folder
#$ -wd /mnt/scratch/users/*******/file

hisat2-build -p 8 --ss genome.ss --exon genome.exon genome.fa genome_tran

The errors I get often look like this:

Settings:
  Output files: "genome_tran.*.ht2"
  Line rate: 7 (line is 128 bytes)
  Lines per side: 1 (side is 128 bytes)
  Offset rate: 4 (one in 16)
  FTable chars: 10
  Strings: unpacked
  Local offset rate: 3 (one in 8)
  Local fTable chars: 6
  Local sequence length: 57344
  Local sequence overlap between two consecutive indexes: 1024
  Endianness: little
  Actual local endianness: little
  Sanity checking: disabled
  Assertions: disabled
  Random seed: 0
  Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
  genome.fa
Reading reference sizes
  Time reading reference sizes: 00:00:10
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
  Time to join reference sequences: 00:00:05
  Time to read SNPs and splice sites: 00:00:07
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
Could not find approrpiate bmax/dcv settings for building this index.
Switching to a packed string representation.
Total time for call to driver() for forward index: 01:00:28
==================================================
SGE job completed on 
==================================================

[WouterDeCoster]

I added code markup to your post for increased readability. You can do this by selecting the text and clicking the 101010 button. When you compose or edit a post that button is in your toolbar, see image below:

In addition, I changed this post to a "question" (rather than a "forum").

[Devon Ryan]

Requesting 3 nodes is a waste of 2 of them. Like most programs, hisat2 can only make use of a single node.

Answer 1

[Devon Ryan]

You're requesting 3 cores, not 3 full nodes (not that hisat2 or any other common bioinformatics tool could use more than one node). Whatever your cluster defaults to for per-process memory is insufficient. Not being an SGE expert, I would presume the following would be more reasonable settings:

#$ -pe smp 8
#$ -l h_vmem=3G

That should result in 24GB of memory and 8 cores, if I remember SGE syntax correctly. You can, of course, increase the memory allocation up to what's available on a single node...though that'd be over-kill.

[i.sudbery]

On every SGE system I have worked on specifying h_vmem does not limit memory usage for the task, rather it tells the queue controller not to kill the job for using too much memory.

That is if you specify h_vmem=24GB using 25GB will not result in an out of memory error (as long as the memory exists on the node), rather your job will be terminated by the controller and the script would not return SGE job completed

My guess is that some other memory hungry job is running on the node requested, potentially using more memory that it requested (not all SGE installs actually enforce memory limits). As far as SGE is concerned the job is using less memory than the default max mem, but when the job actually comes to try and use it, it is not free.

[Devon Ryan]

Could be, I assumed the SGE implementation in use here was somehow actually limiting the memory availability, but your explanation seems more likely (we have our slurm controller set to kill jobs that use more memory than they request to prevent this sort of thing).

[emcc]

Thank you Devon, I didn't realise about the nodes, I was kind of "throwing the kitchen sink at it" by requesting as much processing memory as possible as various other attempts have not worked.

I have used your exact instructions, and:

   #$ -pe smp-verbose 8
    #$ -l h_vmem=64G

And further variations of this to request more RAM, but I still get the same error message and an incomplete file set.

[i.sudbery]

Two things about your SGE spec there:

1) You are asking for 64GB per core and 8 cores. This means you are asking for 512GB. However, 2) you are also asking for this as virtual memory rather than physical memory, and HISAT can't use virtual memory.

Couple of things you might like to try: adding a free -g before you run HISAT - this should tell you how much physical memory is available on the node. Secondly add a ulimit -a, this will tell you if the OS is limiting the amount of RAM you can access.

If you need to increase the amount of physical RAM available, you need to ask your local SGE admins how to do this, I think it differs from system to system.

[emcc]

Thank you

I needed to request access to a 1TB node and the settings below then worked. Thanks again for your help :-)

>     > #$ -pe smp-verbose 8
>     > #$ -l h_vmem=125G
>     > #$ -l himem=true