I read the manual carefully and adjusted the memory parameters in vmop files up to 100gb RAM. This was already done and I therefor I asked the question here. So has anyone out there a helpful answer?
Hey everybody,
At the moment I try to index a bunch of sequences with malt. I've indexed all viral proteins from refseq without a problem. But now I wanted to index all bacterial ref-seq sequences at once. The file is about 9.8gb great and contains 5242 sequences. I used the command:
./malt-build -L /ifs/data/nfs_share/sukmb241/home.julian/malt/MEGAN5-academic-license.txt --input references/bacteriall_nucleotide --sequenceType DNA --index index/bacteriall_nr/
But after allocating the hash table the program spits out the following error:
Allocating: 75.6 GB
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(Unknown Source)
at java.nio.ByteBuffer.allocate(Unknown Source)
at jloda.E.H.<init>(Unknown Source)
at jloda.E.D.<init>(Unknown Source)
at malt.data.O.A(Unknown Source)
at malt.MaltBuild.run(Unknown Source)
at malt.MaltBuild.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
at com.install4j.runtime.launcher.UnixLauncher.main(Unknown Source)
I changed the -Xmx parameter in the vmoptions files as described in the manual and my last try to get this to work was with 120g but it crashed with the exact same error message as above. At the moment I have no way to increase the memory level. Has anybody experience with the indexing of greater files in Malt? Did you encounter the same problem?
With best regards and thanks in advance,
Julian
3 answers
The manual here says that maximum memory availability is set at setup time.
Quote from manual:
The most important performance-related option is the maximum amount of memory that malt-build is allowed to use. This cannot be set from within the program but rather is set during installation of the software
There also seems to be another option:
The installation dialog will ask how much memory the program may use. Please set this variable carefully. If the amount needs to be changed after installation, then this can be done by editing the files ending on vmoptions in the installation directory.
Please, always read the manual before asking on forums.
- You did not mention that you already tried this. If you did, I would not have suggested this.
- There's no need to be dismissive. Even if my answer were not relevant (which is not the case here because of point #1), you should acknowledge it better.
- Your response belongs in a comment to my answer, not an answer of its own. I've moved it to the appropriate location now.
If you have enough memory, you might try something this:
export JAVA_OPTIONS="-Xmx200g"
Not sure about malt but in general it will override any subsequent attempt to set the -Xmx flag.
Thank you for that. I'll try it out tomorrow.
And maybe let me rephrase the question a bit. I'm not entirely sure if this is just a pure problem of RAM or a Java error and therefore I posted the question here. Because even if I give the task 100gb of RAM it says after the hash table that 75,6GB are allocated. And therefor I thought it has another reason.
Java does not use 100% of the memory for data; a large fraction is reserved for garbage-collection. Being able to use 75% sounds pretty typical.
Log in to answer this question.