This is a test version of Biostars. For the public version, visit https://www.biostars.org.
HaploView 4.2: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

I'm trying to run HaploView 4.2 on one chromosome of a genome in order to compute haplotype blocks. The chromosome is approx 600Mbp and I'm looking at about 6k markers and 400 samples. Looking at the marker data it is clear that some of the blocks should be in the >200Mbp range, meaning that I do not want HaploView to limit itself to the default 500kb window used. Using the standard window or a subset of markers I'm able to successfully generate blocks, but using the full chromosome as window with all markers I end up with various java errors. Most recently this is what I get:

java -jar /tmp/Haploview.jar -memory 2000 -nogui -pedfile plink.ped -info plink.info -out outfile -check -blockoutput SPI -skipcheck -maxdistance 0
************************************                                          
Haploview 4.2   Java Version: 17.0.20
*****************************************************                                          


Arguments:      -nogui  -pedfile        plink.ped        -info   plink.info    -out     outfile    -check  -blockoutput    SPI     -skipcheck      -maxdistance    0


Skipping genotype file check
Max LD comparison distance = 0kb  
Using data file: plink.ped                                                              
Using output fileroot: outfile                                                                    
Using marker information file: plink.info                                               
File outfile.CHECK already exists and will be overwritten.                                        
Writing output to outfile.CHECK                                                                   
Fatal Error:                                                                                   
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Attempting to increase the memory further I get an overflow error:

java -jar /tmp/Haploview.jar -memory 4000 -nogui -pedfile plink.ped -info plink.info -out outfile -check -blockoutput SPI -skipcheck -maxdistance 0
*************************************
Haploview 4.2   Java Version: 17.0.20
*****************************************************


Arguments:      -nogui  -pedfile        plink.ped        -info   plink.info    -out     outfile    -check  -blockoutput    SPI     -skipcheck      -maxdistance    0


Skipping genotype file check
Max LD comparison distance = 0kb
Using data file: plink.ped
Using output fileroot: outfile
Using marker information file: plink.info
File outfile.CHECK already exists and will be overwritten.
Writing output to outfile.CHECK
Writing output to outfile.SPINEblocks
Exception in thread "main" java.lang.NegativeArraySizeException: -1710227456
        at edu.mit.wi.haploview.EM.create_super_haplos(EM.java:1155)
        at edu.mit.wi.haploview.EM.full_em_breakup(EM.java:569)
        at edu.mit.wi.haploview.EM.doEM(EM.java:407)
        at edu.mit.wi.haploview.HaploData.generateHaplotypes(HaploData.java:1002)
        at edu.mit.wi.haploview.HaploData.generateBlockHaplotypes(HaploData.java:909)
        at edu.mit.wi.haploview.HaploText.processFile(HaploText.java:1626)
        at edu.mit.wi.haploview.HaploText.processTextOnly(HaploText.java:1369)
        at edu.mit.wi.haploview.HaploText.<init>(HaploText.java:245)
        at edu.mit.wi.haploview.HaploView.main(HaploView.java:1600)

I've seen speculation online that this is a matter of some 32 bit memory limitation, but I also see people saying that java jars aren't limited in this way. Looking at the code resulting the the NegativeArraySizeException it seems like an int is overflowing - simply changing it to a long does not work as it is being used to index an array, and apparently java doesn't accept long indexes.

One way of solving this is accepting that this is a limitation of HaploView and using a smaller number of markers, maybe even running multiple iterations of different marker sets and attempting to merge these in some way. What other solutions do y'all propose?

haplotype blocks java haploview
• 197 views • link
updated by cmdcolin 448 • | written by nikos 0

1 answer

That NegativeArraySizeException is the giveaway - -1710227456 is a wrapped int, so this isn't a heap problem at all and no amount of -memory will touch it. create_super_haplos sizes an array off the haplotype count for a block, which goes as 2^n in the number of markers, so once one block gets past about 31 markers you're through 2^31 and it wraps negative. Setting -maxdistance 0 removed the only thing that was keeping blocks small, so you hit it almost straight away. Algorithmic ceiling rather than a tuning one.

(For the earlier OutOfMemoryError, heap has to go on the JVM itself: java -Xmx8g -jar Haploview.jar ...)

For the actual job I'd switch to plink --blocks. Same Gabriel confidence-interval algorithm, C++, and it takes --blocks-max-kb, so set that to 250000 and let it run the whole chromosome in one go.

Side note: 6k markers over 600 Mb is one per 100 kb, which is thin for placing block boundaries. I'd sanity check whatever comes out against plain LD decay before believing 200 Mb blocks.

• 0 views • link
written by SeqBench 41

Thanks, I'll have a look at plink! Do you know if plink's implementation would be capable of handling large blocks? Any thoughts on the Gabriel confidence interval algorithm relative to e.g. the LD spine method?

I understand your skepticism of the big blocks! To provide some additional context, I'm working with oats. Plotting the marker data, my data looks similar to panel b in this figure. What I'm really interested in is capturing the structure that's visible here in a good way. If you have recommendations for other ways of capturing this that would also be welcome! :)

• 0 views • link
written by nikos 0

JBrowse 2 could potentially help.

JBrowse 2 can calculate LD from VCF directly, which works for smallish VCF datasets, or it can also load linkage disequilibrium data that is outputted from PLINK, these are so called '.ld' files. The result should be quite similar to the haploview program.

We also can draw the 'multi-sample variant matrix' similar to the figure you linked from the oat pangenome paper.

Please see https://jbrowse.org/jb2-staging/docs/tutorials/ld_human/ which covers both of these things

We are just about to release a new version that contains a lot of new functionality and that tutorial is based on the beta, but i'm hoping to make it an official release very soon

• 0 views • link
written by cmdcolin 448

Log in to answer this question.

More posts like this