This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Picard Markduplicates Error: Requested Array Size Exceeds Vm Limit

Hi, everyone,

I am working with wheat exome data, which are Illumina PE 100 reads. I analysed the data followed the GATK pipeline. Read were aligned to references using BWA and the BAM files were sorted by samtools. But when I tried to mark PCR dupication using Picard, I got error:

 java -Xms16g -Xmx256g -jar MarkDuplicates.jar I=IN.bam O=OUT.DeDup.bam METRICS_FILE=OUT.dedup REMOVE_DUPLICATES=false MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=1000 VALIDATION_STRINGENCY=LENIENT

Error :Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit

I also tried to other options:

java -Xmx160g -jar MarkDuplicates.jar I=IN.bam O=OUT.DeDup.bam METRICS_FILE=OUT.dedup REMOVE_DUPLICATES=false MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=1000 VALIDATION_STRINGENCY=LENIENT

Error:java.lang.OutOfMemoryError: Java heap space

The BAM file is 4G, java version is 1.7.0_11, picard is version 1.92 What can I do to solve this problem?

Thank you !

Xiaoyan

picard markduplicates
• 10,766 views • link
updated by Jordan Jordan • written by xiaoyanli82 • xiaoyanli82

3 answers

I'm just going to guess here. Did you try increasing your PermGen space? If your perm gen space is low, no matter how much heap space you increase, it does not matter.

Try this:

-XX:MaxPermSize=1g

Or how much ever you think is necessary.

• 0 views • link
written by Jordan Jordan

It should be "-XX:MaxPermSize=1g", but I don't think this is an issue. Allocating (trying) so much memory is the one, IMO.

• 1 views • link
written by Pavel Senin Pavel Senin

Oh yea. Sorry about. Fixed it. And it could be like you said too. But I'm curious to know how much RAM his system actually has!

• 0 views • link
written by Jordan Jordan

Sounds good then. What is OS and JVM implementation you are running? And what would be the output if you specify -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps?

• 1 views • link
written by Pavel Senin Pavel Senin

I has the same problem for human genomes and adding "MAX_RECORDS_IN_RAM=5000000" helped me. Maybe you can try that, here's the code that I use

 java -Xmx16g -jar MarkDuplicates.jar    \
    I=IN.bam \
    O=OUT.bam \
    METRICS_FILE=dupmetrics.txt \
    REMOVE_DUPLICATES=true \
    MAX_RECORDS_IN_RAM=5000000 \
    ASSUME_SORTED=true \
    VALIDATION_STRINGENCY=SILENT \
    TMP_DIR=$TMPDIR \
    CREATE_INDEX=true \
    OPTICAL_DUPLICATE_PIXEL_DISTANCE=10
• 0 views • link
written by NB NB

The error is due to Java heap space. The user tells the tool that it has 256 Gb of RAM that I don't think is normal. So I dont think "MAX_RECORDS_IN_RAM" is an issue.

• 1 views • link
written by Ashutosh Pandey Ashutosh Pandey

I tried again following your suggestion, the problem was fixed, thank you very much!

• 1 views • link
written by xiaoyanli82 • xiaoyanli82

Does your computer has 256 Gb RAM ? because you specified it on your command line -Xmx256g . If your bam file is 4 Gb, then I wont even care about changing that parameter and go with the default one.

• 1 views • link
written by Ashutosh Pandey Ashutosh Pandey

Log in to answer this question.

More posts like this
  • VCF files
    written by Sanazarghami4444 • Sanazarghami4444
    Hello, I run this code for produce VCF files of samples that i work on it : ``` java -jar picard.jar MarkDuplicates \ INPUT=/home/zr/SRR3229050/SRR3229050_sorted.bam \ …
  • Picard MarkDuplicates VM warning: fix the library
    written by Arthur Melo • Arthur Melo
    Dears, I'm running Picard to MarkDuplicates in a `_merged.sorted.bam` file from RNA-seq data. My script is: ``` java -Xmx5g \ -jar /usr/local/picard-tools-1.119/MarkDuplicates.jar \ INPUT=merged.sorted.bam \ …
  • markDuplicates: Histogram not produced
    written by dariober dariober
    <p>Hi All- Picard/<a href="http://broadinstitute.github.io/picard/command-line-overview.html#MarkDuplicates">markDuplicates</a> sometimes produces the <a href="http://sourceforge.net/p/picard/wiki/Main_Page/#q-what-is-meaning-of-the-histogram-produced-by-markduplicates">histogram</a> of sequencing vs gain and sometimes doesn&#39;t. Does anybody know how/why markDuplicates decides this?</p> <p>(I haven&#39;t …
  • Java Samtools Api: Write Bam To Stdout
    written by dariober dariober
    <p>Hello,</p> <p>I'm trying to use the Java API for picard/samtools to write a <a href='http://samtools.sourceforge.net/SAM1.pdf'>BAM</a> file to standard output.</p> <p>With the code below I can …
  • Sam To Bam - Loss Of Data Or Just Great Compression?
    written by Clare Clare
    <p>I use picard tools to take a <a href='http://bio-bwa.sourceforge.net/'>BWA</a> alignment <a href='http://samtools.sourceforge.net/SAM1.pdf'>sam</a> file and convert it into a sorted <a href='http://samtools.sourceforge.net/SAM1.pdf'>bam</a> file. Normally this works …
  • Markduplicates Creating A Loss Of Mate Pair
    written by DG DG
    <p>This is the first time this problem has ever happened for me, after dozens of times of running my exome sequencing mapping, post-processing, and variant …
  • Picard Estimatelibrarycomplexity
    written by komal.rathi komal.rathi
    <p>I am trying to run EstimateLibraryComplexity.jar on a <a href='http://samtools.sourceforge.net/SAM1.pdf'>BAM</a> file obtained from GSNAP:</p> <pre><code>java -jar EstimateLibraryComplexity.jar INPUT=/home/krathi/Downloads/C00060_filter_sort.bam OUTPUT=output </code></pre> <p>but I am getting this …
  • Sam/Bam File Samfilereader
    written by mad.cichlids mad.cichlids
    <p>Hi, there</p> <p>I was trying to add read groups to my alignments and take advantage of GATK to call variants. I first did my cleaned …
  • Realignertargetcreator Error: Not Enough Memory
    written by xiaoyanli82 • xiaoyanli82
    <p>Hi, everyone</p> <p>I use GATK to realignment the <a href='http://samtools.sourceforge.net/SAM1.pdf'>bam</a> files, which is about 400M, the command I ues is :</p> <pre><code>java -Xmx24g -jar GenomeAnalysis.jar …
  • Improving Performance Of Picard For Markduplicates
    written by Brett Mccann • Brett Mccann
    <p>I'm trying to improve the performance of MarkDuplicates when processing a BAM file. I am running on a 12 core box with 64GB of RAM. …