This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Speed up Picard MarkDuplicates or any better software?

I use picard software to mark duplicates, and here is my command:

java -d64 -server -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Xms8g  -Xmx16g  -Djava.io.tmpdir=tmp -jar ./picard.jar MarkDuplicates I=input.bam O=out_markdup.bam METRICS_FILE=out.metrics ASO=coordinate VALIDATION_STRINGENCY=LENIENT

It works well but when the input.bam file gets bigger, the speed is very slow! I found that the picard MarkDuplicates doesn't support multiple threads. So, is there anyway to speedup picard? Another way, is there any better software to do the same as picard MarkDuplicates but with less time? I know elprep is another choice, but it needs very large memory!

Besides, I found that samtools can also remove duplicates, but according to my search, samtools can not remove the duplicates cross different chromosomes, so picard is more universe.

Any reply will be much appreciated!

genome alignment sequence
• 8,797 views • link
updated by GenoMax GenoMax • written by lghust2011 lghust2011

MarkDuplicates supports multiple GC threads..

-XX:ParallelGCThreads=<number of threads>
• 0 views • link
written by James Ashmore James Ashmore

3 answers

So, is there anyway to speedup picard?

1) picard stores data in memory until it needs to flush them on disk. The bigger the memory is (option MAX_RECORDS_IN_RAM) , the less you need I/O operation:

2) use sambamba-markdup http://lomereiter.github.io/sambamba/docs/sambamba-markdup.html (not tested)

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

I have this toolkit which I am currently developing and is therefore not to be "browsed too much", where the deduplicate module is completed for the paired-end reads. I would be interested in a test from another user, if you want to give it a try! https://github.com/MatteoSchiavinato/SPIQR

The principle is that it removes PCR duplicates by sequence identity and not by mapping position, so it will speed us the mapping step as well and not only the deduplication. We tried with our data and the intersection of reads deduplicated by this tool and by picard was ~99%.

• 0 views • link
written by Matteo Schiavinato Matteo Schiavinato

Use clumpify.sh from BBMap suite (Introducing Clumpify: Create 30% Smaller, Faster Gzipped Fastq Files. And remove duplicates. ). You do not need to align the data. Clumpify can address all types of duplicates (PCR, optical and so on).

• 0 views • link
written by GenoMax GenoMax

Log in to answer this question.

More posts like this