This is a test version of Biostars. For the public version, visit https://www.biostars.org.
what is the fastest way to sort big bam files? (300Gb)

What is the fastest way to sort a very big bam? (300GB)

I have read this Efficient And Fastest Way To Sort Large (>100Gb) Bam Files? but it is a bit old post. sambamba seems to outperform samtools.

What's your experiences with sorting big bam files now?

Thanks,
Ming

bam
• 5,381 views • link
updated by Ram Ram • written by Ming Tommy Tang Ming Tommy Tang

1 answer

I haven't used it, but DNAnexus has a samtools fork that uses Facebook's RocksDB to do an external memory sort. They claim a 5x speedup over samtools. Their blog post about it is here, and the code is here.

Bcbio-nextgen uses sambamba, which was also mentioned in the old thread, and is still a good option.

• 0 views • link
updated by Ram Ram • written by matted matted

+1 for samtools rocksort.

• 0 views • link
written by george.ry george.ry

I got curious and tried sambamba and I'm quite impressed! A quick test with an unsorted bam of ~78M reads (4.3G):

samtools Version: 1.1 (using htslib 1.1):

time samtools sort -@ 15 tmp.bam tmp.samtools.sorted
[bam_sort_core] merging from 45 files...

real    6m13.660s
user    18m40.911s
sys    0m46.348s

Now sambamba:

time sambamba_v0.5.8 sort --tmpdir ./ -t 15 tmp.bam

real    1m38.581s
user    18m11.722s
sys    0m39.935s

sambamba also uses considerably less memory. Finally, it appears that leaving the output filename as default you get as a bonus the index file for the sorted output (!!)

• 0 views • link
written by dariober dariober

Log in to answer this question.

More posts like this