Hello everyone,

We are excited to introduce FastDup, a new high-performance tool for marking duplicate reads in NGS data, recently published in Bioinformatics.

Duplicate marking is a critical step in variant calling pipelines (e.g., GATK Best Practices), but standard tools like Picard MarkDuplicates can be a bottleneck due to single-threaded implementations or high memory requirements. We developed FastDup to solve this scalability issue without sacrificing accuracy.

• Paper: https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btaf633/8362221
• GitHub: https://github.com/zzhofict/FastDup
• Bioconda: conda install -c bioconda fastdup

What makes FastDup different? FastDup uses a novel "speculation-and-test" mechanism. Instead of relying on expensive global sorting or holding massive amounts of data in memory, FastDup processes data blocks in parallel to speculate potential duplicates and then verifies them.

Key Features & Benchmarks:

• High Speed: FastDup achieves up to 20x speedup compared to Picard MarkDuplicates and is significantly faster than Sambamba and GATK MarkDuplicatesSpark.
• Identical Accuracy: A major concern when switching tools is reproducibility. FastDup guarantees 100% identical output to Picard MarkDuplicates (the current gold standard).
• Low Memory Footprint: Unlike Spark-based tools, FastDup is highly memory-efficient, making it suitable for large WGS datasets.
• Scalability: It shows near-linear scalability up to 16 threads.

Performance Comparison In our benchmarks (WES and WGS datasets ranging from 100M to 1.5B reads), FastDup consistently outperformed existing tools:

• vs. Picard: ~20x faster
• vs. GATK Spark: ~8x faster
• vs. Sambamba: ~4.5x faster

We welcome any feedback, bug reports, or suggestions on our GitHub repository. We hope this tool helps accelerate your genomic analysis pipelines!