This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Removing duplicate reads from Illumina before hybrid de-novo assembly/ before using for correcting PacBio reads or Pac-Bio only assemblies

Hi All,

I was wondering how important is to get rid of exact duplicate Illumina reads before --

  1. Before using it for correcting PacBio reads (planning to use ProovRead)
  2. Before using it to polish a Pac-Bio only assembly using Pilon (Assembly was done using uncorrected PacBio reads - miniasm)
  3. Before using the reads to do a hybrid de-novo-assembly using PBcR

Some of my Illumina libraries have significant amounts of reads duplicated >10 times. What are your recommendations to handle these duplicate reads considering the scenarios mentioned above?

Many thanks in advance!

assembly proovread pbcr pilon duplicate reads
• 4,461 views • link
updated by Brian Bushnell Brian Bushnell • written by VS VS

2 answers

It's not a good idea to remove duplicate reads unless your libraries are amplified. If they are amplified, and you have reads appearing 10+ times, I highly recommend you change to an unamplified protocol, because you are wasting sequence. And by duplicates, I mean that both read 1 and read 2 of pairs are duplicates... otherwise the pairs are not, in fact, duplicates.

But - if you have a situation in which you are using an amplified library, and duplicate pairs occur, I recommend eliminating all duplicates and replacing them with a single copy of their consensus, in any situation other than quantification (e.g. RNA-seq).

• 1 views • link
written by Brian Bushnell Brian Bushnell

Thanks, this seems reasonable.

• 2 views • link
written by VS VS

You can try seqkit with the rmdup subcommand ( usage):

seqkit rmdup --by-seq --md5 reads.fq > reads.uniq.fq

For a 2.2G test dataset:

$ seqkit stat dataset_C.fq 
file           seq_format   seq_type    num_seqs   min_len   avg_len   max_len
dataset_C.fq   FASTQ        DNA        9,186,045       100       100       100

Remove duplicate reads:

$ memusg -t seqkit rmdup -s dataset_C.fq > dataset_C.uniq.fq
[INFO] 501303 duplicated records removed

elapsed time: 53.893s                                                                        
peak rss: 2.68 GB

The memory is a little bigger than size of the file.

We can use MD5 digest of the sequence to represent the sequence content, which uses less memory but more time:

$ memusg -t seqkit rmdup -s -m dataset_C.fq > dataset_C.uniq.fq
[INFO] 501303 duplicated records removed

elapsed time: 1m:24s
peak rss: 1.51 GB

However seqkit can only handle single file, this will produce unpaired reads for PE reads, I don't know whether it affect the down-stream assembly.

You can also have a look at Digital Normalization using khmer.

• 0 views • link
written by shenwei356 shenwei356

The question wasn't how to remove duplicated reads, but whether it's required/added value in OPs approach.

• 1 views • link
written by WouterDeCoster WouterDeCoster

:(. How can I delete my comment?

• 1 views • link
written by shenwei356 shenwei356

chose moderate then delete.

• 1 views • link
written by Medhat Medhat

Log in to answer this question.

More posts like this