Thanks for your help. I have added examples in the question.
Is there any efficient tool to remove substring reads or duplicate reads from NGS data set? I know that readjoiner could remove the duplicated reads, but seems not work on substring reads. Thanks.
Example: Duplicates: read1: AGTCAT read2: AGTCAT In this case, only one read will be kept.
Substring: read1: GTCA read2: AGTCAT In this case, read1 will be removed.
2 answers
The most efficient tool for this purpose is Dedupe from the BBMap package. However, it requires all reads to be stored in memory, so it needs a lot of memory. Can you explain in more detail what you are trying to do?
Just noticed that SGA tool is able to do this. Collapse Reads That Are Substrings Of Other Reads In Same Library
Log in to answer this question.