This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Remove duplicate sequences with same id from a fasta file

Dear all, there are many posts about remove duplicate sequences in a fasta file (https://www.biostars.org/p/3003/), but I want to remove only the duplicate sequences with the same ids.

I have many duplicate sequences in my fasta file, but with different ids and I want to keep them.

How to remove only same id sequence duplicates? I have protein sequences and my sequences are split in different lines.

sequence fasta remove duplicate
• 9,639 views • link
updated by Alex Reynolds Alex Reynolds • written by CB • CB

BBMap's Dedupe utility has a "requirematchingnames" flag. This will make it only remove duplicates that have identical sequence and identical names. For example:

dedupe.sh in=file.fasta out=deduped.fasta ac=f requirematchingnames

One copy of each duplicate set will remain, unless you add the "uniqueonly" flag.

• 0 views • link
written by Brian Bushnell Brian Bushnell

2 answers

http://bioinf.shenwei.me/seqkit/usage/#rmdup

• 0 views • link
written by shenwei356 shenwei356

It worked very well. It is very easy to use. Thanks!

• 0 views • link
written by CB • CB

Make your FASTA files single-line (one line for header, one line for sequence): Multiline Fasta To Single Line Fasta

Then:

$ awk '{ if (($0 ~ /^>/) && (!seen[$0]++)) { print $0; printSeq=1; } else if (($0 ~ /^[^>]/) && printSeq) { print $0; printSeq=0; }  }' in.fa > out.fa
• 0 views • link
written by Alex Reynolds Alex Reynolds

Log in to answer this question.

More posts like this