This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Removing Redundant Amino Acid Sequences From Fasta - Tools And Algorithms

Is there any tool that can be used to remove redundant amino acid sequences from Fasta? I know fastx_collapser from fastx-toolkit (How to remove the same sequences in the FASTA files?), but it works on nucleotide sequences only. I am looking for a similar tool.

Are there any widely known algorithms for that purpose? Is using Levenshtein distance, pair distance or similar a good idea?

fasta sequence distance

All the answers are great, thank you guys. However, I would like to know more about algorithms that can be used. Share your ideas.

4 answers

If using prove-that-you're-academic-ware doesn't bother you, you should definitely try UCLUST. Its author claims UCLUST is definitely superior to CD-HIT, and while I would take it with a grain of salt, much smaller requirements at much higher speed are hard to ignore.

Many of the answers to the question that you mentioned will work fine for amino acid sequence.

You should definitely look at CD-HIT. Although not updated for some time it works well and is still widely-used; for example, to create NR databases at UniProt and the PDB.

Days ago, I wrote a Python script to remove redundant nucleotide sequences from FASTA. Here is the script:

import re

pattern_newline = re.compile(r'([A-Z-]{80})(?=[A-Z-])')

seqs = {}

with open('example.fasta') as fin:
    data = fin.read()
    for m in re.finditer(r'>(\w+)[^\n]*([^>]*)', data):
        seq = ''.join(m.group(2).split())
        id = m.group(1)
        if seq not in seqs: seqs[seq] = []
        seqs[seq].append(id)

with open('unique.fasta', 'w') as fout:
for seq, ids in seqs.iteritems():
    fout.write('>%s\n%s\n' % (ids[0], pattern_newline.sub(r'\1\n', seq)))

It works on amino acid sequences, too. Wish it helps.

Here is my free program on Github Sequence database curator (https://github.com/Eslam-Samir-Ragab/Sequence-database-curator)

It is a very fast program and it can deal with:

  1. Nucleotide sequences
  2. Protein sequences

It can work under Operating systems:

  1. Windows
  2. Mac
  3. Linux

It also works for:

  1. Fasta format
  2. Fastq format

Best Regards

Log in to answer this question.