This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to remove duplicate sequences in fasta file using python?

I have a two fasta files, file 1 and file 2 ,they have a lot of overlapped sequences but not all of them, here I want to merge these two files into one file file 3 and remove the duplicate part, just keeping the unique one, is there code example for python use? Well the duplicate here means the exact same query name and sequences, like these two:

>YP_204112.2
MEHYISLLVKSIFIENMALSFFLGMCTFLAVSKKVKTSFGLGVAVVVVLTIAVPVNNLVYTYLLKENALV
AGVDLTFLSFITFIGVIAALVQILEMILDRFFPPLYNALGIFLPLITVNCAIFGGVSFMVQRDYNFAESV
VYGFGSGIGWMLAIVALAGIREKMKYSDVPPGLRGLGITFITVGLMALGFMSFSGVQL

>YP_204112.2
MEHYISLLVKSIFIENMALSFFLGMCTFLAVSKKVKTSFGLGVAVVVVLTIAVPVNNLVYTYLLKENALV
AGVDLTFLSFITFIGVIAALVQILEMILDRFFPPLYNALGIFLPLITVNCAIFGGVSFMVQRDYNFAESV
VYGFGSGIGWMLAIVALAGIREKMKYSDVPPGLRGLGITFITVGLMALGFMSFSGVQL

Many thanks!

python
• 13,401 views • link
updated by michau • michau • written by horsedog • horsedog

Is there a reason you seem to want a python solution everytime? CD-HIT is meant for this sort of application.

• 0 views • link
written by GenoMax GenoMax

Well thank you very much! I'll take a look into CDHIT! no reason just I'm practising python recently so I would like to see how people solve problem by python.

• 0 views • link
written by horsedog • horsedog

6 answers

Shorter option via awk:

$ cat one.fa two.fa | awk -vRS=">" '!a[$0]++ { print ">"$0; }' - > answer.fa

If you want to replicate this with Python, you could look at using a dictionary to store unique keys.

• 0 views • link
written by Alex Reynolds Alex Reynolds

Are you using GNU awk? If you're using OS X, install GNU awk via Homebrew: brew install gawk

• 0 views • link
written by Alex Reynolds Alex Reynolds

Hi again, well you're right it works now, but it's so weird cuz in the "answer.fa" it contains even more lines than file1 plus file2, which means that seems it didn't 'remove' but 'add'? (I used wc -l to count lines)

• 0 views • link
written by horsedog • horsedog

Maybe grab the first few lines of both files and try it out on those test files. I'm not sure why you get that result; this is a pretty common use of awk.

• 0 views • link
written by Alex Reynolds Alex Reynolds

I like the simplicity! This method almost worked for me, but was printing empty lines and one line with only '>'. I'm not an expert in awk, so filtering results by using grep:

cat one.fa two.fa | awk -v RS=">" '!a[$0]++ { print ">"$0; }' - | grep -Ev '^\s*$|^>\s*$' > answer.fa
• 1 views • link
written by souzademedeiros • souzademedeiros

This task can be accomplished with FASTA/Q Collapser quickly.

• 0 views • link
written by tiago211287 tiago211287

One liner using seqkit

zcat fasta.fa.gz | seqkit rmdup -s -i -m -o clean.fa.gz -d duplicated.fa.gz -D duplicated.detail.txt
• 0 views • link
written by lakhujanivijay lakhujanivijay

Fast and do the job nicely. The inspection of the duplicated is very helpful. Best option for me so far.

• 0 views • link
written by gildas.lepennetier • gildas.lepennetier

Learn to use Biopython library. It's handy as hell. You can use any format as in/out

from Bio import SeqIO

with open('output.fasta', 'a') as outFile:
    record_ids = list()
    for record in SeqIO.parse('input.fasta', 'fasta'):
        if record.id not in record_ids:
            record_ids.append( record.id)
            SeqIO.write(record, outFile, 'fasta')
• 0 views • link
written by michau • michau

It should be noted this only checks for duplicates based on their IDs, which is not always going to be the most robust way to do it. It would probably be best to check for duplicated sequences which is what most of the other solutions are doing.

• 0 views • link
written by Joe Joe

This is a python based alternative to your issue that uses BioPy

https://github.com/MJChimal/BiologPy/blob/master/drop_unique_records.py

Hope it still helps! :)

• 0 views • link
written by pauley-p • pauley-p

This python code will combine all the input fasta files (any number of files), and output 1 file with duplicated sequences removed. Note: if your input fasta files are too big to load into memory, this code will fail. 

input_files=[list of input file names]
output_file='output_file_name'

holder=[]
for file in input_files:
    with open(file,'r') as file:
        rec=file.read().split('>')[1:]
        rec=['>'+i.strip()+'\n' for i in rec]
    holder.extend(rec)

total='\n'.join(list(set(holder)))

with open(output_file,'w') as out:
    out.write(total)
• 0 views • link
written by shoujun.gu shoujun.gu

Log in to answer this question.

More posts like this