This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to remove duplicate fasta sequences from a file

Hi, I have total 250 files which contain scaffolds in fasta format, however several scaffolds are duplicate sequences with different headers in between files. Therefore, I want compare these files and make a file of unique scaffolds sequences. Please see following example files:

File 1:

>NODE_265_length_56_cov_170 [gcode=11] [organism=Escherichia species] [strain=strain]
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
>NODE_266_length_56_cov_121 [gcode=11] [organism=Escherichia species] [strain=strain]
GGTTCATCGATAGGAATTTAAATCCCCAAAAGACTAAAAAAGCATCACAAAACGGA
>NODE_267_length_56_cov_67 [gcode=11] [organism=Escherichia species] [strain=strain]
ATTATTTTTGTGGAGCCGGAGGAAACAAACCAGACGGTTCAGATGAGGCGCTTACG
>NODE_268_length_56_cov_43 [gcode=11] [organism=Escherichia species] [strain=strain]
TCAGATTGAACGCTGGCGGCAGGCCTAACACATGCAAGTCGAACGGTAACAGGAAG

File 2:

>NODE_250_length_56_cov_292 [gcode=11] [organism=Escherichia species] [strain=strain]
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
>NODE_251_length_56_cov_157 [gcode=11] [organism=Escherichia species] [strain=strain]
GGTTCATCGATAGGAATTTAAATCCCCAAAAGACTAAAAAAGCATCACAAAACGGA
>NODE_252_length_56_cov_86 [gcode=11] [organism=Escherichia species] [strain=strain]
ATTATTTTTGTGGAGCCGGAGGAAACAAACCAGACGGTTCAGATGAGGCGCTTACG
>NODE_253_length_56_cov_29 [gcode=11] [organism=Escherichia species] [strain=strain]
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

OUTPUT:

File 1:

>NODE_265_length_56_cov_170 [gcode=11] [organism=Escherichia species] [strain=strain]
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
>NODE_266_length_56_cov_121 [gcode=11] [organism=Escherichia species] [strain=strain]
GGTTCATCGATAGGAATTTAAATCCCCAAAAGACTAAAAAAGCATCACAAAACGGA
>NODE_267_length_56_cov_67 [gcode=11] [organism=Escherichia species] [strain=strain]
ATTATTTTTGTGGAGCCGGAGGAAACAAACCAGACGGTTCAGATGAGGCGCTTACG
>NODE_268_length_56_cov_43 [gcode=11] [organism=Escherichia species] [strain=strain]
TCAGATTGAACGCTGGCGGCAGGCCTAACACATGCAAGTCGAACGGTAACAGGAAG
>NODE_253_length_56_cov_29 [gcode=11] [organism=Escherichia species] [strain=strain]
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
alignment sequence assembly
• 2,671 views • link
updated by finswimmer finswimmer • written by Kumar Kumar

2 answers

Concatenate the fastas and use Dedupe or CD-HIT.

• 0 views • link
written by h.mon h.mon

I tried the following command:

cd-hit-454 -i /home/kumarm/CD-HIT/phy_D-scaffold-marge.fasta -o /home/kumarm/CD-HIT/454_reads_95 -c 0.99 -M 0 -T 7

error: Fatal Error: in diag_test_aapn_est, MAX_DIAG reached Program halted !!

• 0 views • link
written by Kumar Kumar

Use seqkit:

$ cat file1.fa file2.fa | seqkit rmdup -s -o out.fa
• 0 views • link
written by finswimmer finswimmer

please let me know how to install seqkit?

• 0 views • link
written by Kumar Kumar

I strongly recommend using bioconda.

The first part of this tutorial by me, might be useful for you.

• 0 views • link
written by finswimmer finswimmer

Log in to answer this question.

More posts like this
  • Prokka setupdb issue
    written by DanielC DanielC
    Dear Friends, I am trying to annotate an assembled phage genome using prokka. I made a hmm model using VFAM and adding some protein sequences …