This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Remove duplicate in fasta file based on sequence

Hello,

I would like to remove duplicate in fasta file based on sequence, not header. The file is a large.

For example:

>A01968:16:HJM3MDSX3:1:1101:7654:1125 1:N:0:ATCACG
GCGTCTGTAGTCCAACGGTTAGGATAATTGCCTTCC
>A01968:16:HJM3MDSX3:1:1101:31096:1141 1:N:0:ATCACG
CTCAGTTTTGTAGTAGGACTCCCACTCTGACATGTT
>A01968:16:HJM3MDSX3:1:1101:27552:1204 1:N:0:ATCACG
CTCAGTTTTGTAGTAGGACTCCCACTCTGACATGTT
>A01968:16:HJM3MDSX3:1:1101:29830:1297 1:N:0:ATCACG
CTCAGTTTTGTAGTAGGACTCCCACTCTGACATGTT
>A01968:16:HJM3MDSX3:1:1101:6017:1329 1:N:0:ATCACG
ACGGGGCATTGTAAGTGAGATCGGAAGAGCCACGTC

and I would like to obtain a file containing only:

>A01968:16:HJM3MDSX3:1:1101:7654:1125 1:N:0:ATCACG
GCGTCTGTAGTCCAACGGTTAGGATAATTGCCTTCC
>A01968:16:HJM3MDSX3:1:1101:31096:1141 1:N:0:ATCACG
CTCAGTTTTGTAGTAGGACTCCCACTCTGACATGTT
>A01968:16:HJM3MDSX3:1:1101:6017:1329 1:N:0:ATCACG
ACGGGGCATTGTAAGTGAGATCGGAAGAGCCACGTC
fasta linux
• 1,918 views • link
updated by madalton • madalton • written by martta95 • martta95

2 answers

Use clumpify.sh from BBMap suite --> Introducing Clumpify: Create 30% Smaller, Faster Gzipped Fastq Files. And remove duplicates.

It will work with fasta files. You will need to adjust memory allocation (-Xmx parameter) depending on size of your input file.

 clumpify.sh -Xmx8g in=input.fa out=deduped.fa dedupe
• 0 views • link
written by GenoMax GenoMax

I like seqkit for basic fasta/q manipulation.

seqkit rmdup --by-seq -o deduped.fa your_file.fa

You can set a number of threads with -j

• 0 views • link
written by madalton • madalton

Log in to answer this question.

More posts like this