How to remove duplicated sequences without using any software

Hello

Please i have question i have contigs file tha i want to annotation using prokka but i get this error msg saying that contains duplicate sequence ID: scaffold36|size13034 it makes sense because i merge some assembly files and i eliminate duplication using cd-hit and seqkit and i think that they didn't the work perfectly

so what i need is eliminate duplication sequences 'manualy' (or using another software )

so basically whta i want to do is

i have file like this :

>scaffold1|size1334
ACTGATGATACAGATACAGAAAGTAGAGATCGATGATAGA..
>scaffold2|size23034
ACAGATGAGACAGATTGACAGATAGAGATAGAGGATAGGACAG..
>scaffold3|size11654
ATAGCGCTCGCGCGCCGCGCGGCGGGGTAGAGAGATCTTTTGAGAGAGA..
>scaffold4|size3034
TGGGGTAGAGAGAGAGAGAGAAGAGGAAGAGAGGAGAGAGGA..
>scaffold2|size23034
ACAGATGAGACAGATTGACAGATAGAGATAGAGGATAGGACAG..
>scaffold100|size304
AAAAAAATACAGATAGAGAGAGAGAGGAGAGAGAGAG..
>scaffold67|size2400
ATAGAGAGAGAGAGAGAGAGAGAGAGAGGAGAGAGAGAGA..

i want to eliminate the duplicated scaffold (in this case is scaffold 2 the line >scaffold2|size2304 and its sequence because is repeated two times

so the out put will be

>scaffold1|size1334
ACTGATGATACAGATACAGAAAGTAGAGATCGATGATAGA..
>scaffold2|size23034
ACAGATGAGACAGATTGACAGATAGAGATAGAGGATAGGACAG..
>scaffold3|size11654
ATAGCGCTCGCGCGCCGCGCGGCGGGGTAGAGAGATCTTTTGAGAGAGA..
>scaffold4|size3034
TGGGGTAGAGAGAGAGAGAGAAGAGGAAGAGAGGAGAGAGGA..
>scaffold100|size304
AAAAAAATACAGATAGAGAGAGAGAGGAGAGAGAGAG..
>scaffold67|size2400
ATAGAGAGAGAGAGAGAGAGAGAGAGAGGAGAGAGAGAGA.

each scaffold is repeated just one time Thank you

alignment assembly sequencing sequence

• 340 views • link

updated 6.3 years ago by Mensur Dlakic

• written 6.3 years ago by Bioinfo •

2 answers

I think Seqkit can make this for you:

seqkit rmdup -s your_file.fa -o your_output_file.fa

• 0 views • link

written 6.3 years ago by lumal29

yes i already used it but it didn't work 0 duplicated records removed and i still get the same error msg in prokka

• 0 views • link

written 6.3 years ago by Bioinfo •

It's weird, maybe try the option "-n" instead of "-s", it removes the sequences by duplicate names instead of duplicate sequences.

• 0 views • link

written 6.3 years ago by lumal29

Ahh yes it works now , thank you :D

• 0 views • link

written 6.3 years ago by Bioinfo •

You should read the answer of Mensur Dlakic, because he may be right. Maybe you have different sequences but with the same name, so you should rename the sequences as Mensur suggested to avoid getting rid of sequences that can be useful.

• 0 views • link

written 6.3 years ago by lumal29

If seqkit found no duplicates, that means that you are mixing and matching completely different files, such that sequences with identical IDs are not the same. That's not a good practice for a variety of reasons, and I have a feeling that whatever you did when merging your assembly files shouldn't have been done.

All that aside, what you want to do is not the solution, because removing sequences with identical IDs actually removes different sequences and therefore throws away potentially useful information. Maybe a better solution is to rename all the headers so they all start with scaffold_:

seqkit replace -p .+ -r "scaffold_{nr}" old_file.fas > new_file.fas

• 0 views • link

written 6.3 years ago by Mensur Dlakic

Hello , thank you very much for your reply and your suggestions , and sorry for late reply

Yes my principal aim was to improve assembly results for that i merged results of assembly (the contigs files ( because i had Hiseq data and Miseq Data) after that i removed duplicated sequences using cd-hit , but when i performed prokka , it gives the error msg above , and i used seqkit to eliminate the duplicated sequences that have the same id ( after checking that there are exactly the same ) , in general i don't know if this is the best solution , pls if you have any other suggestions tell me

• 0 views • link

written 6.3 years ago by Bioinfo •