This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to Deduplicate files

Hi,

I have list of 160111 protein files. Some of the files are duplication as GCA and GCF id contains same protein sequnces. How I can deduplicate the list on the basis of ASM102201v1? 

Enterobacter_hormaechei-158836#GCA_001022015.1/GCA_001022015.1_ASM102201v1_protein.faa
Enterobacter_cloacae-550#GCF_001022015.1/GCF_001022015.1_ASM102201v1_protein.faa
sequence
• 1,142 views • link
updated by finswimmer finswimmer • written by saadleeshehreen saadleeshehreen

1 answer

Try this:

$ awk -v FS="#" '{match($2, /(ASM[^_]+)/, asm)} !seen[asm[1]]++' input_file
• 0 views • link
written by finswimmer finswimmer

Log in to answer this question.

More posts like this