This is a test version of Biostars. For the public version, visit https://www.biostars.org.
File Without Redundancy Using Awk

Hi people,

i have two fasta file with redundant reads. I want one file with all reads without redundancy, using awk. Someone help me?

awk reads
• 4,249 views • link
updated by Prakki Rama Prakki Rama • written by thiagomafra thiagomafra

Seems the answer is no, I cannot :)

• 0 views • link
written by Neilfws Neilfws

4 answers

If the IDs of the sequences (the bit after the >) are the same for identical sequences, you could do something like this:

cat file1.fa file2.fa | awk '{if($1 ~ /^>/){name=$1}else{print name"\t"$1}}' | sort | uniq | awk '{print $1"\n"$2}'

If the IDs are not the same, and you're only interested in the sequences themselves, you could get those with sed:

cat file1.fa file2.fa | sed -n '2~2p' | sort | uniq
• 11 views • link
written by Matt LaFave Matt LaFave

Hi Matt, your first command line could be simpler:

awk '{printf (/^>/) ? $0"\t" : $0"\n"}' file1.fa file2.fa | sort -u | tr "\t" "\n"

You can use a conditional expression to shorten the awk command; sort has a -u option to remove duplicates. In your second example, you can use awk to select only the sequences:

awk '! /^>/' file1.fa file2.fa | sort -u

• 13 views • link
written by Frédéric Mahé Frédéric Mahé

Good to know - thanks!

• 0 views • link
written by Matt LaFave Matt LaFave

Try CD-Hit or Usearch See the similar question: Generating a non-redundant gene set

• 0 views • link
written by Rm Rm

not sure if you are looking for tools to remove duplicate lines, if so, using vim this can be done by: in command mode enter :sort ,hit enter to run and then enter :g/^(.*)$\n\1$/d

• 0 views • link
written by ewre ewre

This is not awk, but can be used for the same purpose. See FASTQ/A Collapser.

• 0 views • link
written by Prakki Rama Prakki Rama

Log in to answer this question.

More posts like this