This is a test version of Biostars. For the public version, visit https://www.biostars.org.
how to unique a file row-wise?

Hello all I have a input file like this: 

3 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  786 0  0
19 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  787 0
1 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  786 0  0  0  0
0 9 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  787 0  0  0  
11 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  786 0  0  0  0

I want to make it unique row-wise. The output , i want to get should look like

3 786 0
19 787 0
1 786 0
9 787 0
11 786 0

Can anybody help me out?

awk perl sed linux
• 2,635 views • link
updated by DareDevil DareDevil • written by sharmatina189059 • sharmatina189059

Hello sharmatina189059,

how is this question related to bioinformatics? Is the order of the numbers per row in the output important? If so, how is the relationship to the input?

fin swimmer

• 0 views • link
written by finswimmer finswimmer

It is actually a frequency table I got it using Prophecy (Emboss module). I need to check the number of isolates in which these variations exist. That's why I want to make it sort so that I can read it easily. Can you please help? I could not understand how to make it unique row-wise?

• 0 views • link
written by sharmatina189059 • sharmatina189059

2 answers

use awk

awk '{ n=split($0,a,FS); $0=""; j=1; delete u; for (i=1; i<=n; i++) if (!u[a[i]]++) $(j++) = a[i]; print }' < file.txt

Using while loop

while read -r line; do echo "$line" | grep -o '[^ ]*' | sort -h | uniq | paste -s; done < file.txt
• 0 views • link
written by DareDevil DareDevil

Using GNU datamash and parallel.

Sending every input line to parallel, which calls sed for converting space-delimited values to tab-delimited, and datamash for transposting column-wise data to row-wise, sort -u is for sorting and saving unique values.

cat t.txt \
    | parallel --pipe --recend "\n" -N 1 -k  \
        'sed -r "s/\s+/\t/g" | datamash transpose | sort -u  | datamash transpose '
0       3       786
0       19      787
0       1       786
0       787     9
0       11      786
• 0 views • link
written by shenwei356 shenwei356

Thanks for the reply

perl -MList::Util=uniq -alne 'print join " ", sort { $a <=> $b } uniq @F' out_new_prophecy > perl_out_prophecy

This also works!!

• 0 views • link
written by sharmatina189059 • sharmatina189059

Log in to answer this question.

More posts like this