This is a test version of Biostars. For the public version, visit https://www.biostars.org.
awk select duplicated lines

Dear all,

I have a file like this:

chr1 1720466 1720593 - 0 0 2
chr1 1795684 1795915 - 0 0 2
chr1 2404708 2405331 - 0 0 0
chr1 2791001 2791133 + 0 0 1
chr1 2839789 2849732 + 2 0 2
chr1 3220352 3220462 - 0 0 4
chr1 3278790 3278835 + 0 0 4
chr1 3278790 3278835 + 0 0 4

I want to print out all the duplicated lines (what I mean duplicate is the first 3 column is identical!!), by awk. Could anyone give me a example code?

Thanks
Jun

awk
• 14,581 views • link
updated by Ram Ram • written by juncheng juncheng

So in this case, the last two lines are duplicated. You want to print just one or them or both the lines? Do you have lines where the first three columns are duplicated but the last three columns are unique?

• 1 views • link
written by komal.rathi komal.rathi

I want to print all duplicates, this just a example list.

• 0 views • link
written by juncheng juncheng

I have lines like this:

chr1 7860756 7973866 - 1 0 0
chr1 7860756 7973866 + 2 1 0
chr1 7860756 7973866 + 2 1 0

I consider this three lines are duplicated. So, I want print all them out.

Thanks.

• 0 views • link
written by juncheng juncheng

3 answers

This is how you find duplicated lines based on first three columns:

awk '{if (x[$3]) { x_count[$3]++; print $0; if (x_count[$3] == 1) { print x[$3] } } x[$3] = $0}' sample.txt

output:

chr1 3278790 3278835 + 0 0 4
chr1 3278790 3278835 + 0 0 4

Source: simple google search

• 0 views • link
written by komal.rathi komal.rathi

Thanks for helping!

• 0 views • link
updated by Ram Ram • written by juncheng juncheng

I'm sorry, this script has bug:

test this:

chr1 11904306 11904806 - 0 0 0
chr1 11904306 11904806 - 0 0 0
chr1 11904306 11904806 - 0 0 0
chr1 11904389 11904841 - 0 0 0
chr1 11904574 11904806 - 0 0 1
chr1 11904686 11904806 - 0 0 0
chr1 11904687 11904806 - 0 0 0
chr1 11904687 11904840 - 0 0 0

Output:

chr1 11904306 11904806 - 0 0 0
chr1 11904306 11904806 - 0 0 0
chr1 11904306 11904806 - 0 0 0
chr1 11904574 11904806 - 0 0 1
chr1 11904686 11904806 - 0 0 0
chr1 11904687 11904806 - 0 0 0

The last three rows are not replicates. Could you help me improve that?

• 0 views • link
updated by Ram Ram • written by juncheng juncheng

You are only tracking the third column. You can make a string by concatenating first three columns and use it as an identifier.

• 0 views • link
written by Ashutosh Pandey Ashutosh Pandey

A sloppy but easy way when your columns are not sorted (chromosome). Assuming it to be a tab separated file.

awk '{print $1"\t"$2"\t"$3}' data.txt | sort | uniq -d|grep -F -f - data.txt
• 0 views • link
updated by Ram Ram • written by Ashutosh Pandey Ashutosh Pandey

The correct way is, in case anyone need:

awk '{if (x[$1$2$3]) { x_count[$1$2$3]++; print $0; if (x_count[$1$2$3] == 1) { print x[$1$2$3] } } x[$1$2$3] = $0}'
• 0 views • link
updated by Ram Ram • written by juncheng juncheng

Log in to answer this question.

More posts like this