This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Filter unique values from col1 and print the dataset using unix_large dataset

I have a large dataset more than >6L rows. In Col1, I have some duplicates in Col1 and want to remove the duplicates and their values present in corresponding rows as well. How to do in unix commands ?

snp gene next-gen sequence genome
• 535 views • link
updated by massa.kassa.sc3na massa.kassa.sc3na • written by sofie_carolina • sofie_carolina

Hi sofie_carolina, I'm sure you can find your answer on StackOverflow. Just remember to present your input data and expected output.

• 0 views • link
written by ahaswer ahaswer

Hello sofie_carolina!

We believe that this post does not fit the main topic of this site.

More suitable at StackOverflow, see below related SO posts:

For this reason we have closed your question. This allows us to keep the site focused on the topics that the community can help with.

If you disagree please tell us why in a reply below, we'll be happy to talk about it.

Cheers!

• 0 views • link
written by zx8754 zx8754

To filter out rows based on value in Col1 do awk '!x[$1]++' < YOURFILE the $1 being the first column (not the column name). Beware that $0 checks if whole row is unique. So if you want to filter based on second column you would write awk '!x[$2]++' < YOURFILE.

See man awk for more information.

• 0 views • link
written by massa.kassa.sc3na massa.kassa.sc3na

0 answers

No answers yet.

Log in to answer this question.

More posts like this