This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Unique lines among 200 files

I have 200 txt files with 1 line containing the name of the read. The command line below finds the intersection.

cat *.mapped.txt | sort | uniq -d > intersection.out

How to find unique reads among these 200 files?

My files are called:

accepted.name.mapped.txt
...

The reads are like this: 

HISEQ1:105:C0A57ACXX:2:1105:12172:84568
HISEQ1:105:C0A57ACXX:2:1108:17762:41110
HISEQ1:105:C0A57ACXX:2:1204:3007:9349
HISEQ1:105:C0A57ACXX:2:1204:11087:160507
HISEQ1:105:C0A57ACXX:2:1301:18982:79651
HISEQ1:105:C0A57ACXX:2:1307:3766:23853
bash ngs
• 1,091 views • link
updated by Pierre Lindenbaum Pierre Lindenbaum • written by User000 User000

1 answer 

cat *.mapped.txt | sort | uniq -u

??

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

Yes, I was thinking about this, but I wasn't sure....will it be very slow on my 200 files 800 MB each?

• 0 views • link
written by User000 User000

should be faster:

cat *.mapped.txt | LC_ALL=C sort -T .  --buffer-size=5G | LC_ALL=C uniq -u
• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

sort -m *.mapped.txt | uniq -u > uniques.txt may be this one works as well...

• 0 views • link
written by User000 User000

Log in to answer this question.

More posts like this