This is a test version of Biostars. For the public version, visit https://www.biostars.org.
greping unique reads from fastq file

I have a fastq file with reads, but there are duplicates. Can you tell me how I can get the unique entires in the 4-row fastq format

sequencing next-gen
• 6,315 views • link
updated by Ram Ram • written by sumithrasank75 sumithrasank75

You might want to clarify what you mean by "duplicate" in this case. Do you mean that they have the same sequence or that you have a single read from the machine duplicated multiple times?

BTW, the former situation is addressed by RAM's answer, the latter by Pierre's.

• 0 views • link
written by Devon Ryan Devon Ryan

3 answers 

gunzip -c in.fq.gz | paste - - - - | LC_ALL=C sort -t '\t' -k2,2 -u | tr "\t" "\n" | gzip > out.fq.gz
• 0 views • link
updated by Ram Ram • written by Pierre Lindenbaum Pierre Lindenbaum

When I run this I get an error:

sort: multi-character tab `\\t'

Also, I have a plain .fastq not the compressed fastq.gz

• 0 views • link
updated by Ram Ram • written by sumithrasank75 sumithrasank75

Try

cat in.fq | paste - - - - | LC_ALL=C sort -t$'\t' -k2,2 -u | tr "\t" "\n" | gzip > out.fq.gz

Waiting for someone to write uuoc :)

• 0 views • link
written by Sukhi Singh Sukhi Singh

Yeah, I used '\t' to show you it's a tab....

• 0 views • link
updated by Ram Ram • written by Pierre Lindenbaum Pierre Lindenbaum

Thanks for clarifying this

• 0 views • link
updated by Ram Ram • written by sumithrasank75 sumithrasank75

With the BBMap package:

dedupe.sh in=reads.fq out=nodupes.fq

The output will contain exactly 1 copy of every unique sequence. It's extremely fast, but may take more memory than other solutions - the amount of memory is proportional to the number of unique reads (rather than, say, the total input size).

• 0 views • link
updated by Ram Ram • written by Brian Bushnell Brian Bushnell

PRINSEQ should solve your problem. Check it out here: http://prinseq.sourceforge.net/manual.html#QCDUPLICATION

A bit of digging should get you the command line options for the feature.

• 0 views • link
written by Ram Ram

Log in to answer this question.

More posts like this
  • searching for sequences in a fastq file
    written by sumithrasank75 sumithrasank75
    I have a list of 10 base-pair sequences say ACTTTTTTTT,TGGTTACTGA,.. in a textfile called id.txt. Using biopython,I want to iterate through a fastq file (say …
  • extracting reads from fastq file based on read_id
    written by sumithrasank75 sumithrasank75
    I have a file with a set of read ids called tmp. I want to extract reads with these read ids from another fastq file …
  • Quorum error corrector - fastq files as output?
    written by VS VS
    <p>I am attempting to error correct my illumina short reads using QuoRum error corrector (part of Masurca genome assembler; but I have installed only QuoRum …
  • comparing two fastq files
    written by sumithrasank75 sumithrasank75
    I have 2 fastq files F1.fastq and F2.fastq. F2.fastq is a smaller file which is a subset of reads from F1.fastq. I want reads in …
  • reporting the best alignment(s) only in pairwise2
    written by sumithrasank75 sumithrasank75
    I have a fastq file of reads, say "reads.fastq". I want to align the sequences to a string saved as a fasta file ref.faa. I …
  • filtering reads based on read counts
    written by sumithrasank75 sumithrasank75
    I have parsed out a fastq file and have a text file like ``` read_id read-count aaaa 10 bbbb 1000 ccccc 10000 dddd 1 ``` …
  • searching reads with a certain sequence in fastq file
    written by sumithrasank75 sumithrasank75
    I have a file say master.fastq which looks like : ``` @M00990:202:000000000-ADM27:1:1101:21678:1536 2:N:0:291 CCTTTTACCGACCCGCTCTTTCTCTCCTACGCTTATTTCCGTCTACCCTTCTCTTCACTCGCTATTTCTATTCTTAAAACTATCTTAATGTTCTGCCTTTGCTCTTTTCTTTTTTCTATAACCTCTCTACAGCCAACTCACCCATCTCCTTCCCTGCTACGCTATTCCTCTGTTAGTTTTTCTTCATCATACTTTTCTCATCTCACACTACCTTTGCACTTCTTCCTTTCCACGTCCCCTTTCTCCTACC + -----,,6;6,@+8++6+,,,<C5@,,,,,,+8,,;,,<,,,,,,,;,,;,,,<C,,,;,,8,,8CC,C,,,5,C,,,,99,+,+4,,,3,,9,,6,@<,,,,,,,9,,,,,,,4),,0**,,5),,5**,)59*0),,*)5,)))*,9,3,0++))5D:)))))5;+;+0)*;+*6++++),******3**50,6,+++**,0*,,31)88*0*1*5*1)0*:*7>C;3,035:0)))8).*2**.*:) @M00990:202:000000000-ADM27:1:1101:22685:1539 2:N:0:291 CTTATCACCGACTCTCTCCTTCTCTTCCAAGTTTATTTCCGACTCCCCTTATCTTCACTTGCTATTTCTATTCTTAAAACTATCTCGACCTTTCACCTTTCCCTCTTTCCTTCTTTTCTCTCCTTCTACACTCCCACCCACTCTTACTTCTTTCTTGTCACCGTTTCCATATTATACTTTCTTCTCTTACATAATTTTCTTCCTGCAAACTATTTAAGCAATCTCTTTCTTTCACCCCTTTTATCTCGCC + -----,,<;67@+B,,6,,,,;C5@,,,,,66<,,6,,<,,,66,,;,66,,;;C,,,;,4<,,<CC6C,;,,,;,,6,:,6?A9=,,+++2,5,,,9<?D,,,,,:C?,@,@,,5,?*3*,9**,,0**,*)93))4+))19*0**,,52,56+**5*03*)3)))42+2***5*+=3+,*4*2****,**2*,3,+++*0,50,,**5*****0****)5)***0**,,***3*)0)))3***0*))) ``` I want only …
  • How To Cluster Mate Pair, Paie End And Single End Reads From Single File??
    written by HG HG
    <p>Hi, I made mate pair illumina sequencing. In fastq file i checked mate pair, paie end and single end reads are present. Can any one …
  • How Can I Get All Coordinates With A Coverage Higher Than 5 Reads?
    written by sanchezcavani sanchezcavani
    <p>I have a <a href='http://samtools.sourceforge.net/SAM1.pdf'>SAM</a> format file. Would anyone tell me how I should get all coordinates with coverage higher than 5 reads within a …
  • Ab Solid Fastq To Illumina Fastq
    written by Varun Gupta Varun Gupta
    <p>HI Everyone</p> <p>I have AB solid data The fastq file looks like this</p> <pre><code>@SRR334700.1 solid0527_20110502_FRAG_BC_ReidWT_bcSample1_ReidWT_1_5_16 length=50 T120202210232000020002.00301000012.100...00........ +SRR334700.1 solid0527_20110502_FRAG_BC_ReidWT_bcSample1_ReidWT_1_5_16 length=50 !/&lt;%2/:%*)-%%0'--'')/.!%('1'%),+/%!&amp;',!!!'+!!!!!!!! </code></pre> <p>I have dealt …