I can extract the reads from fastq.gz file as follows.
gunzip -c in.fastq.gz | awk '(NR%4==2)' > out.seq
Is there anyway, that I only extract the unique reads and discard any duplicate reads?
•
written
by
abdul.karim •
3 answers
Are you sure you want to do this at the fastq level? (I don't understand why you want to do this at all) Do you really want to count every sequence with a one-off error as a unique sequence?
The typical thing to do would be to align your reads to their reference, then use picardtools MarkDuplicates.
But if you really want to get unique sequences in the raw fastq:
zcat my.fastq.gz | awk 'NR%4==2' | awk '!x[$0]++' > unique.txt
Use clumpify.sh from BBMap suite. You can use fastq data as is. I suggest you do no other manipulations. See: A: Introducing Clumpify: Create 30% Smaller, Faster Gzipped Fastq Files
You can choose to allow one or more errors. Separate PCR/optical duplicates.
Log in to answer this question.
-
syntax error near unexpected token `('
written by hiroyuki.kato.911 •
-
16S amplicon sequencing unequal read lengths
written by Earendil •
-
extracting DNA sequnces from multiple fastq.gz files
written by abdul.karim •
-
Extatrcting only sequnces from fastq.gz
written by abdul.karim •
-
Using flowcell name as file name for FASTQ file
written by Ric
-
Extract unique and multimapped reads from bowtie 1 sam/bam output
written by Konstantinos Yeles
-
Awk command for fastq read length and number of reads?
written by oars
-
Pipe output from find to input of awk
written by fiona.newberry
-
Trinity run failed -- Phase 1
written by ddowlin
-
How to remove a list of reads from fastq file?
written by Tao
How do you define a duplicate read? Same sequence? Same identifier? Same sequence and quality? All of those? How did you end up with duplicate reads?
I don't know if I understand your question but you can use Picard's MarkDuplicates (check on manual) to remove duplicated reads!