3 answers
Hi Mike,
If you want to deduplicate raw paired fastq files, I recommend trying dedupe.sh from the BBMap package. You can run it like this:
dedupe.sh in1=read1.fq in2=read2.fq out1=x1.fq out2=x2.fq ac=f
That will also print the exact number of duplicates removed.
•
written
by
Brian Bushnell
Oh, sorry, my mistake. So, I would suggest you to map first. In my opinion, it is much better to map first and remove duplicates then. But, if you want to remove duplicates first, you could try fastx_collapse, remove the duplicates and count how many of them have you lost.
Unless you want to use these for assembly, it's generally fast enough to just align and remove/mark duplicates from the resulting BAM file.
It should be noted that fastx_collapse only works on single-end reads (this is pretty common for tools like this).
HI , have you found any solutions to extract duplication reads from paired fastq files ?
BBMap's dedupe program has an "outd" flag that will capture duplicate reads:
dedupe.sh in1=read1.fq in2=read2.fq out1=x1.fq out2=x2.fq ac=f outd=dupes.fq
Alternatively, you can use Clumpify:
clumpify.sh in=reads.fq out=clumped.fq markduplicates allduplicates
This command assumes paired reads are interleaved in a single file, although the upcoming release supports paired reads in twin files. The "allduplicates" flag will mark all copies as duplicates; if you remove that, all but one copy will be marked as duplicates (which is probably better for most purposes). The "optical" flag will mark only optical duplicates (rather than, say, PCR duplicates). Anyway, "clumped.fq" will contain all of the reads, but the duplicates will be marked with " duplicate". So you can then separate them like this:
filterbyname.sh in=clumped.fq out=dupes.fq include=t names=duplicate substring
filterbyname.sh in=clumped.fq out=unique.fq include=f names=duplicate substring
Log in to answer this question.
-
How to calculate total and average read counts from paired-end and singleton FASTQ files?
written by Sachin •
-
is there a tool to recover corrupted fastq files
written by pt.taklifi
-
Samtools flagstat result
written by kiran •
-
How to find the mutations of a bacterium without reference genome from paired-end short-reads?
written by eli_bayat •
-
Remove all entries with duplicate names from fastq file?
written by wormball •
-
Cancer driver mutations Identification from VCFs
written by David_emir
-
Remove hundreds of probes
written by Floydian_slip
-
genomeCoverageBed for paired end reads
written by Vanilla
-
Downsampling dataset with more than 60 million reads
written by mike
-
Need Script Or Software To Remove Unpaired Reads From Paired End Reads
written by yangfangisok •
