This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Duplicate/identical reads in fastq file

I got paired end sequencing results from core. I was checking for duplicated reads as part of QC. I got read entries like this from R1 file (headers):

@NB501140:187:HHVLCBGX9:2:23206:9999:2086 1:N:0:CACCTTAC
@NB501140:187:HHVLCBGX9:2:23206:9999:2086 1:N:0:CACCTTAC

@NB501140:187:HHVLCBGX9:2:23206:9999:7430 1:N:0:CACCTTAC
@NB501140:187:HHVLCBGX9:2:23206:9999:7430 1:N:0:CACCTTAC

Is this duplication possible?

Further querying file, printed following read information:

 zgrep -iw -A 3 "NB501140:187:HHVLCBGX9:2:23206:9999:2086 1:N:0:CACCTTAC" R1.fastq.gz 

@NB501140:187:HHVLCBGX9:2:23206:9999:2086 1:N:0:CACCTTAC
AGCATCAGAGGCACCCACCTGAGGAAAGTCCTCGCTGTCCATGGCCTGCAGAGTCTGGTTGGCTGTCTCCAGAAGCTCCTCGGAGCTCTCCAGGGCCCGCGTGCAGGCAGCCAGCTGGTTCTGTGGATACCAGGCACCAAAGGAGGGGACA
+
AAAAAEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEAAEEEAAEEEEAEEEAEE<E/
--
@NB501140:187:HHVLCBGX9:2:23206:9999:2086 1:N:0:CACCTTAC
AGCATCAGAGGCACCCACCTGAGGAAAGTCCTCGCTGTCCATGGCCTGCAGAGTCTGGTTGGCTGTCTCCAGAAGCTCCTCGGAGCTCTCCAGGGCCCGCGTGCAGGCAGCCAGCTGGTTCTGTGGATACCAGGCACCAAAGGAGGGGACA
+
AAAAAEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEAAEEEAAEEEEAEEEAEE<E/
sequencing fastq
• 9,816 views • link
written by cpad0112 cpad0112

So, what you are concerned about is identical headers + identical sequence or ANY ONE of these 2?

• 0 views • link
written by lakhujanivijay lakhujanivijay

All of them. Al three are exactly identical (read name, read sequence and base qualities) between duplicated sequences, which is kind of hard to believe. I wanted to know if this is fairly common as I haven't come across identically duplicated reads, in the past Vijay Lakhujani

• 0 views • link
written by cpad0112 cpad0112

Well, I for one have never encountered this before

looks very dodgy to me as well

• 0 views • link
written by lieven.sterck lieven.sterck

Thanks Vijay Lakhujani I have used this for duplicate read identification. Since I had duplicate read names i used '-n' instead '-s'. 

$ seqkit rmdup R1.fastq.gz -n -o clean.fastqc.gz -D duplicates.txt
• 0 views • link
written by cpad0112 cpad0112

Hi!

I also used the same command, however, it's producing a truncated fastq file for some reason. 

    10KIVNT_S1_L001_R1_001.fastq.gz: unexpected end of file
@A00804:41:HNJ53DSXX:1:2125:14055:13244 1:N:0:TCCGGAGA+CCTATTCT
CAGGATGTGGAGTGCAGCAGTCCCACGGAGATCTGTTAACTCTGTGACTCTGAGCATGTACAGAACATAATCTCTGAGCCCATTCTCTCTTCTGAGAGAAAAAATAAACACACCTATTCAATGTTTTTGCTGCTTAATAAGTGATAATCTC
+
FFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFF,FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF,FFF,FFFFFF
@A00804:41:HNJ53DSXX:1:2125:14470:13244 1:N:0:TCCGGAGA+CCTATCCT
GTGTGGCATTTCCAATCAGCATTGATAGATGTAATTAGGGTTCCACTCTACAAATTGTTCACAGTGAGGGTAAGAAGGTAAACAAGAATCAAAGGGTCTTGTCTGTGATGAAGATTCATTTTTCCCCTGGGAAAAGGGAAAATAACTGGA
+
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFF:FFFFFFFF:FFFFFFFFF:FFFFFFFF::FFFF,FFFFFFFFFF
@A00804:41:HNJ53DSXX:1:2125:14959:13244 1:N:0:TCCGGAGA+CCTATCCT
TTTAGTAGATTTAGTAATACTATCTTCCCAAGCCCTAAAATGCTCAAAACCTGCCAGCCAAAAATGGTGAGGAGGGACAGATAGGAACTCTGTGTGGCACTTGGTTATTAGCCTGG
• 0 views • link
written by rohitsatyam102 rohitsatyam102

You very likely have a corrupted data file to begin with. See if you can download a fresh copy. Ask for md5sum values to confirm you have a complete copy.

• 0 views • link
written by GenoMax GenoMax

Hi Genomax. Thanks for your input. We have sent a request to check if there was some issue at the demultiplexing step. However, I have a doubt that maybe I am missing something. Isn't it the job of markduplicates to mark such duplicates and not throw an error. In the case of PCR duplicates, isn't it like PCR duplicates are assigned the same read name. If not, how and why the read names of PCR duplicates different from their primary alignment?

• 0 views • link
written by rohitsatyam102 rohitsatyam102

isn't it like PCR duplicates are assigned the same read name.

There should be no duplicate read names in a normal illumina read file. Each sequence that passed QC cam from a unique cluster that originated from a single library fragment. PCR duplicates are generally identified by identical end-to-end sequence in both pairs of reads.

• 0 views • link
written by GenoMax GenoMax

I've never seen this. Are the duplicated reads randomly placed? Or duplicates immediately follow the original? Are the reads repeated on R2 as well?

• 0 views • link
written by h.mon h.mon

This definitely looks like data/file corruption. If the original files don't have this problem then you would need to get a fresh copy (ask for md5sums to compare). Otherwise the core will have to re-run demultiplexing and regenerate the data.

• 0 views • link
written by GenoMax GenoMax

Hi, I am facing the same issue. Is there a turnaround for this? I am unable to resolve this issue.

Because of two reads having same header I am facing issue at Markduplicates step.

• 0 views • link
written by rohitsatyam102 rohitsatyam102

0 answers

No answers yet.

Log in to answer this question.

More posts like this
  • READ GROUP in GATK
    written by Payal Payal
    My fastq files for a sample with their header line looked like this: ``` HHNG7DSX5_19417170_S118_L003_R1_001.fastq.gz @A00428:335:HHNG7DSX5:3:1101:5466:1000 1:N:0:NGATGTTT+NTCAATTG HHNG7DSX5_19417170_S118_L003_R2_001.fastq.gz @A00428:335:HHNG7DSX5:3:1101:5466:1000 2:N:0:NGATGTTT+NTCAATTG HHNG7DSX5_19417170_S118_L004_R1_001.fastq.gz @A00428:335:HHNG7DSX5:4:1101:2302:1000 1:N:0:NGATGTTT+NTCAATTG HHNG7DSX5_19417170_S118_L004_R2_001.fastq.gz @A00428:335:HHNG7DSX5:4:1101:2302:1000 …
  • Renaming reads
    written by vvs.hazia • vvs.hazia
    The chalange is to append read names with UMIs, that stored in a separate file. The sceme of files with reads: @name 2:N:0:indx #or 1:N:0 …
  • Extract UMIs using UMI-tools
    written by I.Kim • I.Kim
    Hi guys, For the first time, we used UMIs for the RNAseq. So I don't know exactly how to deal with that. I would really …
  • Decipher 3' 10X paired-end reads
    written by tvibhaps • tvibhaps
    How to read barcode and umi in 3'v.3.1 10X read1, when **GTGGATCAAA+CAGGGTTGGC** is same in read 1 and read 2 both? Thanks **Read1 ** zcat …
  • Merge fastq files
    written by BM BM
    I have RNA-seq fastq files, each sample has multiple files from different lanes: A10_S4_R1.fastq.gz A10_S8_R1.fastq.gz A10_S40_R1.fastq.gz A10_S4_R2.fastq.gz A10_S8_R2.fastq.gz A10_S40_R2.fastq.gz then A11 up A40 I am …
  • add UMI sequences to fastq read name
    written by User6891 User6891
    Dear all, I have paired-end fastq data generated with Illumina bcl2fastqv2.19 & sequenced on a Novaseq.The i5index is 7bp long, the i7 8bp long R1.fastq.gz …
  • Error with demuxbyname.sh
    written by mb2subi • mb2subi
    Hi everyone, I am trying to demultiplex a fastq.gz paired-end reads with demuxbyname.sh but I obtained an empty output, I attach the log : demuxbyname.sh …
  • Is it possible/recommended to trim RNA-Seq reads to a specific length
    written by komal.rathi komal.rathi
    Hi everyone, I have paired end RNA-sequencing samples where the mates in the two paired files are of unequal lengths: For e.g.: Original reads R1: …
  • error in bowtie2 alignment
    written by prasoonagar • prasoonagar
    Hi I have done the trimming of my reads using cutadapt after trimming i did FASTQC and find that three modules * Per Base Sequence …
  • repeated read ids from Illumina MiniSeq fastq.gz files?
    written by 14134125465346445 14134125465346445
    Has anyone experienced repeated read ids from Illumina MiniSeq fastq.gz files? I have seen a few cases where the fastq.gz files produced from the bcls …