This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Adding a UMI tag to the UMI sequences in FASTQ files

I'm struggling with finding a way to append the following string "UMI_" to the 8th field in every header in my FASTQ files, and to edit the gzipped files in place. I used bcl2fastq's UMI trimming tool to trim the first 23 bp of read 2 which contains the QIAseq 12bp random UMI sequence followed by the 11bp common ligation sequence. I'm also wondering about trimming the latter 11bp as it may cause issues with read collapsing if they are not all identical.

I'm starting with:

@NB551615:187:HWHW5BGXF:1:12379:1239:ATGCACAGCCTGAAACGAGTCCG 1:N:0:ATCTCAGG+CGGAGAGA

going to this:

@NB551615:187:HWHW5BGXF:1:12379:1239:UMI_ATGCACAGCCTGAAACGAGTCCG 1:N:0:ATCTCAGG+CGGAGAGA

I've looked into using awk and sed but awk is messy with printing the other lines it is not supposed to edit and sed has trouble finding the 8th field with the UMI sequence.

Thanks, Ivan

fastq
• 4,307 views • link
updated by i.sudbery i.sudbery • written by idedios • idedios

What is the ultimate goal here? Perhaps we can suggest an alternate solution.

• 0 views • link
written by GenoMax GenoMax

2 answers

Its not possible to edit the gzipped files in place. You you can try:

zcat myreads.fastq.gz \
    |  sed -E 's/([0-9]+):([ATGC]{11})/\1:UMI_\2/' \
    | gzip > myreads_processed.fastq.gz
• 0 views • link
written by i.sudbery i.sudbery

Thanks! Works like a charm

• 0 views • link
written by idedios • idedios

Sorry to necro this thread, but it looks like not every read in my FASTQs is getting the UMI_ added with this. Below are a couple of reads that show this inconsistency. 

@NB551615:187:HWHW5BGXF:1:11101:25141:1161:GCCTCAATATNG 1:N:0:ATCGCAGG+CGGAGAGA
GGATGCCTCTCATGAATGGCCTTATTGGATCCAGTCCTCATCTCCCACATAATTCTTTGCCACCTGGAAGCGGACTGGGAACTTTCTCTGCAATTGCACAATCCTCTTATCCTGATGCCAGGTACAAGCCTTATTTTCTATGGAACCTTA
+
AA/AAEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEE/EEEEEEEEAEEEEEEEEEEEEEAE<AEEEEE<EEEEEEEEEEEEEEEEEEE/EEEEEAEEEEEEEEEEEE</EEAAEEAEEEE/EE<EEE/EEAAEEEA<AEAEA/EEEE
@NB551615:187:HWHW5BGXF:1:11101:17990:1164:UMI_GGGGAGATGAAT 1:N:0:ATCTCAGG+CGGAGAGA
CTCACTGACGTCGAAGGCTGCCTTCAGTGCCTGGATGTCCGTGGCCACACCGGACACGCGGTAGATGCCCACCTCCTCCATGCCTCGGCGCTCGATCTCCTCCACGCACTGGCGCACGATGTAGGGCACCTTGGACCTCTCTCTCCTGTGG
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAAAEAEEAEAEEE

EDIT: I found the cause was due to N bases in the UMI sequences, so I edited the sed command to look for ATGCN bases.

• 0 views • link
written by idedios • idedios

That sed command won't match if there's an N in the 11 bases, of course

• 0 views • link
written by swbarnes2 swbarnes2

Yeah, I guess you add N to the regex in sed:

zcat myreads.fastq.gz \
    |  sed -E 's/([0-9]+):([ATGCN]{11})/\1:UMI_\2/' \
    | gzip > myreads_processed.fastq.gz
• 0 views • link
written by i.sudbery i.sudbery

You can probably tell bcl2fastq to ignore those 11 bases. Do you really need it to say UMI in the read name? Some tools, like umi_tools will not be expecting that.

• 0 views • link
written by swbarnes2 swbarnes2

UMI tools doesn't need it to say UMI, but saying UMI wouldn't be a problem.

• 0 views • link
written by i.sudbery i.sudbery

Looking back it would have been better to use bcl2fastq to ignore those 11 bases and this is what I can do for future runs. Especially since for a lot of reads, most of those bases are called as Ns and have very low phred scores.

• 0 views • link
written by idedios • idedios

Log in to answer this question.

More posts like this

No similar posts found.