This is a test version of Biostars. For the public version, visit https://www.biostars.org.
bash script for merging R1 and R2 reads

Hi,

I am sure this question was already asked, although I could not find any information, I am new to sequences processing and I would like to know if there is a way to merge fastq.gz files:

I have this set of files:

CACTCTGAGT_S7_L001_R1_001.fastq.gz CACTCTGAGT_S7_L001_R2_001.fastq.gz
CACTCTGAGT_S8_L001_R1_001.fastq.gz CACTCTGAGT_S8_L001_R2_001.fastq.gz

I have 33 files and I would like to merge the R1 and R2 files with a for loop or another way that is not in bash.

Thanks a lot!

fastq bash illumina
• 5,776 views • link
updated by Ram Ram • written by ieie • ieie

2 answers

merging data may be one of the following.

  • Do you want to merge each pair of R1/R2 reads (if they overlap) to generate a longer representation? In this case you can use bbmerge.sh or PEAR or a similar program to merge the reads.
  • Or are you referring to interleaving R1/R2 data so the order of reads becomes R1_1,R2_1,R2_1,R2_2,R3_1,R3_2 etc in single data file. Individual reads are kept intact. In this case you will use reformat.sh -Xmx2g in1=R1.fq in2=R2.fq out=interleaved.fq.

While the second case is not technically merging, it would combine the data into one file.

• 0 views • link
written by GenoMax GenoMax

for example, I am following the vsearch example pipeline:

Process samples                                                               
for f in *_R1_*.fastq; dor=$(sed -e "s/_R1_/_R2_/" <<< "$f")
s=$(cut -d_ -f1 <<< "$f")

echo
echo ====================================
echo Processing sample $s
echo ====================================

$VSEARCH --fastq_mergepairs $f \
    --threads $THREADS \
    --reverse $r \
    --fastq_minovlen 200 \
    --fastq_maxdiffs 15 \
    --fastqout $s.merged.fastq \
    --fastq_eeout

however, i am new to the for loops so although I understand more or less the steps I cannot make it work with my data because I cannot make the for loop work.. maybe with the vsearch example I can explain better what I want to achieve with merging the pairs

• 0 views • link
written by ieie • ieie

Ah I see. Here VSEARCH is actually doing the read merging as described in point #1. You should be able to use this code as is.

You can try this to see what the loop is actually is doing (assuming VSEARCH can use gzipped files, I am not a VSEARCH user)

for f in *_R1_*.fastq.gz 

do
r=$(sed -e "s/_R1_/_R2_/" <<< "$f")
s=$(cut -d_ -f1 <<< "$f")

echo
echo ====================================
echo Processing sample $s
echo ====================================

echo $VSEARCH --fastq_mergepairs $f \
    --threads $THREADS \
    --reverse $r \
    --fastq_minovlen 200 \
    --fastq_maxdiffs 15 \
    --fastqout $s.merged.fastq.gz \
    --fastq_eeout

done
• 0 views • link
written by GenoMax GenoMax

great thanks! at the end of the for loop should there be a done?

because I am running the script as it is i the terminal but it gets stuck.. Thanks again!

• 0 views • link
written by ieie • ieie

Use ctrl+c to kill the job. I simply copied what you had above. There was a done missing (and one more mistake). I corrected those. See if the new version above is better. This will just print all the command lines to screen.

If all looks good then remove the word echo before $VESEARCH to actually run the analysis.

• 0 views • link
written by GenoMax GenoMax

great thanks a lot for your help!

• 0 views • link
written by ieie • ieie 
cat *R1*.fastq.gz > merged.R1.fastq.gz
cat *R2*.fastq.gz > merged.R2.fastq.gz

is super simple. Why "not bash" and "not loop" (although loop is not required here)?

• 0 views • link
written by Ram Ram

Thanks for the answer. I would need a for loop because I have multiple files and I was wondering if a part from bash maybe there is another way. However, if I use cat am I merging the R1 and R2 pairs or just the R1 files and R2 files? Now I have R1 and R2 but I would like to have just one file with these two merged.

Thanks again!

• 0 views • link
written by ieie • ieie

I would like to have just one file with these two merged.

Merging R1/R2 files with cat would simply append the contents of R2 file at end of R1 file. Are you sure that is what you want to do? No program can use data in this format.

• 0 views • link
written by GenoMax GenoMax

Thanks! No I don't want to append, I would like to merge the R1 and R2 reads in the files to be processed later.

• 0 views • link
written by ieie • ieie

Log in to answer this question.

More posts like this
  • BWA-MEM2 and samtools using for loop
    written by rodolfo.peacewalker • rodolfo.peacewalker
    Hi! I'm trying to align some paired-end fastq files derived from target sequencing using **BWA-MEM2** and **samtools** to filter and sort mapped reads. To avoid …
  • How to read fastqs from different sequencing runs rather than merging?
    written by Vasu Vasu
    I have fastqs of samples from the first sequencing and second sequencing runs and they are kept in different directories like below: First Run: Data1 …
  • How to run MLST with multiple fastq files
    written by Kumar Kumar
    Hi, I am trying to run a bash script for MLST at CGE (https://cge.cbs.dtu.dk/services/MLST/). I have fastq files and downloaded the MLST program. When I …
  • How to cat 2 paired end reads files together in one file using loops for multiple libraries ?
    written by v.berriosfarias • v.berriosfarias
    Hello let's say I have a directory that have multiple paired end reads and want to create a file containing the corresponding pairs, so this …
  • Print based on character strings
    written by selplat21 • selplat21
    I have a series of fastq files in different folders like so: Historic_WestFork/80_S74_L002_R1_001.fastq.gz Historic_WestFork/80_S74_L002_R2_001.fastq.gz Historic_WestFork/82_S75_L002_R1_001.fastq.gz Historic_WestFork/82_S75_L002_R2_001.fastq.gz Historic_WestFork/83_S76_L002_R1_001.fastq.gz Historic_WestFork/83_S76_L002_R2_001.fastq.gz Historic_WestFork/84_S77_L002_R1_001.fastq.gz Historic_WestFork/84_S77_L002_R2_001.fastq.gz Historic_WestFork/85_S78_L002_R1_001.fastq.gz Historic_WestFork/85_S78_L002_R2_001.fastq.gz Historic_WestFork/86_S79_L002_R1_001.fastq.gz Historic_WestFork/86_S79_L002_R2_001.fastq.gz Historic_WestFork/88_S80_L002_R1_001.fastq.gz …
  • Merge fastq files
    written by BM BM
    I have RNA-seq fastq files, each sample has multiple files from different lanes: A10_S4_R1.fastq.gz A10_S8_R1.fastq.gz A10_S40_R1.fastq.gz A10_S4_R2.fastq.gz A10_S8_R2.fastq.gz A10_S40_R2.fastq.gz then A11 up A40 I am …
  • R1 and R2 matching
    written by ieie • ieie
    Dear all, I have a number of X_R1_001.fastq.gz X_R2_001.fastq.gz and I would to match them. I have seen a pipeline where they give this: # …
  • BBMerge saving unmerged reads
    written by from the mountains from the mountains
    Is there any way to save the unmerged reads to separate files when using bbmerge? I am attempting to merge reads with overlap using bbmerge, …
  • Run multiple fastq.gz files with FastQC and yield single report?
    written by pbigbig pbigbig
    Hi, I would like to run FastQC for multiple fastq.gz files (both R1 and R2 of multiple lanes), but this program run in queueing manner, …
  • bash loop for alignment RNA-seq data
    written by Paul Paul
    Dear all, Could you help he please to modify my loop in bash to multiple alignment my FASTQ files? I have a many fastq files …