I'm trying to find a fast way to cat multiple fastq files in a directory, but keep the output merged files separate for each samples. For example, if I have 2 samples split across multiple lanes in a directory:
Sample1_C_S4_L001_R1_001.fastq.gz Sample1_C_S4_L001_R2_001.fastq.gz Sample1_C_S4_L002_R1_001.fastq.gz Sample1_C_S4_L002_R2_001.fastq.gz Sample1_L_S4_L001_R1_001.fastq.gz Sample1_L_S4_L001_R2_001.fastq.gz Sample1_L_S4_L002_R1_001.fastq.gz Sample1_L_S4_L002_R2_001.fastq.gz
I've tried something like:
for f in Sample1_*.fastq.gz; do
cat $f > $f-merged.fastq.gz; done but I can't get it to work, the output I want is two files named Sample_1_C-merged.fastq.gz and Sample_1_L-merged.fastq.gz. I don't need to keep the R1 and R2 reads separate
3 answers
Careful, the solutions from both Prakash and swbarnes2 merge the paired-end data into a single file which is probably not what should be happening even though OP does not explicitely says so. I am reasonaby sure you want forward and reverse read being separated for proper alignment. @OP, this is paired end data so combining both into one file would double-count each sequenced fragment since every of the two paired-end reads comes from the same piece of DNA. Are you aware of that? Better do:
for f in Sample*.fastq.gz
do
Basename=${f%_L00*}
## merge R1
ls ${Basename}_L00*_R1_001.fastq.gz | xargs cat > ${Basename}_merged_R1.fastq.gz
## merge R2
ls ${Basename}_L00*_R2_001.fastq.gz | xargs cat > ${Basename}_merged_R2.fastq.gz
done
I thought that was what I wanted. But I confirmed with the sequencing centre that there is little to no overlap between the forward and reverse reads as the bulk of each library has fragments with inserts >300bp. Also I will be using these reads for metagenomic analysis (metaphlan2) which doesn't require merged reads.
Or am I going about this the completely wrong way, should I be keeping the forward and reverse reads separate during the trimming and filtering step?
I am not into metagenomics so I do not kow for sure but for most other analysis like RNA-seq, ChIP, ATAC etc you always want to keep them separated. In your specific case it might also work that way but as said I do not know how the metagenomics field works. Others might comment on that. What could be a problem is that when you do any aalysis with tools such as Picard (e.g. to get rate of duplicates) it might be messed up since you have two reads with the same name not being flagged as paired. I'd keep them separated as this is how paired-end sequencing is typically interpreted by most tools.
If you don't need to merge your reads to run metaphlan (I can't remember what form of data it expects to ingest), then don't.
If you need all of the input reads in a single file, then cat them together with the suggestions above. It's not clear to me what the difference between your L and C samples are, so its hard to say whether it makes sense to 'pool' them or not, since apparently everything is Sample1.
Or am I going about this the completely wrong way, should I be keeping the forward and reverse reads separate during the trimming and filtering step?
If your reads don't overlap (sounds like they mostly don't) then you should keep them separate. Some programs may expect interleaved data (Read1_R1, Read1_R2, Read2_R1, Read2_R2 etc). In that case you can interleave the data using reformat.sh from BBMap suite.
I agree with you ATpoint, paired-end data R1 and R2 should be kept separate. Thanks for highlighting this point. Few things are still no clear to me such as
no overlap between forward and reverse reads
merged reads are not required
also as Joe highlighted
difference between L and C samples
I hope @m.radz would be able to choose better.
Log in to answer this question.
-
Should I process two complete sets of 10x single-cell multiomics sequencing files from one donor to…
written by Wu-Sheng Zhang •
-
Issues with GBS pipeline, samtools and bwa specifically
written by dimeresia •
-
Out of Sequencer Nanopore Raw Data
written by SomeOne •
-
bcftools merge of over 9000+ vcf files
written by Vanish007 •
-
How to concatenate multiple fastq files (located in different directories) for each sample
written by salehm •
-
Bowtie2 to map multiple fastq.gz files?
written by karlitavasco87 •
-
STAR multiple FASTQ alignment, can you use with a variable $?
written by science_lizard •
-
Snakemake - moving, re-naming, unzipping and merging files for downstream analysis
written by camerond
-
Help Understanding PairedEnd read merging
written by m.radz •
-
how can I align several fastq files with hisat2 and get htseq-count
written by Learner
Can you rerun demultiplexer and not split by lane.?