This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Fastq Files From Different Flowcells

Hi,

For a single sample, i have several paired-end fastq files from four different flowcells. i.e. fastq files from different lanes from each flowcell. Instead of processing individual fastq files from different flowcells, can i merge all the forward reads(from different flowcells and different lanes) into a single fastq file and all the reverse end reads into another fastq file?

Thanks

fastq
• 20,307 views • link
updated by Pierre Lindenbaum Pierre Lindenbaum • written by hellbio hellbio

2 answers

Yes, you can (see BruceyB's answer) but that's usually a bad idea.

You can process the fastqs in parallel using , for example make with the option -j (number of parallel tasks), and merge the SAM files later.

enter image description here

• 186 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

well i have fastq files from 8 lanes i.e. 8pairs of forward and reverse reads. IF we map them individually, we will end up with 8 sam files which has to be merged. In this case it becomes so complex with 8 different sam files to be merged. So, would it be wise to concatenate the fastq files and then generate a single sam/bam file?

• 0 views • link
written by hellbio hellbio

if time is not problem, concatenate your FASTQs. If you can align the 8 pairs of fastq , convert to BAM and sort 8 jobs in *parallel, then you'll get your result faster.

" it becomes so complex.." : why ? A makefile will solve your problems.

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

could you please elaborate how a makefile will solve the problem? just curious...

• 0 views • link
written by Sebastian Kurscheid Sebastian Kurscheid

Could you please provide a sample make file, which you have been using. Make file might make life easier in case WGS data.

• 0 views • link
written by hellbio hellbio

I would also like to mention that my data is paired-end data

• 0 views • link
written by hellbio hellbio

why ?

that's usually a bad idea.

any thing else beside speed and RG

• 1 views • link
written by Medhat Medhat

Is processing Lane separately faster than using bwa on the merged fastq file with thread option ? Which one is faster ? Strategy A or B ? I think those strategies are equivalent.

Strategy A : Makefile

    bwa lane1.fastq
    bwa lane2.fastq 
    bwa lane3.fastq
    bwa lane4.fastq

Strategy B : merged

  bwa all.lane.fastq -t 4
• 0 views • link
written by sacha sacha

Yes, you can. The simplest way of doing this is with 'cat' on the terminal. This will concatenate the files you choose into one FQ file. E.g. cat R1_001.fq.gz R1_002.fq.gz ... R1_n.fq.gz > R1_combined.fq.gz

• 287 views • link
written by BruceB BruceB

So it can be done by concatenating all the forward reads to 1_fastq.gz and reverse reads to 2_fastq.gz and then mapping the paired-end files to a single bam file.

• 0 views • link
written by hellbio hellbio

Yes, that is exactly what I would do (and have done in the recent past). Once concatenated, you would never know they came from different lanes.

• 0 views • link
written by BruceB BruceB

Not exactly, lane number is also represented in the sequence identifier, see http://support.illumina.com/help/SequencingAnalysisWorkflow/Content/Vault/Informatics/Sequencing_Analysis/CASAVA/swSEQ_mCA_FASTQFiles.htm

Each entry in a FASTQ file consists of four lines:
• Sequence identifier
• Sequence
• Quality score identifier line (consisting of a +)
• Quality score

Each sequence identifier, the line that precedes the sequence and describes it, needs to be in the following format:

@<instrument>:<run number="">:<flowcell id="">:<lane>:<tile>:<x-pos>:<y-pos> <read>:<is filtered="">:<control number="">:<index sequence="">

• 0 views • link
written by chen chen

Log in to answer this question.

More posts like this