Hello,

I've downloaded paired-end Illumina sequencing files from a repository (.fq.gz). Some samples are sequenced in multiple runs. I therefore want to merge forward reads of all runs per sample, and the reverse reads of all runs per sample. In the end I thus want to end up with 1 forward and 1 reverse reads file per sample.

e.g. of sample 1 I have files:

• 1_HJTTNDRXX.1.fq.gz #fw run 1
• 1_HJTTNDRXX.2.fq.gz #rev run 1
• 1_HJVG7DRXX.1.fq.gz #fw run 2
• 1_HJVG7DRXX.2.fq.gz #rev run 2

To merge the samples, I use following script:

for sample in $(ls *.fq.gz | awk -F'_' '{print $1}' | sort | uniq); do #the ID before the first "_" is the sample name
  echo "Processing sample: $sample"

  # Merge .1 files for the sample = forward reads
  cat ${sample}_*.1.fq.gz > ./../merged_data_16S/${sample}.R1.fq.gz #the * depicts the run ID

  # Merge .2 files for the sample = reverse reads
  cat ${sample}_*.2.fq.gz > ./../merged_data_16S/${sample}.R2.fq.gz
done

This would yield for e.g. sample 1: 1.R1.fq.gz (merged fw reads sample 1) and 1.R2.fq.gz (merged rev reads sample 1).

However, after merging, some files turn out corrupted:

1. I get a BadZipFile error when importing into QIIME2.
2. When I run" for file in *.fq.gz; do gzip -t $file; done, for some files I get gzip: 190.R2.fq.gz: invalid compressed data--crc error or gzip: 242.R2.fq.gz: invalid compressed data--length error

When I run this for loop on the original unmerged files, no error occurs, which gives me the idea it results from the merging.

I think re-merging the files results in different files being corrupted, so this also would confirm that the original files are OK.

Why does this corruption happen? How can I accomplish correct merging of these files?