Hi,
I received RNA seq data for 55 samples run by illumina sequencer Nextseq500. Each sample has 4 fastq files and each file is in a separate directory. So I have a total of 220 directories, each directory has only one fastq file. Now I need to concatenate each 4 files (belong to their respective sample) in a single fastq file. I used to use this command:
"for i in $(find ./ -type f -name ".fastq.gz" | while read F; do basename $F | rev | cut -c 22- | rev; done | sort | uniq) do echo "Merging R1" cat "$i"_L00_R1_001.fastq.gz > "$i"_ME_L001_R1_001.fastq.gz done"
However, it needs that all files to be in one directory. My files are now in 220 directories. So I am wondering if there is a way to modify this command to look for files in different directories. Or if there is a command, I could use to move each file in the individual directories to a single directory.
Thank you for your help.
•
written
by
salehm •
2 answers
See this answer for inspiration: C: Concatenating fastq.gz files across lanes
GNU parallel solution that is untested but should work (which will probably summon Ole Tange to provide a better version). This will search recursively through directories from the parent directory, which is the only requirement.
parallel --dry-run -j1 cat {} '>>' '$(basename {} | rev | cut -c 22- | rev)'_ME_L001_R1_001.fastq.gz ::: $(find . -type f -name ".fastq.gz")
Remove --dry-run if it looks good. -j1 means to run the command for one file at a time, but you can increase that for parallelization (or remove it to use all available cores).
Log in to answer this question.
-
How to execute one command on all files within a directory
written by Kumar
-
Size difference using gunzip-cat-gzip and zcat
written by dazhudou1122
-
bash loop to python loop for google colabs
written by svilla •
-
Merge fastq files
written by BM
-
How to cat fastq files but keep different samples separate
written by m.radz •
-
Bash scripting FastQC for multiple fastq files in multiple directories
written by rc1253 •
-
RSEM on single replicate with multiple fasta files
written by gozrom •
-
Merge fastq file from same lane and start bowtie alignment
written by giorgiocasaburi
-
shell script for bowtie/bwa alignment pair end reads
written by Paul
-
cat files with GNU Parallel
written by tayebwajb
