This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to use BBtools repair.sh on multiple files

Hello Friends,

I have have 30 paired forward and reverse fastq files that I need to run BBtools repair.sh on. As of now I can only run one pair at a time. Is there a way to run repair.sh on each pair without me having to do it individually?

bash repair.sh in1=DW2_S12_L001_R1_001.fastq_16S.fastq in2=DW2_S12_L001_R2_001.fastq_16S.fastq out1=fixed1.fastq out2=fixed2.fastq outs=singleone.fastq repair

Cheers,
Danny

software-error next-gen-sequencing
• 6,978 views • link
updated by Ram Ram • written by dw845316 • dw845316

No issues with your way of execution. Since repair on one set of files (R1 and R2 of one sample) are independent of another set (R1 and R2 of another sample), this work can be parallelized. Try parallel in linux:

$  parallel --plus echo {%R._001.fastq_16S.fastq}  ::: *.fastq  | parallel 'repair.sh in1={}R1_001.fastq_16S.fastq in2={}R2_001.fastq_16S.fastq out1={}fixed_R1_001.fastq_16S.fastq out2={}fixed_R2_001.fastq_16S.fastq outs={}single.fastq repair'

(assuming that all your files have (R1 and R2) have same extensions: R1_001.fastq_16S.fastq and R2_001.fastq_16S.fastq after sample names and parallel version is GNU parallel 20171022)

• 0 views • link
written by cpad0112 cpad0112

I have file names in the following format: HB-7R-25L0.clean.R1.fastq.gz, HB-7R-25L0.clean.R2.fastq.gz I tried to use parallel, as you suggested but dynamic replacement is not happening.

parallel --plus echo {%R..fastq.gz}  ::: *.fastq.gz

This echos the entire filename and does not capture the sample name HB-7R-25L0.clean, hence it does not pass to the next step as desired. Instead it replaces with in1=HB-7R-25L0.clean.R1.fastq.gz.R1.fastq.gz, in2=HB-7R-25L0.clean.R2.fastq.gz.R2.fastq.gz

• 0 views • link
written by arunprasanna83 • arunprasanna83

It would help

  1. If you can open a new post on this
  2. If you can post example input file names for R1 and R2
  3. Parallel version in use

you have an issue with the command: parallel --plus echo {%R..fastq.gz} ::: *.fastq.gz (extra dot before R is necessary)

$ ls *.gz
HB-7R-25L0.clean.R1.fastq.gz  HB-7R-25L0.clean.R2.fastq.gz

$ parallel --plus echo {} {=s/R1/R2/=} {%.R1.fastq.gz} ::: *R1.fastq.gz
HB-7R-25L0.clean.R1.fastq.gz HB-7R-25L0.clean.R2.fastq.gz HB-7R-25L0.clean

$ parallel --plus echo {%.R..fastq.gz} ::: *.fastq.gz
HB-7R-25L0.clean
HB-7R-25L0.clean

$ parallel --plus echo {%.R..fastq.gz} ::: *R1.fastq.gz
HB-7R-25L0.clean

I would suggest digit instead of dot after R as following:

$ parallel --plus echo {%.R\[1-2\].fastq.gz} ::: *.fastq.gz
HB-7R-25L0.clean
HB-7R-25L0.clean

$ parallel --plus echo {%.R1.fastq.gz} ::: *R1.fastq.gz
HB-7R-25L0.clean

is this what you are trying to do?

$ parallel --plus --dry-run  'repair.sh in1={} in2={=s/R1/R2/=} out1={%.R1.fastq.gz}.fixed.R1.fastq.gz out2={%.R1.fastq.gz}.fixed.R2.fastq.gz outs={%.R1.fastq.gz}_singletons.fastq repair' ::: *R1.fastq.gz

repair.sh in1=HB-7R-25L0.clean.R1.fastq.gz in2=HB-7R-25L0.clean.R2.fastq.gz out1=HB-7R-25L0.clean.fixed.R1.fastq.gz out2=HB-7R-25L0.clean.fixed.R2.fastq.gz outs=HB-7R-25L0.clean_singletons.fastq repair
• 0 views • link
written by cpad0112 cpad0112

Thanks. The issue is not resolved. Opening a new post now with more details.

• 0 views • link
written by arunprasanna83 • arunprasanna83

Yeah ! :-). Now it is clear, where the mistake is. I have been trying to feed the first echo{} values and it feeds the entire output.

• 0 views • link
written by arunprasanna83 • arunprasanna83

1 answer

Use a for loop. Something like this should work:

for i in `ls -1 *R1*.fastq | sed 's/_R1_001.fastq_16S.fastq//'`; do repair.sh in1=$i\_R1_001.fastq_16S.fastq in2=$i\_R2_001.fastq_16S.fastq out1=$i\_fixed_R1_001.fastq_16S.fastq out2=$i\_fixed_R2_001.fastq_16S.fastq outs=$i\_single.fastq repair; done
• 0 views • link
written by GenoMax GenoMax

Log in to answer this question.

More posts like this