BWA script on partially executes

Question

[matthew.dambrosio.1]

I'm trying to run multiple alignments on my school's pbs cluster. Running the script as an interactive job works perfectly fine, I get all the files .sam files I need in about an hour. When I run it as a non-interactive job it outputs about a 9000 kb .sam file and then finishes the job which is far from complete.

#!/bin/bash
#PBS -l nodes=1:ppn=28,walltime=02:00:00
#PBS -N mapping
#PBS -q short

cd /home/workingdir/

module load shared
module load anaconda/2

i=1

for j in {1..10};
  do
    bwa bwasw -t 3 "mypath/ref.fasta"  'CH2009_split_'$j'_R'$i'.fq.gz' > map/'CH2009_'$j'_R'$i'.sam' &
done

This is the same script I run in the interactive and non-interactive node. I should also say that I've split up one large .fasta file into smaller ones so I can align them simultaneously in the for loop

Answer 1

[Philipp Bayer]

Some PBS systems don't like the & - the jobs get all sent to background, the first job starts writing, but since all jobs are in the background the PBS job finishes.

Have you tried adding a single wait at the end of the last line? Like in this example: https://stackoverflow.com/a/13283453

And since you have so many jobs, it's probably better to learn about PBS job arrays (one job per 'slot' in the array) https://arc-ts.umich.edu/software/torque/job-arrays/