I'm confused as to why the two bowtie lines are running at the same time? I was under the impression that everything in unix is run top to bottom and the second bowtie line wouldn't start running until the first is completed. The reason I ask is these commands are running simultaneously and I'm having memory issues of sorting two large bam files at the same time.. I'm assuming it has something to do with the piping but I am confused. Script below
(I have changed the sample names per privacy)
!/bin/bash
echo BOWTIE LOG > ../BAM/bowtie_log.txt
mkdir ../BAM/
bowtie2 --very-sensitive -x /PATH/TO/genome -X 2000 2>> ../BAM/bowtie_log.txt -p 1 -1 trimmed/Sample1_R1_001_val_1.fq.gz -2 trimmed/Sample1_R2_001_val_2.fq.gz | samtools view -Sb - | samtools sort - -o ../BAM/Sample1.bam
bowtie2 --very-sensitive -x /PATH/TO/genome -X 2000 2>> ../BAM/bowtie_log.txt -p 1 -1 trimmed/Sample2_R1_001_val_1.fq.gz -2 trimmed/Sample2_R2_001_val_2.fq.gz | samtools view -Sb - | samtools sort - -o ../BAM/Sample2.bam
Unless told otherwise (e.g. with &), bash processes lines sequentially waiting for the first to finish before starting the next one. In the case here, if the two bowtie2 commands run concurrently, something in the first line must have forked/backgrounded itself.
Oh this is a good point. I did start the command and then put it in the background by control-z and bg, would that have an effect??
ok, case closed. with that you've made the programs run in parallel - I also moved the comment to answer.
Got it. Sorry for the confusion and not saying everything in my original post
Log in to answer this question.
(Edited my original comment based on @Jean-Karim's comment and some other searches): It may be the pipes in the command that are backgrounding something and allowing both jobs to run. There are no logical/control operators.
Adding
{ }around the entire command (as suggested by @steve below) may prevent that backgrounding from happening (if it works, my assumption).(see this, specifically A.2 logical operators).
Wow learn something new about bash everyday. Thanks for the input
I don't follow this. This has to be more subtle than that. The newline is a terminator and the shell should wait for the command to complete before continuing on the next one.
I think the OP does not have a new line at all and the text wraps around and shows as if there were a newline.
I added a comment to Jean's answer? Does this bg have this effect?
does the same thing happen if you write it as
{ bowtie2 .... | samtools ... ; } && { bowtie2 .... | samtools ... ; }?What is making you think that both are running concurrently?
Reason I thought that was because Sample2 tmp merge files were being made before Sample1 was merged and got a memory error