Dear all,
I need help with parallel command. I have one script in shell. And I would like to run it 12time in same time. But for each script I need different name of output. output is .tsv and the name is same like name of input, could you help how to do that?
Thanks a lot
2 answers
an example with echo:
seq 1 12 | parallel echo "Hello" '>' 'result.{}'
for input in *.bam; do out=`echo $input | awk -F"." '{ print $1}'; bin.sh $input $out.tsv & done
To understand:
for input in *.bam; #for each bam file
do
out=`echo $input | awk -F"." '{ print $1}'` #get the uniq output prefix
bin.sh $input $out.tsv & #run the srcipt and push it to background
done
•
written
by
GouthamAtla
won't work if there are not enough cores.
Yes. But the for loop helps me a lot in other cases where the operation is computationally not expensive.
•
written
by
GouthamAtla
... and if something wrong happens, you'll have to (quickly) find & kill your PIDs ...
I used to do that.
•
written
by
GouthamAtla
I am trying to understand why people still use for loops for independent jobs.
Is it readability? Is the for-loop really easier to read than 'parallel bin.sh {} {.}.tsv ::: *.bam'? Or if the jobs were bigger/more complex using a function:
myfunc() {
bin.sh "$1" "$2"
#more stuff here
}
export -f myfunc
parallel myfunc {} {.}.tsv ::: *.bam
For computationally cheap jobs I really do not see the benefit of a for loop.
The only advantage I can think of is that GNU Parallel may not be installed. But that advantage can vanish in just 10 seconds: wget -O - pi.dk/3|bash
@Geek_y can you enlighten me, what you see as the advantage?
I am used to use for loop. But definitely will shift towards parallel. Started reading your tutorial on parallel. I am from biology background and a kind of beginner in core bioinfo, hence, need some time to learn best practices.
Log in to answer this question.
-
Nextflow: split a FASTA file into 5 parts to enable parallel processing
written by neng •
-
How do I use GNU parallel with two inputs?
written by fb143 •
-
snakemake how to get or create wildcards from newly generated files
written by gb
-
How to run different fasta files in a same process with its output respective
written by diego1530 •
-
Blast locally on multiple fasta files with multiple database
written by fec2 •
-
Parallel for bwa mem - problem with -R argument for ID and SM
written by Korsocius
-
running a perl script multiple times; input and output files are required
written by Ana
-
Plot graphs in R by loop and save it like jpeg
written by Korsocius
-
Mean read length
written by Korsocius
-
Cufflinks output with filename prefix
written by Ron
please post your standard shell command line;
the parallel command would be somthing like
parallel scriptname <hardcoded options for the script> {1}.input {1}.output ::: prefixes
see Gnu Parallel - Parallelize Serial Command Line Programs Without Changing Them
Show some example of what is input and output.
I have script with name bin.sh where is input 1.bam and output will be 1.tsv. Every input is in the different folder with same name from 1-12. In this folder are 1.bam (imput), 1.bai (input) => they are reading with shell script. Output will 1.tsv and etc. for each bam file.
parallel myscript {} {.}.bai '>' {.}.tsv ::: */*.bam