This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How do I use GNU parallel with two inputs?

I am using GNU parallel on the following shortbred script. The script is supposed to take two input files and output one .tsv file but I am getting both _1.tsv and _2.tsv files and directories as an output. How do I get the output something like :

SRR059331_dir

SRR059331.tsv

SRR059339_dir

SRR059339.tsv

I did bash for loop and it outputs the above expected results but it is very time consuming. How do I convert this bash script into parallel. This is my bash script:

for i in test_files/*_1.fastq.gz; do
    F=`basename $i _1.fastq.gz`;
    mkdir test_out/"$F"_dir;
    python shortbred/shortbred_quantify.py --markers markers.faa --wgs "$F"_1.fastq.gz "$F"_2.fastq.gz --results test_out/"$F".tsv --tmp test_out/"$F"_dir --usearch ./shortbred/usearch;

done

This is my script so far for GNU parallel

#!/bin/bash

#Create a sub-directory for ShortBRED output
mkdir test_out

time parallel -j 10 \
"python shortbred/shortbred_quantify.py \
--markers markers.faa \
--wgs {1} {2} \
--results test_out/{1/.}.tsv \
--tmp test_out/{1/.}_dir \
--usearch shortbred/usearch" ::: test_files/fastq/*_1.fastq.gz :::+ test_files/fastq/*_2.fastq.gz
parallel gnu

Have you determined what the time consuming part of the process is? Spawning multiple instances of your python process may not be as economical as simply increasing the threads that usearch is using for example.

2 answers

See the parallel manual on input sources:

https://www.gnu.org/software/parallel/parallel_tutorial.html#input-sources

for example:

parallel --link echo {1} and {2} ::: A B C ::: D E F

will print:

A and D
B and E
C and F

I think parallel --link is deprecated. I tried different methods but didn't work. I ended of concatenating to a single file and passed a single argument. Thank you both for your help.

--link is fully supported, but what you may be thinking of is :::+ which links two inputs, and which is newer. --link will link all inputs and is thus less flexible.

Compare:

parallel --link echo ::: a b c ::: d e f ::: g h i
parallel echo ::: a b c :::+ d e f ::: g h i

For readability I would always use a function:

doit() {
  python shortbred/shortbred_quantify.py \
    --markers markers.faa \
    --wgs "$1" "$2" \
    --results test_out/"$3".tsv \
    --tmp test_out/"$3"_dir \
    --usearch shortbred/usearch
}
export -f doit

time parallel -j10 --plus doit {} {/_1.fastq/_2.fastq} {/.} ::: test_files/fastq/*_1.fastq.gz

Log in to answer this question.