This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Paralellize samtools piped with awk

Dear all,

I have question. I would like to parallelize one line command. Could you help me please?

ls 4*.bam | parallel --pipe -j 4 -k samtools view -h {} |  awk '(length($10) < 180 || $1 ~/^@/)' | awk '(length($10)>35 || $1 ~/^@/)' | samtools view -bS -o  '5{}'

Is it possible to do that like this or I have to separate samtools and awk? Thank you

parallel samtools

1 answer

You can write a shell script doing what your pipe is doing and using parallel with that script.

An expansion to Michael's script idea, you should do something like this:

ls 4*.bam | while read BAM
do
   [parallelised code executed on "${BAM}"]
done

Alternatively, you could split the output of ls into chunks and then run each in separate terminals:

ls 4*.bam | split --lines=4

This will create file 'chunks' called xaa xab xac, etc. You can run the same loop above (less parallel) multiple times in multiple sessions, and in this way it's parallelising in a different sense.

Log in to answer this question.