Parallelizing Read Trimming/Filtering And Genome Assembly!!

Question

[Rahul Sharma]

Hi all,

I want to make a parallel shell script, which could run the trimming processes parallel on different processors and later do the genome assembly. I am trying the following script:

Before doing this, one could also split the whole read file into small chunks (using split command) and use for loop to do the trimming on multiple processors.

#!/usr/bin/bash

STEP1: Library1 read Filtering/Trimming & # Use any software of your choice

STEP2: Library2 read Filtering/Trimming &

STEP3: Library3 read Filtering/Trimming &

wait

velveth demo 65 -short -fastq Library1 -short2 -fastq Library2 -short3 -fastq Library3

velvetg demo -exp_cov auto

Dose anybody have better way to write this? Which would also take care of successful completion of first 3 steps. using their process Ids.

I would really appreciate your inputs. Thanks in advance!

Best regards,

Rahul

Answer 1

[Manu Prestat]

Your 2 inquiries are quite different:

1 - the trimming step is very easily parallelizable as you don't need the information in the other sequences to process one, I think the best way to do it in bash would be by using GNU parallel (take a look here: GNU Parallel - parallelize serial command line programs without changing them ) . However, I've been processing quite a lot of big datasets but I've always figured it was fast enough using a single processor and I'm wondering if this worths doing it in parallel.

2 - the assembly step is much more complex and I think you cannot parallelize an assembler which has not been designed for. Until recently I did not know about any assembler capable to do it, but now there is Ray and maybe (but not sure it'd fit your needs) Minia, but I personally don't have any experience with any of those.

Answer 2

[Pierre Lindenbaum]

use a makefile with option -j (num parallel jobs)

VELVET2: VELVET1
    velvetg demo -exp_cov auto

VELVET1: Library1 Library2 Library3
    velveth demo 65 -short -fastq Library1 -short2 -fastq Library2 -short3 -fastq Library3


Library1:
       statements1

Library2:
       statements2

Library3:
       statements3