This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to parallelize fastq-dump command when reading SRA IDs from a .txt file?

How to paralellize fastq-dump command when reading SRA IDs from a .txt file?

here is my working code without paralell, it downloads a pair of fastq files:

    list=`cat SRAIdFromPythonInput.txt` # list of the SRA record file  IDs.
     for i in $list
     do  echo $i
    ./fastq-dump --split-files $i -v
     done

How to rewrite it using parallel GNU to make it download all the data with SRA IDs written in .txt file, not a single pair of fastqs? How to apply pattern "cat list | parallel "do-something1 {} config-{} ; do-something2 < {}" | process-output" to these codes?

paralell gnu shell fastq-dump sra
• 6,493 views • link
updated by ole.tange ole.tange • written by bioinform • bioinform

I'm too lazy to check/test: what would be the generated files for one given ID ?

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

2 fastqs with SRA ids as the names

• 0 views • link
written by bioinform • bioinform

what would be the names ? ID.fq.gz ? ID.fastq ? ID_R1.fq ? ID_R1.fastq.gz ?

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

ID.fastq a pair of them, I use renaming code in the next step

SRR5656566_1.fastq and SRR5656566_2.fastq

• 0 views • link
written by bioinform • bioinform

2 answers

using a Makefile

IDS=$(shell cat SRAIdFromPythonInput.txt)

%_2.fastq: %_1.fastq
    touch -c $@

%_1.fastq:
    ./fastq-dump --split-files $* -v && touch -c $@

all: $(addsuffix _2.fastq,$(IDS)) $(addsuffix _1.fastq,$(IDS))

invoke with make and the number of parallel jobs. e.g:

make -j 16
• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

thank you for your efforts, could you please write these codes in a manner of the pattern of the GNU parallel: cat list | parallel "do-something1 {} config-{} ; do-something2 < {}" | process-output, why do you use Makefile? and is there any tutorial, article or a chapter on using it in bioinformatics? I have never used Makefile for NGS data processing. I found one at http://bsmith89.github.io/make-bml/

• 0 views • link
written by bioinform • bioinform

could you please write these codes in a manner of the pattern of the GNU parallel

no

why do you use Makefile?

because it works, it's easy , standard, ubiquitous, universal , etc...

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

thanks, need code examples using GNU parallel, however,

• 0 views • link
written by bioinform • bioinform

It is unclear to me what SRAIdFromPythonInput.txt contains. Can you give a couple of lines as example?

doit() {
  i="$1"
  echo "$i"
  ./fastq-dump --split-files $i -v
}
export -f doit
parallel doit :::: SRAIdFromPythonInput.txt
• 0 views • link
written by ole.tange ole.tange

It contains a column of SRA IDs:

 SRR5656566
 SRR5656567
 SRR5656518
 SRR5656500

thx

• 0 views • link
written by bioinform • bioinform

Log in to answer this question.

More posts like this