This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Converting a bunch or SRA files using fastq-dump split-files

I have recently downloaded a bunch of sra files. and i like to convert them to fastq paired reads. It works by doing this :

./fastq-dump --split-files /Users/medsmit/ncbi/public/sra/SRR3501908.sra

But I need a way to convert them all together.

I was trying this

for i in  `ls /Users/medsmit/ncbi/public/sra/*.sra' ; do ./fastq-dump -- split-files $f; done

But definitely doing some silly mistake as its not working. Can anyone please help me? Thank you, Suparna

sra split-files fastq-dump ncbi
• 5,501 views • link
updated by ATpoint ATpoint • written by Mitra • Mitra

Do you have a space between -- and split-files in the loop? I would also use ls -1 so only one file is fed to fastq-dump for each iteration of the loop.

• 1 views • link
written by GenoMax GenoMax

Thanks Genomax. Yes I do have space between -- and split-files in the loop. Also I tried with ls -l After I pass this code bellow: 

 medsmit$ for i in  `ls -l /Users/medsmit/ncbi/public/sra/*.sra' ; do ./fastq-dump -- split-files $f; done

I only see 

>

As if it entered in any interface. Not sure what wrong I am doing. Thanks, Suparna

• 1 views • link
written by Mitra • Mitra

You can't have a space between --split-files. That was also a 1 (number one) not l(L) in the ls command.

And two additional mistakes noted by @jean below.

• 1 views • link
written by GenoMax GenoMax

Alternatively, you can always check the ENA for your files, which are typically mirrored there directly as fastq, or use parallel-fastq-dump (python3) if the sra files are big (tens of Gb).

• 1 views • link
written by ATpoint ATpoint

1 answer

  1. You have ' instead of `
  2. You need $i instead of $f

  3. You need --split-files not -- split-files

    for i in `ls -1 *.sra` ; do ./fastq-dump --split-files $i; done

• 1 views • link
written by jean.elbers jean.elbers

jean.elbers Thanks for pointing these errors to me.. part of which just got introduced when I wrote this post. Now it seems like its creating fastq files, but unfortunately also creating some strange error: 

./fastq-dump : 2.9.0

2018-05-24T12:10:07 fastq-dump.2.9.0 err: item not found while constructing within virtual database module - the path '1' cannot be opened as database or table
2018-05-24T12:10:07 fastq-dump.2.9.0 err: item not found while constructing within virtual database module - the path 'medsmit' cannot be opened as database or table
2018-05-24T12:10:08 fastq-dump.2.9.0 err: item not found while constructing within virtual database module - the path 'staff' cannot be opened as database or table
2018-05-24T12:10:08 fastq-dump.2.9.0 err: item not found while constructing within virtual database module - the path '25754388' cannot be opened as database or table
2018-05-24T12:10:08 fastq-dump.2.9.0 err: item not found while constructing within virtual database module - the path '18' cannot be opened as database or table
2018-05-24T12:10:08 fastq-dump.2.9.0 err: item not found while constructing within virtual database module - the path 'May' cannot be opened as database or table
2018-05-24T12:10:08 fastq-dump.2.9.0 err: item not found while constructing within virtual database module - the path '15:16' cannot be opened as database or table
Read 134818 spots for /Users/medsmit/ncbi/public/sra/SRR3502002.sra
Written 134818 spots for /Users/medsmit/ncbi/public/sra/SRR3502002.sra

Trying to understand why are they there . Thanks,

• 1 views • link
written by Mitra • Mitra

Yes the spots looks correct. So I get the fastq results. But I am not sure what are all these messages though !

• 1 views • link
written by Mitra • Mitra

Are you still using a l (lower case L) instead of 1 (number one) in your ls command?

• 1 views • link
written by GenoMax GenoMax

yes I am using for f inls -l /Users/medsmit/ncbi/public/sra/*.sra; do ./fastq-dump --split-files $f; done as my command now.

• 1 views • link
written by Mitra • Mitra

You need to use number 1 instead of lower-case letter l.

• 1 views • link
written by GenoMax GenoMax

Thank you Genomax. Sorry I couldn't reply you yesterday as Biostars restricted my daily comments limit. This time with your suggestion it works :) can you please tell me what is the exact difference in l and 1. I can see from man page -l use a long listing format and -1 list one file per line. But what I don't understand is why -l wouldn't work. Sorry for asking all these question. Actually I am a self learner. Thank you again.

• 1 views • link
written by Mitra • Mitra

Not a problem. With the long-listing (l, character lower-case L) you are getting additional information about unix permissions/group ownership/file size etc in the listing. You don't want that to be used an an input for fastq-dump so listing just the file paths one line at a time is the way to do this with 1 (number one).

• 1 views • link
written by GenoMax GenoMax

Great ..that is really helpful. I understand now thats the reason with -l I was getting result but also additional error msg as probably fastq-dump was not dealing well with all that extra information. Thank you very much, S :)

• 1 views • link
written by Mitra • Mitra

Log in to answer this question.

More posts like this