This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to analyze SRA metagenomic data without downloading large files locally (Galaxy or cloud alternatives)?

Hi everyone, I am currently learning metagenomic data analysis using QIIME2 (MOSHPIT) and Galaxy. I am working with a public dataset from: BioProject: PRJNA1236379 Example run: SRR32701636 The dataset is shotgun metagenomics (paired-end), and each sample can expand to several GB after extraction. My problem is: I am using a personal laptop with limited storage, and I cannot download or process these datasets locally (I encountered “not enough space” errors when using fasterq-dump). I would like to ask:

  1. Is there a way to analyze SRA data without downloading it locally?
  2. Are there recommended platforms (Galaxy, cloud, or others) that can handle this efficiently?
  3. Is it possible to stream or directly process SRA data from NCBI without full download?
  4. What is the best practice for beginners working with large metagenomic datasets? My goal is to follow a metagenomics tutorial (including QC, taxonomic classification, and possibly MAG reconstruction), but I am blocked by storage limitations. Any advice or recommended workflows would be greatly appreciated. Thank you!
metagenomics

2 answers

There are no truly free compute resources (galaxy may work if your can work within its storage limits or https://cyverse.org/ may be another option) that would allow one to work with large sequence datasets. Generally people will have access to institutional resources to do this kind of work. It is possible to subscribe to cloud compute (if you are able). One can't analyze datasets by streaming from SRA.

If you eventually intend to create datasets of your own in future please make sure you can arrange the compute resources beforehand before you invest time/effort into creating those datasets.

Perhaps you (or your mentor) can try and collaborate with another institution/lab, which may have the compute resources available.

The truth is, as you progress, you’ll need more and more storage space.

One strategy I’ve found works is to use Google Drive + Google Colab Pro. This way, you can download your reads directly to Google Drive and link it to Google Colab. It’s slow, but it works.

However, I’d recommend you buy a portable SSD straight away and store your files on it. Next, you’ll need to access an HPC or institutional server.

I hope it helps

While google drive+collab may work with smaller datasets, it would not be a good way to analyze larger datasets. It would also be difficult to properly record analysis steps to ensure reproducibility, which would be important for publishing the results.

PS: Biostars SPAM filter had flagged your message. It has now been restored.

Log in to answer this question.