This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to download NCBI fastq data to local machine where access type is "Use Cloud Data Delivery"

I have setup AWS and Google cloud accounts with billing. But both are overwhelmingly complicated for a simple user. What I would like to do is download multiple fastq files from NCBI that have gs:// or s3:// locations, to my local machine. I realise there are instructions but I cant find a simple explanation of what I need to do.

If we take AWS. I have aws cli installed. When I try:

aws cp s3://sra-pib-src-7/thesra/thefastq

I get:

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

What exactly do I need to do to sign up for with AWS and setup on AWS to allow this data download?

Do I need to create an EC2 instance, log into console, s3 copy to the EC2 instance, then download from my home machine via s3 copy - or is there a way a bypass the process of copying to an EC2 instance and direct download to home machone from the s3 bucket location on NCBI?

delivery ncbi cloud
• 5,520 views • link
updated by GenoMax GenoMax • written by daewowo • daewowo

to get data from NCBI use fasterq-dump from the SRA-toolkit, or check if the data already is sync in ENA

• 0 views • link
written by JC JC

I have used SRA download and fasterq-dump before but I want to download the original fastq files as I want to obtain the original machine read header information. SRA download doesnt keep the original machine id in the header.

I just checked and ENA files also have the original machine id/flow cell etc details overwritten by the SRA number and an index. I need to obtain the original fastq

• 0 views • link
written by daewowo • daewowo

Can you post an example SRA# you are trying to download?

When data is set to "requestor pays" having an account on AWS/GCP would be needed (which you said you have). You should be able to directly copy data to your EC2 instance.

• 0 views • link
written by GenoMax GenoMax

1 answer

If you want to download the data to your local computer then you could do so using aws/GCP command line utils. You could also directly copy the data to your EC2 instance, if you are planning to analyze it there.

With google you are able to copy the data by doing (use real values for your project etc)

gsutil -u your_project cp gs://sra-pub-src-9/SRR10620024/BDR5-2_S3_L007_I1_001.fastq.gz.1 copy_location

Something similar should be possible with AWS.

• 0 views • link
written by GenoMax GenoMax

See response from GenoMax which worked, thanks!

• 0 views • link
written by daewowo • daewowo

Log in to answer this question.

More posts like this