This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Download of FASTQ files from the European Nucleotide Archive (ENA) using Aspera

Hi there, I am trying to retrieve sequencing data from ENA using the Aspera client and following the tutorial:

Fast download of FASTQ files from the European Nucleotide (ENA)

The tutorial has been really helpful and I followed each step. I get a download.txt file which is a list of items looking like:

ascp -QT -l 300m -P33001 -i $HOME/Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

I then followed the tutorial and ran:

while read LIST; do
$LIST; done < download.txt

This leads to either two errors

Startup failed, exit
ascp: no remote host specified

Or:

-bash: $LIST: ambiguous redirect

I also tried running:

$HOME/Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh

And get a permission denied error.

I'm new to this and haven't found a solution yet.

rna-seq sequencing
• 9,036 views • link
updated by ATpoint ATpoint • written by r.castellan • r.castellan

This leads to either two errors

What are you doing differently to get the two errors?

• 0 views • link
written by mbelmadani mbelmadani

If I hit return after do I tend to get the second error and if I don't the first one. Thanks!

• 0 views • link
written by r.castellan • r.castellan

I am getting following error while downloading fastq files with ascp

(base) [nikhil@hpc04 dmd_SRA]$ ascp -QT  -l 300m -P33001 -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/ERR353/008/ERR3537668/ERR3537668_1.fastq.gz ./

ascp: Target address not available 
Startup failed, exit

Please guide me in this regard

• 0 views • link
updated by ATpoint ATpoint • written by nikhilshinde0909 • nikhilshinde0909

The command runs smoothly on my machine. Can you try:

ascp -QT -l 300m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/ERR353/008/ERR3537668/ERR3537668_1.fastq.gz .
• 0 views • link
written by ATpoint ATpoint

Just wanted to mention that the problem persists even after specifying the home directory with $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh.

• 0 views • link
written by sbstevenlee • sbstevenlee

Hello, did you ever figure out what caused the issue? I encountered the exact same issue while running on server (let's call this server A):

(fuc) [sbslee@bdcm03 ~]$ .aspera/connect/bin/ascp -QT -l 100m -P33001 -k 1 -i .aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/run/ERR323/ERR3239277/NA06986.final.cram .
ascp: Target address not available 
Startup failed, exit

The funny thing is, the exact same command works just fine in my local Linux PC:

(fuc) [sbslee@localhost Desktop]$ .aspera/connect/bin/ascp -QT -l 100m -P33001 -k 1 -i .aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/run/ERR323/ERR3239277/NA06986.final.cram .
NA06986.final.cram                                                                                                                      10% 1539MB  1.3Mb/s    38:04 ETA

So I was thinking it might be a server-related issue, but I can't think of any reason why. I also confirmed that in a different server (call it server B), the command works OK. So it got to be an issue in server A, but I didn't change any configuration, etc., so I'm not sure why I'm seeing this problem. Server A is connected to Internet for sure (I can download stuff such as $ git clone). Any help would be much appreciated!

• 0 views • link
written by sbstevenlee • sbstevenlee

I would talk to the server admins and discuss potential problems. It is obviously a specific problem with connection on that particilar server. Maybe some kind of firewall that is in place or ports that are not open, idk.

• 0 views • link
written by ATpoint ATpoint

1 answer

Thanks for reporting the issue. I could reproduce the error and think it is due to the whitespace in the name of the Aspera folder. Could you try to rename Aspera Connect into something without whitespace e.g. AsperaConnect.app, then adjust the download command accordingly and try again. I guess that is Mac-specific because (wisely) Linux does not use a whitespace in the folder name. I Did it on my machine and the errors disappeared, still I got a time-out message. Not sure if this is due to an error at ENA which happens at times, so if you cannot download, maybe wait a day and try again. Connection errors are rather frequent in my experience. I will double check when I've access to my workstation tomorrow.

==> Edit: The whitespacing is not an issue if properly escaped. I updated the command in the tutorial so that the awk command does not mess it up. It should be fine towards that initial error you got.

• 0 views • link
written by ATpoint ATpoint

Thanks a lot, will try that now and update.

• 0 views • link
written by r.castellan • r.castellan

Just checked on both Linux and Mac and the same problem, I think this is due to server problems at ENA. I will update the tutorial towards this whitespace problem, thanks again for reporting.

• 0 views • link
written by ATpoint ATpoint

Oh right! Sorry only saw your reply now. No worries, thanks for helping

• 0 views • link
written by r.castellan • r.castellan

So I changed it and I get a new error:

ascp: Private key file not found at path /Users/Raphael/.ssh/$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

I'm novice, not sure it was worth pointing out, but does this path look strange?

I also ran the following: 

$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

I get the permission denied error still

• 0 views • link
written by r.castellan • r.castellan

Is there a file at the location? What do you get if you do ls -l $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh? If your SSH key is in some other location then you will need to use that path.

Are you running macOS catalina BTW?

• 0 views • link
written by GenoMax GenoMax

I get: -rw-rw-r--@ 1 Raphael staff 668 8 Nov 15:12 /Users/Raphael/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

I am running Mojave still

• 0 views • link
written by r.castellan • r.castellan

It is normal that you get the permission denied when calling this file, it is also not an executable so no point in trying to run it. Your problem is here:

/Users/Raphael/.ssh/$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

Remove /Users/Raphael/.ssh/ from it as $HOME is already there.

Try and it should work unless there is something odd going on with your environment:

ascp -QT -l 300m -P33001 -i $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

• 0 views • link
written by ATpoint ATpoint

So when I ran the code by itself it worked. However, it fails when I run:

while read LIST; do
$LIST; done < download.txt

when I head download.txt I get:

ascp -QT -l 300m -P33001 -i $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .
• 0 views • link
written by r.castellan • r.castellan

What is the error mesage?

• 0 views • link
written by ATpoint ATpoint

sorry I had to wait 6 hours as I had put up 5 posts...

the error message is the same as before: 

ascp: Private key file not found at path /Users/Raphael/.ssh/$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh
• 0 views • link
written by r.castellan • r.castellan

It is puzzling that your $HOME variable is expanding correctly to /Users/Raphael when you use it here ls -l $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh but when you try to run the loop it seems to change to something strange /Users/Raphael/.ssh/$HOME and does not properly expand.

What do you get when you echo $HOME?

What happens if you simply try one file like this? Does this work?

ascp -QT -l 300m -P33001 -i /Users/Raphael/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .
• 0 views • link
written by GenoMax GenoMax

So echo $HOME returns:

/Users/Raphael

The following worked, so I guess it's a problem in the loop which adds the extra bits?

ascp -QT -l 300m -P33001 -i /Users/Raphael/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .
• 0 views • link
written by r.castellan • r.castellan

It is indeed odd that $HOME seems to improperly expand. I used $HOME in the tutorial commands to provide a generic command, but glad it works now. You can simply edit the awk command that created the download.txt by replaceing the $HOME with /Users/Raphael/. This should be independent of what the loop does as it only calls what is inside the download list.

• 0 views • link
written by ATpoint ATpoint

It's running nicely, the session is timing out every now and then on certain files, is there a way to automatically determine which ones have failed and re-run them? Maybe matching the output files with the download.txt file?

• 0 views • link
written by r.castellan • r.castellan

Log in to answer this question.

More posts like this