Biostar Beta. Not for public use.
Aspera ENA download: errors and permissions?
0
Entering edit mode
21 months ago
f.a.galkin • 40
Germany

I am trying to download 20TB of metagenomic data from ENA. However their standard ftp service hardly ever reaches 400KB/s and it would take me literally ages to download it all

I've been looking into the Aspera alternative, but am currently facing problems with it.

I've installed it and tried to download ERR1729192 run:

ascp -v -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 1 -T -l 100M -L- fasp@fasp.sra.ebi.ac.uk/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz ./

And I get the following log:

LOG Configuration: using v2 configuration file "/home/galkin/.aspera/connect/etc/aspera.conf", user -
LOG Initializing FASP version 3.8.1.161147, license max rate=(unlimited), account no.=1, license no.=1 product=6
LOG Configured symlink actions: create=1, follow=1, follow_wide=0, skip=0
LOG Aspera Connect version 3.8.1.161274

LOG Alternate log directory: "-"
ascp: no remote host specified
Startup failed, exit
ERR no remote host specified
ERR Missing mandatory arguments
LOG FASP Session Start uuid=cfe1b5c8-890d-40c8-a93f-466f37d0b51c op=recv status=failed errcode=106 errstr="Usage error" source= dest= source_prefix=- local=0.0.0.0:0 peer=0.0.0.0:0 tcp_port=22 os="Linux 4.4.0-62-generic #83-Ubuntu SMP We" ver=3.8.1.161147 lic=6:1:1 peeros="-" peerver=- peerlic=0:0:0 proto_sess=20003 proto_udp=20000 proto_bwmeas=20000 proto_data=2000d
LOG FASP Session Params uuid=cfe1b5c8-890d-40c8-a93f-466f37d0b51c userid=0 user="-" targetrate=10000000 minrate=0 rate_policy=fair cipher=none resume=0 create=0 ovr=0 times=0 precalc=no mf=0 mf_path=- mf_suffix=.aspera-inprogress partial_file_suffix=- files_encrypt=no files_decrypt=no file_csum=NONE dgram_sz=1492 prepostcmd=- tcp_mode=no rtt_auto=no cookie="-" vl_proto_ver=1  peer_vl_proto_ver=0 vl_local=0 vlink_remote=0 vl_sess_id=0 srcbase=- rd_sz=0 wr_sz=0 cluster_num_nodes=1 cluster_node_id=0 cluster_multi_session_threshold=-1 range=0-0 keepalive=no test_login=no proxy_ip=- net_rc_alg=alg_none exclude_older/newer_than=0/0
LOG FASP Session Stop uuid=cfe1b5c8-890d-40c8-a93f-466f37d0b51c op=recv status=failed errcode=106 errstr="Usage error" source= dest= source_prefix=- local=0.0.0.0:0 peer=0.0.0.0:0 tcp_port=22 os="Linux 4.4.0-62-generic #83-Ubuntu SMP We" ver=3.8.1.161147 lic=6:1:1 peeros="-" peerver=- peerlic=0:0:0 proto_sess=20003 proto_udp=20000 proto_bwmeas=20000 proto_data=2000d
LOG FASP Session Params uuid=cfe1b5c8-890d-40c8-a93f-466f37d0b51c userid=0 user="-" targetrate=10000000 minrate=0 rate_policy=fair cipher=none resume=0 create=0 ovr=0 times=0 precalc=no mf=0 mf_path=- mf_suffix=.aspera-inprogress partial_file_suffix=- files_encrypt=no files_decrypt=no file_csum=NONE dgram_sz=1492 prepostcmd=- tcp_mode=no rtt_auto=no cookie="-" vl_proto_ver=1  peer_vl_proto_ver=0 vl_local=0 vlink_remote=0 vl_sess_id=0 srcbase=- rd_sz=0 wr_sz=0 cluster_num_nodes=1 cluster_node_id=0 cluster_multi_session_threshold=-1 range=0-0 keepalive=no test_login=no proxy_ip=- net_rc_alg=alg_none exclude_older/newer_than=0/0
LOG FASP Session Statistics [Receiver] id=cfe1b5c8-890d-40c8-a93f-466f37d0b51c delay=0ms rex_delay=0ms ooo_delay=0ms solicited_rex=0.00% rcvd_rex=0.00% rcvd_dups=0.00% ave_xmit_rate 0.00Kbps effective=0.00% effective_rate=0.00Kbps (detail: good_blks 0 bl_total 0 bl_orig 0 bl_rex 0 dup_blks 0 dup_last_blks 0 drop_blks_xnf 0) (sndr ctl: sent 0 rcvd 0 lost 0 lost 0.00%) (rcvr ctl: sent 0 rcvd 0 lost 0 lost 0.00%) (rex  ctl: sent 0 rcvd 0 lost 0 lost 0.00%) (progress: tx_bytes 0 file_bytes 0 tx_time 0) rex_xmit_blks 0  xmit_total 0 rex_xmit_pct 0.00%

Using enaBrowserTools does not help:

enaDataGet -a -f fastq ERR1729192

Downloading file with md5 check:fasp.sra.ebi.ac.uk:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz
/home/galkin/.aspera/connect/bin -QT -L /home/galkin/ERR1729192/logs -l 100M -P33001  -i /home/galkin/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz ./ERR1729192
/bin/sh: 1: /home/galkin/.aspera/connect/bin: Permission denied
Error with Aspera transfer: [Errno 2] No such file or directory: './ERR1729192/ERR1729192.fastq.gz'
Downloading file with md5 check:fasp.sra.ebi.ac.uk:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz
/home/galkin/.aspera/connect/bin -QT -L /home/galkin/ERR1729192/logs -l 100M -P33001  -i /home/galkin/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz ./ERR1729192
/bin/sh: 1: /home/galkin/.aspera/connect/bin: Permission denied
Error with Aspera transfer: [Errno 2] No such file or directory: './ERR1729192/ERR1729192.fastq.gz'
Failed to download file after two attempts
Deleting directory ERR1729192
ERROR: Something unexpected went wrong please try again.
If problem persists, please contact datasubs@ebi.ac.uk for assistance.

I have contacted our administrator and he says that our firewall settings are ok.

Meanwhile, when I am using this example I have found on the Internet, everything works fine and I can download at 90MB/s

ascp -QTr -l 400M -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh  anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByStudy/sra/SRP/SRP009/SRP009247 .

Has anybody faced such a problem? What permission am I lacking? Is that a problem on my server side, or I need to ask for some kind of permission from ENA to use Aspera? What does "no remote host specified" mean?

ADD COMMENTlink
0
Entering edit mode

You're missing a colon after the server name: fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz

ADD REPLYlink
2
Entering edit mode
11 weeks ago
ATpoint 17k
Germany

You may try following the steps of my recent tutorial.

In your case there are two things:

1) You use the wrong path. Rather than fasp.sra.ebi.ac.uk is must be era-fasp@fasp.sra.ebi.ac.uk and

2) there must be a ":" after .uk, so era-fasp@fasp.sra.ebi.ac.uk:/vol1(...)

This command worked for me (note that I changed the path of -i to my $HOME

ascp -QT -l 300m -P33001 -i "/home/a/a_toen03/.aspera/connect/etc/asperaweb_id_dsa.openssh" era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz ./
ADD COMMENTlink
0
Entering edit mode

Thank you for the quick reply. I will make sure to study your tutorial asap. But have you used enaBrowserTools? It is supposed to take in only a run accession and look up all the needed links by itself, and I still fail at downloading

ADD REPLYlink
0
Entering edit mode

No, I never used this one. Always downloaded via command line as described in the tutorial. I also modified my answer.

ADD REPLYlink
0
Entering edit mode

Wow, thanks a lot Your line works absolutely fine for me too. All that's left is to put up a script together

ADD REPLYlink
0
Entering edit mode

You're very welcome. If the answer helped you, please consider to mark it as accepted to help others in the future ;-)

ADD REPLYlink

Login before adding your answer.

Similar Posts
Loading Similar Posts
Powered by the version 2.1