This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Ensembl FTP Mirror in USA? Slow Downloads

Hello I work on a fairly good cluster at an R1 university in the USA. I am downloading the VEP 101 indexed cache files and it's slow as Christmas. I mean AOL speeds slow.

If I download a similar size file from AWS or NCBI it's extremely fast.

Is there a mirror for the USA containing the VEP cache files. I would only need hg38 merged VEP 101.

10 hours to download a 17GB file at 268KB/s

Give me a break, I get you Europeans don't like the USA but come on guys ;)

ensembl vep
• 4,653 views • link
updated by jessime.kirk • jessime.kirk • written by QVINTVS_FABIVS_MAXIMVS QVINTVS_FABIVS_MAXIMVS

1 answer

Perhaps the problem is local? Is your firewall stepping in? I just tried the download.

$ curl -O ftp://ftp.ensembl.org/pub/release-101/variation/indexed_vep_cache/homo_sapiens_vep_101_GRCh38.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 12 14.2G   12 1882M    0     0  32.4M      0  0:07:30  0:00:58  0:06:32 33.9M
• 0 views • link
written by GenoMax GenoMax 
$ curl -O ftp://ftp.ensembl.org/pub/release-101/variation/indexed_vep_cache/homo_sapiens_merged_vep_101_GRCh38.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 16.4G    0 13.8M    0     0   733k      0  6:31:51  0:00:19  6:31:32  690k

Not sure, I am on a dedicated data mover node and can obtain 10x speeds on NCBI FTPs and AWS buckets. So I am not sure why it's painfully slow for Ensembl. Thanks!

• 0 views • link
written by QVINTVS_FABIVS_MAXIMVS QVINTVS_FABIVS_MAXIMVS

AFAIK Ensembl ftp site has no mirrors (unlike main Ensembl site which does). May want to check with your local networking folks to see if they can shed some light. On east coast no problems.

• 0 views • link
written by GenoMax GenoMax

Ever get a response? Having the same issue.

• 0 views • link
written by osmith • osmith

Not seeing a problem.

$ curl -O ftp://ftp.ensembl.org/pub/release-101/variation/indexed_vep_cache/homo_sapiens_vep_101_GRCh38.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3 14.2G    3  532M    0     0  21.5M      0  0:11:17  0:00:24  0:10:53 12.3M
• 0 views • link
written by GenoMax GenoMax

May depend on time of day. I'm getting abysmal speeds on local, remote server, and cloud in eastern and western USA.

• 0 views • link
written by trodriguez2011 • trodriguez2011

Just to add another example to the pile:

❯ networkQuality
==== SUMMARY ====
Upload capacity: 31.633 Mbps
Download capacity: 494.663 Mbps
Upload flows: 12
Download flows: 20
Responsiveness: Medium (216 RPM)

My overall network quality, versus trying to grab the VEP cache:

❯ curl -O https://ftp.ensembl.org/pub/release-108/variation/indexed_vep_cache/homo_sapiens_vep_108_GRCh38.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  1 21.2G    1  283M    0     0   406k      0 15:13:50  0:11:52 15:01:58  405k
• 0 views • link
written by jessime.kirk • jessime.kirk

Log in to answer this question.

More posts like this