This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How can I speed up wget from UniProt for UniRef90/50 fasta?

How can I speed up the download here? https://ftp.uniprot.org/pub/databases/uniprot/uniref/uniref90/uniref90.fasta.gz

I'm running wget on our compute servers, on the head node, and my personal computer which I know has a fast internet I/O but it's VERY slow taking days to download by the estimates.

How can I speed up this download process?

wget download uniprot uniref
• 3,852 views • link
updated by Mensur Dlakic Mensur Dlakic • written by O.rka O.rka

1 answer

I think you need a program that can create multiple download streams. Here is one example:

https://github.com/aria2/aria2

With aria2, a week ago I downloaded a compressed UniProt90 file in ~12 hours.

• 0 views • link
written by Mensur Dlakic Mensur Dlakic

Does it need to run multi-threaded?

• 0 views • link
written by O.rka O.rka

It doesn't run multithreaded. It simply opens multiple download connections (something like --max-connection-per-server=5).

• 0 views • link
written by Mensur Dlakic Mensur Dlakic

A few questions:

  • What can I use for a dropin replcement with this command? wget -v -P ${DATABASE_DIRECTORY} https://data.gtdb.ecogenomic.org/releases/release207/207.0/auxillary_files/gtdbtk_r207_v2_data.tar.gz
  • How can I specify the maximum amount of connections the server will allow?
• 0 views • link
written by O.rka O.rka

For someone of your experience in this field, I think it is lazy to keep asking these types of questions. I already answered your original question, and the rest comes down to typing aria2c -h and going through the options. Nobody can tell you without testing how many connections any given server will allow.

• 0 views • link
written by Mensur Dlakic Mensur Dlakic

Touché. I'll look into it. Do you usually use --max-connection-per-server=5?

• 0 views • link
written by O.rka O.rka

Log in to answer this question.

More posts like this