This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Large file transfer

Hi

I need to transfer large files between Linux servers in two locations. Is there a better approach than scp and rsync.

Thanks Philge

linux big data sequencing
• 1,714 views • link
updated by Jean-Karim Heriche Jean-Karim Heriche • written by philgev2 • philgev2

Better in what sense? Faster? Safer?

You may want to compress the files before the transfer, or use the -C switch with scp or -z with rsync, both of which will compress the stream.

• 0 views • link
written by Mensur Dlakic Mensur Dlakic

rsync is probably as efficient as it gets when it comes to freeware, so if you experience slow download rates then you probably have a bandwidth problem. Commercial software would be Aspera but depending on the size of the files you indeed could simply pack it onto a harddrive and ship it. That is fairly common, Novogene for example does that all the time.

• 0 views • link
written by ATpoint ATpoint

1 answer

rsync with compression is the standard way to go. I've also used lftp to parallelize transfer but be aware that this puts some pressure on the receiving end and some may have limits. You don't say what large is for you and where the locations are but for long distance transfer of several tens of TB, it may still be faster and safer to send copies on disks.

• 0 views • link
written by Jean-Karim Heriche Jean-Karim Heriche

Log in to answer this question.

More posts like this