This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Get wget to download only new items from a list

I'm using wget to download a list of items.

Here's a sample of the long list of links in 'cram_download_list.txt':

ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR323/ERR3239319/NA12750.final.cram
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR323/ERR3239324/NA12763.final.cram
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR323/ERR3239480/HG00118.final.cram
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR323/ERR3239480/HG00125.final.cram

And the wget command, which works well:

wget -i cram_download_list.txt

But the pipe breaks part way through the download. I don't want to re download the ones I've already got. Is there a way to get wget to ignore the files I've already got without having to remake the 'cram_download_list.txt' file?

wget linux
• 1,858 views • link
updated by Ram Ram • written by michael.flower.14 • michael.flower.14

try using -r and --tries options for wget.

• 0 views • link
written by cpad0112 cpad0112

2 answers

Actually, what I needed was

wget -c -i cram_download_list.txt

This continues partially downloaded files, rather than skipping them!

• 0 views • link
written by michael.flower.14 • michael.flower.14

I suppose wget -nc -i cram_download_list.txt is what you need

• 0 views • link
written by Hood • Hood

In that case, accept their response as well.

• 0 views • link
written by Ram Ram

Log in to answer this question.

More posts like this