This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Download all files from a web directory which requires login ID and password by using command line (Linux).

I have a web directory which requires login and password for access. I need to download multiple data files from the web directory by using linux command line but I don't know how to get the web links to all files in the directory as well as how to use a list of web links in wget command. Any help will be appreciated.

Thanks

linux web directory
• 9,202 views • link
written by eDNAuRNA • eDNAuRNA

In addition to @Alex's solution below you may need to use -r and -A option to get all files (of a kind).

• 0 views • link
written by GenoMax GenoMax

and -np and -nd

• 0 views • link
written by cpad0112 cpad0112

Thanks a lot Alex.

For getting the list of links, can we have a solution in Linux?

• 0 views • link
written by eDNAuRNA • eDNAuRNA

Please use ADD COMMENT/ADD REPLY when responding to existing posts to keep threads logically organized. This comment belongs under @Alex's answer.

SUBMIT ANSWER is for new answers to original question.

• 0 views • link
written by GenoMax GenoMax

1 answer

If the site uses basic authentication, you could use wget --user=<user> --password=<password> "<url>" to download the specified url.

The wget tool is for requests. It won't give you a list of links. You could use Python requests with scraping libraries: https://blog.hartleybrody.com/web-scraping-cheat-sheet/

• 0 views • link
written by Alex Reynolds Alex Reynolds

Log in to answer this question.

More posts like this