I am working on nematodes and want to download ESTs from Mark Blaxter's website: http://www.nematodes.org/NeglectedGenomes/NEMATODA/Steinernema/index.html
The programm that curates the database seems outdated, so I guess I need to download them manually. Is there a solution for automating this?
e.g. download all pages in this format: http://www.nematodes.org/NeglectedGenomes/NEMATODA/Steinernema/wwwPartiGene_cluster.php?cluster=SCC*&chosenorganism=SCC
1 answer
This will be painful ... did you try to write the author an email? If he could simply send you the data you would save yourself some time.
If not, it seems like you would have to do html scraping e.g. here you could find more information but yep, it is painful.
Do you know what "clusters" and "organisms" exist on this page? If you do you could try to construct the URL yourself - so change the variables behind the ? accordingly and browse through all the pages you need that way. However, it all is painful.
P.S.: the example you posted doesn't seem to work
P.S.S: php is NOT a database!
Ah, I see. What I did was to search with the box empty, then you get a site with all the clusters. It is obviously impossible to link to this page. However, this would be where I would start to scrap the page. BUT, check the post above form genomax - maybe you can get the data from somewhere else? It sounds like a lot of effort that is not really connected with your work to try to scrap this from the page, so finding another way would be beneficial.
Log in to answer this question.
NCBI has them as well. Should be less painful to get.
Yup, if there is another way to get this data, I would also recommend to go that way.