I agree, I use mysql for the same job, it's very easy to use and you can also download refseq on your computer. It requires almost no computer power, works fine on laptop too.
I want to compare the quality of different human genome assemblies by looking at their inclusion of the RefSeq genes.
On the UCSC browser I can call the locations of RefSeq genes by their accession numbers in any assembly--for example, https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg16&position=chr22%3A17007506-17034714&hgsid=381962759_r6crkXh3VMlFtCa2rnXaO5BTAjcH.
However, as a newbie at programming in general, I'm unsure how to scrape these inclusions or lack thereof. Anyone have tips?
4 answers
download the raw data:
http://hgdownload.cse.ucsc.edu/goldenPath/hg16/database/refGene.txt.gz
http://hgdownload.cse.ucsc.edu/goldenPath/hg19/database/refGene.txt.gz
Don't escape it. You can query their mysql server directly:
mysql --user=genome --host=genome-mysql.cse.ucsc.edu -NB -e 'select * from refGene'
As a former Ensembl team member, I just want to emphasise that scraping websites is absolutely NOT DONE!!! I know of people who were scraping the Ensembl Genome Browser website and were given IP bans because of this (which were lifted again after Ensembl spoke with them and told them how to get the desired data without slowing down / bringing down their production webservers). So, please be aware of this! As already indicated in the other responses, there are many other ways to get the UCSC data (mysql, downloads, Table browser).
Log in to answer this question.