Thanks David. As a future reference , mapping NG_ ids can be done using this file ftp://ftp.ncbi.nih.gov/refseq/H_sapiens/RefSeqGene/gene_RefSeqGene

Thanks David. In case any one needs it , mapping NG_ ids can be done using this file ftp://ftp.ncbi.nih.gov/refseq/H_sapiens/RefSeqGene/LRG_RefSeqGene

It looks like this file has been split into two now, so the URLS are:

ftp://ftp.ncbi.nih.gov/refseq/H_sapiens/RefSeqGene/refseqgene1.genomic.gbff.gz ftp://ftp.ncbi.nih.gov/refseq/H_sapiens/RefSeqGene/refseqgene2.genomic.gbff.gz

UPDATE: as of 30th Apr 2011, there are three files ftp://ftp.ncbi.nih.gov/refseq/H_sapiens/RefSeqGene/

the code has a bug,there was a extra "&" before geneid should be:

 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=gene&id=' + gids + '&retmode=json';

Hi, Michal, thank you for your awesome work. I've tried your script and run it as you put, however something just went run. Can you do me a favor?

below is how I run it and the output

gongjing@hekekedeiMac ..tion-corrrelation/results/biomart/ucsc % ll 

total 4.0K
-rwxr--r-- 1 gongjing staff 2.0K Aug 21 16:02 ucsc_download.sh 

gongjing@hekekedeiMac ..tion-corrrelation/results/biomart/ucsc % source ucsc_download.sh 

gongjing@hekekedeiMac ..tion-corrrelation/results/biomart/ucsc % get_whole_genome_table summary.tsv.gz genes refGene hgFixed.refSeqSummary gzip
--2017-08-21 20:54:51--  http://genome.ucsc.edu/cgi-bin/hgTables Resolving genome.ucsc.edu... 128.114.119.134, 128.114.119.135,
128.114.119.133, ... Connecting to genome.ucsc.edu|128.114.119.134|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: 'STDOUT'
-                                                               [  <=>                                                                                                                                       ]  40.56K   109KB/s   in 0.4s
2017-08-21 20:54:52 (109 KB/s) - written to stdout [41537]
hgsid=604153833_Jsj2HlaA2zgpZxEPKxAxKQ0XgqWL&jsh_pageVerPos=0&posiion:chr21=33031597-33041570&clade=mammal&org=Human&db=hg19&hga_group=genes&hga_rack=refGene&hga_able=hgFixed.refSeqSummary&hga_regionType=genome&hga_oupuType=primaryTable&boolshad.sendToGalaxy=0&boolshad.sendToGrea=0&boolshad.sendToGenomeSpace=0&hga_ouFileName=oupu&hga_compressType=gzip&hga_doTopSubmi=ge+oupu
--2017-08-21 20:54:52--  http://genome.ucsc.edu/cgi-bin/hgTables Resolving genome.ucsc.edu... 128.114.119.135, 
128.114.119.133,
128.114.119.136, ... Connecting to genome.ucsc.edu|128.114.119.135|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: 'summary.tsv.gz'
summary.tsv.gz                                                  [  <=> ]  41.50K  85.1KB/s   in 0.5s
2017-08-21 20:54:54 (85.1 KB/s) - 'summary.tsv.gz' saved [42497]

gongjing@hekekedeiMac ..tion-corrrelation/results/biomart/ucsc % ll 
total 48K
-rw-r--r-- 1 gongjing staff  42K Aug 21 20:54 summary.tsv.gz
-rwxr--r-- 1 gongjing staff 2.0K Aug 21 16:02 ucsc_download.sh

We at SolveBio have also run into this problem while parsing these flat files. There are some odd exceptions and omissions in these flat files. For instance, sometimes records appear with no summary or there are duplicate records. There are even instances where the gene is just absent. Here is an expansion on the code written by David Quigley that tries to account for some of this.

import sys
import gzip

def run(filepath):
    with gzip.open(filepath, 'rb') as f:
        locus2comment = {}
        in_comment = False
        first_time_symbol = False
        first_time_entrez = False
        real_gene = False
        for line in f:
            if line[0:5] == "LOCUS":
                real_gene = False
                first_time_symbol = True
                first_time_entrez = True
                locus = line.split()[1]
                comment = ""
            # elif line[0:7] == 'VERSION':
            #     print line.split('\n')
                # locus2comment[locus] = line.strip().split()[0]
            elif line[0:7] == "COMMENT":
                in_comment = True
                comment += line.split("    ")[1].replace("\n", " ")
            elif line[0:7] == "PRIMARY":
                in_comment = False
                try:
                    locus2comment[locus] = comment.split("Summary:")[1]
                except:
                    locus2comment[locus] = "Remove Me"
            elif line[0:9] == '     gene' and 'complement' not in line:
                real_gene = True
            elif line[0:27] == '                     /gene=':
                if first_time_symbol and real_gene:
                    locus2comment[locus] += \
                        '\t' + line.strip().split('/gene=\"')[1][:-1]
                    first_time_symbol = False
            elif '/db_xref="GeneID:' in line:
                if first_time_entrez and real_gene:
                    locus2comment[locus] += \
                        '\t' + line.strip().split('GeneID:')[1][:-1]
                    first_time_entrez = False
            elif in_comment:
                comment += line.split("            ")[1].replace("\n", " ")

    for locus in sorted(locus2comment):
            if "Remove Me" in locus2comment[locus]:
                del locus2comment[locus]

    with gzip.open(filepath[:-8] + '.tsv.gz', 'wb') as outfile:
        for locus in sorted(locus2comment):
            if locus2comment[locus] == "Remove Me":
                continue
            outfile.write(locus + '\t' + locus2comment[locus] + '\n')

if __name__ == '__main__':
    run(sys.argv[1])

SolveBio parses and versions this dataset along with many others popular in bioinformatics with a full API for easy access. Check it out, you may find it saves you a lot of time. https://www.solvebio.com/library/RefSeqGene