This is a test version of Biostars. For the public version, visit https://www.biostars.org.
N-Gram Plots Using Pmc Or Pubmed Abstracts

I am looking at a way to visualize distribution of a set of keywords over the years in PubMed. I am sure that there must be a tool to do that. An ideal solution will be similar to Google books Ngram Viewer. Here is an example plot using 2 key words.

alt text

Do you know about such a tool ? Please share !

text data visualization
• 7,034 views • link
updated by Biostar Biostar • written by Khader Shameer Khader Shameer

I think this is a great question but as a geneticist, I am puzzled by one aspect of that graph. It appears that google is showing citations for the word "gene family" before the word "gene" was coined. I can't think of a reason for this but it may be something to keep in mind when doing these searches.

• 0 views • link
written by SES SES

This may be due to i) wrong publication date assigned to some google books entries ii) default smoothing of the graph. This is the case for a "peak" of 'bioinformatics' term use around year 1900 :-)

• 0 views • link
written by Jan Kosinski Jan Kosinski

SES & Jan thanks. I know the Google n-gram plot is not correct from a scientific context and this specific example have lot of false positive and low specificity :). The words could have come from different contexts, not exactly biology. You can click on the interval link given in that page to see the corresponding books that have these keywords.

• 0 views • link
written by Khader Shameer Khader Shameer

Another issue with the Google data is incorrect values due to OCR errors (conversion of scanned documents to text). Frankly, I'm amazed at how little attention many people pay to n-gram data quality; it seems they are dazzled by the "big data" aspect.

• 0 views • link
written by Neilfws Neilfws

3 answers

Nothing particular comes to mind aside from searching PubMed and downloading all the citation in CSV format, extracting the column with all the years into 'years.txt' and then running the following linux scripts: 

sort -n years.txt | uniq -c

This will give you a count how many citings you have in a year. I did this few months back for the word "metadynamics" and I got the following plot (plotted in libreoffice.org).

alt text

• 292 views • link
written by dimkal dimkal

Thanks dimkal, this is helpful. I am specifically interested in an n-gram style plot from a text-mining perspective. Wanted to know how my key-words of interests are compare to other keywords in PMC full-text or PubMed abstracts

• 0 views • link
written by Khader Shameer Khader Shameer

Do you need full-text search? If you're content with just title and abstract, Neil has you covered. If you have trouble with his examples, I've tweaked his code to do similar things (see below) and may be able to help.alt text

• 252 views • link
written by Chris Miller Chris Miller

Thanks Chris, this is nice. Are you suppose add any code in the answer ? I am specifically interested in an n-gram style plot from a text-mining perspective. Wanted to know how my key-words of interests are compare to other keywords in PMC full-text or PubMed abstracts.

• 1 views • link
written by Khader Shameer Khader Shameer

Neil's site (that I linked to) has the basic code you'll need (some Ruby and some R).

• 0 views • link
written by Chris Miller Chris Miller

Although it works on Arxiv, here's a recently published tool that might be of interest to you: http://arxiv.culturomics.org/

and related news on NYtimes: http://www.nytimes.com/2012/03/25/business/words-by-the-millions-sorted-by-software.html?_r=1&src=tp

• 182 views • link
written by B. Arman Aksoy B. Arman Aksoy

Thanks Arman, this is a nice tool to look the prevalence of keywords in "Quantitative Biology" articles in Arxiv.

• 0 views • link
written by Khader Shameer Khader Shameer

Log in to answer this question.

More posts like this