This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Xml Returned Using Efetch Differs To That Downloaded From Query At Pubmed Website

If I go to PubMed and enter this query, which currently returns around 2290 results:

"Retraction of Publication"[Publication Type]

then select "Send to File", format = XML, "Create File", the download generally takes a few seconds and returns a file with only one DOCTYPE line, as expected:

grep -c DOCTYPE ~/Downloads/pubmed_result.xml
# 1
grep DOCTYPE ~/Downloads/pubmed_result.xml
# http://www.ncbi.nlm.nih.gov/corehtml/query/DTD/pubmed_120101.dtd">

If I perform an equivalent query using the BioRuby implementation of EUtils:

require "rubygems"
require "bio"

ncbi    = Bio::NCBI::REST.new
Bio::NCBI.default_email = "me@me.com"

retmax = ncbi.esearch_count("Retraction of Publication[ptyp]", {"db" => "pubmed"})
search = ncbi.esearch("Retraction of Publication[ptyp]", {"db" => "pubmed", "retmax" => retmax})
result = ncbi.efetch(search, {"db" => "pubmed", "retmode" => "xml"})

File.open("pubmed_result.xml", "w") do |f|
  f.write(result)
end

it takes significantly longer to return the XML file, the file has a slightly different size and it contains multiple DOCTYPE lines which breaks XML parsing:

grep -c DOCTYPE pubmed_result.xml
# 23

It appears that Efetch returns separate, complete XML "chunks" and concatenates them into one file. This does not occur if a smaller subset of the variable search, e.g. search[0..4] is passed to efetch. So:

  1. Is this issue due to passing too many IDs to efetch?
  2. Have other people observed it using other implementations of Eutils?
  3. Can it be resolved using e.g. a POST query as suggested in the Eutils documentation?
pubmed eutils xml
• 8,641 views • link
updated by wdiwdi wdiwdi • written by Neilfws Neilfws

On further investigation: I forgot to pass retmax as a parameter to efetch. However, even when this is done, results seem to be returned in batches of 100. Does not occur when the Bioperl Eutils library is used, so may be a bug in (my version of?) BioRuby.

• 0 views • link
written by Neilfws Neilfws

On even further investigation: it seems that adding the parameter "step = retmax" to efetch solves the problem.

• 0 views • link
written by Neilfws Neilfws

4 answers

Here is a thread with similar problems: http://comments.gmane.org/gmane.comp.python.bio.general/6962

EFetch doesn't seem to cope well with fetching large amounts of records at once. It looks like with your query, EFetch is breaking it up into chucks of 1,000 records. Maybe they've fixed it since then to automatically break up queries into chunks of 1,000 when the returned number is large?

It is interesting though that their web implementation doesn't have this problem. You would assume they use the same EFetch system for their web interface...

• 360 views • link
written by Damian Kao Damian Kao

Seems more like chunks of ~ 100 records? 23 DOCTYPE lines for 2290 records.

• 0 views • link
written by Neilfws Neilfws

To answer my own question: efetch in BioRuby can take the parameter step= for maximum number of records to retrieve at one time.

So this line works in my code:

result = ncbi.efetch(search, {"db" => "pubmed", "retmode" => "xml"}, step = retmax)

Documented here.

• 0 views • link
written by Neilfws Neilfws

I can add some evidence to support that it is the Ruby library not properly aggregating chunked results, and not a problem on the NCBI end. You can try this query out manually from the web interface:

It does take significantly longer to download the XML this way, as you mentioned; so that's strange. But the resultant document, when it finally does arrive, only has one doctype declaration.

• 0 views • link
written by Chris Maloney Chris Maloney

Efetch does indeed have problems when the return data size gets too large (it's the total data size, not the record count), after which there may be timeouts between various internal components of the Eutils system which can lead to incomplete or corrupted results. For queries returning only UIDs in XML format, my experience is that the safe limit is about 2.5 mil records. So it is better to split the retrieval into chunks.

• 0 views • link
written by wdiwdi wdiwdi

The issue here is not incomplete or corrupt data, just how the chunks are returned.

• 0 views • link
written by Neilfws Neilfws

Yes. That was supposed to be a comment on the first and second answer.

• 0 views • link
written by wdiwdi wdiwdi

Log in to answer this question.

More posts like this