This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to retrieve large datasets for proteins

Under Entrez Programming Utilities Help, Application 3 says that you can RETRIEVE LARGE DATASETS. Here Chimpanzee example is given and we can retrieve mRNA sequences. But my interest is to retrieve protein sequences of my analysis. I tried the same script in PERL, in place of mRNA, i made protein and also .faa.gz files.

But i cannot get the output. Please guide me.

Thanks Naresh

sequence
• 2,883 views • link
written by Naresh Naresh

If you have all the gi list of your protein of interest. you can use eutility's EFETCH option to retrieve n number of sequences

• 0 views • link
written by Prasad Prasad 
use LWP::Simple;
$query = 'Schizosaccharomyces pombe[orgn]+AND+biomol+mrna[prop]';

#assemble the esearch URL
$base = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/';
$url = $base . "esearch.fcgi?db=nucleotide&term=$query&usehistory=y";

#post the esearch URL
$output = get($url);

#parse WebEnv, QueryKey and Count (# records retrieved)
$web = $1 if ($output =~ /<WebEnv>(\S+)<\/WebEnv>/);
$key = $1 if ($output =~ /<QueryKey>(\d+)<\/QueryKey>/);
$count = $1 if ($output =~ /<Count>(\d+)<\/Count>/);

#open output file for writing
open(OUT, ">Schizosaccharomyces pombe.fna") || die "Can't open file!\n";

#retrieve data in batches of 500
$retmax = 500;
for ($retstart = 0; $retstart < $count; $retstart += $retmax) {
        $efetch_url = $base ."efetch.fcgi?db=nucleotide&WebEnv=$web";
        $efetch_url .= "&query_key=$key&retstart=$retstart";
        $efetch_url .= "&retmax=$retmax&rettype=fasta&retmode=text";
        $efetch_out = get($efetch_url);
        print OUT "$efetch_out";
}
• 0 views • link
updated by GenoMax GenoMax • written by Naresh Naresh

The opening and closing tags in the regexes don't match e.g. you have <webenv> and <\/WebEnv>. Make sure that they match what's actually returned in $output otherwise, you won't get anything.

EDIT: For proteins, you need $query = 'Schizosaccharomyces pombe[orgn]'; and $url = $base . "esearch.fcgi?db=protein&... and $efetch_url = $base ."efetch.fcgi?db=protein&WebEnv=$web";

• 0 views • link
written by Jean-Karim Heriche Jean-Karim Heriche

This is the output..

C:\Users\Naresh\Documents>perl C:\Users\Naresh\perl_tests\hello_worldpl
Bareword found where operator expected at C:\Users\Naresh\perl_tests\hello_worldpl line 6, near ""esearch.fcgi?db=protein&$efetch_url =$base . "efetch"
        (Missing operator before efetch?)
Operator or semicolon missing before &WebEnv at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
Ambiguous use of & resolved as operator & at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
String found where operator expected at C:\Users\Naresh\perl_tests\hello_worldpl line 6, near "open(OUT, ""
        (Missing semicolon on previous line?)
String found where operator expected at C:\Users\Naresh\perl_tests\hello_worldpl line 6, near "faa") || die ""
Bareword found where operator expected at C:\Users\Naresh\perl_tests\hello_worldpl line 6, near "") || die "Can't"
        (Missing operator before Can't?)
Precedence problem: open file should be open(file) at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
String found where operator expected at C:\Users\Naresh\perl_tests\hello_worldpl line 6, near "$efetch_url = $base .""
        (Missing semicolon on previous line?)
Operator or semicolon missing before &WebEnv at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
Ambiguous use of & resolved as operator & at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
String found where operator expected at C:\Users\Naresh\perl_tests\hello_worldpl line 6, near "$efetch_url .= ""
        (Missing semicolon on previous line?)
Operator or semicolon missing before &retstart at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
Ambiguous use of & resolved as operator & at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
String found where operator expected at C:\Users\Naresh\perl_tests\hello_worldpl line 6, near "$efetch_url .= ""
        (Missing semicolon on previous line?)
Operator or semicolon missing before &rettype at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
Ambiguous use of & resolved as operator & at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
Operator or semicolon missing before &retmode at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
Ambiguous use of & resolved as operator & at C:\Users\Naresh\perl_tests\hello_worldpl line 6.
String found where operator expected at C:\Users\Naresh\perl_tests\hello_worldpl line 6, near "print OUT ""
        (Missing semicolon on previous line?)
String found where operator expected at C:\Users\Naresh\perl_tests\hello_worldpl line 6, at end of line
        (Missing semicolon on previous line?)
syntax error at C:\Users\Naresh\perl_tests\hello_worldpl line 6, near ""esearch.fcgi?db=protein&$efetch_url =$base . "efetch"
Can't find string terminator '"' anywhere before EOF at C:\Users\Naresh\perl_tests\hello_worldpl line 6.

C:\Users\Naresh\Documents>
• 0 views • link
updated by GenoMax GenoMax • written by Naresh Naresh 
use LWP::Simple;
$query = 'Schizosaccharomyces pombe[orgn]';

#assemble the esearch URL
$base = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/';
$url = $base . "esearch.fcgi?db=protein&$efetch_url =$base . "efetch.fcgi?db=protein&WebEnv=$web";

#post the esearch URL
$output = get($url);

#parse WebEnv, QueryKey and Count (# records retrieved)
$web = $1 if ($output =~ /<WebEnv>(\S+)<\/WebEnv>/);
$key = $1 if ($output =~ /<QueryKey>(\d+)<\/QueryKey>/);
$count = $1 if ($output =~ /<Count>(\d+)<\/Count>/);

#open output file for writing
open(OUT, ">Schizosaccharomyces pombe.faa") || die "Can't open file!\n";

#retrieve data in batches of 500
$retmax = 500;
for ($retstart = 0; $retstart < $count; $retstart += $retmax) {
        $efetch_url = $base ."efetch.fcgi?db=nucleotide&WebEnv=$web";
        $efetch_url .= "&query_key=$key&retstart=$retstart";
        $efetch_url .= "&retmax=$retmax&rettype=fasta&retmode=text";
        $efetch_out = get($efetch_url);
        print OUT "$efetch_out";
}
• 0 views • link
updated by GenoMax GenoMax • written by Naresh Naresh

Don't just copy/paste code from a web page. It may not be properly formatted. For example, things like 'assemble the esearch URL' are comments not code so they should be written as proper perl comments. Also you still haven't corrected the regexes. The difference between upper and lower case is meaningful. If you don't know how to program in perl, I suggest you at least have a quick look at a tutorial. Also when posting code, please try to format it for readability.

• 0 views • link
written by Jean-Karim Heriche Jean-Karim Heriche

I never did Perl. I will learn now. Sorry for not formating it for readability.

• 0 views • link
written by Naresh Naresh

0 answers

No answers yet.

Log in to answer this question.

More posts like this