Hi all, I have downloaded the whole NT database locally for running BLAST. During my search, I miss some sequences in the local NT database but are found from NCBI website. These are some of the accessions which could not be found in local NT:
Error: [blastdbcmd] Entry not found: NC_019090.1
Error: [blastdbcmd] Entry not found: NC_019424.1
Error: [blastdbcmd] Entry not found: NZ_CP021711.1
Error: [blastdbcmd] Entry not found: NZ_CP021210.1
Error: [blastdbcmd] Entry not found: NC_020278.2
Error: [blastdbcmd] Entry not found: NC_019095.1
Error: [blastdbcmd] Entry not found: NZ_CP029734.1
Error: [blastdbcmd] Entry not found: NZ_CP016389.1
Error: [blastdbcmd] Entry not found: NZ_CP029974.1
Error: [blastdbcmd] Entry not found: NZ_CP015072.1
Error: [blastdbcmd] Entry not found: NZ_CP007652.1
Error: [blastdbcmd] Entry not found: NC_024954.1
Error: [blastdbcmd] Entry not found: NC_019163.1
Error: [blastdbcmd] Entry not found: NZ_CP024879.1
Error: [blastdbcmd] Entry not found: NC_015872.1
Error: [blastdbcmd] Entry not found: NZ_CP016037.1
Error: [blastdbcmd] Entry not found: NZ_CP010880.1
Doubting if I had downloaded all the files, I checked the downloaded file number (nt.1..nt.60) and confirmed with my .nal output which looks like this:
$ cat nt.nal
#
# Alias file created 08/08/2018 12:50:38
#
TITLE Nucleotide collection (nt)
DBLIST "nt.00" "nt.01" "nt.02" "nt.03" "nt.04" "nt.05" "nt.06" "nt.07" "nt.08" "nt.09" "nt.10" "nt.11" "nt.12" "nt.13" "nt.14" "nt.15" "nt.16" "nt.17" "nt.18" "nt.19" "nt.20" "nt.21" "nt.22" "nt.23" "nt.24" "nt.25" "nt.26" "nt.27" "nt.28" "nt.29" "nt.30" "nt.31" "nt.32" "nt.33" "nt.34" "nt.35" "nt.36" "nt.37" "nt.38" "nt.39" "nt.40" "nt.41" "nt.42" "nt.43" "nt.44" "nt.45" "nt.46" "nt.47" "nt.48" "nt.49" "nt.50" "nt.51" "nt.52" "nt.53" "nt.54" "nt.55" "nt.56" "nt.57" "nt.58" "nt.59" "nt.60"
NSEQ 49266009
LENGTH 188943333900
I randomly checked the md5sums also of NT files, and they found to be same with md5sums available in the NCBI FTP page. Am I missing something here? Many thanks for your comments in advance.
•
written
by
Prakki Rama
I contacted NCBI about this issue lately and it seem they're having problems formatting the nr/nt databases that they offer via ftp. The ones on their own website are indeed OK. They are working on this issue they mentioned.
The large number of volumes of these public databases makes their maintenace a big challenge. The FASTA requires extra steps and spaces to archive, which adds significant strains to the limited resources we have on hand.
I see that the FASTA has a newer date than the preformatted, which indicate some issue in their update. Given that the FASTA has newer time stamp, it is not a surprise for that to have newer data that are not present in the preformatted versions.
I will check with our developers and ask them to look into the issue. Your patience and understanding will be greatly appreciated.
Regards,
NCBI User Services
The fasta files they provide on their ftp are OK as well, so you can download the fasta file and format the DB locally
Thanks for posting NCBI's reply @lieven.sterck. Let me check the accession in the fasta file then.
I am trying to download fasta file from FTP. But, connection is timed out. I also tried downloading using axel from terminal. Still had the same problem. Any other ideas to access the fasta sequence data?
you might give one of the mirrors a try? we often use the following one: http://mirrors.bi.vt.edu/mirrors/ftp.ncbi.nih.gov/
Sure Thanks much @lieven !! I will try it and post again!
OK, from what I can see, it seems they updated the (preformatted) nr DBs yesterday ( / this morning) (Oct 10th)
File:nr.00.tar.gz 316659 KB 10/9/2018 9:46:00 PM
The nt is still lagging behind :/ (though they updated the fasta file of nt yesterday as well)
File:nt.00.tar.gz 828406 KB 8/12/2018 2:40:00 PM
But I still can't understand why it's that difficult to update the preformatted DBs along with the fasta files ? if they don't have 1 cpu and a few Gb of RAM to spare at NCBI to accomplish this, I start to get worried.
yeah, OK , overnight might indeed be pushing it, but something like once a week should be possible, no?
Moreover, it would also be OK if they told that from now on they will not provide this anymore (and for instance only the fasta files), at least then we know what to expect.
Any thing is better than offering different versioned fasta and DB files!
Same approach here. And indeed once a week is more than frequent enough.
But an annoying issue we have in our lab is that for some resources we need to start from fasta files and for others we download the pre-made DBs.
Perhaps we should consider switching to just downloading the fasta files and build all the DBs from that.
Do you always need the fasta files? Can those be out of sync from the pre-formatted db (as you have discovered)? So basically two separate downloads.
Wonder if recreating the fasta file from the pre-formatted DB is faster than creating the DB from the downloaded fasta.
huh, ... that might be a brilliant idea (technically) to recreate from the DB rather than downloading it.
However, the pre-made DB will always be lagging behind the fasta file (as you need the fasta to create the DB), as is the case now and not with a few days but with nearly 2 months for nt . So from an 'being up-to-date' point of view we'll be better of with the fasta file
I actually did it this way but noticed that if I execute the following command:
blastdbcmd -db nt -entry all > nt.fa
Sometimes a sequence got two headers in nt.fa like this:
>accession|header1>accession|header2
AGTAGATAGAGAGACGACACTAGCATCA
Maybe the command is wrong... I do not remember the version I used back then
@gb These are merged data entries in NCBI databases.
Here is the link for the official explanation from NCBI: A: non redundant protein sequence database
Making a huge blast db takes a lot of time. Sure, they could hold the release of the fasta files until they have formatted and compressed the db files, but what would be gained? I don't really understand why they even offer the fasta files..
formatted it here myself recently, so yes indeed it takes a few hours (on a single core, no multi threading possible) but I can't see that it might be an issue for a player such as NCBI
yes, with the parse_seqids option on (always do that) it took about 4 hours, on a single core (no idea you even could parallelize anything for this?)
Log in to answer this question.
nr/nt (the one on NCBI website) is not the same database as nt which you can download from their ftp..
and what's the difference then?
and
ok, yes, that I know.
I thought the statement was that the nr (or nt) available from the ftp is different then the one from the ncbi blast page itself
That
nrdefinition is for the protein db. I don't know what exactly is different betweennr nt(the one on the website) andnt(the one of the ftp), but right nownr nthas48,336,722 seqswhereas OP'sntis slightly larger with49,266,009 seqs. I tried a few identifiers from OP and they were all RefSeq sequences. Could it be that those seqs are inntbut not with the RefSeq identifiers but GenBank identifiers, e.g. fromNZ_CP016037.1toCP016037.1, fromNC_019095.1toJF927996.1, etc.Edit. like the README states
OK, right, never noticed that before but indeed it says
nr/ntfor the non-redundant DB in blastnfrom what I can see it's a different 'state' of non-redundancy :
from the ftp README:
from the NCBI blastn page:
Though I totally agree this even adds the confusion
I assumed they are same.
makes two of us
Actually, this is not straight forward. Downloading large files is timing out. So, for now, I just used NCBI eutils to download my sequence from NCBI online:
This also has problem sometimes downloading the sequence if the internet connection is slightly off.
You are in Singapore so bandwidth should not be an issue unless you are behind some restrictive scanning firewall.