Thanks. I am interested in gz files or any flat file data. Is there any tool/method of just only downloading changes!
What are the best methods to maintain up-to-date biological database?
I used a gnome from NCBI which was a large gz file. Now there is a newer release available in gz format. Every-time there is a change in database I have to download it completely and reanalysis the data. Same is the problem with some other databases. Is there a way to just only get the information which have been updated only?
2 answers
It depends on which kind of analysis you do. if your outputs are simple data representation such as .bed or .bedgraph files the liftOver tool from UCSC can help keep everything up to date. it might require more work in generating the chain files if you work with unusual organisms. this method works for almost any form of feature-based annotation (or even .wig files), the only thing you need is to convert it into a .bed or .bedgraph.
A brief word on how liftOver works. the program is designed to convert a set of genomic coordinates (with respective values if present) between different assemblies of the same organism. in order to do this, liftOver requires a special "chain file", a file containing the differencese between the two target assemblies.
a number of pre-computed chainfiles for a number of different organisms, along with liftOver itself, is downloadable here.
For UniProtKB, you could explore the advanced query mechanisms by date and use them programmatically (cf http://www.uniprot.org/faq/28 and http://www.uniprot.org/help/query-fields ), e.g.
lwp-mirror "http://www.uniprot.org/uniprot/?query=reviewed:yes+AND+created:[current+TO+current]&format=txt" new_seq.dat
lwp-mirror "http://www.uniprot.org/uniprot/?query=reviewed:yes+AND+sequence_modified:[current+TO+current]&format=txt" upd_seq.dat
lwp-mirror "http://www.uniprot.org/uniprot/?query=reviewed:yes+AND+modified:[current+TO+current]&format=txt" upd_ann.dat
Log in to answer this question.