I worked with public datasets before in my internship and kept running into the same issue that important metadata fields were missing from NCBI records (like Biosample, genbank, etc.), but the information often existed somewhere else like in the associated paper, supplementary tables, or methods sections.
After spending way too much time manually tracing accessions back to publications, I just built a small tool called OpenBioData to automate some of the process.
The tool on my github here: https://github.com/vy-phung/OpenBioData
So briefly what it does:
- Traces accessions back to source publications and supplementary materials
- Extracts metadata that may be missing from the record itself
- Provides a confidence score for extracted values
- Includes direct citations (PMID + table/section) so the source can be verified
I'm still actively improving it and would love feedback from others who work with public genomics datasets.
Has anyone else encountered this problem? If you have a few troublesome accessions, feel free to share them and I'll test them with the tool and post the results. Curious to hear how others currently handle problem of NCBI metadata curation.
(Also posted this on r/bioinformaticstools: https://www.reddit.com/r/bioinformaticstools/s/STnOIjJLSF, cross-posting since I'm genuinely trying to reach people who work with this kind of data day to day, not just farming engagement on one platform.)
P.S. Upfront: I built this with real help from Claude Code, especially on the extraction layer and disclosed in the README, not hidden. I wrote the core pipeline and logic myself. If that's a dealbreaker for you, fair enough:)) but I'd rather say it directly than have someone find it in the commit history and wonder why I didn't mention it.
0 answers
No answers yet.
Log in to answer this question.
-
Tool: Tool to recover missing / inconsistent NCBI metadata - feedback welcome
written by Vy •
-
Tool: 17 pre-configured pixi environments for microbial genomics — built by a biologist, not a deve…
written by Bharat K. C. Patel •
-
Herald: The Biostar Herald for Monday, October 23, 2023
written by Biostar
-
What tools are available for downstream analysis of 10x scATAC-seq in entirely new organisms?
written by Chilly •
-
Herald: The Biostar Herald for Monday, June 13, 2022
written by Biostar
-
Herald: The Biostar Herald for Friday, August 27, 2021
written by Biostar
-
hg19 or hg38 for variant calling
written by jsneaththompson
-
Job: Bioinformatician/Data Scientist Fellowship at the Centers for Disease Control and Prevention (…
written by bis3 •
-
Job: Bioinformatician Fellowship at CDC-Atlanta in Microbial Transmission Networks, Atlanta, GA, USA
written by bis3 •
-
fastq-dump stream to named pipe (fifo) to Trinity
written by glarue •
How does this tool perform compared with other tools, such as nf-core/fetchngs, pysradb, and pachterlab/ffq?
yes this is a good point, everything should be framed relative to existing tools and techniques
Thanks for the question. I did try to find any tool out there doing metadata curation before building this one. I knew about nf-core/fetchngs but didn't know about the others, so just did quick research and looked again to make sure I answer it right (just correct me if I am mistaken). So fetchngs, pysradb, and ffq pull metadata and raw data that's already recorded in SRA/ENA/GEO's own structured database, and fetchngs in particular is built to feed straight into other nf-core pipelines. None of them go beyond the database record itself by looking for the published papers associated with the samples (or papers cited by the samples) to cross-check whether the metadata is correct. Sometimes authors mistype their metadata or only submit minimal metadata to the database, while the richer and more accurate metadata is in the publication, supplementary materials, etc. So, if I were a user, I would use the tools you recommended to get the list of accessions, and if I wanted richer, recovered, or cross-checked metadata (or simply cared about metadata quality), I would then input those accessions into my tool and get the Excel output. Of course, the tool is still early and buggy, so I'm actively developing it and really need feedback to improve it (and contributors to the GitHub are always welcome too!).
This tool looks like a bunch of little scripts strewn all over a repository.
After spending 5 minutes with the docs, I still don't quite understand what it does.
At the very start, demonstrate your tool's utility with a simple command. It should be like this:
Hey people, aren't you annoyed when you can't get the metadata for a run? Look Ma! Look how easy it is with my magnificent tool:
and wow, look at how much cool stuff I get back in return that you would have a hard time getting otherwise
Instead, what you have is lots of seemingly disorganized scripts, explanations are too long and hard to read,
people are left with no idea what the tool does
PS. This problem is very common with software developers. Don't take it personally. Programmers are so enthralled with their own tool that they have spent so much time with it that they are unable to step back and look at it with the eyes of someone new.
That is really helpful feedback! Thanks and I’ll definitely fix the docs and try the example case you just suggested. I’ll circle back to this comment (or add a new one) once it's updated.
How are you doing this? Via a PubMed search for the accession? While journals have become more strict about including SRA accessions, this was not always the case. And it was likely the old accessions that are missing the metadata.
Thanks for the questions. Yeah, it's more layered than I made it sound (I'll definitely update the docs so it's easier to understand and can answer questions like these).
For tracing accession back, it first checks NCBI's own record for a structured publication link (not a search). If that's empty, it falls back to a Europe PMC search for the literal accession string, then a raw NCBI cross-reference lookup.
Separately, there's also an always-on search step that runs regardless of whether the above already found something. It searches Google (via the Serper API) the accession first, then PubMed, then Europe PMC again, and finally a direct URL if the user provides a specific link to search. Bare BioSample accessions with no parent BioProject skip the structured database check entirely and use this search process only.
So the real gap is that every fallback, in both places, searches for the literal accession string as text. A paper that never restated the accession in its own body (which was common before journals required it) won't be found by any of them. Yeah, that's still not solved yet.
I'm thinking about pulling in cross-references from other linked databases and relying more on citing-paper searches instead of just the original deposit paper. I'll open an issue on GitHub for this (or if anyone wants to contribute, feel free to open an issue or submit a PR!).