I'm Struggling to automate bulk downloads of ~36k+ metagenome assemblies from JGI's IMG database. The web interface works but isn't feasible at scale.
What I've Tried:
- jgi-query (Python)- https://lnkd.in/erXHH5SM
- get_jgi_genomes (Perl)- https://lnkd.in/e_5keBQx
- Custom scripts using JGI's XML API
Issues I'm hitting:
"Portal not found" errors for many IMG IDs
Authentication cookies expiring mid-download
Tape storage timeouts
Variable portal naming (IMG_, Ga0, Gp0*)
I know others have faced this (Biostars thread), but haven't found a reliable solution for 2025.
Has anyone:
- Successfully automated bulk IMG downloads recently?
- Found updated tools that work with current JGI API?
Any guidance appreciated!
•
written
by
Ananaya •
1 answer
Hey Ananaya,
The JGI Data Portal (https://data.jgi.doe.gov/) unifies access across portals (including IMG) and has an API for bulk automation: https://files.jgi.doe.gov/apidoc/. It handles auth via session tokens (no expiring cookies) and lets you stage files from tape storage in advance.
Try this Python script for bulk metagenome pulls—it uses the API, splits huge requests (>10TB), and generates curl/Globus commands: https://github.com/WrightonLabCSU/JGI-API-Request. Input your IMG OIDs in a TSV, get a request ID, and download via Globus to dodge timeouts.
jgi-query (https://github.com/glarue/jgi-query) is still solid for CLI but regex portal names (e.g., IMG_|Ga0|Gp0*) and add retries (-n 5). Perl script looks unmaintained.
This worked for me on 30 000+ assemblies last month - please message if stuck.
Kind regards, Kevin
Log in to answer this question.
Please post direct links to software you mention. Not sure why you are using LinkedIn short URL's.
Historically downloads from JGI have been riddled with problems as noted in prior posts.
Looks like there is a new data portal now online at JGI : https://data.jgi.doe.gov/ Try your luck there.