Tool: In-memory streaming downloader for massive NCBI/EBI datasets (bypasses disk I/O)
Downloading massive genomics datasets (e.g., multi-gigabyte .fna.gz or .fastq files from NCBI/EBI) often hits two bottlenecks:
- Single-connection limits on high-latency networks (e.g.,
wgetcapping out at low speeds). - SSD wear-and-tear from multi-connection tools (e.g.,
aria2) that require writing out-of-order chunks to disk before processing.
HydraStream is an open-source CLI tool built to solve this specific pipeline issue.
Core Mechanism:
It fetches chunks concurrently via multiple TCP connections but uses a sequential reordering min-heap buffer in memory. This allows it to yield a strict byte stream directly to stdout, completely bypassing the disk.
Pipeline example:
hs "https://ftp.ncbi.nlm.nih.gov/.../genome.fna.gz" -t 20 --stream --quiet | zcat | grep -c "^>"
Additional specs:
- Pure Python (
uvloop+httpx). - AIMD rate limiting to prevent server IP bans.
- POSIX compliant (UI/status logs to
stderr, data tostdout).
Installation:
pipx install hydrastream
or
uv tool install hydrastream
Repository & Architecture details: https://github.com/HydraStream/HydraStream
Any code reviews, feedback, or bug reports are highly appreciated.
• 555 views
•
link
updated
by
GenoMax
•
written
by
Valentin Zhukovetski •
•
written
by
Valentin Zhukovetski •
0 answers
No answers yet.
Log in to answer this question.
New tools are always welcome since they could be of use to someone at some point.
Trying to imagine a use case for this. Aligners would need a copy of the genome stored, even if you could stream and build the index on the fly. As for sequence data, single end datasets would work but paired-end data would not, if you were aligning on the fly.
Did you build the tool to address a specific need?
You are absolutely right regarding the constraints with aligners and paired-end sequence data.
To be completely transparent, I didn't build this for a specific internal pipeline. My background is in wet-lab operations, but I am currently focused on systems engineering. I built HydraStream primarily as an architectural challenge to solve a fundamental infrastructural problem: the notoriously slow, single-connection bottlenecks of NCBI/EBI servers on high-latency networks, and the heavy SSD I/O required by multi-connection tools like aria2. While the stdout streaming mode has practical limitations for complex paired-end alignments, it is highly applicable in other data-engineering scenarios:
1.On-the-fly parsing & filtering: Streaming massive uncompressed databases (e.g., BLAST nt/nr FASTA or UniProt XMLs) directly into a custom parser, script, or local SQLite/Postgres DB without extracting hundreds of gigabytes of intermediate files to an SSD.
2.Single-end / Long-read QC: Streaming Oxford Nanopore or PacBio FASTQ files directly into filtering tools (like Filtlong, seqtk, or grep) to calculate stats or subsample on the fly.
3.High-speed standard downloads: Even without the streaming mode, using it simply to write to disk (-o flag) allows saturating a network connection via 20 multiplexed TCP streams, combined with exact-byte connection recovery when an FTP/HTTP server inevitably drops the connection.
Essentially, it is a network-level optimization tool first. I really appreciate the practical feedback!
P.S. To be completely honest, my English is currently quite rough, so I am using an AI assistant to help translate and format my replies clearly. But the technical logic and the code itself are 100% my own.
Number 3 above will be a generally useful application (good to know that the tool will output to local disk) since speeding up downloads can be a time saver. SSD degradation may be a consideration on individual computers but many folks use institutional resources with high performance storage solutions.
Thanks for sharing the background information on the origin of the tool. Impressive to see someone on the web-lab side of things venture into network engineering.
Actually, I’ve been thinking more about your point regarding paired-end data and on-the-fly alignment. It occurred to me that while a single stream wouldn't work, we can actually solve this using bash process substitution.
You are absolutely right that building the FM-index for a reference genome requires random access, so for the reference, HydraStream should stay in standard disk-write mode.
However, for paired-end reads, we can technically stream them without touching the SSD by spawning two parallel instances like this:
By piping the raw bytes directly from the network to /dev/fd/*, we bypass the I/O overhead entirely for those massive 100GB+ files. I'm curious - have you ever experimented with process substitution for high-volume streaming in your pipelines, or do you find that local caching is still safer for reproducibility?
If reproducibility is a concern then that is a different use case. One can use clever unix constructs, if efficiency is the goal.
Did you confirm if the above command line works. Doing something like this would llikely not allow one to use multiple cores with the aligner (or can one do that, never tested this myself). On other hand, using a single core should help with deterministic output.
The concept for direct downloading has been fully validated as functional attached is a screenshot of a successful test run.
However, during implementation, a significant technical hurdle was uncovered regarding the security layers of the ftp.sra.ebi.ac.uk servers. My current stack, based on the httpx library, is being flagged and blocked due to a TLS Fingerprint (JA3) mismatch. In short, the server detects the specific 'handshake' of the library, identifies it as a bot, and triggers an immediate IP ban.
To ensure the solution is stable and doesn't lead to mass blacklisting, I need to refactor the networking module - likely switching to curl_cffi, which can accurately impersonate a real browser's TLS signature.
I apologize for the delay, but I will be able to fully implement this only in version 1.2. I need extra time to test this bypass across various mirrors to provide a truly reliable tool. Thank you for your patience!
I've been reflecting on our discussion over the last day, and it struck me that while the 'cat-and-mouse' game with public firewalls is an interesting technical challenge, the real-world value of HydraStream lies in intra-institutional workflows.
In a local network or an HPC cluster environment, where you can easily whitelist internal IPs, this tool completely removes the need to 'hop' data from one storage node to another. You can stream directly from the central data repository into the compute nodes' memory.
Essentially, it turns the network into a virtual filesystem. There's no need for intermediate FASTQ files, no SSD wear, and zero waiting for a massive transfer to finish before the alignment starts. It’s about infrastructural efficiency, and that’s the core vision I’m pursuing.
At larger institutions people tend to use high performance clusters. Generally on these clusters there are one or more high performance storage clusters/systems (e.g. isilon/netapp). These storage systems are accessible in parallel on all compute nodes so there is no need to use a tool like hydrastream within a cluster. If people are accessing data from a central storage location then that could be a potential internal use case.
That said, is the issue you described above specific for EBI servers or does it affect NCBI as well. Perhaps you could send a ticket in to EBI and work with their support to see if you can convince them about the utility of your tool to allow some sort if whitelisting of the traffic.
Unfortunately in the world we live in today, ensuring security and accessibility are a balancing act so good intentions may not always be enough.
Have you heard of a software called
asperafrom IBM that accelerates network transfers? In case you have not, you may want to look it up. It is commercial software that requires a license for the server but users can use the client for free. EBI and NCBI use it to facilitate user uploads of data.