I contribute to an open source Rust project called HuskHoard. It was originally built for large scale data archiving, but after talking to a few people about the large datasets in computational biology (and the recent shift toward Parquet), I was told this might solve a major I/O bottleneck in bioinformatics pipelines.
The Problem From my understanding, a lot of genomics and single-cell data is moving to cold/object storage (like AWS S3) or Parquet files. However,many legacy bioinformatics tools, or even modern Python/Polars/DuckDB scripts, expect a local POSIX filesystem. If you want to query a specific column or range from a 500GB file on S3, you often have to download the whole thing to expensive local NVMe just to run the tool.
The Solution: HuskHoard acts as a transparent data-tiering layer using the Linux fanotify kernel API. It creates a lightweight stub file on your local disk that looks and acts exactly like the real file.
When your tool (which is completely unaware of S3 or HuskHoard) attempts to read that local file, HuskHoard intercepts the call, pauses the process, fetches only the requested byte ranges via HTTP Range requests from cold storage, and passes the bytes to the tool.
Why it might be useful here:
- Zero-Disk Extraction for Parquet: If you query a 100GB Parquet file of variant data with Polars or DuckDB looking for a specific column, HuskHoard will only fetch the exact byte range chunks required for that column. The other 99GB is never downloaded, and your local disk space is untouched.
- No Code Rewrites: You don't need to rewrite legacy C/C++ bioinformatics tools to support S3 APIs or AWS SDKs. As long as it can read a POSIX file, it works.
- Fast & Safe: Written entirely in Rust.
A Quick Example:
1. Create a local stub pointing to a massive remote dataset
huskhoard stub --remote https://your-bucket.s3.amazonaws.com/big_dataset.parquet --local ./local_data/
2. Run your standard script or tool against the local directory
The tool thinks it's reading a local file. HuskHoard intercepts and streams only the necessary chunks into memory. python query_variants.py --input ./local_data/massive_dataset.parquet
I am not a bioinformatician by trade, so I am very eager to hear from this community. Does this fit into your current workflow architectures (like Nextflow/Snakemake)? Are there specific edge cases in bioinformatics file formats (like BAM/CRAM indexes) where a tool like this would be particularly useful?
Any feedback, feature requests, or GitHub stars would be massively appreciated!
Website: huskhoard.com GitHub: github.com/huskhoard/huskhoard
0 answers
No answers yet.
Log in to answer this question.
-
Tool: Just-DNA-Lite, an open-source platform for personal genome annotation
written by Anton •
-
Tool: In-memory streaming downloader for massive NCBI/EBI datasets (bypasses disk I/O)
written by Valentin Zhukovetski •
-
Tool: polars-bio - blazing fast Python DataFrames for genomic operations
written by Marek •
-
Blog: Rust in Bioinformatics monthly newsletter
written by Andrew •
-
Random Access remote BAM files
written by Lucas R.F. •
-
pybiomart sqlite3.InterfaceError
written by ngarber •
-
How to convert PLINK outputs into Parquet files?
written by telroyjatter
-
Java code for indexing fasta file
written by dariober
-
index a large file to get lines faster
written by sacha
-
Tool for random access to indexed BAM files in S3?
written by donfreed
Was it you who recently asked on r/bioinformatics reddit whether bioinformatics had a niche for Parquet, and like everybody said no, as the field as tremendous legacy burden towards its formats?
it was not, thanks for making that clear. Parquet or no, the format is irrelevant. the compressing of data into searchable frames is where the power is here. You create an index which is held in the database and is searchable. you can extract segments that you are interested in rather than the entire file. Also auto moving cold data off of your hot tier is more economically sound and improves performance.
I'm not sure thats entirely true @ATpoint. We use parquet a lot. Hail, used by Gnomad etc, is a parquet based system, although they also provide their own tooling.
One of the benefits of systems like this should be that it allows people to move to new formats without haveing to deal with the legacy burden.
Like, most often if we need to use Gnomad data, we download the VCFs, which are massive, and is a real pain. Samtools does allow remote access to ranges of BAM files already, but I can see uses for an interface that exposes these as files without download - say if you were working on a large number of TCGA bam files which are stored on S3.
I'd be interested to know how this tool compares to Fusion from seqera (https://seqera.io/fusion/). My feeling is that this is file based while Seqera is file system based.
Can HuskHoard do write as well as read?
Yes. Since it archives files (to tape/cloud etc) leaving a local storage stub.
Huskhoard sounds like an open source implementation of similar tools that are available in enterprise storage data tiering (at significant cost) for a while. But the added benefit here may be selective reads from remote storage.
There is a penalty to accessing data in cold storage in terms of cost. I assume this tool is mainly for use with data that is not in "glacier" type storage.