Tool: polars-bio - blazing fast Python DataFrames for genomic operations

polars-bio: Blazing-Fast Genomic Interval Operations on Python DataFrames

Title for Biostars: polars-bio – fast, scalable genomic interval operations on Polars/Pandas DataFrames (now published in Bioinformatics)

Hey everyone,

Wanted to share a tool we've been working on that some of you might find useful — polars-bio, a Python library for fast genomic interval operations. It was recently published in Bioinformatics (Wiewiórka et al., 2025) and is under active development with new features landing regularly.

What is it?

polars-bio is an open-source Python library for performing common genomic interval operations — overlap, nearest, count_overlaps, coverage, and merge — directly on DataFrames in Python.

Why Polars over Pandas?

Most bioinformatics tools in the Python ecosystem still rely on Pandas, but if you've worked with large genomic datasets you've probably felt its limitations: high memory usage, single-threaded execution, and eager evaluation that loads everything into memory at once. Polars is a modern DataFrame library written in Rust that addresses these pain points head-on:

Native multi-threading — Polars parallelizes operations automatically, no need for multiprocessing hacks
Lazy evaluation — build query plans that Polars optimizes before execution, reducing unnecessary computation
Apache Arrow memory model — columnar, cache-friendly, and zero-copy interop with other Arrow-based tools
Significantly lower memory footprint — Polars avoids the overhead of Python objects per cell that plagues Pandas

polars-bio builds on top of this foundation and extends it with Apache DataFusion — a fast, extensible query engine also written in Rust — to provide genomic-specific operations like interval joins using cache-oblivious interval tree (COITrees) algorithms. The result is a stack that runs natively in parallel with out-of-core/streaming support for datasets that don't fit into memory.

That said, if your existing pipelines use Pandas, polars-bio works with Pandas DataFrames too — you don't have to migrate everything at once.

It also reads common bioinformatics formats (VCF, BAM, GFF, BED, GTF, FASTA, FASTQ) natively and supports federated reads from cloud storage (S3, GCS) via Apache OpenDAL — no need to download files locally first.

Performance: Range operations

We benchmarked polars-bio against PyRanges, GenomicRanges, and Bioframe on the three most common genomic interval operations. Here's how it stacks up:

Overlap — GenomicRanges wins on small inputs (~2.1–2.3x), but polars-bio takes over from medium-sized datasets onward and dominates on large ones, where PyRanges falls far behind. On real-world data (10M vs. 1.2M intervals), polars-bio is 6.5x faster than Bioframe.

Nearest — polars-bio leads decisively at every input size; speedups over other libraries grow with dataset size (orders of magnitude on large data). 15.5x faster than Bioframe on real-world intervals.

Count overlaps — the biggest gap: 38x faster than Bioframe, with up to 282x speedup when scaling to 8 CPU cores thanks to native multi-threading.

Coverage — 15x faster than Bioframe on real-world data.

In streaming mode, memory usage drops dramatically — up to 90x less memory for overlap and 60x less for count_overlaps vs. Bioframe.

Full benchmark details and reproducible scripts: polars-bio-bench.

Performance: File format parsing

Reading bioinformatics file formats is often the bottleneck in real pipelines. In v0.23.0 we overhauled the parsing layer (datafusion-bio-formats 0.5.0):

VCF — up to 3.6x faster parsing by eliminating per-record heap allocations (writing directly into Arrow StringBuilder/PrimitiveBuilder arrays instead of intermediate Vecs). The biggest gains are in VCF with INFO field parsing.
BAM — auxiliary tag parsing now appends directly to Arrow builders instead of collecting into temporary Vecs, cutting per-record allocations from O(t) to zero.
BGZF decompression — enabled libdeflate across all BGZF dependencies, yielding ~30–50% faster gzip decompression for both BAM and bgzipped VCF reads.
FASTQ — similar allocation-reduction optimizations.

No API changes — just pip install --upgrade polars-bio and enjoy the speedup.

What's new in v0.26.0?

The latest release is v0.26.0 — check the release notes and blog for full details.

Install

pip install polars-bio

Pre-built wheels for Linux, macOS, and Windows — no Rust toolchain needed.

0 answers

No answers yet.