Hi.
I resigned from Illumina last year and foolishly embarked on a navel gazing exercise by trying to build a tauri genome browser. Probably because introverting into code is a good way to hide from real world issues :-)
Anyway as part of that effort I developed a Rust crate that will query fasta, bam, tabixed and bigwig/bigbed files, locally, or at an http, s3, azure or google cloud URL, using local environment variables as credentials. It relies heavily on the fantastic object store crate (I since discovered noodles :-/ but haven't tried it)
Im keen to know if anyone thinks this is useful, wants to contribute, or has any other opinion about it. The git repo includes a command line interface that wraps the core crate, and a rocket server implementation that does the same.
All have very simple APIs. The core crate I think is fairly stable but I don't have a lot of test data so it would be good if others could crank it on real world files.
I benchmarked against samtools and its negligibly slower.
Im planning to debug the genome browser for another 2 weeks then I’ll open that up for everyone here as well. Its not completely finished but I think its a time suck and I should work on something people really need.
Andrew
3 answers
I would recommend a Python interface, which lots of tools provide.
Rust, as a programming language, is too abstract for the vast majority of people who need bioinformatics.
In a way, Rust is like the concept of "recursion" - as I understood it, a surprising number of people cannot ever comfortably use recursion as an algorithmic process, it is a barrier they cannot get over.
The world needs a new biopython that is a free of the many design mistakes and inefficiencies of the current biopython.
Making a better BioPython that runs on Rust behind the scenes would be game changing.
BioPython carries two decades of suboptimal decisions - think of any programming library you know of ... now imagine the API has been the same for 20 years ... how bad that API is ...
many people reach for it when you have to parse a file format but even at that has lots of problems ... for example, even after two decades, it does not have an efficient GFF parser - for what is basically a simple tabular format .... let alone a VCF parser
On learning Rust vs Python, if you are not computationally well inclined.
Think of it this way: if you are a European, how hard is it to learn another European language versus the effort to learn an Asian language
It is just not the same order of magnitude in complexity
Well its not exactly what you want but its a start: A PyPi Package that wraps the Rust API Im not gonna redo the whole of BioPython, but I think I will create a version of this that streams files efficiently from start to end, rather than the current form, which only does range requests.
Would you like to contribute to polars-bio We already support streaming reads from the main public cloud providers with datafusion-bio-formats ?
If this is about ranges then VCF makes more sense than FASTA. tabix can do this already so not a huge lift.
bam, vcf, and bigwig all require different tools to query (samtools, tabix, bigwigToBed). Some of them support s3 but setting up for azure and google cloud storage is not simple (samtools and tabix) BigwigToBed only works on local files to the best of my knowledge. Ranges are 100% useful for fasta, bam, gff, bed, bigwig.
yes correct, everything is possible to do today - just it is neither comfortable nor seamless
tabix is awesome and all that but far from being usable for the public,
I just went through this a few days ago, so I got a bitter reminder how antique the system still is, you get a file, you try to slice it with tabix,
but then the file is gzipped but not bgzipped, ok so I rezip with bgzip, then it turns out the file not sorted, now I need to sort it, is easy with unix sort - oh but then the standartd sort works only 90% of time, in the others you need a topological sort by parent, now you have to hunt down methods to do all that
I was able to do this efficiently only because I immediately from the error messages what the problem is ... it was super tedious
Log in to answer this question.
