This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Tool: microBioRust: fast, lightweight bioinformatics for microbial genomics in Rust & Python

Hi,

I'd like to introduce an open-source project we’ve been developing called microBioRust, along with its Python wrapper microbiorust-py. The goal is to provide blazing fast high-performance, memory-safe building blocks for microbiology workflows!

Key Features: Functionality:

  • Conversions between formats gbk, embl, DNA fasta (fna), protein fasta (faa), embl & gff3

  • Parsing of BLAST XML and tabular (-m8) format

  • Parsing MSA - subsetting, get_consensus etc.

  • Sequence Metrics: count amino acids, amino acid percentages, hydrophobicity etc.

Performance: The core engine is written in Rust for big time speed-up and reliability.

Python Integration: Using PyO3, microBioRust-py allows you to drop these high-performance modules directly into your existing Python/Jupyter scripts without even installing Rust (rustc).

pip install microbiorust

in Python:

from microbiorust import gbk_to_faa
result = gbk_to_faa("filename.gbk")
for r in result:
    #print each protein sequence from the genbank file 
    #   with id line, newline, sequence fasta (faa)
    print(r)

Lightweight: Designed to be a focused alternative for tasks where larger libraries might feel like overkill

Speed: check out the benchmarking comparison with pure Python

  • counting sequences converted from gbk_to_faa here

We are in the process of expanding the functionality and would love to get feedback from the community, whether it’s feature requests, bug reports, or potential collaborations.

rust microbial bioinformatics python genomics
• 872 views • link
updated by Wayne Wayne • written by Lcrossman • Lcrossman

Opinions will differ, but I don't think any of the tasks you describe is a major bottleneck. Whether a BLAST XML file is parsed in 7 seconds or half a second should be of no consequence to most people.

While I think it is nice to have a collection of these tools in one place, you may need to find another selling point that goes beyond speeding up something that already doesn't take a long time.

• 0 views • link
written by Mensur Dlakic Mensur Dlakic

Thanks for your feedback Mensur Dlakic I appreciate receiving it, however, I would like to bring this to your attention which exactly happened to me after building these scripts and that is why I made them available as open source.

Whilst I was converting 10,000 gbk files to protein fasta - this took 4 hours in Python but only 25 minutes in microBioRust. I could have downloaded all the files in protein fasta format - but then I would have to store it, and it is more traffic for the databases.

Even if so another way of looking at it is - if 10,000 people convert 1 gbk file to protein fasta in microBioRust instead of Python = a large impact in compute resource. Someone is saving $.

That is why we are working to get it speedier which we have already pinpointed a further inefficiency, along with adding other functionality.

L

• 0 views • link
written by Lcrossman • Lcrossman

Even if so another way of looking at it is - if 10,000 people convert 1 gbk file to protein fasta in microBioRust instead of Python = a large impact in compute resource. Someone is saving $.

Each one of those 10,000 people i s saving money, to the tune of $0.00000001 or so. As to your 10,000 files, it is my guess that not many people have to do that conversion. Even if they do, it can't be very often. Good luck selling a software to people that saves them a second or two and an amount of money that can't be calculated without a scientific calculator.

• 0 views • link
written by Mensur Dlakic Mensur Dlakic

Alternative view - it's great to see many more fast tools written in Rust these days which make my life easier as a pipeline author by often being a single binary. Thanks!

• 0 views • link
written by colindaven colindaven

Thanks colindaven !

Actually, it's important feedback to find out what people are thinking on this, so we can clearly state our point.
For the single user, it is a question of scale as well, for example, if I put a single plastic bottle in the recycling, this does not directly benefit me as a single user at that one time at all. It might even be annoying to have two separate bins/trash cans. The benefit is much further down the line.
Yet someone sold that idea to everyone.

If you pay for your compute directly as I do, saving compute is more directly a benefit but saving time and compute is still green. So we are aiming to build this repo to include other fast tools!

• 0 views • link
written by Lcrossman • Lcrossman

1 answer

I tried the tool, first problem, the documentation is incorrect, so make sure your examples work it should be

from microbiorust.gbk import gbk_to_faa

The Python documentation is severly lacking though, I had a hard time with it, there are few examples, and even those seem incorrect. I could not figure out how to covert to GFF from Python, the error messages are cryptic.

Now on the positives.

The mere act of being able to convert between file formats is massively useful!!!

There are very few tools that can go from genbank to other formats.

I would recommend that internally you convert the GenBank/EMBL to JSON then expose that as an API call, and from that, you or anyone else can easily go to any other format. With that you make the tool universally useful.

In addition have the option to convert to so-called line-oriented JSON, where the output is a line-oriented generator, and each line is a complete JSON record, but the whole thing is not a valid JSON. Internally keep everything as a generator.

This format will allow data to be streamed without waiting for the whole data to be rendered. This can be very handy when we need to filter millions of genbank files while looking for the first N hits etc.

Then make sure to demonstrate all use cases in Python.

I see this as an interesting and exciting project with a lot of potential - it is addressing some of the seemingly intractable warts of bioinformatics.

• 0 views • link
written by Istvan Albert Istvan Albert

@Lcrossman , Istvan's excellent comments could have been probably handled by offering something that had everything already set up for the curious user so that 'it just worked'. From what I can tell, this doesn't need a lot of computational power, at least for these small-scale examples, and so they are quite amenable for offering via Jupyter's MyBinder service that integrates very well with GitHub repos.

To demonstrate, I forked the repo and made some configuration files in a directory named binder. So now if you go to my fork and hit a launch badge, you'll get a session where everything is already set up to run.
To save those curious from an extra step, one can just press the launch badge on the next line to get a remote session with two notebooks available to demonstrate the package:
Binder

That content is just mainly placeholder content to show you what is possible. Hopefully with it, you can see how you could set up things to do as Istvan says 'make sure to demonstrate all use cases in Python'.
Aside from making it easy for introducing new users to your package, going this route quickly helps highlight where you, as the developer need to focus because you get a new machine each session. Furthermore, it eliminates the 'on my machine' issue that will arise often with users filing issues. Offering a level playing field can help in a lot a ways.

• 0 views • link
written by Wayne Wayne

Hi, Sorry I missed this until now! There was an initial problem with the __init__.py which worked locally but not externally (causing the broken docs with the module call). Appreciate it would have been better to provide better working docs @Wayne @Istvan. I will be sure to investigate the binder project.

We have recently spent some time refactoring to improve efficiency with our Python InterOp and the docs also got out of sync with the new methods. The efficiency is now much improved, and we are now updating the docs properly which will be complete by 3rd June and also push a new version to PyPi.

Briefly the original method had a problem with the handoff between Rust and Python, causing Python to have to allocate a separate string for every protein sequence across the genome (noted by devs in Rust/Python InterOp). Now fixed.

JSON format: It's a very interesting point - actually the INSDB sequence databases debated using JSON at great length, but they decided not to implement it in favour of GFF format because they have big storage space issues and JSON is larger than each of gbk, embl or gff due to having replicated key requirement in the text.

On the other hand, having it as an internal format could be very different and useful, so we will debate this, and welcome to join should you have time.

• 0 views • link
written by Lcrossman • Lcrossman

Hi,

Just a note to state that after looking into it, the JSON valid format was a fairly simple implement. Will look into the line-oriented

So valid JSON is now done and published a new version to PyPi (0.1.6) for

pip install microbiorust

import microbiorust as mb


#example: convert the genbank file to protein fasta

collection = mb.gbk_to_faa("Rhiz3841.gbk")

#save the collection to variable as a json string

json_str = collection.to_json()

#save the collection as a dictionary
import json

data = json.loads(json_str)

#access specific entry

data_list["AM236080|RL0281"]

Output example of the last protein entry in the file as json string:

 { 

  "AM236080|RL0281": {

  "locus_tag": "AM236080|RL0281",

  "faa": "MNAVQPIPHASSAIDAPRGGLPAPRDGLRDLLTRLWPTVIAYSVILAAAILITKLSGATDYVGPDNDDGMRLVEVRDFLAGQGWFDLMQYRLGLDGGTLMHWSRLIDLPIASLIWFFGLFAPRETAEALALAVWPVSLILPAMLAMAVAGRRIGGVAAMHISLGLTSLAIYTGNRFAPGAIDHHNVQLALVATMIAMLLDPERRAWSYAVAGIAAAVAIAIGAETTPFVAAVCLTIALLWAWEGENFAAAAKAFGLALAIAISILFFATVPPRLYSTVTCDNLSLGYYSLAAIGGGLLLFSAVFASRLRRPLRFAALAVVGAGVLGSAIVIAPQCLSDPLAGLDPMLVELWLRNVSEAQSIFALGRTDLFSIGGFYAAGLFGIAVCIFRVVQRDRVQIHLVLLFLLMTSWAIALVQVRGSTFSNLISILPLALLIIDVRRISNGDSENVAAAFVYIVTVLASAPAAWAVGGGFVSLQMQNAAQKKAAEPAKKISCTSQAALAPLAGLPAGLVSAPSEMGVPILRFTANRVLSAPYHRNQGGMLTELHIGMAEPQEAEAFLKGTGVTVLAFCPSDLQTREITKLKPDGLYAELGKSNIPPYLEPLPKAADVGVQFFRYRPATN"

  }

}

Output example as dictionary format.

{'locus_tag': 'AM236080|RL0281', 'faa': 'MNAVQPIPHASSAIDAPRGGLPAPRDGLRDLLTRLWPTVIAYSVILAAAILITKLSGATDYVGPDNDDGMRLVEVRDFLAGQGWFDLMQYRLGLDGGTLMHWSRLIDLPIASLIWFFGLFAPRETAEALALAVWPVSLILPAMLAMAVAGRRIGGVAAMHISLGLTSLAIYTGNRFAPGAIDHHNVQLALVATMIAMLLDPERRAWSYAVAGIAAAVAIAIGAETTPFVAAVCLTIALLWAWEGENFAAAAKAFGLALAIAISILFFATVPPRLYSTVTCDNLSLGYYSLAAIGGGLLLFSAVFASRLRRPLRFAALAVVGAGVLGSAIVIAPQCLSDPLAGLDPMLVELWLRNVSEAQSIFALGRTDLFSIGGFYAAGLFGIAVCIFRVVQRDRVQIHLVLLFLLMTSWAIALVQVRGSTFSNLISILPLALLIIDVRRISNGDSENVAAAFVYIVTVLASAPAAWAVGGGFVSLQMQNAAQKKAAEPAKKISCTSQAALAPLAGLPAGLVSAPSEMGVPILRFTANRVLSAPYHRNQGGMLTELHIGMAEPQEAEAFLKGTGVTVLAFCPSDLQTREITKLKPDGLYAELGKSNIPPYLEPLPKAADVGVQFFRYRPATN'}

more details in the docs, including how to access on a per contig basis rather than a multicontig file as flat (CONTIG_NAME|LOCUS_TAG) if that is preferable.

I couldn't get on with binder because it takes a long time to spin up but here below is a Google collab link with a jupyter-notebook like interface. If you'd like to modify, you can save a copy to your own google space with File > save a Copy

Link here: Colab

• 0 views • link
written by Lcrossman • Lcrossman

very cool thanks for following up, I'll evaluate and comment

• 0 views • link
written by Istvan Albert Istvan Albert

The MyBinder builds can require patience at the outset.
As to Colab vs MyBinder: Using MyBinder will allow testing with actual, current Jupyter and something more like users will have on their system. Plus, I think(?), it is possible for the session to be in developer mode from the outset so it can encourage contributor development. (At least I think I set this in the postBuild. Or should be able to...)
The big thing is that it is all set up, installed, and ready to go without need to login or install. Anyway, you can try it here. Or just click the following badge: Binder . (I've updated things to your more current version of microbiorust.)

(I tried to keep a bit of the old demo below the new one you had at Google, but you'll see the old part is broken. I didn't investigate too much to fix it at this time.)

With Google Colab, you need a Google account, it isn't actual, current Jupyter, and you're are at the mercy of their limited benevolence & support. Check out the Google Graveyard if you need inspiration.

Minor:

  • Even in your Google Colab you should be encouraging users to install with %pip install microbiorust. You are suggesting outdated methods in the first code cell of your Colab demo. It only works reliably in Google Colab because they controlled the system to make it work. The magic installs, %pip install (and related %conda install) will work reliably for more users in more Jupyter, read more about it here. (For more background, the second paragraph here goes into more details about why the exclamation point may lead to issues in some places, and so even Google Colab has adopted support the magic variation.)

  • Your current README still goes to all the old support stuff that doesn't match the new version, I think.

• 0 views • link
written by Wayne Wayne

Log in to answer this question.

More posts like this

No similar posts found.