Hacker News

Training mRNA Language Models Across 25 Species for $165

112 points by maziyar 4 days ago | 28 comments

We built an end-to-end protein AI pipeline covering structure prediction, sequence design, and codon optimization. After comparing multiple transformer architectures for codon-level language modeling, CodonRoBERTa-large-v2 emerged as the clear winner with a perplexity of 4.10 and a Spearman CAI correlation of 0.40, significantly outperforming ModernBERT. We then scaled to 25 species, trained 4 production models in 55 GPU-hours, and built a species-conditioned system that no other open-source project offers. Complete results, architectural decisions, and runnable code below.

seamossfet 6 hours ago

The problem with models like this is they're built on very little actual training data we can trace back to verifiable protein data. The protein data back, and other sources of training data for stuff like this, has a lot of broken structures in them and "creative liberties" taken to infer a structure from instrument data. It's a very complex process that leaves a lot for interpretation.

On top of that, we don't have a clear understanding on how certain positions (conformations) of a structure affect underlying biological mechanisms.

Yes, these models can predict surprisingly accurate structures and sequences. Do we know if these outputs are biologically useful? Not quite.

This technology is amazing, don't get me wrong, but to the average person they might see this and wonder why we can't go full futurism and solve every pathology with models like these.

We've come a long way, but there's still a very very long way to go.

stardust2 4 hours ago

How do we get more verifiable protein data? So even if we had better data, we don't yet understand how the structure impacts the biology?

maziyar 4 days ago

full article: https://huggingface.co/blog/OpenMed/training-mrna-models-25-...

pfisherman 5 hours ago

Nice work! Here is an article you may find helpful if you have not already come across it.[0]. You may also want to consider benchmarking against some non ML methods.[1]

0. https://pubmed.ncbi.nlm.nih.gov/35318324/

1. https://www.nature.com/articles/s41586-023-06127-z

xyz100 10 hours ago

What makes this dataset or problem worth solving compared to other health datasets? Would the results on this task be broadly useful to health?

CyberDildonics 9 hours ago

What other "datasets" are you talking about? How do you "solve a dataset" ?

colingauvin 5 hours ago

HN's blindspots never cease to amaze me.

I am a structural biologist working in pharmaceutical design and this type of thing could be wildly useful (if it works).

rubicon33 9 hours ago

Can someone explain what one might use this model for? As a developer with a casual interest in biology it would be fun to play with but honestly not sure what I would do

colechristensen 9 hours ago

You can get your feet wet with genetic engineering for surprisingly little money.

This guy shows a lot of how it's done: https://www.youtube.com/@thethoughtemporium

Basically you can design/edit/inject custom genes into things and see real results spending on the scale of $100-$1000.

com2kid 6 minutes ago

We actually did this in my highschool genetics class back in 1999! We made bacteria change color by splicing in a gene. Awesome stuff.

The (public!) school had a grant from one of Seattle's biotech boom companies.

someuser54541 9 hours ago

Is there something like this in text/readable format?

_zoltan_ 6 hours ago

My main concern is using fungi. If it ends up in my lungs I'm most likely screwed, right?

colechristensen 5 hours ago

This is the classic meme https://www.reddit.com/r/labrats/comments/mmv2ig/lab_strains...

Lab strains of things tend to be extremely sensitive and not human adapted. You shouldn't study and modify human-infecting organisms in your basement anyway. While you shouldn't ignore protective equipment and proper procedure... paranoia about infecting yourself with a lab leak isn't warranted.

nurettin 5 hours ago

Yes, but most students produce their best work while infected.

khalic 10 hours ago

> In Progress: CodonJEPA

JEPA is going to break the whole industry :D

digdugdirk 9 hours ago

Can you explain this? I haven't heard of JEPA, and from a quick search it seems to be vision/robotics based?

khalic 9 hours ago

It’s a self supervised learning architecture, and it’s pretty much universal. The loss function runs on embeddings, and some other smart architectural choices allover. Worth diving into for a few hours, Yann LeCun gives some interesting talks about it

lukeinator42 9 hours ago

https://openreview.net/pdf?id=BZ5a1r-kVsf

simianwords 10 hours ago

What makes these Domain specific models work when we don’t have good domain models for health care, chemistry, economics and so on

colechristensen 9 hours ago

>we don’t have good domain models for health care, chemistry, economics and so on

Who says we don't?

simianwords 9 hours ago

Examples please?

colechristensen 8 hours ago

No, it's really simple to search for domain specific models being used "in production" all over the place

simianwords 8 hours ago

I didn’t find a single one that outperforms a general model.

colechristensen 7 hours ago

Ok, alphafold.

simianwords 7 hours ago

It’s not a large language model

yieldcrv 9 hours ago

Distributing the load on this will probably be infinitely more useful than “folding at home”

HocusLocus 10 hours ago

gray goo of the future

skyskys 7 hours ago

hmmmm seems like some fake hype.