Show HN: I made an audio-to-MIDI model 60x more efficient
181 points by hidude562 4 days ago | 34 comments
I released dettato, my open weight audio to midi transcription model. It beats muscriptor 1.3b on every set and is roughly 60x smaller, without training on copyrighted recordings. Feel free to ask questions

dekhn 16 minutes ago
I tried this on a Phish guitar track that I have and while GPU failed with an error, CPU eventually finished and it did a pretty good job! The recording is 45 minutes long, so getting GPU working is my next interest.

Unfortunately, Phish live recordings are copyrighted, so it's unlikely you can put them in the training corpus. Demonstrating that you can deconvolute and transcribe multiple different instruments with highly overlapping frequencies and sounds faster than an expert human would be a great demonstrate of the capabilities.

reply
gw32 39 minutes ago
This is fantastic. I love the commitment to train only on PD recordings. In my opinion, that is an essential to maintaining the favor of musicians - building technology not to take from their work and replace them, but rather to celebrate human-written music by making it accessible.

This is why I believe (AMT, OMR) and MIR-adjacent projects to be so much more meaningful than generative models (e.g. Suno). I'm looking forward to hearing more from kobi music - congrats on the launch!

reply
hdb2 12 minutes ago
I just signed up and fed it a piano solo from a Dixie Dregs tune, and it nailed it. this is very impressive for players like myself to transcribe things very, very quickly.
reply
peaseagee 6 hours ago
Congrats on training on open and public domain recordings. Thanks for proving this kind of work can be done in a more ethical way.
reply
piltdownman 6 hours ago
TBF it's by far the best corpus for this use case anyway.

Classical Music as an academic concern is very pedantic in the best way, and has the lions share of the research hours and funding compared to any other genre. As a result the best 1:1 approximations of multi-instrument Audio into multi-channel MIDI files is done simply by OCR of the established Gesamtausgabe in each case. Computer based scoring and playback has been solid since Finale/Sibelius/Score came on the market, and democratized at a wide consumer level somewhere between Amiga mod-trackers and Macintosh Quicktime 2.0.

reply
thih9 3 hours ago
Congrats on the launch!

The top illustration looks like an Angry Birds character with an ear. Fine for a one time blog post but if you plan to reuse it then you may want to change the design.

reply
ohmahjong 2 hours ago
It is a reference to an Angry Birds meme: https://knowyourmeme.com/photos/3226156-angry-bird-ear-angry...
reply
epiccoleman 5 hours ago
That is awesome! I'll definitely have to check it out. I was very impressed with muscriptor when I used it in a recent project (a conference talk I gave). It was an AI talk with a musical component, and the final "scene" in the talk involved my own guitar playing. I used Muscriptor to transcribe that playing for categorization as part of a "guitar solo generator" that recut a new solo out of my stock phrases each time. Really fun little demo for a bigger project.

Anyway, it would be awesome to be able to do those sorts of transcriptions better and faster - so I'll definitely give this a whirl. Maybe I can dig up the pipeline scripts from processing my licks for that talk and see how dettato slots in.

reply
SoleilAbsolu 2 hours ago
Curious what's the performance for both note duration, and note pitch? How does it do for tunings/temperaments outside 12TET? Very refreshing to not violate copyright in this day and age amongst all the greedy grabbers!
reply
locusofself 2 hours ago
Very cool, I have been working on a DAW for several months that is getting close to a beta release. I'd love to incorporate this.

As a guitarist who can't play keys well, I'd love to use something like this to help me transcribe string arrangements.

reply
dimmuborgir 2 hours ago
Is there a way to force it to a single instrument? I uploaded audio files of pure piano music and the output midi files are split into piano, guitar and drum parts.
reply
rogerdickey 4 hours ago
all the examples seem to be classical music or largely instrumental stuff. how well does it work for things like pop music, rap?
reply
pier25 3 hours ago
Very cool. This is the sort of stuff I want to see from AI.
reply
simedw 2 hours ago
Very cool, I wonder if my MIDI generation model would improve if I trained it on the MIDI files your model generates.
reply
oever 6 hours ago
Audio-to-MIDI sounds very ambitious. How do you verify that the midi fits the audio? Does it result in useful beat detection? E.g. I'd very much like to have a way to map audio offset in a recording to labels in sheet music. But even beat detection in fairly standard jazz recordings fails often.

Does the model know when a piece (whole piece, not a section) is 3/4 or 4/4 th and can it give the onsets of all measures? Or all the midi notes just based on time and not organized rhythmically?

reply
gunalx 4 hours ago
Its definetly a problem where generating the trivial version of synthetic data is easy though.
reply
jsphweid 6 hours ago
I've been out of the loop since Magenta's MT3 model. This feels a lot better and significantly smaller. Very impressive, dude. And thank you!
reply
gpugreg 5 hours ago
Nice! Worked surprisingly well.

You might want to implement deduplication for your website so impatient users do not convert the same mp3 multiple times (at multiple times the price) when they click "run audio to midi" and the server takes a bit longer to respond.

reply
cocodill 4 hours ago
I guess impatient users could run audio to midi local.
reply
ch_sm 5 hours ago
Just tried it with some home-recorded songs (pop/rock) — this works great!
reply
gregzem 3 days ago
Congrats with SOTA model. How good is it in recognition of the instruments which are overlapping by frequencies? Does is provide note offset and velocity in the transcribed MIDI? Thanks!
reply
JodieBenitez 6 hours ago
I need this in Bitwig !
reply
pier25 3 hours ago
yeah I was thinking that DAWs should be implementing this
reply
thatmf 3 hours ago
my exact thought. hmm. weekend project?
reply
mycall 6 hours ago
I'm still waiting for someone to convert beatboxing to midi in realtime. That would be a game changer for that art.
reply
maxgraf96 6 hours ago
You should check out https://vochlea.com/ :)
reply
mycall 5 hours ago
> Trained on your own percussive sounds

This is probably where I would find it limiting range of sounds can be huge, much more than just 6 sounds (10000+) and not all of them percussive. Worth an eval though, thanks!

reply
chahmed312 6 hours ago
Is it fully open source?
reply
awinter-py 4 hours ago
at long last we have solved karaoke
reply
eduardoagarcia 4 hours ago
super interesting, I'll have to check this out tonight when I get home!
reply
th1nhng0 3 hours ago
just in time when I need it. Thank you so much
reply
IAmGraydon 5 hours ago
Excellent! I'll try this tonight. The timing couldn't be better as I've been researching audio->MIDI models for some personal projects recently.
reply
pikann22 6 hours ago
crazy how a 20m param model actually beats the 1,3b muscriptor. awesome work!
reply
hongyong 4 hours ago
nice, this is what i've been looking for
reply
saascostlab_off 5 hours ago
[flagged]
reply
grd_dev 4 hours ago
[dead]
reply
staff000 4 days ago
[dead]
reply