Continuous Diffusion Language Models (CDLM's)
37 points by peter_d_sherman 3 hours ago | 8 comments
NickNaraghi 2 hours ago
I wonder if we’ll get something like CDLMs for automated harness engineering, sort of piloting the LLM underneath.
replyamelius 2 hours ago
"Attention is all you need" should be renamed into "Attention is sufficient but not necessary".
replyViscountPenguin 44 minutes ago
A quick look at the continuous diffusion models linked in the post shows lots of transformer models still
reply
I disagree with this. Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked.
(The only exception I will make is encoder-decoder models which now are often done by decoder-only.)
But what made it go mainstream was RL. RLHF at first, then other improvements like DPO that were less of a pain in the ass to set up. Adding diffusion on top of that would be an even bigger pain in the ass.
Before ChatGPT there really wasn’t much of a concept of pre-training and post-training. It was all pre-training. Post training was what made the bots conversational and not just “continuing the thing you wrote to them”.
So in short, diffusion never took off because it was just a more complicated way to generate tokens, and the real problem was getting tokens in the right distribution.
This is how ridiculous this industry is. Regulation-seeking panic over nothing. Drama in search of a moat.
Everything is "too dangerous". GPT2 is going to invent a time machine and break crypto and genetically engineer super rabies.
They sell knives, guns, combustible materials, and multi-ton heavy machinery in stores. That's what's actually dangerous.