Perplexity and Intelligence
44 points by acossta 49 minutes ago | 17 comments

MrCheeze 35 seconds ago
I always had the impression that model capabilities were stagnating around GPT-5.0, and this finally broke around the release of Claude 4.5 later that year. Seems like their own graph corroborates that things were pretty stagnant then. I wonder what changed?
reply
jrflo 8 minutes ago
This trend is pretty shocking to me. I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL, it's wild that the trend is so clear. Really nice to have such a simple benchmark instead of the increasingly convoluted ones that are getting spun up every day.

Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.

reply
derac 4 minutes ago
I got the feeling it was too large and inefficient at the time.
reply
qphe95 5 minutes ago
I wouldn't say perplexity is great metric because GPT-4 and o3 have similar perplexity on this chart but o3 was a massive step improvement. Pachocki has an incentive to push perplexity as one of the creators of the original LLM paradigm but I would say we have kind of moved on from that.
reply
amluto 11 minutes ago
> The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot).

Why would OpenAI expect any meaningful relationship between their models’ ability to predict the next token of supposedly held-out chunks of their internal codebase and the overall capabilities of the models? I expect both to improve, but that’s about the extent of it.

reply
AIShill101 5 minutes ago
Gah man do I hate AI ethics and stuff. I really hoped the lead scientist at openAI wouldn't be so screwy but they're all part of that cult aren't they?

I really hope we go full speed ahead on AI development.

reply
derac 45 seconds ago
https://x.com/sama/status/2124551883130114403

^ Sam Altman statement, basically saying that's not their intention. I want better AI too, but it seems prudent to move forward with caution.

reply
bayindirh 19 seconds ago
> Gah man do I hate AI ethics and stuff.

Why? You hate ethics in general, or only when applied to AI?

reply
Leary 20 minutes ago
Looks like their latest internal model performs somewhat similarly to model 4 of anthropic.
reply
eulgro 11 minutes ago
Didn't they hit a data wall roughly two years ago and had to rely on artifical datasets for pretraining i.e. LLM-generated?

I would expect the entropy of artificial datasets to be lower, thus easier to predict and scoring better on "perplexity"?

reply
jrflo 6 minutes ago
Yes, they started to hit a data wall, but I thought that much of the progress of the last couple years came from reinforcement learning after pretraining, which is mostly building complex environments for the models to reason in, rather than accumulating piles of raw text.
reply
rfgplk 45 minutes ago
What's the perplexity of the average person?
reply
kingstnap 4 minutes ago
Beating even a very small LLM at next token prediction is basically impossible.

[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2

reply
jrflo 19 minutes ago
Probably high because humans aren't trained on lowering cross entropy loss. Kind of funny to think that next token prediction is actually an emergent capability of the human brain...
reply
Retro_Dev 10 minutes ago
Saying that perplexity is a metric of intelligence is misleading. Inkling-2-Mini was released recently, and I've measured its perplexity at 5.7 on a small varied text dataset. For reference, the graph OpenAI is glazing is plotting log(PPX), which would put this model at a 0.756 on the Y Axis... Which is apparently between GPT 6.2 Astra and their internal model. Perplexity can also give the same number for a model which understands text as to one that memorizes text... And as I'm pretty sure Inkling-2-Mini is smaller than GPT 6.2 Astra - and it's generally acknowledged that number of parameters equals memorized knowledge, this shows how weak this advertising stunt for OpenAI is.
reply
jrflo 4 minutes ago
The dataset matters a lot here. You'd need to run Inkling on the same holdout set that OpenAI uses here to get any useful information to compare to this graph. If you measured the perplexity on any model on reciting the declaration of independence it would probably be zero.
reply
ThouYS 4 minutes ago
depends entirely on the dataset...
reply