Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.
Why would OpenAI expect any meaningful relationship between their models’ ability to predict the next token of supposedly held-out chunks of their internal codebase and the overall capabilities of the models? I expect both to improve, but that’s about the extent of it.
I really hope we go full speed ahead on AI development.
^ Sam Altman statement, basically saying that's not their intention. I want better AI too, but it seems prudent to move forward with caution.
I would expect the entropy of artificial datasets to be lower, thus easier to predict and scoring better on "perplexity"?
[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2