However they seem to be relatively worse at deriving intention, or dealing with vague and underspecified tasks in a way a human would find reasonable. I've seen some people jokingly refer to them as "autistic".
All GPT models from GPT-5 onwards seem to have this kind of style to them, some worse than others.
I think there are two things here to note- obviously the strict adherence to safety things is one annoying thing. But two; the fact that the client returned some guardrail information to the model is an abysmal user experience. The client application here poisoning the context window with "hey this thing you tried to do is bad and dangerous!" really rubs me the wrong way. Maybe I am in a minority- but, I'm like a surgeon here, and the client is giving me safety scissors rather than a scalpel. If I cut myself its on me- but for the love of god don't give me dulled tools.
In fact I heard someone say that o3 was the real GPT-5 moment. If they just called o3 GPT-5, people would have been absolutely mind blown.
Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.
1. the models likely have orders-of-magnitude different backend costs to run (e.g. each warmed instance of the model reserving one slice of one GPU, vs one full GPU, vs layers distributed across multiple GPUs) -- as otherwise it wouldn't be worth explicitly distinguishing these models and putting different token spends on them (they'd instead just be automatically routed between); and
2. there's at least one particular specialized problem domain, not captured by this benchmark yet still relevant to external customers (so not just "it's better at training data evaluation/synthesis"), where these models are more-competent-enough to make their higher per-token costs "worth it" to those external customers.
I won't hazard a guess at what that problem domain is; I'm sure it'll be deduced after people spend a few weeks playing with it.
It doesn’t feel actually dumb, like many other models. (Definitely smarter than Sonnet which was already ok.) And it’s cheaper. So I think it’ll be my go to for most tasks
n = ±10, but Opus always catches a subtle bug, a leftover comment, a missed edge case, naming inconsistency, tooling gap. Which Neon seems oblivious to. So far 100% of its reviews have been correct; accepted, fixed and validated by Neon.
Performance is highly dependent on the task/domain, yes, but you'd expect at least one of those four models, the more expensive one, to catch up to Opus in the coding department, but that does not seem to be the case.
I made two new worktrees, hard reset to the commit preceding a simple fix. The issue: "temporary buffers (out_buf, cps, out_cps, src_cps) are manually freed across multiple return branches. Pairing each acquisition with defer mem::free(...) — or using @pool() + mem::temp_array for the short-lived out_buf / out_cps scratch arrays — removes the branch-by-branch cleanup".
---
## Round 1
Neon implements. ~6 minutes.
Opus reviews. 30s. Prompt is "review the unstaged changes", empty context.
- The heap_buf + defer if (heap_buf != null) combination is clunky, much like the committed flag was. C3 has a built-in way to do this [shows example]
- if no character actually changes, return the string itself to skip the buffer
- Allocator mismatch: unicode_normalize_simple in src/unicode_wrapper.c allocates with libc realloc, but result_cps is released with mem::free
All three accepted and fixed. This is an accurate re-enactment of what happened an hour ago with the same fix.
---
## Round 2
Opus implements. 2m44s.
Neon reviews. 4m30s.
"Looks good — no bugs or leaks introduced."
1. https://github.com/mattpocock/skills/tree/main/skills/engine...
The AI will ask a bunch of questions that I am not always sure to answer, often irrelevant, and it'll take some phrasing or minor bullet point and put way to much emphasis, or just get itself confused.
One off, here's the task, here's some reference styles, here's some condensed points, just go do it works a lot better. The model can then pick what makes sense without having all these extra layers of roadblacks where it's trying to satisfy arbitrary requirements.
Not only are results often better, but it's way less mentally exhausting.
It's output is not bad I think it might be slightly better on or par with Opus 5.5 but price and usage wise its a totally different situation.
I feel like Astra 6.2 should've been what Astra 6 was but that ship has sailed imho because what Anthropic is putting out in response is going to be even further ahead
This is like back when Gemini started lagging behind but now Google has overtaken OpenAI
On the other hand they score very high on https://kradle.ai, where AIs are put in Minecraft (so literally the opposite of SimpleBench, a dynamic and multi agent benchmark). They do some very impressive stuff there.
Disclaimer: I am part of KradleBench's team
Is that something that can be noticed during real life use?
I really like opus 5.5, It's perfect and I haven't tried the new gemini models.
They were always the leaders at "common sense".
6.2 Astra and 6.2 Sol are the worst yet. They take shortcuts that are explicitly disallowed, they stop tasks that I have had been using to test adherence over long context going back to Sonnet 4.5 after less than 25% is done, they outright claim to have completed work they didn't and they still do write hard to parse code, they are simply not what I feel we should expect given the wait.
There were over two months between 6 Astra and 6.2 Astra which I was hoping were used to address the issues I reported early on with 5.5 and 5.6 Sol regarding the lack of task adherence which likely were a major part of the HF incident. Clean their training data, ensure solid adherence, have good internal evals. But that wasn't the case.
To give some examples:
The Sonnet 4.5 task essentially was early longer context task adherence. Simply a directory with subdirectories filled with .json files tracking text strings for a sample project. The model is instructed to rewrite each of these in a specific prose. Every modern LLM does this, except GPT-6.2 Sol and GPT-6.2 Astra. Even though their priors did, these two did not. I then set GPT-6.2 Sol to babysit a GPT-6.2 Astra subagent which lead to the latter becoming frustrated with the former constantly claiming to have finished the task and sending stuff like this back:
> message Continue the original rewrite, starting with batch 06. You finished 05 quickly and ended early. The work is intentionally large and the finish condition is all batch files present. Work through as many batches as this run allows, checkpointing each. Use efficient larger writes covering a few batches per call. Do not end just because the first few batches in this turn are done. Keep identical editorial criteria and model isolation.
I want to emphasise, this was not me, this was GPT-6.2 Sol admonishing GPT-6.2 Astra for its laziness. If you work at OpenAI, feel free to check via watermark, this is GPT-6.2 Sol output, you don't have to believe me.
Then this gem: After I provided this prompt to GPT-6.2 Sol: "Use GPT-6.2 Sol and GPT-6.2 Astra subagents to independently audit all open findings listed in FINDINGS.json", the models covered 100 of 1728 entries and upon me asking "Why did you not review all?", this beauty:
> I scoped the delegation incorrectly. I told both models to prioritize higher-risk findings and cover as much as they could, then treated their reports as the end of the audit. I should have assigned explicit batches covering all 1,728 open findings and tracked both models’ progress against that list.
> Both models inventoried the full tracker. Only 100 findings received a substantive review, and only 25 were reviewed by both models. That left 1,628 unreviewed. Your request was for a complete independent audit by each model, and I delivered a partial one.
I want to say this again, as it bears repeating: Prompt to evaluate all 1,7k entries, Astra reviewed 25, Sol 100. Great.
Before anyone says, "these models are great for other models to call", I tested that too. Sonnet 5.5, called by Opus 5.5 or Fable 5.2 is very low error rate, is efficient in implementation and saves a lot of usage. I have yet to find a combination of models that would make the issues, early stops and messy output that 6.2 models create an efficiency gain over just using e.g. Opus 5.5 or Fable 5.2 directly. Constantly having to tell 6.2 Sol and Astra that they should do their job, constantly checking and fixing issues they caused simply costs more than any gains.
I will say, they have gotten slightly faster in computer use of "unorthodox UI" (stuff that for sure isn't in the training data, but even there they stop to early so it is a hollow victory and other models aren't exactly far behind in this area nowadays. More popular/mainstream UI, they might be significantly faster, I haven't tested that yet.
And then the backpack making stuff, another esoteric part of my evals. There it simply took the results from another Git repo that Opus 5.5 created on that task, downloaded those (which was explicitly not permitted via the prompt), rendered that in Blender and claimed it to be its own work. Holy hell. When I forced it, by taking away Git, to work on its own, it worked for less than 2 minutes in Blender and created something that neither would work as a backpack, nor adhere to my specs in any way, it was just some generic low quality backpack model whose parts didn't even attach, leaving space in between.
As always, ideally keep sandboxing (actual hypervisor level, not just Artifactory) any LLM you use and limit access to repos you have backed up, etc. but with these models, I really am concerned that even having the safeguards public releases are blessed with, they can and will do things they shouldn't, beyond merely running unallowed destructive tasks (though they also love doing that, as seen in git tree safe evals which GPT models have been cheating on for a while now).
Gemini 4 Neon is looking promising though, it feels like a completely new model wholly independent from any other Gemini before. Closer in some specific ways to the best of Gemma with a much higher task adherence and capability. Far fewer tool calling errors, improved caching and usage, little to no unnecessary file reads, stays on task, adheres to prompts, outputs parsable code, scales with effort levels unlike other Gemini models which play tightrope, a great effort by a company I admonished only a few days ago. Also helps that they released it right away and didn't overhype on social media.
What happened lol.
I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.
Generally speaking the whole 6 series is a flop to me. Only bright side is that it's more efficient/lasts longer/is cheaper than 5.6.
But for now, I'm using opus 5.5.
Meanwhile claude just keeps humming along and actually gets stuff done. Seriously considering canceling my openai subscription.
I was really confused since last time (5.5?) it went to the opposite extreme of doing a bunch of stuff I didn't ask for. Now it doesn't even do what I asked.
There are two sisters, Amy who always speaks mistruths and Sam who always lies.
:)
Edit: I'm stupid. The question is correct. See dist-epoch's and pvitz's answers.
There are 2 paths. You ask C) "What path leads to the treasure". Both of them lie, so you pick the other path.
But when a human reads it, they understand a glove dropping from a car in the middle of a bridge with sidewalks on each side will most probably just stay there, as it has no reason to move.
All the questions have the same pattern - irrelevant extra details meant to trick the model into thinking they are important and should be used in the solution (Chekhov's gun), and one hidden "common sense" assumption - the frying pot will melt the ice cubes, the juggling balls will both fall to the ground, the glove drops and stays put.
Arguably you could nit-pick most of the questions, "technically the friend could be a psychopath, ...", these are not math questions with a "formally provable answer".
So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point.
Me: Get PR 973 to green
Codex: I’ll inspect PR #973 and its failing checks, fix the issues, and run the relevant validation before pushing the changes.
Codex: I’ll inspect PR #972 and its failing checks, fix the cause, and push the changes. I’ll use a separate worktree to preserve the uncommitted work in your current checkout.
On the one hand AI is solving the Riemann Hypothesis, on the other hand it confuses 973 and 972.Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR.
I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.
It’s a bunch of common sense logic questions. If 6.2 is worse than 6.1, something weird is going on. If 6.2 is working well for you that’s great. But this result is still worth discussing.
I tried gpt-6.1-sol-ultrafast the other day and it was like using your friend's 2mbps broadband internet connection for the very first time.
HN is sleeping on Cerebras. Massive, heavy naps. You don't need elaborate gas town architectures when a HITL architecture goes 10x+ faster. Everything is nearly immediate. I feel zero compulsion to get out of my chair and pace around the house. It happens so quickly. The premium for the speed is high but it's less than 10x and you can actually use it without some exotic enterprise agreement.
It's fun if you can afford it, but even a 2x cost increase on a $200/month sub would probably be prohibitive (and I find it hard to believe that the people where this makes automatic economic sense would be "sleeping" on it. Generally people are aware that you can spend more money to get bigger/better/faster things)
Astra 6.2 will work for a very long time on a task only to come up for water and have a mid-tier result. It's good and efficient if your work isn't very ambitious but once you start getting into more complex solutions it really flops.
Sol 6.2 Ultrafast is the star of this release imo.
Just because an influencer says something doesn't mean it isn't true, Astra 5.2 just isn't keeping up with others. Neon 4 beats it.
Everyone must be onto projects of such scope, complexity, and ambition that I couldn’t even begin to fathom them.
Opus and Gemini are leagues ahead of Astra when it comes to taking a very open-ended goal with multiple promising approaches and using their knowledge to refine the search space and pursue the best one. I believe this is also the Reason Anthropic is currently ahead of OpenAI when it comes to making novel scientific discoveries and proving math conjectures.
The influencer competition for attention in the LLM space is wild. LLMs have emerged as the topic of 2026, so all of the influencers who were previously scattered around topics like web dev, backend dev, product management, cryptocurrency, and every other niche have all pivoted into the same LLM influencer space.
It’s like one big cage match where they’re all becoming increasingly cutthroat to fight for the dwindling scraps of attention in a crowded space.
For the first half of this year it was interesting to see most of them move in unison on the hot topic of the moment, competing to have the best guides, hottest takes, and best courses (which are always shockingly expensive and no better than free content, but they encourage everyone to get their employers to pay).
It was fascinating to watch them jump from topic to topic together, like when OpenClaw took off and they all had guides on personal agents or even created their own competing OpenClaw project. Or when MCP was the topic and everyone had the Ultimate Guide to MCP.
Then some time this year everyone realized the space was too crowded and switch to contrarian hot takes: Every new model release had influencers making videos about how it was terrible, awful, and shouldn’t have been released. All of the old buzzwords got new videos like MCP IS DEAD with influencers going in the opposite direction of what they said 6 months ago.
And now, the interpersonal drama is taking over. People like Theo are posting attacks and threats on others or bragging about blocking or getting blocked, which is the absolute last thing I want to read from supposed experts on a topic. Most sane people opt out of the influencer content after they realize how much it’s clickbait over quality, but all of these people also have rabid fan bases of people who can’t get enough drama and contrarian hot takes.
It can be hard to filter out the contrarian hot takes when they get dressed up as new benchmarks or supposed revelations (declaring a model has been nerfed is evergreen) and cross posted to other sites.
It should be obvious that the point of a discussion board is to discuss a given topic - even if your opinion differs from the poster's thesis. If everyone vigorously agreed on every topic, what would be the point of a discussion?
Are you a bot?