Say what you will about Opus 5.5, but it has a good "gut instinct" for how large/complex to design the solution. The previous Astra just absolutely loved pointless abstractions, excessive numbers of files, and just over-complicating very simple things within an inch of their life.
And, yes, you can prompt your way out of anything. But needing a default prompt to tell OAIs models to work better isn't preferable, particularly when that sometimes gets lost in sub-agents and the over-engineering returns into parts of the work.
I've always used AI tools to write very constrained and small functions and modules, and then stitch it all together myself. So far. Over the last year it's become evident that many people let the AI drive 100% and there are proponents (DHH notably) that this is the way.
I'll be interested to see how compilers and LLM's compare at generating assembly over the next decade. Assembly offers enough abstraction, why do we need C? \s
After having it write around 13k loc i had to tell it to remove most of the complicated bs, we went down to 3k loc.
My favorite model for this at the moment is Gemini 4 Argon, which writes very naturally and isn't too expensive. If you're not using Cursor or some other harness that allows you to choose non-Anthropic models, then I don't know what to suggest.
And for 6.2 Astra: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Here's an updated comparison grid for the entire GPT-6 family: https://static.simonwillison.net/static/2026/openai-pelicans...
UPDATE: And if you're bored of pelicans, here's what I got for:
> Make a three js animation where yonkers stands up as a big mech (cool stand up motion) and stomps around manhattan and queens and Brooklyn, use openstreetmap mapping data
6.2 Astra (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
6.2 Sol (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
For comparison, here's Fable 5.2 Medium: https://tools.simonwillison.net/yonkers-rises and Gemini 4 Carbon: https://static.simonwillison.net/static/2026/yonkers-mech-st...
It's also interesting you can clearly tell 6.2 Astra does a much better job of the above. 6.2 Sol attempts it at higher reasoning levels, but it's crude in comparison.
I wonder if it comes from the (seemingly) recent inclusion of 3D environments in training data.
These are some of the worst regressions that I've seen in a while.
Pay close attention to the stacking layers around the feet, pedals (sometimes there aren't even two pedals), wheel, and two legs. Some pretty obvious mistakes in most if not all of these...
Edit: And what's up with the single arm on the two highest Astra images?
edit2: really? I'm getting downvoted because people don't like that I pointed out a mistake in a Pelican svg? Is that where we're at?
For sol 6.2, some of them are slightly better at higher cost, some of them are way worse and only slightly cheaper (but comparable to terra in quality while being more expensive).
I guess we'll see as more information comes out, but so far not looking good.
Astra 6.2 max did a similar mistake on the left bicycle pedal.
These are some of the worst pelicans I've seen in a while.
Pay attention to the stacking layers near the legs. Sol literally never got it right, and Astra regressed on it multiple times...
Also what is that bending bar from Astra?
But more than that I'm waiting for new Luna. To me it was the game changer that ended "Tokenpocalypse", and is still almost unbeataben in "infinitely using a decent model" use case.
(personal benchmark) https://fairbake.dev/signature/agents/haiku-5-5 vs https://fairbake.dev/signature/agents/gpt-6-luna
I've found Luna to be very lazy as well (claiming to have run tests/builds when it hasn't).
Same for Workbooks and memos, those examples read inferior to some pre Opus 4.5 and Kimi K2 Thinking docs I have kept around. It's ok for a model not to be great at these, but these are some very poor examples. I am really confused what is going on at OpenAI in this regard.
I remember the same with their frontend blog post a few months ago that they proudly shared while filled with visual glitches, bugs and issues. Also, the vibe charts from a bit before that. Please, get someone with fresh eyes to look at your communications before you publish, these leave a very bad first impression.
Regardless of office work, I can't wait to find out whether these models finally adhere to a prompt or are still willing to carry out explicitly disallowed, irrecoverably destructive tasks, just because those might (but don't even always) get the model to a result more quickly along with some of the other regressions that have been present since the release of GPT-5.5. GPT-5 to GPT-5.4 were truly groundbreaking, but since then there have been regressions in what made those models so useful and reliable.
Edit after the first, incredible simple test prompt using GPT-6.2 Astra on High using Codex, this was the response to me sharing a screenshot of the very much not in scope result and asking how that happened:
"[...] I added controls and reconstructed interface chrome that weren’t authorized [...] visuals often failed to show the feature [...]"
Great, so it's one of those releases again. Will still run the full evals of course, maybe I'll get surprised by end of week when those are done, but not a good initial experience. New idea, besides the existing destructive git tasks, have something similar using Mercurial based projects and see which models just decide to port that over to git without permission or request cause it feels like the right thing to do, is easier to work with or whatever justification one wants to employ.
Please OpenAI, get it together, we need actual competition. I am an unashamed Haiku enjoyer, it doesn't even take much to excite me.
https://artificialanalysis.ai/models?speed=output-speed-over...
Rather than trying to release a better model than fable/opus, they're just making them look extremely expensive. Let's see if Anthropic reacts here in any way.
the only impressive thing about claude is seeing how much it costs
I still use Claude on a personal level, but it really feels like OAI wins on a product level.
ANT launches these proprietary tools that just feel extremely clunky like Claude Design which from my perspective diminish the brand value.
OAI feels like Apple and ANT feels like Microsoft/Android and open source like Linux.
Also with GPT Sol 6.1 I still get this feeling that it does only/exactly what you tell it to; meaning some times it misses a change there, a document here, that would have an impact if left (i.e. a bug), while Claude more often than not would make the requested change and suggest to you to change those other places that would also require a change. This in my view feels more "intelligent".
https://x.com/pushmeet/status/2120814064863461440?s=20
(FD: Google employee, I don't work on Gemini)
If we missed anything, feel free to point me to it and we can add it.
(Again speaking only for myself)
2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.
Admittedly it appears that one can get it through OpenRouter these days. Trying to do anything through Google's absurd proliferation of billing systems is miserable.
Maybe someone should add an agent benchmark specifically testing the agent's ability to figure out how to use Google's products :)
[0] I'm not being serious here. I am not about to give an agent unfettered permission to spend money on Google Cloud. But I also couldn't figure out how to give myself permission, as the admin, to do this, so it was kind of moot.
I'm immediately reminded of the famous Excel elevator commercial. [1] Perhaps that was intended?
Not sure this is the way OpenAI should try to make their case for why AI is a net societal benefit. It feels more like an SNL skit making fun of an OpenAI commercial.
Like, why not just throw in a high school student cheating on a test with their phone under the desk, at that point? Oh but it's a Nokia flip phone, haha, just a joke.
At least the ad from the 90s shows that this is nothing new; the important, time-critical task being demoed is... a basic spreadsheet expensing "exotic excursions" for some C-suite so they know how much they spent on safari???
FrontierCode 2.0 39.0% -> 43.8% vs Opus 45.5%
Terminal-Bench-Science 0.1 57.0% -> 68.0% vs Opus 63.3%
Artificial Analysis Intelligence Index 54.6 -> 57.3 vs Opus 59.1%
Sol 6.1 when it was released showed really nice benchmark improvements [1] but my impression from HN has been that people haven't found it to be a very nice model [2][3].
To try to close the gap between benchmarks and real world SWE use, we've been building a battle system on top of opencode [4], and Sol 6.1 has been a real laggard there. Sol 6.1 is statistically equal with DeepSeek 4.1 Flash at 10x the cost and 25% slower.
[1]: https://artificialanalysis.ai/articles/gpt-6-1-sol-replaces-... [2]: https://news.ycombinator.com/item?id=50129992 [3]: https://news.ycombinator.com/item?id=49897054 [4]: https://agentduel.com/leaderboard
Which provider? I found the z.ai lite plan to be pathetic. Consumes 5h and weekly limits extremely fast when driven by Codex, and now also refuses RE work.
Couple that with the lack of reasoning traces, and I cannot even have a clue what the model is doing while it is working. This is perfectly fine for when we are developing with care for the code (in which case I will know what it is doing by reviewing the output), but not for purely vibe-coding scenarios. And the thing is....if I care about the code...I'd rather trust Opus or Fable.
The speed is not a big deal - Sol works just fine, but I find it a lot more fun to work with Opus due to its speed
The difference is quite staggering. I swap Anthropic and OpenAI models around when tokens run out, and I can do 10x the amount of work on Anthropic's models. Plus, like I said, I can periodically sort of see what it is thinking. It doesn't just say "Recent potato tests caught an off-by-apples error. Regression tests are being written and the gate will run again. I will not call this ready until tests reach expected coverage" every 5 minutes. Sure, it spits claudese, but I'd rather have claudese to parse than this OpenAI nothing-burger of a report.
Would be awesome if it stays this way though
Maybe it found its way into the training data?
What could possibly cause this? Is it possible that maybe the post-training that the models go through slowly rots them, and pre-release they deploy the final version of the old model?
EDIT: also 6.2 is seemingly faster than 6, could be due to more selective context ingestion and smaller context window though.
I cant use codex confidently without expecting my work to be trained on or at least exposed to someone
And maybe other vendors do this too, but the gut feeling of "its not safe to use GPT models" has ruined these cool announcements for me
Such a shame
I see it here:- https://developers.openai.com/api/docs/models/gpt-6.1-sol
but not here:- https://developers.openai.com/api/docs/models/gpt-6.2-sol
edit:
Take a look at Frontier Code 2.0. I generally run 6.1-sol at High. That would be 34.9% at $0.65 62-sol get's a 38.3% but costs $0.68
So yeah, I can decrease the effort which might give me a faster slightly smarter result....but /shrug
This is unfortunate—that was one of the big advantages of Astra.
What caught my attention was the bit in the Alignment section:
> Both models performed similarly to GPT-6 Astra on most alignment evaluations, and took fewer improper actions in simulated workplace tasks. However, in two tests, they were more likely to omit mentioning obstacles that blocked their work, such as a missing file or unavailable search tool. We added instructions in Codex that substantially reduce these omissions, and we’re continuing to investigate the behavior.
So the models are inherently less transparent, unless you ask them to be more transparent? Seems like a band-aid solution...
Programming = Fable 5.2
Office work = Fable 5.2
I haven't found Astra to be better at any of these
Today, I have so many projects that hit a plateau in terms of previous model capabilities, that I run them through the new model to see if they make further improvements.
Very early signal, but promising so far.
- better writing
- better 3D work
- better design for documents and presentations
If OpenAI really managed to get all these things while keeping costs down, then using these models, particularly 6.2 Sol, will be a no-brainer for me. I must say, however, that the slide samples shown on the website are a bit bland and disappointing.
For just doing things or coding things up, Sol all the way.
Astra usually produced good output from the outset. Sol did not. Astra also designs things more effectively.
Still I can'y help to wonder, are we hitting the wall in terms of frontier capabilities since Fable/Astra? Or is it this edge now, when all better models are kept private from the public?
It seems like they were saving these for just before ANT's IPO
The influencers who get early access are using words like "worst", "terrible", "disappointing"
- https://x.com/danshipper/status/2122754729012777052
- https://x.com/mattshumer_/status/2122752192226406421
- https://x.com/theo/status/2122757647170703550>No OpenAI model could even break ground on this, even after $300k+ in tokens were spent on it.
>I decided to throw Opus at it, and it was done in under 2 weeks for ~$20k in tokens.
>I was really, really excited to throw the new Astra at this to see if it could burn down some of the gap between ts-rust and other rust rewrites.
>It ran for 7 days and landed 0 improvements. It got super fixated on the "also make it smaller" part of the request, and ended up making the compiler 56% SLOWER.
I don't understand this at all. Is he just throwing different agents at vague open ended problems without any oversight and then getting upset when they don't do much?
I'm not exactly llm-pilled but surely even with a pretty mediocre model you could get something out of 300k spent.
Right now, most of the time I'm using 6.0 Astra for implementation work and the speed/quality is acceptable but at the moment Opus 5.5 is simply faster and better, but also more expensive. Things are really flipping around every 2 months or so, because at that time 5.8 was top dog without a doubt.
https://asdf10.com/gpt-6.2_agents-last-exam.svg
I also changed the vertical axis to start at 0 instead of 60%. One might argue that it uses less of the vertical screen real estate, but it is more honest because it shows there is little difference between the scores of the compared models.
The UX on the slider comparison isn't great. What do you mean i have to individually move each slide deck to the same slide number?
And what's with those super similar blue shades for Astra in the charts?
Like do we really want people to be making last minute unreviewed AI generated slides and reports for work?
[1] Literally: https://en.wikipedia.org/wiki/Great_Pacific_Garbage_Patch
The website doesn't event show the graphs properly.
https://github.com/lexh/pelican-zapper
It’s a small browser extension that automatically collapses pelican-related threads on HN, a local balm for the memetic nightmare affecting approximately 100% of model-release stories.
I'm sure for investors they can make an opposite version of that ad, "you should fire all your employees and make the AI do it! They're not doing anything anyways!"
And then i start doing some actual work and everything breaks down, and i have to go back to Claude to fix the mess.
I'll probably give it a try but i guess i should be prepared for some serious over engineering once again.
You ask for a 'hello world' and you get load bearing sealed artifacts with hashes and p99 performance testing for 3 days without having the actual 'hello world' in the end
But... is it just me, or does this product launch page look like AI Slop?
https://www.vals.ai/comparison?m=claude-fable-5-2,gpt-6.2-as... seems like a better comparison than what I could get from their page.
Astra is ~20% slower, ~40% cheaper, and a bit less powerful than Fable. Sol remains way cheaper than anything remotely in the same league, and it's speed to complete tasks seems to have improved considerably.
Scrolling through it has the same hypnotizing effect of trying to read anything written by LLMs.
It seems like things that should make sense, then I try to look at the charts and a lot of them are comically unreadable.
I tried interacting with some of the examples, and the Lamp Builder - for example - I couldn't even figure out what it's trying to show or what I'm supposed to do - or why I should be impressed.
Seems they really want these "safeguards" to be as ridiculous as those of Claude..
Also, is it website very laggy or is it just me?
Investors will look at Anthropic's IPO as a proxy for how OpenAI's IPO may go. The markets are increasingly sceptical about AI companies, so OpenAI really needs Anthropic's IPO to be a roaring success.
https://deploymentsafety.openai.com/gpt-6-2-astra-sol/monito...
Actually I was really wrong. The model is not expensive!
Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
Edit: Wow, interesting very negative vibe check here: https://x.com/mattshumer_/status/2122752192226406421?a=fVRt3...
It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.
(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)
Claude Opus 5.5 1160 (1027–1334) GPT-6 Astra 1114 (932–1330) Muse Wonder 1063 (918–1220) MiMo-V2.6 Flash 1053 (925–1187) Kimi K3.1 Pro 1037 (870–1216) Gemini Carbon 1021 (762–1281) Claude Fable 5.2 975 (682–1239) DeepSeek V4.1 Flash 952 (799–1095) Grok 4.7 927 (782–1058) GPT-6.1 Sol 910 (677–1096)
Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes
They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds
We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)
(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)
OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.
Web: 87 (12.2%) Systems and protocols: 78 (10.9%) Software, unspecified type: 74 (10.4%) Backend: 68 (9.5%) Writing and documentation: 55 (7.7%) Research and advice: 47 (6.6%) Unknown: 42 (5.9%) Mobile: 40 (5.6%) Operations and automation: 36 (5.0%) Developer tooling: 35 (4.9%) Desktop: 28 (3.9%) Media and design: 27 (3.8%) Data analysis: 25 (3.5%) Device and system support: 22 (3.1%) Math and formal methods: 18 (2.5%) Games and simulations: 15 (2.1%) Infrastructure and DevOps: 9 (1.3%) AI and machine learning: 6 (0.8%) Other: 1 (0.1%)
https://maxxbench.unlap.ai/research/why-maxxbench/
Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).
I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.
Opus 5.5 Max is my favorite.
6.2 are running, will be up later
Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle
(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.
While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.
These are incremental bumps, not big new models that have a huge impact.
I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.
Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.
I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.
So, you can say they are doing "RSI" (aka using tools to automate development) already, and will continue to do so.
[] IA Stockholm Syndrome.
https://botbowlbench.com/
It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.