They should be doing what this test did, and generalizing and significantly varying the training environments (and ensuring some of them are unexpectedly complex and unexpectedly simple).
Further I suspect the smaller models are especially overfitting to these sorts of simpler RL envs (benchmarks tend to be much simpler than the hardest RL env used today) because they are more likely to be scoring at ~20-40% success rate (which is sort of what you want to get good training signal when doing RL runs).
The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.
Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.
All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.
Especially since this benchmark is still mostly DeepSWE, and the only reason the open source models are doing decent is because much of the information hasn't changed. The open source models have been benchmaxxed to death.
Astra does seem close.
I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra despite the benchmarks (obviously much closer than Flash, though).
I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...
Something seems amiss.
Edit: I think that's the answer. This page lists the score as 53.3% which is close to the 54.4% reported here https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 for the 0731 update. The original from April was called preview, doesn't list the score, but in the comparison with the update it scores 7.3%.
Every time a Chinese lab releases a new model, I give it a solid try that same weekend and drive it hard by giving it the exact same tasks that I give to Claude/Codex models. And the new releases all have the same issues: they start out strong, but start deviating after about ~30 minutes and I quickly find myself in a quagmire where the model makes mistake after mistake while fixing its previous mistakes. Sometimes these are minor. Occasionally they can be pretty bad. Like, would-cause-data-loss-if-merged bad.
Ultimately I also reject the notion that Chinese labs are only a few months behind US labs. We haven't had an actual Fable-level model come out of China. AFAIK only Kimi 3 was claimed to come close, and then it turned out that not only had they heavily distilled Anthropic models, but, unbeknownst to their users, they were routing some percentage of their requests to Fable.
[0] https://news.ycombinator.com/item?id=50373961 [1] https://news.ycombinator.com/item?id=50374111
This is the same stupid argument trotted out repeatedly in the thread, and it doesn't hold up to reality. Would you care to explain how adding specifications helps a model that demonstrably ignores specifications? A model that cannot even handle the trivial specification of "the filename must be lowercase" is not going to get better when you add more specifications it ignores. I would guess that you're actually significantly underspecifying relative to me. Perhaps you're happy with whatever slop you get and don't notice Luna's overwhelming incapability to actually provide what one requires, because you don't have requirements.
Wrong. What I want Luna to be able to do is ~deterministically make small edits or generations of 1~10 LoC according to a specification, and it literally cannot even do that. It violates specifications no matter how small you aim.
> there is nothing deterministic about using llm's for codegen
And if you're okay with that, you're more like a farmer eating the slop meant for his pigs than an engineer. Using Sol Max, you can get near-deterministic output for a much wider range of tasks and specifications. Not all specifications; at a certain degree of sufficiently detailed specification it will begin failing again. But some, which is better than none.
again, you're expecting to use it without steering. that's not how you get the most out of the smaller models. i don't want to say you're using them wrong, but you clearly are. it could be deficiencies in your harness as well which the stronger models mask. try pi.
> you're expecting to use it without steering
If I have to manually "steer" every mistake it makes on every trivial task, instead of being able to deterministically eliminate classes of failures so that fixing the task one time ensures that failure never happens again on similar future tasks, it is in fact faster to just type the fucking code myself.
but your claim was about file name invariants when generating code, especially small amounts of code, which is not what they were doing with DeepSWE, so you've confused two different things.
anyways, this is a pointless argument. if you can't figure out how to effectively use smaller models in your workflow, fine. but don't come on here talking shit about people who can.
>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.
I ran into the same problem on my English exams.
For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.
Moreover, the months I've used anthropic/OAI exclusively my API token costs are in the hundreds (or sometimes thousands). With DS flash as the first agent I reach for, I rarely exceed $40/mo.
Every once in a while I come across comments that just hates DS. I wonder if it's a matter of how people use it or the prompt. I admit pre-v4.1 was pretty bad, hallucinating too much and not getting the abstractions right so I had to correct it often.
I also have $200 Claude subscription (now called "Ultra"). Fable gets abstractions right on highly complex problems (e.g. defining protocol that spans multiple services) that DS simply cannot do. However, DS is comparable to Opus 5.5, although less "intelligent", but DS excels at speed which is what I prefer than waiting minutes on Opus 5.5.
I have access to Claude Code at work (and use it quite a bit), so I feel I can honestly compare them.
You do have to plan things first a bit more with DS (than say Opus can (Opus will have more chance of one-shotting something, although I generally don't think that's a good idea), although it often over-engineers and tries to be too helpful in my experience) and be a bit more explicit at times, but I feel it's quite close to Sonnet in performance for implementing plans, debugging issues and explaining how codebases work.
When you see this your worldview will instantly become much clearer.
I've also been very happy playing around with qwen 3.8 27b running locally for personal projects. It sometimes can't quite get there, and that's when I'll reach for GPT.
Anyway, I use DSF v4.1 on max as much as I use Opus 5.5 on xhigh, Opus is definitely a more versatile model, but I’m not getting any value using it to implement well defined tasks that DSF v4.1 does just as well at for a fraction of the cost.
You see where GLM-5.3 is in this test down but respectable, but this model benches below DSF 4.1. TLDR basically nothing in this benchmark invalidates the Deepseek that everyone is using today.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.
Edit: another comment pointed out it's probably not benchmaxxing due to the chronology.
https://news.ycombinator.com/item?id=50406948
But why should minor changes to the task cause catastrophic performance drops? How is that not evidence of benchmaxxing?
Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.
(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).
If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.
It's not unreasonable to assume Kimi 3.1 is also a distilled model even though it's large (the difference being that it's a distilled from US models). I wouldn't be surprised if it also collapsed on this benchmark.
For instance, I had Qwen3.8 Flash Next reply with "I'm Kilo Code, a coding agent designed to ...".
That was in llama.cpp's UI - no harness whatsoever.