I think those kinds of questions would be more interesting than the raw win/loss ratios.
I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm.
I've open sourced some of GPT-Astra's work (below). It actually did an interesting amount of changes in strategy over time as it's opponents became more difficult.
as far as the on rails comment : there's no good way to know the provenance of an LLM output, but the LLM made bots generally look like very small, modular, sometimes functionalized versions of the human equivalents; the difference being that the LLM is calling the funtions often rather than a conditional loop in a main().
a factorio bot I worked on when Astra was first getting previewed was essentially a directory of very small argument-receiving functional and self-supporting programs that had very limited scope, and an agents.md file that directed the LLM to where such capabilities existed to preempt a directory search every compaction/context-swap.
so you'd have things like "python walk.py N 40" for 40 units of northern travel. a watchdog that waited for HP to drop or movement to become inhibited, etc.
My very first experience with LLM driven bot development in the GPT3 days created huge unmaintainmable monoliths that kind of worked due to TDD but were essentially spaghetti. LLMs have been trending away from those paradigms into small self-encapsulated programs. probably because those are less abusive to ctx count and produce better results since they lose/compact less.
There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).
- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
The customer may not care what the internal contributions are to such a measure. Instead it represents the customers’ overall values in some way.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.
I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.
I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)
TLDR they suck and many default to doing probe rushes.
But right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.
Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.
On a more serious note: all three races play quite differently so it makes sense to focus on just one. And protoss is mechanically the easiest and most straightforward race: fewer but stronger units, a single worker can be used to build multiple buildings at once, versatile basic static defence
Protoss vs Protoss only also helps downscope the problem given the time constraints
[1] https://rolandgao.com/blog/gobench/