Brood War Bench
64 points by benswerd 7 hours ago | 37 comments

pelagicAustral 58 minutes ago
Unrelated to the benchmark...

I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...

I miss those days so much.

Everybody was from a different background back then, and nobody was anything other than a guy that plays StaCraft at the cybercafe... And now, we are in our 40's and I know Math teachers, history teachers, oil rig operators, software programmers, professional gamers, lawyers and more... hahah So crazy to think about it... and I know them, we talk, what a world.

reply
snicky 15 minutes ago
Oh boy, I've spent more hours playing it than I dare to admit. I won over 10000 battle.net games ... on just one of my several accounts :) When the SC2 beta came out, I played about 50-100 games and never bought the full game, because I knew it would be like heroin to me, and I was already an adult that had to take care of himself.
reply
nemo1618 41 minutes ago
Even at Burning Man, in the middle of the desert, there is a camp that hosts a StarCraft tournament every year (on the dustiest setups you've ever seen!) :)
reply
PorciiVorbesc 24 minutes ago
So ... Zerging Man?
reply
benswerd 41 minutes ago
So dope
reply
benswerd 41 minutes ago
I agree.

My first time playing StarCraft was at summer camp around a decade after it came out.

All the smartest people played it so I wanted to too. Great decision, I have been continually impressed with the people who StarCraft introduced me to.

reply
gary17the 29 minutes ago
My best greetings to all Starcraft Elders clanners of yesteryear :).
reply
sidewndr46 52 minutes ago
Later we even listened to Eminem and played violent video games. Most of have never even been charged with a crime, much less abused someone.
reply
AntiRush 15 minutes ago
Back in 2010, during the early days of bwapi, there was a Brood Wars AI tournament held by the Expressive Intelligence Studio at UC Santa Cruz. It's interesting to see how different the approaches were back then, vs this or Deepmind's SC2 work.

https://web.archive.org/web/20091124210529/http://eis.ucsc.e...

There's a great contemporary Ars Technica piece by a competitor:

https://arstechnica.com/gaming/2011/01/skynet-meets-the-swar...

As an undergrad I did a project using genetic programming. It was not very successful, but it was a lot of fun.

https://tomisin.space/archive/starcraft-genetic-programming/

reply
GodelNumbering 51 minutes ago
A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on

[1] https://rolandgao.com/blog/gobench/

[2] https://rolandgao.com/gobench.pdf

reply
c7b 3 minutes ago
At last something that feels properly orthogonal to pelicans on bicycles.
reply
faeyanpiraat 46 minutes ago
There is currently a bot beating everyone on the ladder. Just watched it today on Artosiscasts yt channel.
reply
minimal_action 17 minutes ago
I think we're on the early days of games you connect with your agent to. Human + AI units one versus the other. Like knights with their horses. Not sure which is the horse..
reply
WillMorr 3 minutes ago
I've been running these with friends recently, it's very fun. I ran irl bot tournaments for board games a couple times but the agent era opens up a huge amount of possibilities.

Must recently I built out a MMORPG puzzle box thing, I wrote a general game architecture doc but left the specific puzzle design up to Fable. Nobody is actively playing rn but I left it up at bot.willmorrison.net.

reply
tweakimp 25 minutes ago
If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net/
reply
gadtfly 50 minutes ago
Did it play by looking at screenshots and sending clicks, or was there other mediation/symbolization?

It sounds like it might have been actually played in real time, which would be very important to distinguish.

I have recently seen other harnesses letting agents play real-time games in what seems like discrete time slices, turning eg Portal into something turn-based https://www.youtube.com/watch?v=ruuGXFAmiOE

reply
loeg 22 minutes ago
There's a bot data stream already; it's probably hooked up to that rather than screencap. Yes, I believe these were playing in real time.
reply
Game_Ender 58 minutes ago
Any details about the harness the agents were given? I am curious what representation of the screen and world state was provided to the agents and what tools they had available.
reply
benswerd 53 minutes ago
Oh sorry I should be more clear on that. Will add to report.

For agent harness I did Claude Code, Codex, Grok Build. This was primarily a cost driven decision — I have a lot of free tokens and I didn't want to pay API prices for this.

For game harness I used minimal BW-API issue command and get observation apis as tools. I felt this was the most fair way to do it on my small scale.

In the future I would like to integrate code mode and multiple games/I think if it was a best of 5 where each agent could learn from its past games and build its own automations over time that would be much more interesting.

reply
dschuessler 51 minutes ago
Somewhat related: In 2018, Google DeepMind had already created AIs that were capable of beating professional gamers in StarCraft 2 (the sequel to Brood War): https://www.youtube.com/watch?v=cUTMhmVh1qs
reply
qerghnui 4 minutes ago
AlphaStar beat one retired professional by cheating.

AlphaStar won a showmatch against TLO. TLO was never one of the strongest players in the world. He had been retired for over three years by the time of the match. Google set the rule that their system would have human-like mechanics, but it played several times faster than any human, never issued a wasted action, had a reaction time of single-digit milliseconds, issued commands with perfect accuracy using an API, and could see the entire map at once.

It was later released to the open ladder with more human-level mechanics. Even strong amateurs regularly trounced it. I have beaten it myself. It was strong, but not even close to the level of the strongest human players.

Even the cheater version would have lost handily to Serral or any other world champion.

(It apparently beat MaNa as well as TLO, but those matches were never released to my knowledge. I see no reason to assume Google cheated less flagrantly in private than it did in public.)

reply
benswerd 45 minutes ago
I predict LLMs will reach superhuman level and beat even that model in the next 12 months
reply
orbital-decay 24 minutes ago
Starcraft is APM-dependent. Unless the latency will improve greatly in frontier reasoning LLMs (which is unlikely), it will remain a bit like knitting with an excavator.
reply
jamiequint 3 minutes ago
Should be possible to play this with Jev.
reply
benswerd 22 minutes ago
I predict latency will improve greatly in the next 12 months to more than 4x speed on current frontier tasks
reply
loeg 21 minutes ago
Yeah but Starcraft needs, like, 10-20x the APM these agents are doing.
reply
benswerd 19 minutes ago
I’m not convinced a lot of it can’t be solved with code mode.

Marine staggering for example seems like an ideal code mode task.

reply
adsfgoinoi 17 minutes ago
[dead]
reply
malfist 2 hours ago
This is a great idea for a benchmark. Something all the benchmarks seem to be missing is strategy, tactical solutions in most of the benchmarks are all thats required but here requires actual long term thinking and tactical thinking, balancing and orchestration.
reply
shard972 59 minutes ago
That’s why I tried making https://wrathbench.shard.page
reply
winwang 51 minutes ago
Would be interesting if you could team a fast and slow agent together -- slow model can either act directly or maybe just communicate to the fast model.
reply
benswerd 45 minutes ago
I might open this up to a tournament if enough people want. Any interest?
reply
bee_rider 2 hours ago
A ton of conversations about the game must be in the training set. I wonder, is there any way just from watching how they play, of telling if they tend to pick strategies that people complain or meme about online?
reply
stackghost 2 hours ago
I wonder if there is a library to decipher brood war replay files. Perhaps an agent could learn by watching.
reply
benswerd 52 minutes ago
Not hard to build. I was shocked at how fast/easy this was to pull together.
reply
frutiger 2 hours ago
I haven’t checked for SC:BW but Blizzard has official parsers/replayers for SC2 on GitHub.
reply
benswerd 7 hours ago
+ Playable Agent driven Starcraft
reply
conorcleary 47 minutes ago
Dibs on the fly brain
reply