DaisyRose-27B: 1,200-task RL took Qwen3.8 from 5.6% to 12% on Terminal-Bench 4.1
53 points by de_sty 2 hours ago | 7 comments

jacquesm 7 minutes ago
I always wonder what you lose when you start maxing out some specific benchmark. It is really hard to do this in a balanced way and I would love if everybody that did these kind of efforts would take the time to also list the before/afters of a number of other benchmarks that they did not use in their training.
reply
woah 23 minutes ago
The heavy reliance on eval datasets showcased here and also evidenced by the frontier lab's high payments for such data cast doubt on the claims that AI will become self-improving soon
reply
xlayn 2 hours ago
Snap, no gguf to download, will wait for one to appear to test, any improvement over an already good model is good news.

"18 days on 144 H200 GPUs" insane.. didn't thought so much hardware would be needed to train Qwen.

reply
XCSme 60 minutes ago
Is it post-training or bench-maxxing?
reply
robertkarl 2 hours ago
I look forward to testing this against my private eval set to determine if it's benchmaxxing or actual improvement based on just 1200 tasks. downloading it now
reply
octoberfranklin 52 minutes ago
FWIW that is about $331,000 on low-onboarding-grief clouds (e.g. vast.ai) right now using the SXM variants they used.

You might be able to shave that down using a different flavor of H200 (like PCIe) but it would take more hours, so benefits are unclear.

reply
lumost 10 minutes ago
Savings would likely be larger with alternative accelerators such as trainium, no?
reply