Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses
41 points by stared 2 hours ago | 17 comments
spider-mario 42 minutes ago
> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower.
replyConfidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them (https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run variation (https://link.springer.com/article/10.1007/s10654-016-0149-3 misconception 22).
purpleflame1257 47 minutes ago
There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.
replycivvv 19 minutes ago
Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!
replybellowsgulch 42 minutes ago
Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.
replyUnfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work that gets otherwise rejected.
sroussey 3 minutes ago
Have you tried https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit ? PrismML is the only people i am aware of doing 1bit that is decent.
replyspider-mario 30 minutes ago
> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.
replyHave you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.
Xeoncross 29 minutes ago
I leave it running at night. No danger of burning my token subscriptions and it has hours and hours to run slowly with a manager like: github.com/kunchenguid/gnhf
replyThrowawayTestr 6 minutes ago
I treat it like image gen. Send a prompt then come back in 40 minutes.
reply
Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc),
But on the other hand, the result that I'm assuming the author means ("on these evals, generation quality seems fairly good") is worthwhile?
Because I really struggle with this question at the moment. Am I allowed to draw an adverse inference that "if the writeup presents irrelevant text side by side with the data, then this may be a sign that the author does not understand the task that they are attempting to write up"?