But what is that amount of SRAM for? It simultaneously seems like a lot, because SRAM is so expensive, but still not very much, because what can you fit in 0.5GB? Is there a specific workload that needs something that looks like this?
SRAM wise it doesn't seem too insane. About the same ratio of memory:SRAM as the 5090 (IIRC). Still more than I would have expected. I assume they found it helps process the current layer faster or something.
Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
Raised arithmetic intensity. It would be quite good for LLM prefill, and decode would want to adapt by pushing parallelism higher (saving on memory bandwidth by reusing the same layer weights for multiple streams) or, if unable to do so (e.g. due to KV cache constraints) by resorting to speculative decoding (where the expensive 'validate' step handles tokens in parallel within a single stream). It does not make the platform inherently unviable. It would also do well at other tasks such as image/video generation, which is already quite compute bound.
Qualcomm Cloud AI 100 Ultra has 800 TOPS, almost 10 times more.
Give me 1TB RAM and 5TB/s or something that exciting.
But this gives you significantly more memory capacity for a compute/vram bandwidth tradeoff (i.e. arithmetic intensity) that's ultimately comparable to Blackwell, perhaps even more skewed towards compute. This seems good if you care about being able to store larger sparse models and more KV caches because that ultimately bottlenecks your ability to do inference.
The thing you are describing is much closer to AMD and Nvidia latest generation server offerings.
See MI455X for the absolute cutting edge and MI350P for something you can use outside of a datacenter.
Plenty of businesses want in local inference and are priced out.
Nobody helps China like American corps helps China. Domestic anticompetitive behavior spurs demand for international competition
Splitting the tensors is closer to n times the effective vram bandwidth but there are still overhead, sync waits, and reduce steps that eat away at the overall performance.
I'm starting to believe in Qualcomm! The Snapdragon X2 Elite Extreme is alas only available in a single system, with it's tri-channel on-package (!) memory good for 228 GB/s. But it's a beast. The core is just good, even in the mainstream form in other systems: some of the best single core performance you can get, shy of Apple.
And they are upstreaming their stuff much better! They hired a bunch of really good known FOSS developers and it seems to be working!
Now they are talking up High Bandwidth Compute as their next big thing. And I want to believe, even if it makes me nervous that they too intend to leave consumer computing completely in the dust. https://www.qualcomm.com/news/onq/2026/11/qualcomm-hbc-valid...
This is, as commented, at M4 Max levels of GB/s. It is not impressive. It has some uses! But if this was their modern opening salvo I would not be impressed.
Someone drop the YouTube link of their robot presentation