in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?
good job.
- multi GPU configuration (hetero and homo)
- PCIe gen chooser (3.0/4.0/5.0)
- is nvlink enabled (2 cards, 2 x 2 cards, 3 x 2 cards)
- is PCIe bifurcated or CPU native (threadripper pro for example)
I don't think in general people care about long list of models, they just care about the best one.
Also dual gpu is common setup, there should be ie. 2x 4090.
It won't win benchmarks of course, but 20 tps generation is okay and letting it chug away on my local kubernetes cluster to troubleshoot is pretty nice. First start is pretty slow though, parsing is around 100-200 tps. Which is crap for long questions or harnesses like opencode which start around with thousands of tokens. Once cache is warm, pretty enjoyable.
I think it could have way more use cases where it's just 'good enough'. Email categorization, voice commands for the home. 1 year ago this was still a very miserable experience, but qwen 9B + better cache quantization and more tricks in the latest llm engines makes it viable.
Will it do difficult architectural refactors by just giving a one-shot prompt like opus allows, nope, not gonna happen.
Depending on who you sign up for, that's within the introductory credits (usually like 5-10$) so it's actually $0.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.
Currently on my 9 year old laptop with a GTX 1070 Max-Q 8GB vram, 16 gb ddr4 ram and i7 8750h. I am running a finetuned Qwen 3.5-35B-A3B (Tiel-Coder-35B-A3B-GGUF-MTP) with 4-bit quant on a llamacpp fork (Turboquant) with 131k context at around 16-19 tk/s and 350-500 tk/s prefil using Pi harness. It is amazing for writing scripts and doing medium coding tasks and summarizing text for me. After around 100k context it goes down to 9 tk/s but that is to be expected.
If you ever ask any frontier LLM or these types of benchmarks, they would probably say that the best model I can run on my system is a crappy 9B model or something, which is just completely false. So I really wonder how much more you can pull out from more powerful systems and much better open weight models.
I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.
At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.
Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.
Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.
I might write more about this setup soon.
RAM is going to be the big bottleneck for MOE (at least for now) as it will determine the quantisations you could run.
On a 3080 10GB and 32GB of RAM I can run the qwen-next coder model and that is a 127B model and it gets 40-50 Token/s with 65k context. That is a lot more usable than a Quantised Qwen3.8-27B in my experience both because its faster and more capable.
I am looking forward to Qwen4 soon, I am hoping we get some more 27-35B MOE models, those are going to be really good for local hardware.
I tried llama.cpp with offloading "all what could be" to my GPU (GTX 750 Ti 2 GB). I could run local Qwen2.5-Coder quite fast (I do not remember but something like 25-30 t/s). However this model generates bad code (I tried Python and JavaScript). So, for my setup it is better to chat with free ChatGPT and do copying-pasting between web chat and code editor.
Is it even worth for trying something on my GPU? What do you think?
And yes, I use Claude Code (Opus 5.5) and Codex in my job but for personal stuff I would prefer trying free solutions (it is the reason I tried local LLM). I do not want to buy subs, at leats for now.
I believe in your case the model quality matters a lot. To my (trust me bro) experience any MOE model, finetuned or not, with params < 19B even with Q4-quant always give lackluster results. So any Qwen2.5 wouldn't suffice to even simple coding tasks. For your case maybe you can bump it up to a Gemma4 E2B/E4B or Qwen3.5-9B. Also make sure that your current build of llamacpp actually uses your GPU with CUDA.
I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.
I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.
I must be doing something wrong.
Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".
llama-server \ -m ~/models/tiel-coder/Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf \ -ngl 20 \ -c 16384 \ --jinja \ --alias tiel-coder
using it in opencode
{ "$schema": "https://opencode.ai/config.json", "provider": { "llamacpp": { "npm": "@ai-sdk/openai-compatible", "name": "Local llama.cpp", "options": { "baseURL": "http://127.0.0.1:8080/v1" }, "models": { "tiel-coder": { "name": "Tiel-Coder 35B A3B", "limit": { "context": 16384, "output": 4096 } } } } }, "model": "llamacpp/tiel-coder" }
If you drop the thinking level to low or even off the context will go further at the cost of the models intelligence. That might also help you get some answer but since such a question will read so many files into that context its the wrong approach. This is one of the bigger limitations of local LLMs, you can't do work that spans across a lot of files because you are almost always context limited. There is a limit of 1M even with the frontier so even that can't review an entire project. Focussing in on individual files or small groups is more achievable.
So just to be specific the -c needs to be >65536 and depending on the size of what is in the folder might very well not be enough.
Not sure what llamacpp build you are using but on mine using this fork: https://github.com/TheTom/llama-cpp-turboquant on commit 407f3237bfb3eeaff61546797de3d8c1a96be748 compiled for Pascal, I am able to push my current system to 131k context with minimal loss in quality. For reference this is my config.ini file for this model (Cyber is the better uncensored version btw):
[*] # Global defaults jinja = 1 metrics = 1 threads = 12 threads-batch = 12 flash-attn = on sleep-idle-seconds = 99999 timeout = 99999 no-mmap = 1 mlock = 1 ctk = turbo4 ctv = turbo2
[qwen36-a3b] model = /root/llm/models/Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf alias = qwen36-a3b n-cpu-moe = 32 temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0.0 presence-penalty = 0.0 repeat-penalty = 1.0 reasoning = on n-predict = 32768 reasoning-format = deepseek reasoning-budget = 16384 reasoning-budget-message = "Proceed to final answer" reasoning-preserve = true swa-full = true np = 2 batch-size = 4096 ubatch-size = 2048 cache-reuse = 256 ngl = auto ctx-checkpoints = 33 ctx-size = 262144 no-mmproj = true no-mmproj-offload = true spec-type = draft-mtp spec-draft-n-max = 2
I am also certain that these flags could be improved somehow depending on your system.
I would go ahead and give it at least 64k but the more the better really!
I run the kv cache at q8 for keys and q4 for values, but going below q8 is generally not recommended.
would drop it from fp16 to 8 bits. You don't want to go below 8 bits if you can possibly avoid it and if you really do need to squeeze just drop the v value to q4_0 but leave the key alone. Unlike model quantisation this really makes things worse quickly.