I'm interested in hearing from folks who are hosting their own GPU to run coding models. So far my own results are... not great. Seems like frontier models are needed via the big providers?
(Submitted title was "A self hosted AI software factory")
1) I have knowledge DB that I query, spend a lot of time to get "query N models, stream to me all N results, let me pick which one" - only to find out that qwen beat all other contenders (those were picked before I got R9700, so the rest is =<8B parameters)
2) Setup OpenCode to use it and gave it a few tasks:
- first task (new feature to my MCP) took 40 minutes to complete with zero input from me, while it took 20 for sonnet-5 and sonnet-5 kept bugging me. Results are near identical.
- second task (port a specific version of a package to my flake) it got stuck in a hilarious loop where model already been told what hash to use by nix itself, but it wanted to figure our how to get hash another way for some reason.
- third task (another task, but much harder than first one with most of the discovery already done), kept doing discovery and running out of context, went through 3 compactions (256k context is what I can fit on R9700). Room got too hot, so I stopped it.
3) virtual assistant like Hermes but my own: no notes, works great.
The Qwen models are decent, but they don't come close to the full Claude experience I've come to expect. As such, I only use the local models for specific tasks where it makes sense to do so. Really the setup is that my Claude-powered agents are able to incorporate my local model into work it builds out. The agents can perform inference against the Ollama API as they see fit, and I encourage them to do so for tasks where (a) the low-level capacity of the local models make sense and/or (b) where costs can become a concern.
It seems to work well when it comes into play (like having Claude drive a web browsing session but letting Qwen handle much of the actual browser interactions, image analysis, etc.). Still, Qwen just isn't smart enough (or fast enough on my machine) to handle anything agentic that isn't non-trivial.
The Hermes subreddit is a curious place where every second person has quick their $300k a year job and is making a living off of Hermes doing... something? They treat themselves as the CEO of a bunch of agentic employees and have AI generated infographics of their "Stack" (all hail the mighty Stack) and everybody stands in a circle and applauds the most convoluted Hermes setups you've ever seen with not a single word how any of this is supposedly making anybody any money.
Are people developing like this? Are people asking money for something they one-shot instructions to an orchestration agent which delegated to fifteen other agents in Kanban and then spat out something that "works"? Every project I've ever had an LLM do a majority of the work for me has been strictly for my personal use; I'd never let anyone else use this stuff because it doesn't pass the vibe check. When a new frontier model comes out I'll pass the previous frontier model's work past the new model and let it tear it to shreds and see what improvements could be made for shits, giggles, and to waste a week's worth of tokens in the course of thirty minutes, but Reddit is overflowing with seeming non-coders who are passing this 100% organic slop off as sellable product?
I need fewer morals.
https://joeldare.com/creating-a-minimal-dark-factory