If you don't run your own LLM proxy, how do you know whats going on?
39 points by amrrs 6 hours ago | 24 comments

kennywinker 2 minutes ago
> Own your infrastructure

...proceeds to use codex.

I can't wait for hardware prices to fall so we can all actually own our own infrastructure. Until then, it's for the lucky and the rich.

reply
variety8675 31 minutes ago
I’m not sure what people are doing with these subscriptions to feel the need to monitor what’s going on this closely. A $100 OpenAI subscription feels effectively infinite for my usage at home
reply
TuxedoBuddy 16 minutes ago
Some people are fine with an ISP-provided router that just works, and some would rather run pfSense, if even just to look at the internals.
reply
NeutralWanted 40 minutes ago
I'd love to do something like this, or use API keys, but I still don't understand how people justify the cost of using API keys vs. subscription.

I tested it once, I'm in on the 20x claude max ($200/month) and I either max my weekly limit every week or come really close. But it only took 2-3 days to hit $200 spend with api keys.

reply
mrheosuper 5 minutes ago
They are betting on people don't max out their subscription.

Someone who pays $20 a month may only use 10% of their max budget.

reply
denkmoon 10 minutes ago
The drug dealer gives you a free hit because it is profitable for him in the long term for you to be addicted to his product, not because he is your friend.
reply
dcss_gardener 26 minutes ago
The current crop of flash models are more comparable to sonnet than opus but they are very affordable.

The reality is that the subscriptions are heavily subsidized. Pretty much every sub that's not run by one of the money-burning frontier labs has had to drop it or dramatically increase prices and lower limits in the last few months. Cheap tokens are an acquisition tool, not a real product. It sounds dramatic but it really is the drug dealer sales funnel.

At some point they're going to drop the real prices in your lab and you're going to have to figure out how to pay that much or pay someone else a lot less. Better to get a handle on it now.

reply
tripleee 8 minutes ago
It's not obvious the subsidization will end. Loss leaders are a thing. Most enterprise at this point are paying API pricing

To better predict what they'll do with it we'd need to know the proportion of token usage via API vs subscription. Keeping an affordable, slightly unprofitable subscription to get people in isn't a bad idea. They also get marketing benefits from the word of mouth

reply
gruez 12 minutes ago
>At some point they're going to drop the real prices in your lab and you're going to have to figure out how to pay that much or pay someone else a lot less. Better to get a handle on it now.

So just wait until that happens? What's the point of scrimping and saving now? Not to mention you're paying for a premium in the meantime.

reply
soared 2 hours ago
One man’s fun is another man’s “idk what any of these words mean, I pay $20/mo for some company to abstract this complexity away”

For clarity, I do like blog posts like these.

reply
jubilanti 44 minutes ago
Yeah you should run an LLM proxy, but imagine a world where LiteLLM and all the other LLM shovels weren't buggy vibecoded messes. And with the velocity of updates I feel there has to be vulnerabilities there. Or am I just holding it wrong? Anyone got a good alternative?
reply
hoss1474489 33 minutes ago
Bifrost has been decent, despite being partially shareware
reply
what 14 minutes ago
Imagine complaining about “buggy vibecoded messes” that you want to use to produce your own buggy vibe coded mess.
reply
ok_major_9889 7 minutes ago
Why not just grab it from the kernel?
reply
hoppp 2 hours ago
Yup. Setting up liteLLMProxy on some servers as we speak and will start playing around.

I can't orchestrate agents in sandboxes without it.

1. It can issue virtual keys, so provider API keys never go near the sandbox

2. I can see everything

3. Guardrails

And much more

Looking forward to it.

reply
DelightOne 2 hours ago
1) What kind of guardrails are you using? Do you do models like prompt injection checkers? Or what guardrails are useful that do not know the environment the agent lives in?

2) I heard Bifrost also works for this. Did you try that as well, pros/cons?

reply
dcss_gardener 20 minutes ago
I'm not the person you asked but yes pretty much. I've found the prompt injection protection models pretty limited, but get good results from gpt-oss-safeguard with a policy describing various prompt injection techniques. More expensive but more flexible and reliable and I only run it on web fetch and web search results. Costs me like 5-20 cents a day depending on what I'm working on.
reply
impure 2 hours ago
OpenRouter does all this for you. It's pretty nice. Although for apps I use another server to lock down the requests even more and prevent the client from seeing my API key.
reply
sroerick 45 minutes ago
Seeing some good stuff in the comments, but how are people doing this?
reply
voodooEntity 5 hours ago
Well that was one of the reasons why i build my own agent/harness - to have actual full observability of what it is doing and why it is doing what it is doing.

Don't really understand why other agents to give those insights but well found my solution.

reply
stavros 2 hours ago
Does anyone here use LiteLLM for this? How well does it do? Can I use it with my Claude subscription?
reply
verdverm 2 hours ago
I use GoModel and its amazing. My favorite features are virtual model, the support for arbitrary request labels, and the nice dashboads / ui. Creator has more sophisticated report generation on the near term roadmap. He's super responsive to feedback and ships fast. (except for his recent trip to the US west coast to meet founders and investors, he's EU based)

You can use and control all.your subscriptions or aoikeys through a gateway like this, so you don't get pwned like what just happened with OpenCode (or an OpenRouter like vendor). GoModel has some ambitions like this, but is open source so you can run it on you own infra, with plans for self hosted enterprise as well (there are a few features already like SSO). We are rolling out the free version at work right now.

I run GoModel at home and have hyper detailed history of every agent interaction, across models running on my sparks and OpenCode sub. You can call me a committed superfan of this project.

reply
dongfutong 30 minutes ago
[flagged]
reply
skeptrune 2 hours ago
most interesting link i've seen on this site in a while
reply