Hacker News

Show HN: Shoehorn – Quantize any model down to run on your machine

62 points by rhgraysonii 4 days ago | 14 comments

Working on Mac, Linux, and Windows now. I include a simple GUI to find new models and get things built and set up. It is working quite well across a few models for me. The GitHub README and DESIGN.md files go into detail of the how/why and it's working remarkably well so far. https://github.com/notactuallytreyanastasio/shoehorn

sscarduzio 27 minutes ago

The project name is perfect!

jedbrooke 2 hours ago

I gotta laugh at some of the models it suggests, for example:

> AnkitAI/Parable-Qwen3-4B-Claude-Fable-5-GGUF

you’re telling me you managed to fit Fable 5 into just 4B?

chompychop 2 hours ago

I gotta laugh at your thought process: knowing Fable 5 is a large frontier model, you're telling me that the first thing that came to your mind on seeing that model name is that it's a quantized version of Fable? As opposed to a distillation/fine-tuning on Fable responses?

unrented7977 2 hours ago

Don't make fun of people you think are ignorant, it's a pretty shitty look

chompychop 49 minutes ago

Well, then don't get all snarky and dismissive of things you might not be knowledgeable about ("you" here referring to OP).

hmokiguess 4 days ago

Reminds me of https://github.com/AlexsJones/llmfit

rhgraysonii 3 days ago

LLMFit tells you what can run on something. I built something quite similar to their search into Shoehorn now.

akshay_akula 3 days ago

This is interesting. I wonder how it could work with something like https://github.com/JustVugg/colibri.

mbuchel-hn 4 days ago

does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?

rhgraysonii 3 days ago

Yes that is exactly what this does.

kennywinker 3 days ago

Could you explain what happens when you try to shoehorn a 2.4T parameter model into a 24gb m4 mac?

akshay_akula 3 days ago

Wondering the same thing but for 48gb M5 Max.

jaylane 4 days ago

tried it out but based on the model sizing result i got i got an insufficient memory error when the server started running

rhgraysonii 3 days ago

If you could post an issue if you still have the error around that would be awesome.

kelvo_ran 2 days ago

[dead]