Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
51 points by marcobambini 5 hours ago | 14 comments
herf 27 minutes ago
So if this Mac uses 30-50W, that's 40-60 tok/Wh...vs maybe 80k for a modern GPU cluster? So that's about 1000-2000x more power for the SSD streaming, unfortunately.
replyjpecar 2 hours ago
Where can this 1tb k3.waste be downloaded?
replymarcobambini 2 hours ago
It is not yet available, the only way is to download the official Kimi K3 model and then convert it:
reply# 1. preflight: reachable? how big? does it fit? tools/fetch_weights.sh --dest /Volumes/staging/k3 --dry-run
# 2. download — resumable, safe to kill, safe to re-run tools/fetch_weights.sh --dest /Volumes/staging/k3
# 3. convert into a container uv run --with torch --with safetensors python tools/convert.py \ --src /Volumes/staging/k3 \ --out ~/models/k3.waste --jobs 3
cjbprime 2 hours ago
Does it not use Metal, on macOS? Would it be faster if it did?
replymarcobambini 2 hours ago
We tried to use Metal, but for that specific project it was slower than just using NEON ARM optimizations. It is all documented in the docs.
replylogicallee 54 minutes ago
Interesting project. The headline number (29 GB of RAM) is for 4k context.
replyFrom what I've read elsewhere, Kimi K3 is quite verbose in its thinking. At the quoted rate, it would generate only a total of 1.8k tokens in 1 hour. Is that enough for it to get any thinking done and produce output on more complicated prompts?
>...
>claude
You don't need to presume. If someone is so lazy that they tell claude to commit their code (ie. they're too lazy to run git commit themselves), the chances they reviewed the code is slim.
Personally I've mostly given in to letting it commit for me now, though I do occasionally take over and hand-write the messages if it's a particularly important concept and Claude's is too verbose.
Codex/GPT-x defaults to one-line commit messages, which are too short. Claude likes to write several paragraphs, which is usually too long.
If you tell it how to commit properly once per session it will stick with your standards for the rest of that session, and you can put that in AGENTS.md if you can be bothered to.
I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.
How bout we make a new rule: only complain about LLM writing when the product as zero relevents to use with LLMs.