Not that this really takes away from the substance of the article, but the first two paragraphs are giving heavy Claude smell. Semicolons, em dashes, "That sequencing matters"... I guess I'm just a little surprised that anyone could be arsed to take on a hardware project like this but can't be arsed to write their own introduction.

Out of curiosity, what are you training with these cards?

What a time to be alive, I remember 10 years ago as a poor student waiting to buy a ATI Radeon X1600 Pro with 256mb, yes 256mb of RAM.

It cost about £190 in 2006.

Now we have GPUs that are in tens of thousands of pounds with insane performance, but what would their price be without the AI and Datacentre squeeze?

It's difficult to speculate as to the exact failure from blurry pictures but the solder on that choke (inductor) looks terrible.

Something went wrong in manufacturing. The solder should have wicked to cover the entire pad, not just a small square, and there should be no (brown) discoloration.

I wonder whether those cards ran the model that wrote the nonsense about the forces involved.

Hint: when you have a piece of metal stuck with thermal goop to a lot of components, the force doesn’t “concentrate” on one of them. You need to detach it from each one with however much force is needed to detach it from that component.

Ok, how are people powering these things? 2.4kW is well beyond a standard circuit in the US. Are people having 240V/30A circuits installed? Are they hijacking the dryer plugs? EV charger plugs? Hottub circuits?

Cool post. FYI you might be better off getting one big fan for your "radiator" instead of lots of little fans. Big fans don't need to spin as fast as small fans to push the same amount of air. So they run a lot quieter.

Is that little computer training LLMs from scratch all by itself? That must take years to get any kind of progress, given the scale of training other providers do. Where do you get the training data from?

Complete side note, but I can’t work out how the author managed to mistype “at” as “Δt”.

Edit: reading fail on my part, nothing to see here.

Ditch the tiny DC fans. Build a shroud and switch to a single ac-powered industrial blower / duct fan.

Converting four RTX PRO 6000 Blackwell cards to waterblocks, finding a VRM choke loose on the workbench, and getting back to 41k tok/s.

If you want ready, well engineered, water-cooled multi-GPU research workstations, my colleagues at https://comino.com build and sell them. Or you can purchase fitted waterblocks from them for many GPUs, and build your own.

AI slop post.

Why does this post sound like it's an AI story based on the inputs from the engineer?

The phrasing is very claude like:

"That cracked joint is the whole story. The card had passed initial bring-up and ran fine at light loads for a week."

"That sequencing matters — it’s why we have a story to tell. The pilot card failed, taught us a lesson, and the lesson is the reason the other three went on without incident."

"Driver swaps, CUDA reinstalls, and inference-engine theories were dead ends I spent hours on. The failure pattern itself told the story — listen to it earlier."