NUMA is one of those amazing things that trip you up in all sorts of ways at unexpected times. The amazing "invisible" performance killler (invisible because unless you're already aware of NUMA, or remember to check, you won't know it's there potentially crippling you.)

It has been a source of routine conversations with customers and engineers of all kinds, and often one of those things you don't know about until too late.

I don't know if the kernel has improved this behaviour in the several years since last tested, but a coworker realised that the linux page-cache wasn't fully split by NUMA node. They were benchmarking mysql running it in each NUMA node, and noticed the second NUMA node was noticeably slower. Then discover after a reboot the second node was fast, and the first was slower. After a bit of thinking and tinkering they discovered that libmysql was ending up in the page cache in the same NUMA node as the benchmark client was run in first, so even though they were pinning the benchmark tool and mysql process to the NUMA node, the benchmark client was causing the OS to reach across the NUMA node to get at the page cached library.

Something I didn’t see mentioned was that this unequal memory access time also affects pcie I/O. If your thread on CPU A needs to get data in or out of a nic on CPU B, your throughput/latency will be impacted.

We have to explain this to customers of our software all the time, it’s something that’s easy to miss.

NUMA can cause really crappy performance. We deployed a Go based LLM gateway in Kubernetes deployed on a server with hundreds of CPU cores. We didn't explicitly set GOMAXPROCS so Go runtime scheduled goroutines over different CPUs and it constantly used 200% CPU and GC was causing latency spikes. Then we set GOMAXPROCS 8 and all performance issues went away. Until recently Kubernetes didn't work well with NUMA.

[dead]