We turned off Pub/Sub and nobody noticed
40 points by patrickhamann 6 days ago | 8 comments
rorymalcolm 6 days ago
Disclaimer: I work at the incident company
replyThis is one of those projects where the end goal is to build the scariest button possible. You cannot complete the project until you’ve pressed that button. You do all the work to build the button, then you have to press it.
Never not scary, never not eventually fun, but type 2 fun for sure.
shintoist 6 days ago
NATS is such a cool piece of technology that deserves more hype than it gets
replyjdon 6 days ago
Given their recentish argument with the CNCF[0] not sure how much I would trust it.
reply[0] https://www.cncf.io/blog/2025/05/01/protecting-nats-and-the-...
DavyJone 2 hours ago
I really like NATS and I agree, but im generally hesitant to use it in critical systems due to https://jepsen.io/analyses/nats-2.12.1
replyWhat is your take on this?
A system, like Pub/Sub, being a critical dependency or hard dependency does not make it a SPOF. Otherwise any system is a SPOF, creating absurd architectures that end up with more complexity and downtime than the original one. For example, I assume incident.io are yet to see if their added complexity of multiple event streams does not cause worse availability issues than what they had, by introducing complexity like problems with data consistency.
If the the underlying "system" components are not redundant and highly available, then you have a SPOF. eg. A single machine is what provides Pub/Sub.
In this case, what you have is a hard-dependancy with an avaialbility lower than what you want to achieve, but there are always shared hard-dependancies that they will have. There is no way around this. What we should aim for is that any system that is a critical dependancy or a hard dependancy with such a big blast radious, to be redundant and robust in itself. Not just the system, but the practices around it.