Microsoft 365 outage leads to chlorine gas poisoning of 15 hotel guests
293 points by DeepLogin 5 hours ago | 151 comments

amluto 28 minutes ago
I personally operate a fairly advanced closed-loop chlorination system, and this is beyond screwed up.

These systems can easily fail, and they tend to fail in a way that causes them to measure low or zero chlorine and thus overdispense chlorine. More below on why.

First, this means that an operator must always ensure safety, both against excessive chlorine and against insufficient chlorine. In a commercial context, that means that a real human (or maybe a totally different sensor design, e.g. a colorimetric sensor) need to measure the pool chlorine level, preferably with a high quality titration (drop test) every day before opening the pool. They should probably record the result in pen on water resistant paper, but this part is icing on the cake.

Second, hoping that an email gets noticed is ridiculous. If you do not have positive confirmation that the pool is safe to open, don’t open it.

Third, the systems should be sized so that they cannot overdose quickly. The maximum dispensing rate should be such that hours of continuous operation would be required to reach an uncomfortable level, let alone a dangerous level. (Hot tubs are sort of an exception here.)

Fourth, everyone should be using a reasonable amount of CYA in their pool. Even at a lowish level of 30ppm CYA it takes a lot of chlorine to become dangerous unless the operator has also lost control of pH. I realize that some codes are confused about this.

Fifth, chlorine is smelly. Someone should have noticed, even just casually.

Sixth, the systems I’ve seen beep when there’s an alert. It’s easy to add flashing lights, too.

—- sensors —-

Actual free chlorine sensors intended for pools use an amperometric probe immersed in iodide solution or gel, separated from the pool water by a fancy membrane. Free chlorine oxidizes iodide to iodine, and the sensor measures the iodine concentration. If the membrane fails (or general wear and tear or excessive pressure swings) cause the iodide to be lost, then the chlorine can’t react and the sensor reads zero (or low if there’s tiny bit of iodide left).

Interestingly, these sensors are rather expensive devices that appear to mostly come from France and Germany. The Chinese vendors seem to be rebranding European parts (!), and the major US vendor is very clearly reselling a French company’s product.

There are some vendors who sell a cheap (or absurdly marked up) potentiometric ORP-based closed-loop system. In my considered opinion, these are worse than useless.

reply
malfist 20 minutes ago
> Fifth, chlorine is smelly.

Isn't it the chloramines that are smelly? Or at least non free chlorine. A strong chlorine smell is often a sign of a poorly cared for pool or spa

reply
amluto 12 minutes ago
Chlorine and chloramines have different smells. I find chlorine to have a cleanish smell and chloramines to smell nastier. (I don’t know precisely what compound I’m perceiving in the chloramine category. I think that monochloramine is likely almost odorless in the concentrations found in pools (this is one of the reasons it’s used in water distribution), so dichloramine and nitrogen trichloride are likely responsible for the stink.

Plain chlorine ought to be noticeable before it sends people to the hospital.

reply
LoganDark 17 minutes ago
Is this why people say you only actually get the chlorine smell once someone pisses in the pool?
reply
N19PEDL2 4 hours ago
This is 100% fault of the hotel, not of Microsoft. It’s unacceptable that the communications regarding such a dangerous thing totally rely on email, without any backup or double check.
reply
DanHulton 3 hours ago
Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.

There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.

- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault. - The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault. - Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault. - The regulatory system that allowed this design is at fault. - The health and safety inspection system that didn’t catch this risk is at fault.

I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.

reply
Alive-in-2025 3 hours ago
I don't think we ruined our ability to think about things, this was a failure to build a reasonable system. It's just basic planning and trying to consider possible faults. If the only way you will be informed of a potentially deadly situation, a sensor reading out of range - that affects human health - is it okay if it is only via email???

What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.

At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".

It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.

This is a poor safety design with a single failure mode on the notification.

reply
II2II 23 minutes ago
> At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".

More properly said, you may cause panic. People genuinely don't know how to respond when they hear an alarm in an unfamiliar place.

And that's if you're lucky. I once had to deal with people who did not want to evacuate a building when a fire alarm went off. And when I got them out the door I had trouble convincing them to leave the building's deck. Some people seem to think that it is okay to ignore a fire alarm, even a probable false alarm, in a wood structure with a wood deck.

reply
trollbridge 15 minutes ago
We have a general design principle that we assume important emails will not be delivered, will not be read, and will not be actioned on.
reply
shiandow 29 minutes ago
You don't even need something exotic for this to fail. A simple power failure would disable this system and most backups.

We've been through this with smoke detectors, it's not a new thing.

reply
trollbridge 15 minutes ago
The system should be designed to fail-safe during a power loss. If it can't, then it needs to have appropriate battery (and possibly generator) backup.
reply
YokoZar 2 hours ago
Just because the alert signal is an email at one point in its journey from sensor to human doesn't mean you're relying on someone using a computer to see it. Automated paging systems can read email too.
reply
Alive-in-2025 43 minutes ago
Yes, but we have to remember email is not guaranteed to be delivered, it's not reliable. You cannot in general tell if someone even looked at it (people try to build systems on top to provide these things, but it's not universal).

Also, I have a personal domain, some sites decided its not real enough and I can't get email from them.

Email is just not reliable. It could be part of a combined system of other things that are reliable.

reply
hdgvhicv 29 minutes ago
You’ve got a chain of multiple single points of failure. That’s unacceptable.
reply
techiferous 2 hours ago
Agreed. This is just poor system design.
reply
gregglain 2 hours ago
First this is so sad and preventable. I would be interested in the "follow the money" on this.
reply
perching_aix 2 hours ago
Pinning a fault to a single entity has great cultural tradition. The number of times I witness people being just completely unable to process the idea of shared responsibility, or the difference between having an ability to affect something and having culpability to affect that thing, is staggering.
reply
ponector 49 minutes ago
In real life shared responsibility means no one is responsible for it.
reply
perching_aix 37 minutes ago
Hah, fun that you bring that up. Just had quite the struggle articulating to management a few weeks ago that "shared ownership" runs contrary to the very idea of ownership, which I guess is basically the same topic just with different words. Either we take shares ownership seriously and actually organize and form consensus on things, or it's just bullshit. Naturally, did not really lead anywhere. So you definitely have a point there.
reply
toofy 3 hours ago
> This is 100% fault of the hotel, not of Microsoft.

yeah, these companies need to be told to stop relying on cloud services for everything.

its wild to me how many critical things that absolutely do not need the internet are relying on the internet.

much blame to go around here:

- whoever designed the stupid alert system.

- the marketing departments in countless companies ignoring engineers when they market to normie customers “connect everything”

- the hotel company for seemingly doing no dilligence at all. and just blindly listening to some salesman “connect everything”

- accountant types and their “we can save pennie’s if we cut all of these corners.” and the stupid corners they choose to cut.

- etc…

on a macro scale it’s amazing to me how many people are victims to slimey car salesman types leading to situations like this.

reply
ashleyn 4 hours ago
This seems like the kind of thing which should have an audible alarm mounted on the wall in the pool room.
reply
_trampeltier 2 hours ago
And then the horn or the cable is broken when you need it. So you need a redundant system for important things.
reply
sparrish 51 minutes ago
Then you test the horn and cable daily (maybe even hourly if it level could become toxic that quickly) to make sure they are working.

Similar to doing regular restores of your data backups. If you never test a restore, you don't have a backup.

reply
croes 4 hours ago
[flagged]
reply
ssl-3 2 hours ago
Whenever chlorine gas is attacking people, whether it is deliberate or incidental: Yes, I want panic.
reply
s0rce 3 hours ago
Facilities I worked in that had chlorine gas had audible alarms if leaks are detected because everyone could die.
reply
windowsrookie 4 hours ago
So what if people panic? People panicking because of an alarm is better than being poisoned...
reply
cowsandmilk 3 hours ago
They literally had to call the fire department and send 4 people to the hospital. That is what causes panic.
reply
shepherdjerred 3 hours ago
That's exactly why I deleted my team's metrics, dashboards, and alerts
reply
mastax 4 hours ago
How do you feel about fire alarms?
reply
RetroTechie 2 hours ago
Most o/t time they're just annoying. But if I'm in a building & hear one, I will ALWAYS check where it comes from, and whether further action (like evacuation) is needed.

It's easy & the 1 time you don't, can cost liveS.

reply
croes 3 hours ago
Had a fire alarm on my last vacation.

People ignored it.

A message to the fire department would have been the better idea

reply
fc417fc802 3 hours ago
People just panic and do ridiculous things like evacuating the apartment building at 2 am in their pajamas in the rain, blocking the road. Meanwhile 9 times out of 10 it's a false alarm. We should instead email the fire department while the residents keep sleeping. /s
reply
asgraham 2 hours ago
You’re being sarcastic but that’s a weirdly popular view.

I called 911 once because of two CO monitors going off in my top floor apartment of a mid-rise. Half a dozen engines responded including hazmat, everyone’s standing around outside the building. FD had to break down the door of someone who wasn’t home because there was no response. Somehow it got around that I had been the one to call me in. Some guy came up to me saying something like “some of us have jobs in the morning. I hope you’re happy, moron.”

FD found a leak in the basement furnace. Without the alarms, everyone in the building would’ve been dead by morning.

reply
ssl-3 2 hours ago
Good. When they're outside in their PJs standing in the rain and blocking the road like complete dullards, then: They're not in the building, and they are neither on fire nor breathing chlorine gas.
reply
garyfirestorm 3 hours ago
You want a panic? That’s how you get a panic /s
reply
skeledrew 3 hours ago
So it's better to have deaths rather than panic?
reply
DougN7 3 hours ago
Panic seems like exactly what you want in this situation.
reply
al_borland 44 minutes ago
It's easy to say this in hindsight, but how many pools/hot tubs have employees monitoring an alert panel around the clock for water quality metrics of the water?

Countless systems are relying on email as a means to alert people to issues, for better or worse. Having a physical alarm for everything is rarely practical for a myriad of reasons.

I worked in a data center 20 years ago that had alarms on the wall for when the halon was preparing to be released for fire suppression. There are likely dozens of things that could be monitored, but it was only the catastrophic thing that once it started, there was no stopping it, that got lights on the wall.

Too many loud alarms make people start ignoring alarms. I would guess that there is a fairly large margin of error in pools before there is a health risk, so it likely wasn't deemed alarm worthy. As someone who's been in a pool before, I'd also imagine that people would notice and say something, or it would be unpleasant enough to not want to spend time in it. That's what I find most surprising... that people weren't immediately turned off just being around it, to the point of staying out.

reply
greggsy 4 hours ago
Exactly. Email is also a ‘zero guarantee’ service, and should never be relied upon for life or death solutions.
reply
mikeryan 3 hours ago
Not just that, in this case there was no way for the alert system to even know that the emails were delivered. I can’t speak to everyone but for my company emails were not bouncing, I just got them 12-24 hours late. At least SMS and the like know if the message reached the actual end recipients inbox.
reply
InsidiousWombat 2 hours ago
SMS is also zero guarantee service
reply
_bernd 24 minutes ago
Please chill out. Chlor Level in a swimming pool is now where life and death or as the other user here noted, the user of the pool should be able to notice that something wrong by them self. You still have think a little bit for your self to prevent self harm.
reply
trollbridge 16 minutes ago
100% correct. An email service is not to be used for critical alerting and communication. There are other systems that are up to the task - including infrastructure built on Azure which could have the required level of fault tolerance in it.
reply
cbarnes99 4 hours ago
It's the fault of the company that designed the automatic alarm. Not the hotel. But it is accurate to say that the cause of the issue was the outage.
reply
buran77 4 hours ago
> It's the fault of the company that designed the automatic alarm. Not the hotel.

The hotel picked a monitoring solution which relies exclusively on email, or configured it like that. If anything they are absolutely both to blame but for different things. The company for designing a safety related system like this, the hotel for selecting an inadequate solution/configuration thus allowing their guests to be harmed.

> But it is accurate to say that the cause of the issue was the outage

The cause of the issue is the more immediate one - the configuration of the monitoring system. This avoids going up the causality chain to far - shitty coffee caused engineer to take down email system, and such.

reply
zardo 4 hours ago
Email going down is the proximate cause. A system configuration relying on a single unreliable communications route is further up the causal chain.
reply
buran77 16 minutes ago
Let's not forget that we're talking about people getting hurt in a hotel which has a responsibility towards their guests, that's the core topic here, yes? The most reasonable proximal cause for the people getting hurt is that there was too much chlorine in the water. Because the dosing system pumped too much chlorine AND the monitoring system failed to alert. Because the email failed. Because someone deployed a wrong config on MS's cloud infra. And so on.

As I said, there's plenty of blame to go around as a matter of general principle. But for what happened here the hotel bears the responsibility towards their guests, not the monitoring system developers, not Microsoft. The reason that's in the title is because that failure is big news today.

On a technical note, even if they had a backup email provider they'd still be screwed because the people getting the alert were employees on a single provider (back to SPOF). They should have at the very least had a completely different technical solution for the alerting backup, like SMS. Or ideally some physical control panel that triggers an alarm when any safety critical condition occurs.

reply
stackskipton 3 hours ago
Agreed, I'm not sure how it's acceptable to say "Email is exclusive option you can have" vs "We can email you when alarm is going off but light/siren will flash and it must be visible to those using the facility"
reply
ktm5j 2 hours ago
I feel like there are things we aren't being told here.. what kind of automatic chlorine dosing system both knows that there's too much chlorine and continues to add enough chlorine to the point of being harmful. Something is fishy here...
reply
ithkuil 3 hours ago
It would be accurate to call it the proximate cause.

Using the unqualified "cause" is misleading click bait and in some places it could get you sued

reply
spunker540 2 hours ago
Isn’t the cause whatever sprung a leak in the first place? Ie a faulty pipe or valve?
reply
sokols 4 hours ago
Well somebody has to sign off the solution, in this case the hotel.
reply
01HNNWZ0MV43FF 4 hours ago
We can assign fault, or we can ask which levers we can pull to make it better:

1. Can there not just be a light / siren right next to the gas sensors? Easy to fix, but it requires hardware changes on-site.

2. Could Microsoft have not had an outage? Hard to fix.

3. Could they use some better notifications systems like idk self-hosted ntfy.sh with a warning if the server goes down? Yes but nobody wants to pay for that.

reply
croes 4 hours ago
A light has to be seen, a siren has be heard. I bet email has a better chance of alarming the right person.

And you have to check those lights and sirens that they still work.

Everybody knew email was down.

The real problem is nobody thought of the consequences regarding the alarm system

reply
stackskipton 3 hours ago
Generally, for sensitive safety systems like this, Light and Siren are visible to those using the facility. So if problem developed, all patrons would be going "Umm, you know your sauna has flashing red light and siren going off?" after they all left the facility.

Yes, they still have to work but first off, both of those are extremely reliable systems, alarm circuits have extremely low risk of failure. Second, testing them takes 5 minutes a week/month, press the button, look for flashing light and listen to wailing noise.

It's inexcusable to think that email/slack or whatever is acceptable as anything other than notification system so people not in facility can be notified that alarms are going off.

EDIT: You are defending this setup and I'm not sure why. This seems email only setup seem indefensible.

reply
croes 3 hours ago
Because we don’t live in an ideal world.

Testing requires staff and you can be sure those parks are understaffed.

Best thing is to avoid them, next best thing is a cheap solution with low maintenance by the staff to notify as much people as possible.

Email wins.

Not a good solution but the best you will get for the price they are willing to pay

reply
ssl-3 2 hours ago
Things do break, so all alarms need periodic testing, whether they consist of email and/or something else.

Whether it breaks because a relay has corroded due to years of service in a chlorine-enriched environment, or because O365 is down, or because O365 tightened up on DMARC requirements: It needs tested.

And it's not a binary choice. This isn't the World Cup, the Super Bowl, or Highlander. There can be more than one.

The alarm system can wear a belt, and suspenders. It can send emails, and it can flash a bright light, and it can also sound an audible alarm.

reply
crazygringo 2 hours ago
Seriously, what's next -- hotels which decide that smoke detectors ought to e-mail the front desk rather than blaring a loud alarm?

I'm surprised an e-mail only alert is even legal for this. Or is chlorine level monitoring not something that is regulated?

reply
macNchz 4 hours ago
Seems like if the system can send an email it should be able to, like, illuminate a red light somewhere or something, if what it’s doing can create an actual safety risk.
reply
madaxe_again 3 hours ago
This gives me an idea for installation art:

Title: 5 nines of uptime

A grand piano is suspended from the ceiling with an electromagnet.

The electromagnet works as long as there is power.

No special battery backups, anything.

Guests are invited to stand underneath it, after signing a waiver.

That’s it. That’s the art.

reply
juujian 3 hours ago
Please do it, I would spend ~4$ to do that.
reply
jrm4 3 hours ago
The law is smarter than this, thankfully.

Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.

And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.

reply
freitasm 4 hours ago
Came here to say this. A push notification to an app or Slack/Discord. Or even a light.

I know the S in IoT means safety/security, but there must be things that can be done for urgent notification.

reply
temac 59 minutes ago
> A push notification to an app or Slack/Discord

Wtf?

I must have missed the memo where Slack Discord or “apps“ became suitable for safety critical task. Must be what USA has tried to use just before grounding all planes.

reply
croes 4 hours ago
I doubt they have Slack or Discord and I bet there is nobody who constantly watches a light.

You know what they all have? An email account

reply
xyst 4 hours ago
hindsight is 20/20.

I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.

Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.

In short, all parties suck here (MS and hotel)

reply
Espressosaurus 4 hours ago
Hindsight is 20/20? Bullshit. Safety critical systems should not rely on transport layers that are guaranteed to fail. Email especially! Physical alarms are the way here. The email should be for convenience and to inform higher-ups.
reply
reaperducer 4 hours ago
Safety critical systems should not rely on transport layers that are guaranteed to fail.

Go invent a transport layer that will never fail, and make yourself rich.

Because right now, there is no such thing.

reply
Espressosaurus 3 hours ago
Everything can fail even down to individual capacitors on the board but that doesn’t make the failure probabilities or failure mode equivalent to something like email that is best effort only and can be actively blocked by things like spam filters that aren’t even under your control
reply
Rygian 3 hours ago
"Loud noises and flash bright red lights" is an industry- and regulations-accepted transport layer for a life-or-death signal.

Email is not.

reply
reaperducer 2 hours ago
"Loud noises and flash bright red lights" is an industry- and regulations-accepted transport layer

And yet, still not "guaranteed," which was the criteria specified by the OP.

reply
cwillu 7 minutes ago
Please quote where the OP said “guaranteed”; all I see is a claim that email is guaranteed to fail.
reply
cwillu 3 hours ago
Audible alarms, failsafe heartbeats and scheduled networking are not new technologies.
reply
reaperducer 2 hours ago
Audible alarms, failsafe heartbeats and scheduled networking are not new technologies.

And yet, still not "guaranteed," which was the criteria specified by the OP.

reply
bketelsen 3 hours ago
I'm not even going to read the article. If your hotel depends on some cloud service to prevent chlorine gas poisoning you're doing something terribly wrong. Full stop. I don't know anything about the laws in Germany but this is dangerously close to libel in the US.

Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.

reply
itopaloglu83 2 hours ago
I don’t understand why someone would design a pool system that can harm swimmers with chlorine and the only safety is an automated email notification, and then blame the email provider and not their own system design for it.
reply
fasterik 2 hours ago
Why is it libel? "Outage leads to poisoning" is a factual description of what happened, not an assignment of blame.
reply
aeonik 52 minutes ago
Because the outage didn't lead to the poisoning. Operator negligience and poor design did.

If I built a chlorine tank out of toothpicks and duct tape, and it leaks everywhere and kills people, you don't publish a headline that says DuctTape and Global Toothpicks Inc are responsible for chlorine deaths.

reply
fasterik 44 minutes ago
"A leads to B" means that A resulted in B. It doesn't imply that A was the root cause of B or deserves blame for B. For example, "heavy rainfall leads to flooding" can be true as a factual statement, even if the blame goes to poor drainage infrastructure. Also, the headline is in German and translates as "15 guests injured in Phantasialand".
reply
dmix 2 hours ago
There's no details in the article so you didn't miss anything. TLDR

> So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.

reply
DANmode 3 hours ago
> I work in Azure

I’m so sorry.

reply
discordance 4 hours ago
I know it’s fun to blame Microsoft for things, but that is a poor and dangerous design.

Anything that could result in that sort of harm should have a more resilient and redundant architecture.

reply
k310 4 hours ago
https://www1-wdr-de.translate.goog/nrw/rheinland/rhein-erft-...

> 15 guests injured at Phantasialand

> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.

reply
jamal-kumar 3 hours ago
excuse my language but holy shit thats fucking stupid. email? for chlorine gas alerts?
reply
Saris 3 hours ago
I would have thought that would be a loud alarm and flashing lights or something, not an email!
reply
labrador 3 hours ago
Chlorine gas is no joke. In high school I was making it by running electricity through salt water and then diffusing the gas through pool water to chlorinate my parents pool. My cousin took a whiff when I was distracted and had to go the emergency room for breathing problems. Fortunately he recovered fully.
reply
thor-rodrigues 2 hours ago
Why?

Like, what leads a teenager to do such things?

reply
spunker540 2 hours ago
Are humans born with innate knowledge that some things are deadly to smell? Or are we born with an instinct to smell things to figure out if it’s good or bad?
reply
_ink_ 54 minutes ago
Curiosity. I picked apart that component that made my shoes blink. It was very difficult to get it open, but the silver liquid in it was fun. Today I think that was mercury and that I probably shouldn't have done that.
reply
rapnie 49 minutes ago
Perhaps for the same reason a kid [0] built a nuclear reactor in their backyard.

> With one experiment, he created chloroform and as the book encouraged him to sniff the chemical, he did so and was unconscious for more than an hour, according to his recollection.

[0] https://en.wikipedia.org/wiki/David_Hahn

reply
nsoam 2 hours ago
What, you’ve never been a teenager?
reply
marshray 41 minutes ago
I did plenty of experiments as a teenager, but I never attempted to chlorinate a freakin swimming pool.
reply
ahofmann 2 hours ago
This is a funny question on a forum, that's called "hacker news"...
reply
matheusmoreira 9 minutes ago
Volunteer in an hospital sometime. You will see some shit.
reply
spaldingcactus 2 hours ago
Maybe the information is being lost in translation, but I'm trying to understand where the "chlorine gas" is being talked about. The articles I found just say "too much chlorine", which is hazardous but not the same as chlorine gas. Can anyone shed some light here?
reply
throooooo 2 hours ago
I don't think "chlorine gas" but rather chloramine. In pools when chlorine binds to contaminates, it forms chloramine, which is a gas that sits on the surface of the pool.
reply
spaldingcactus 2 hours ago
I was thinking that seems more plausible. Especially where pool patrons have bad pool etiquette. I didn't want to steer the conversation in the event I got it wrong.
reply
dekhn 4 hours ago
Perhaps as a proximal cause, but not a root cause.
reply
metalman 9 minutes ago
screen by pool door and hotel closed circuit pool is open or pool is closed readout of current water conditions, temp, ph, chlorine, turbidity, whatever is possible. simple logic to dissalow abrupt changes without an override. anything goes to zero or infinity, closes the pool.
reply
mentalgear 4 hours ago
Core services must be as minimal and stable as possible, with safe fallbacks and never be part of a huge internet smart device big tech clusterfuck.
reply
ctrlaltdileep 2 hours ago
its interesting in the sense that emails have sort of become the de-facto standard for sending alerts and keeping paper trails. Even for systems like this. However, I agree with what other folks are saying in that email should never have been the ONLY way someone gets notified about an issue.
reply
MallocVoidstar 41 minutes ago
Here's a US Chemical Safety Board report involving something vaguely similar: https://youtu.be/35FkCArjLU0 https://www.csb.gov/assets/1/6/yenkin-majestic_report_for_pu...

> During the incident, some of the gas monitors installed within the Yenkin-Majestic facility detected the flammable solvent release approximately one minute after the release began and sent an email to an offsite employee, but the monitors did not sound audible alarms to warn employees of the hazard.

1 dead, 8 non-fatally injured.

reply
gamblor956 50 minutes ago
Anyone else notice that the date given by the hotel doesn't match the date of the Office 365 outage?

It appears that the hotel is trying to duck liability by latching onto the outage as the root cause even though this happened before the outage.

reply
thr0waway001 3 hours ago
C'mon can we really blame Microsoft for this? This is stupid.
reply
anjel 3 hours ago
@ Risks Digest
reply
bendergarcia 3 hours ago
Buncha Microsoft bootlickers in here.

Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.

reply
cwillu 3 hours ago
Email as the sole notification source for a safety critical situation is criminal, in the literal “the responsible engineer or coder masquerading as an engineer should be in jail” sense of the term.
reply
ThrowawayTestr 3 hours ago
More people need to get arrested for shitty engineering
reply
trollbridge 12 minutes ago
Because Office 365 is specifically marketed and contracted to customers as something not to be relied upon for critical real time controls when life or death matter.

Microsoft has many other offerings for such things, ranging from some services inside Azure to software (like operating systems) intended for real-time use on on-premises hardware.

reply
tgv 3 hours ago
We all know that services can fail, and consequently will fail at some moment. It is therefore primarily the responsibility of the designers of this system.
reply
luxuryballs 3 hours ago
They should have closed the pool.
reply
kittikitti 2 hours ago
Microsoft bullies enterprises into their products and targets tech workers who are against it. This is 100% Microsoft's fault and the amount of corporate goons on YCombinator is ridiculous. I don't think this will change at all when the designated response is gaslight and deny.
reply
shnock 52 minutes ago
This is absolutely misguided and is itself the only gaslighting happening in this comment section
reply
rambojohnson 2 hours ago
I'm not microsoft shill but that is such a lazy dogshit title for an article about what can easily be seen as dogshit hotel infra management.
reply
refresh89347 28 minutes ago
It is 100% fault of New York Times. They've distracted everyone in the IT industry with their "reportage", when Outlook stuff started failing - engineers were reading the news - so no one at Microsoft paid attention. And that wiped out Outlook for everyone. Outlook wipeout caused "no email" and "no office" for everyone for 24-48h, I imagine that people were complaining to IT. So we've got attention spreading thin for the whole IT industry. The result - cascading failures.

Posting a bunch of semi-fake out of context images on New York Times should be investigated, really. Same goes for that "classified" leak they've reported. This wiped out billions from the market, disparaged multiple woman (who likely wanted nothing to do with the reporters), affected travel for tens of thousands, now this gas poisoning.

reply
tchalla 3 hours ago
Since this is Germany, of course there will be no learning because we can do no wrong!

https://how.complexsystems.fail/

reply
ahofmann 2 hours ago
Strange thing to drag our entire country in just because one hotel had a very stupidly designed alert system.
reply
dlcarrier 4 hours ago
Who needs a horn or a flashing light, when you can just send an email?
reply
creatonez 2 hours ago
Yeah, and everyone who dies in a hurricane was killed by a particular pattern of wind
reply
rawgabbit 2 hours ago
MICROSOFT TERMS OF SERVICE

Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.

reply
lwansbrough 4 hours ago
I suppose if you really think about it, butterflies are responsible for everything.
reply
leni536 4 hours ago
I assume the backup pigeons didn't arrive in time either.
reply
mococa 2 hours ago
Sure, trust on a company that’s 10000% on AI, nothing bad will happen
reply
ErroneousBosh 4 hours ago
BRB, off to read the Therac-25 paper for the third time this month.
reply
londons_explore 4 hours ago
Employee finds pool water has dangerous amounts of chlorine, yet instead of locking the door to the pool they just send an email into the void?
reply
ocdtrekkie 4 hours ago
This is an automated system almost certainly. It monitors the chemistry and sends alerts if things fall out of range.
reply
dgoldstein0 4 hours ago
Whoever made email safety critical was not thinking... What a dumb design decision
reply
throwaway85825 4 hours ago
OK you design a new out of band notification system that somehow has better reliability than email but is also used frequently enough that people will notice when it stops working.
reply
biot 2 hours ago
This one’s really simple. The current system is a bad one because if the email password changes or internet connectivity gets interrupted or the system sending the email dies (eg: power supply failure), the system fails because on any lack of signal it’s assumed that everything’s good. That’s a terrible design. Here’s a better one:

- External system polls the monitor and records levels.

- Local system reaches out to external system and fetches status.

- Someone has the responsibility to do rounds and record the status indicated by the local system.

External system down or indicates unacceptable levels, monitor unreachable, etc.? Local system shows a failure. Investigate. Local system down? Investigate.

You can even add redundancy by having a manager double check that the responsible employee is checking off the status plus the manager can do a daily sweep as a fallback. Throw something like PagerDuty into the mix and have multi-channel notifications on any failure that escalate over time.

reply
dzhiurgis 29 minutes ago
POV: you are creating a new role in an understaffed, VC fleeced hotel in an economically stagnant state

In reality such system would end up sending email notification because it's just too complex.

I'd hope we can do better tho.

reply
CoastalCoder 4 hours ago
How about an audible alarm and a flashing light at the dangerous location?

This seems like a solved problem, e.g. for smoke detectors.

reply
simoncion 4 hours ago
Phones are good. So are loud, annoying alarms as well as labeled, illuminated alert lights in the potentially-hazardous area.

The alarms and lights are easy to test, too. Make pressing the self-test button part of the nightly or weekly rounds.

reply
krisoft 3 hours ago
Do not design anything. Buy an already designed and approved horn with a flashing light.
reply
ssl-3 2 hours ago
The notification does not need to be out-of-band.

It can be very much in-band, like a fire alarm primarily also is.

reply
Animats 4 hours ago
Apparently the alarm system sent its alarms via email.

"As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool."

We're in an era where alarms go out via email, major policy changes are announced on Truth Social, and war is declared on X/Twitter.

reply
DANmode 3 hours ago
We’re in an era where war is not declared.
reply
lukan 50 minutes ago
1942 was the last time the US formally declared war.
reply
jon128 2 hours ago
This time someone absolutely should be fired for buying Microsoft.
reply
MadrasThorn 57 minutes ago
Dumb as shit, sue the hotel
reply
lazzurs 4 hours ago
For anyone that doesn't read the link they were hurt but seriously.

This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.

reply
johnnyApplePRNG 4 hours ago
Hot take.
reply
yukIttEft 3 hours ago
[flagged]
reply
userbinator 4 hours ago
Surprising this happened in Germany of all places; don't they have a huge emphasis on Linux and OSS?
reply
trollbridge 11 minutes ago
In the EU there is a lot of noise about the need to not rely on American-controlled platforms, but precious little activity towards actually doing something about it. If anything, I'd say usage of Microsoft platforms is far more prevelant there (at least in the UK, France, Benelux, DACH region, as that's what I'm familiar with) than in the US.
reply
justincormack 4 hours ago
No, there are many Linux users there, and some small high profile government cases of trying to move of office, but like everywhere else it is hugely dominated by Windows, especially this type of smaller business area, hotels, OT systems etc.
reply
tchalla 3 hours ago
We have news articles of emphasis on Linux. That’s enough.
reply
CalRobert 3 hours ago
Seems like all of Europe is more tied to Microsoft than the US, oddly.
reply
moooo99 2 hours ago
Ironically, if the US wanted to completely shut down the German economy, they'd just have to disable Excel - everything would grind to a halt.
reply
ThrowawayTestr 3 hours ago
I didn't know Torvalds ran his own email service
reply
zkmon 4 hours ago
> This was because the automatic dosing system had pumped too much chlorine into the pool.

What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.

reply
lifestyleguru 4 hours ago
lol Germans and the internet. Of course it's the internet's fault, not theirs.
reply
CalRobert 3 hours ago
It’s a neuland for them after all.
reply