Maybe AI scanning found these Exchange exploits and after laying off so many developers, they vibe coded an exploit fix and deployed it. But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
Just speculation though
Anyone who has seen connection leaks/pool exhaustion in practice, it looks like an otherwise healthy low utilization server right up until it stops replying, and a restart appears to fix it until it runs into the limit again. Easy to think that your health checks are the problem. Per MS updates, they rabbit holed on this issue and built 'an alternative health check method' which worked in testing (i.e. low number of test connections); but when they tried to apply this change, servers would go unresponsive anyway.
My guess is a rushed update with a connection leak somewhere in the stack. A call that opens, but does not close. Just the kind of thing that AI can miss because it's only trying to hit the goal and not necessarily clean up after itself.
I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?
Exchange and Active Directory are easily my favorite MSFT products. I've always wished they would port them to Linux, even though I know that OS dependencies make that a non-starter (though they surprised me w/ SQL Server for Linux).
I'll concede that the the biggest Exchange environment I ever ran was around 1,000 mailboxes. I've never seen how it fails in really large environments. I was also fortunate that I only ran the earlier versions (pre-2013) in sub-500 mailbox environments. In those settings, though, I found it worked well and wasn't bad to administer.
It was very reasonable to use in small business environments, too (sub-100 mailboxes). I put a lot of little Exchange 2003 boxes out in small offices. It plugged-away very effectively.
Yes-- there were updates here-and-there that had issues, for sure. By-and-large it was a stable and predictable product.
I always tried to be mindful of maintaining free disk space, following MSFT's recommendations for speeds-and-feeds for I/O, and running on solid hardware (ECC memory, redundant power, UPS). I feel like I would have had more problems if I'd scrimped on that stuff, for sure, but I was unwilling to let Customers cheap-out.
With Exchange 2019 and high-availability (database availability groups) I was easily able to do zero-downtime maintenance. That "big for me" Exchange Customer I worked with had more downtime in the last 24 hours than they did the entire time they used Exchange 2019 in production.
I haven't seen the code, so I can't speak to the "nightmare of code". Large parts of it were ported to the .NET Framework in between the 2010 and 2013 releases. The 2013, 2016, 2019, and current Subscription Edition releases appear to be an evolution of the codebase with no signs of major rewrites. I question how much pre-2013 code is even present in the codebase anymore.
I dunno. My experience just hasn't been that bad. I don't want to be an apologist for MSFT. I've never fully gotten over the suspicion I have for their motives going all the way back to the Halloween documents. I don't like how they're forcing everyone to hosted offerings and killing their on-prem products. I've been pleased with the value my Customers realized from Exchange.
How is Samba 4's AD 'emulation' these days?
yup. You're the only one.
When we ran our own mail servers, "some fuckery with client's Exchange server config" was good 90% of mail problems we had.
And my experience with the cloud version management decided we will migrate to has been entirely awful too, simple stuff like "stop spamming the mailing list I get constantly" is entirely impossible to set up, despise both global and per mailbox rules to let it thru, it still lands in spam and the useless support we apparently pay for can't figure out why
As an Exchange admin (and Postfix, Sendmail, qmail, exim) that was my experience, too. I think the popularity of the product, questionable defaults, and the low barrier to entry for admins (read: lacking in experience and knowledge) contributed to that.
I guess past reliability might be irrelevent here if an AI slop patch for AI discovered exploits caused the outage. Just speculation though. Since exchange has so many moving parts and so much legacy code, with even more bolted on to make it a cloud service, I could see how an AI exploit fix would not have all the context needed.
There is no information that it was caused by Exchange either, guess we have to wait on those answers. It is just a reasonable assumption.
Sysadmins who were used to running file servers and thought Exchange was the same thing were the problem. A DBA was actually more apt to understand the product's storage needs because it wasn't wholly dissimilar from the RDBMS they supported.
Any bets on whether it's DNS?
[1] https://x.com/Microsoft/status/2121282013600907325
"The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.
Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration."
All the CEO's AI and All the CEO's Men / Couldn't Put Microsoft Together Again
And, if eventually a big bunch of CTOs starts considering going back so self-hosting/managing people will pop up everywhere saying "even if <big corp> sucks, you cannot do better" or "it will cost you more" or "what about the peak usage".
It's incredible how we ended up paying much more, giving voluntarily all our data, and getting a shitty service.
“Aren’t you glad it’s them and not us?”
The lesson most people learn from this sort of outage is that things bounce back afterwards, and the risk is minimal enough that a day out of every few years isn't enough of a problem to pay more.