Glad someone else appreciates circuit switched technologies though :) . It really does shine in any application that isn't burst-y data transfers.

so you were forced to release updates on weekends? isn't that worse?

Most likely, they HAD to do it.

By people with unreliable systems and a lot of myhts behind.

In a well engineered organization, you can deploy whenever you want. Sure, it will not work 100% of the time, but you figure out why, fix the problem and you will be one step closer to the mythical 100%. And so on. Deploying less frequently is what creates more problems.

Isn't that exactly what they did? If you do major change on Friday at 3pm you likely will work over weekend.

In what sense? It was well reported at the time, so presumably most people know about it. I'd be interested in what makes you think the time and day of week chosen would delay market or public reaction be _weeks_.

But it does not make it 100% resilient to cascading failures or possible Layer2/3 loops that might happen during a software upgrade, especially in a country wide deployment.

The FAA will give a root cause at some point of why this happenned.

No, they mean "had a sexual relationship with a subordinate".

Which is what the linked article says.

>Sorry, but I am tired of the effort to make it look like Jansen's crime was infidelity.

Strawman. Nobody says that, and nobody cares.

Jensen's "crimes" that we know of are:

1. Having a sexual relationship with a person suspected of being a Chinese spy (who, allegedly, influenced NVIDIA's chip sales to China)

2. Having an undisclosed sexual relationship with a direct report

Both are huge liabilities for the company, as well as national security concerns (someone in a critical position putting themselves into a position for easy blackmail).

None are crimes, all are factors contributing to the firing decision.

>If he just went to whore house he would be fine (well, ignoring a possible expensive divorce).

Same goes for the ousted L3Harris CEO.

"it's computer science, if you make a mistake nobody will die!

Unless you're dealing with nuclear plants, medicals, or airlines"

flights departing hawaii and alaska are also allowed

Economy taking a lot of crazy hits this week. They're footgun hits too, not even enemy action.

Edit: ok so one of them was maybe a little bit enemy action.

This thinking caused an extended outage for the biggest mobile network in Australia: https://www.telstra.com.au/exchange/some-mobile-calls-and-da...

Those pair of boxes nobody was brave enough to touch cost them AUD 30 million in fines.

Many forms of "touching" are tests that cannot be accomplished any other way, so untouched is untested, and if something can't be sanely turned back on after it gets turned off, that's just a latent, undetected fault.

If and when things break you'll end up with an even bigger disaster if you don't have people prepared to maintain the old systems. A well managed decommissioning and upgrade is much more preferable.

The parts to maintain the legacy copper infrastructure are no longer manufactured and they are cannibalizing used parts for spares.

Not to mention the expertise for these systems have mostly either retired or in some cases passed away.

My grandparents still have a landline and the number of outages in the last 12 months has been insane. Some were in excess of a week.

Many such cases.

Sometimes the telcos have been shrugging their shoulders and refusing to perform repairs. This is common after a car accident takes out a pole, cabinet, etc. They accelerate the retirement to right now and force the customers onto new technologies.

   Sure, but the purpose of the tech policy office is to
   argue why old systems (which have fixed maintenance costs)
   should be replaced with new systems which goose the
   economy when the federal government pays 10x for what they
   should be paying.

Both things can be true: US infrastructure is in a dire place AND this monumental acute unforced fuck up is the fault of the unqualified, corrupt people who were put in charge of it

That's a pretty low bar.

-- some clever person whose name I do not know

The house where I lived on 9/11 is in the approach for a runway at a local airport. Air traffic was a fixture at that house. Somebody was landing at all hours most days.

It was the strangest thing to have no traffic at all back in 2001. It's odd for the absence of a thing to be jarring but it was.

Edit: I also never noticed balloons before. N804XR is kinda interesting, for example.

Oh-- there are two. N804XR and N807XR. Tethered Customs and Border Patrol aerostats. I'd never heard of these things before.

You'd say "Valid from the thirteenth at two zero five three zulu" over radio.

Unrolling these decisions is a monumental task given the decades of hardware already deployed in the field that will only be upgraded when the current unit fails.

If things blow up because you're upgrading to something supposedly better and it goes wrong, then that's not a time when you can properly fix the new thing. You can only push forward with the transition or go back to the old. I disagree with "you've got nearly a blank check to do whatever's necessary to make sure it never happens again." You have a blank check, but anything you use it for has to be scoped within stopping the bleeding. As soon as you're onto the new or back to the old, the check expires and you likely can't touch anything real anymore because it'll be involved in lawsuits and finger-pointing.

I don't disagree with the core message, but I think it plays out differently for a major upgrade vs a production meltdown on an existing system. With mostly an existing system, you can pour effort into resilience. With an upgrade, I doubt people would let you beef up a system that is still on the chopping block, even if it's getting a life extension. And if you're still switching to the new system, all effort is going to go into getting it up and running and anything that gets in the way of that (as quality improvements are likely to) is going to face some serious resistance. The check is not truly blank.

You can spend years working on a problem but if you have a catastrophic failure in your personal life, you can use that to change dramatically with far less internal resistance.

I was working on upgrading US ATC in 1995. As were lots and lots of other people.

A lot of very hardworking people getting too little pay for something so critical while their employers (some contracting company) take a lot of profit. Eventually they burn out waiting for help and a replacement system and retire or quit, then the systems really start failing because the knowledge leaves with them.

Godspeed for everyone getting to where they need to be, somehow, or at least ending tonight with a roof over your heads.

Most airliners over the ocean also have inmarsat satphones for the pilots.

https://x.com/_whitneywebb/status/2119846209871282200

https://x.com/_whitneywebb/status/2119846209871282200

After 1962 there was too much air travel to continue to shut it down for exercises, and NORAD switched to smaller, regional level things that did not shut down air travel after that. But the results of Sky Shield III in 1962 were the only experience that the FAA had to work from when they did the full ground stop after September 11th. Sky Shield III had tested how quickly air traffic control could clear the sky in the event of a nuclear war- the USAF had put up several hundred T-33 trainers, flying them randomly around the US and Canada, and then at a certain point in the exercise the ATC had to get them all on the ground far away from major metropolitan centers (presumed lost in the war).

“Mr. Mullin said that new technology upgrades overloaded the existing, antiquated system, “overwhelming” the existing equipment.

“It’s going to continue to fail,” Mr. Duffy added. He also outlined many of the additional investments he believes the air travel system needs, like new air traffic control towers, infrastructure and communication networks.”

Of course they are all going to insist it has nothing to do with SMART, that would be incredibly damming, and threatening to ASI. But it’s pretty unfortunate timing no? What other “technology upgrades” have been recently applied to air traffic control/ communications?

The destructive & arrogant behaviors won't be reigned in properly if we all fail to recognize that, just FTR.

What could go wrong really...

yeah... sounds about right.

https://www.nbcdfw.com/news/tech/a-calamity-disrupting-hundr...

Attention: National Radar Outage

https://www.weather.gov/afc/radaroutage

Flight Service uses a different set of radios and can relay messages to ATC, but they've been slowly dismantled over the last two decades and wouldn't have the capacity to step in for an outage (nor should they be expected to - that's not their purpose).

Some aircraft have CPDLC for text-based communication, but it's asynchronous and you can't use that as a replacement for voice communication in the terminal environment. Also, many aircraft aren't equipped to receive it.

Pilots are trained on lost comms procedures, so if you're in the air and the radio fails they'd fall back to those regulations. Basically all the aircraft would fly their last assigned clearance, and attempt to deconflict manually with other aircraft over air-to-air radio (similar to flying into an untowered airport).

So it falls back to local control, more in the style of visual flight rules. Pilots in existing flights will have to be vigilant and do their own deconflictation and new flights are banned out until the network is back up.

All conjecture at this stage of course, but all the radios won't just fail. It is probably going to be a poster child example about the dangers of large scale integration. The attraction of course is that integration is very efficient, the danger is it tends to fail all at once instead of in smaller pieces.

I would hope that airports keep a battery operated radio + antenna in case of emergency with them, to at least communicate with aircraft within visual range of the airport.

Under the hypothesis that some under-reviewed AI generated code made it into prod, the situation could be qualified as "engineers are panicking, because they don't know if they are the cause or not"

I don't doubt that the system needs upgrades, but these people should not be involved.

What am I missing?

It's possible this is the standard "privatize a government function and skim the profits" move but some of this stinks of union busting and trying to reduce the amount of influence air traffic controllers have whenever the organized criminals that run our government want to have a shutdown but keep their private jets flying.

Suffice it to say - there's no reason to trust any of these people when it comes to taking care of critical government functions.

The secretary also mentioned that the new system should be brought altogether, and not piece by piece. Which would lend credence to an incompatibility between both systems.

(https://news.ycombinator.com/item?id=50492957)

But my completely unfounded and more conspiratorial theory, based on zero knowledge of air traffic control systems, is that to better accommodate SMART, L3Harris was strong-armed into to rolling out a hastily written patch on a Friday, which accidentally took down the system.

Transportation secretary Duffy says that the smart system is unrelated. That the smart system is a dashboard for displaying all the data from across the air traffic system. My questions is obviously where is the smart system getting its data from...

I suspect FAA pushed for the new AI system, didnt take into account the increased traffic demand to feed the system. Aging infra isnt sufficient to handle the increased demand. Now Duffy is trying to deflect blame unto someone else.

Should be an easy answer. We saw increased traffic from X and we've implemented Y to ensure it doesn't happen again.

Instead it was vague hand waving around old hardware and funding.

Even if they don't have all the details organized, what was done to fix the problem? How was the 12b spent thus far and what where does the additional 50b number come from?

Or maybe don't start with demanding money. What happened? How was it fixed? What is being done to ensure it doesn't happen again? Then mention this shit needs fixed and we'll be working with Congress for additional funding.

It was interesting that Mullen mention new technology contributing at one point. Sounds like someone made a mistake somewhere and this press conference was purposefully vague or they really don't understand what happened yet (doubtful).

Plenty of critical systems operate reliably on old tech as long as you work within the limitations.

This press conference had maybe 30s of actual info and the rest was this antiquated system is fucked even though it's been working for years and the only way to move forward is $50b. :shrug:

It is possible that this overwhelming was caused by a software or configuration error, no doubt.

I think that if that's the case, rather than failing upwards and going for a full simultaneous upgrade with a 50B congress bill, the brakes should be stepped on a little bit, continue the partial migration as long as there are no issues. Or at least wait until the start of a 4year administration to go for a rehaul, so that you can stay around if things go fubar.

Maybe things are moving fast today and they didn't have time to understand and communicate accurately, but at some point soon I think we deserve a more in depth explanation.

https://www.transportation.gov/sites/dot.gov/files/2026-09/M...

The Senate will be in session for four more days next week before the Thanksgiving holiday, and then back for the first three weeks of December before the 119th Congress adjourns for good on Dec 18th. So basically, after two full years with control of both branches of Congress and the WH, this bill was introduced on the very last week of the regular Congress calendar. After which the lame duck Congress would only meet for about four weeks until adjournment.

Not really stalled, more like procrastinated until days before the election (when everyone is scared to vote on anything), hoping that the lame duck Congress (notoriously unproductive period generally) would force it through in a rush?

Doesn't seem like a great strategy if they actually wanted it to pass. Especially since a budget CR is needed before Dec 11 which will dominate members' attention in the remaining time (plus the permitting bill BAAJA).

But also, this is about when the flight plans expire out of the system if they're not activated for departure - so it's saying unused flight plans won't be deleted for 4 hours, so pilots won't have to refile for departure.

  EFFECTIVE TIME:  132054 - 140430
  SIGNATURE:  26/11/13 20:54