US halts flights at busy East Coast airports, says fiber line cut
97 points by allanbreyes 2 hours ago | 57 comments

cube00 2 hours ago
> When it went to flip into the backup, we discovered that the backup fiber had a break

Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.

I wonder how long it was down? Days, weeks, months?

reply
newhotelowner 21 minutes ago
All these cable/internet service companies are incompetent as it gets.

I am paying, $1000, $1800 & $1900 for the same service at 3 different location (20 mins from each other).

Two locations, I also have old coax lines that are still active, but not paying for it.

When I bought two businesses, I learned that they were paying for a dedicated fiber but using coax service.

At one of the location, we had fiber, and paying for backup coax and wireless. But if you turn off fiber box, it wont fail over to either one.

I wouldn't surprise it was down for weeks and no one bothered about it.

reply
toast0 2 hours ago
Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.
reply
unethical_ban 22 minutes ago
That is negligence, practical if not contractual.

Over ten years ago, my employer was spinning up a new DC across the state and had three links between it and the primary DC. Two were pretty direct, but we needed a third because at one point in the 300 mile path, the two main links went within 400 meters of each other.

So how national-security-adjacent critical systems like an airport system doesn't have a larger set of backup links, and validates that they are geographically separate up until the connections, and have realtime alerting on the connection status, is surprising to me.

reply
Jtsummers 16 minutes ago
> validates that they are geographically separate up until the connections

This isn't relevant in this case. The problems occurred at different places. The backup fiber was, separately from the issues with the primary system, cut by a construction crew. It's not two fiber lines both cut at the same spot.

reply
koolba 58 minutes ago
Put all your backups in one basket, and then pray that nobody crushes the basket.
reply
oasisbob 43 minutes ago
This is a pretty classic network operator story: go to great lengths to provide for physically diverse paths, then not notice when your provider refactors and grooms them onto the same bundle.
reply
konfusinomicon 39 minutes ago
last time this happened in my area a local farmer was burying a cow and took out the whole towns connection
reply
krashidov 54 minutes ago
Could this be sabotage?
reply
kqgnkqgn 47 minutes ago
That seems more than merely plausible given current tensions and the location/timing connection with NYC and UN.
reply
someothherguyy 12 minutes ago
trump's escalator revenge?
reply
pvaldes 24 minutes ago
I would put my money on it.

Similar events happened in Europe disguised as thieves stealing fiber optic. This does not have any sense economically, as the value in the market is zero so... either is an honest accident and is cleared in a few days, or is sabotage

reply
rcostin2k2 3 minutes ago
This used to happen more often in the past, when copper thieves mistook fiber for conductor. There were cases where critical systems were shut down due to fiber optic theft, and the thieves were caught burning the sheathing to expose the... glass.
reply
12876qj 15 minutes ago
Yes please. Russia did it. Please move the patriotic war to the Bering Strait (Strait of America?) and leave Europe alone.
reply
zoomablemind 46 minutes ago
That's very uncharacteristic, especially given all those contingency requirements (backup policies, failovers, testing schedules etc.) imposed on corporations/companies in the wake of 9/11.
reply
pixl97 40 minutes ago
You actually have to ensure that someone isn't just checking boxes the tests passed and the tests actually passed.

I worked for a regional ISP that had a major outage when the redundant fiber provided by the telephone company was cut in one place triggering a full loss of connectivity. It also caused a massive 911 outage for 200,000 people as it isolated the 911 center from the city core.

Turns out the phone company didn't connect one side of the ring topology even though they certified they did. Needless to say lawsuits abounded.

reply
readthenotes1 2 hours ago
A lot of people don't check their backups until they need to restore.

A lot of people are incompetent.

reply
andrewjf 33 minutes ago
Nobody wants backups as a feature. The feature is restore.
reply
thewebguyd 2 hours ago
A backup without a restore test isn't a backup at all
reply
bonoboTP 36 minutes ago
You don't always have a copy of your hardware to restore onto. And the test's entire purpose is that you're not yet sure whether your restore will truly work. So you can't just run a backup and restore on your true prod system, because you're not sure it won't wreck it. So you need extra money to have a second system onto which you try to restore. If you don't have a lot of money, you will want to actually use your disks for storage, not to put them into a second testing server. Of course I'm not talking about very professional companies with super critical data. Just simpler smaller scale places or consumers.
reply
ranger_danger 20 minutes ago
Besides what the others have said, speaking from a neteng perspective, sometimes backup lines (and the core infrastructure in general) are engineered in such a way that it can't easily be tested properly without taking other things down, or manually rolling a truck specifically to test it in isolation with extra equipment.

Not saying that's what is going on here, just that it's possible.

reply
kqgnkqgn 40 minutes ago
Building multiple diverse paths and monitoring for fiber cuts is not hard. Having 2 fiber paths is insufficient for even only moderately important workloads at a tech company. Overlapping fiber cuts happen. For something with significant economic and safety impact this is just crazy.

Sometimes the level of incompetence / lack of care in organizations like this astounds me. I understand issues like this can be complicated and systemic but it honestly makes me think very poorly of the technologists building these systems in government.

reply
woodruffw 31 minutes ago
They had two diverse paths, but apparently did not have a (reliable?) process for ensuring the backup path was functional during fail-over. I imagine there will be some soul-searching over that.

Nothing I've ever seen or experienced with FAA indicates a lack of care; the parsimonious explanation is almost always that people who care a great deal are working within complex systems that don't always have a consistent or externally legible set of priorities. Or more intuitively, the failures we see are the "acceptable" ones versus the unacceptable ones (like planes falling out of the sky).

reply
markdown 12 minutes ago
> Nothing I've ever seen or experienced with FAA indicates a lack of care

Hard to fathom someone still holding that view after the 737MAX disasters.

The NTSB has repeatedly and publicly criticized the FAA for failing to implement their recommendations, including as recently as last week (Amazon Prime crash resulting in 5 deaths). They've explicitly called out the FAA's failure to act as a contributing factor in crashes, multiple times.

Regulatory capture ruined the FAA.

reply
OutOfHere 24 minutes ago
The problem is with considering the second path a backup path. It is a common problem with things that are considered to be a backup. In contrast, both paths should ideally have been in regular use. As a general policy, for resiliency, one should always exercise all of one's routes/suppliers/vendors, even if some are suboptimal, not just the primary one.
reply
woodruffw 17 minutes ago
I completely agree. My comment is not to suggest there wasn't a problem or failure (clearly there was one!), but that we shouldn't assume incompetence versus conflicting priorities under limited resources.
reply
dgellow 20 minutes ago
I’m pretty sure that type of project is contracted to the private sector and isn’t caused by government technologists in any way…
reply
kqgnkqgn 13 minutes ago
That sounds pretty plausible. But you still have to have enough internal competency to oversee a contractor, and ask the right questions/build proper requirements and verify they are meeting them.
reply
sc68cal 2 hours ago
There is also a new ATC system being rolled out, as early as today

https://www.airwaysmag.com/new-post/faa-smart-first-deployme...

reply
whatever1 2 hours ago
I suspect it is vibe coded. Contract won in June. Deploying in September.

Good luck to all of us.

reply
gkedzierski 39 minutes ago
I can guarantee you that it was not. Check out DO-278A, and its standards and guidelines that must be followed.
reply
tencentshill 29 minutes ago
...Or what? The US government is in move-fast-and-break-things mode, and is running on a skeleton crew besides.
reply
macintux 48 minutes ago
Deploying first to DC? Yeah, that’ll end well.
reply
atonse 2 hours ago
I don't quite understand this sort of thing happening. Wasn't the whole point of the internet to be a self-healing network where we route around severed cables, etc?

Or is it that these ATC networks are their own air-gapped network with less redundancy? That just doesn't add up. Or maybe there was only one line going to the ATC, with no multiple "ISPs" like a datacenter would have?

reply
bri3d 2 hours ago
* Yes, TRACON have their own dedicated links. You can look into ASTERIX and STARS to learn some of the cursed ways the data processing and dataflow work.

* There is supposed to be a primary and a secondary link, in this case the primary failed and the fail-over also failed. It's unclear from the reporting if they were damaged in the same incident or if the failover was not tested or monitored adequately.

The Philadelphia TRACON site has been notoriously unreliable and was supposedly improved in 2025, it's also unclear if these issue actually could stem from that implementation.

reply
wrs 59 minutes ago
I’ve had enough double-redundant links fail - and I mean pretty thoroughly double-redundant, different provider, different media, different physical path - that I’m quite surprised something like the air traffic control system only has two links.
reply
Polizeiposaune 23 minutes ago
Contractually dual-path, or actually dual-path? My understanding is that there's enough infrastructure horse-trading going on behind the scenes that it's very difficulty to be certain that two circuits between points A and B don't share the same infrastructure somewhere in between.

First big oops of this form that I remember:

"In December 1986, the ARPANET had 7 dedicated trunk lines between NY and Boston, except that they all went through the same conduit -- which was accidentally cut by a backhoe. "

https://www.csl.sri.com/~neumann/insiderisks06.html

reply
pixl97 34 minutes ago
One thing that needs to be reviewed is if they are dual links or redundant links.

In any system where both sides of redundancy always carry traffic then loss of one link can cause congestion failure if any link fails. This is a very common means of failure in electrical networks that requires load shedding. Well, you can't load shed air traffic.

A more complex system that I like, but comes with it's own set of constraints and implementation issues is a system where both lines carry all the traffic at all times. This way the default state of the system is always working and your first failure isn't invisibly critical.

But this is very hard as we see in TCP when things get out of order and high latency creeps in. You have to manage a lot more state at the data level.

reply
Hikikomori 20 minutes ago
It is, but you need to set it up correctly.
reply
readthenotes1 59 minutes ago
Do you want air traffic control to be on the general internet?

That seems like an extremely foolhardy thing to do.

reply
ssl-3 52 minutes ago
As a tertiary backup: Yeah, maybe I do want that.

It seems like it would present a less-chaotic solution than that provided by having no data communications at all.

reply
kqgnkqgn 46 minutes ago
You certainly don’t need to run this over public Internet to get resiliency over multiple paths.
reply
baggachipz 39 minutes ago
Maybe Sean Duffy was just binge-watching his old reality tv shows and gobbled up the bandwidth?
reply
dctoedt 52 minutes ago
Not to be conspiratorial, but about an hour ago I saw this in The Economist: "Russia’s grey-zone attacks on Europe are growing more brazen"

[QUOTE:]

Russia has been staging covert operations and deniable provocations against European countries for many years. Since its full-scale invasion of Ukraine began in 2022 they have increased. These have included incursions by reconnaissance drones, hacking, arson, parcel bombs targeting cargo flights and plots to kill executives of leading European arms firms. In recent months the pace of such shadowy attacks appears to have risen again. The Russian operations have three main goals, according to analysts and security officials: trying to coerce European countries into halting their support for Ukraine; raising the costs of providing that support; and attempting to make NATO look powerless.

[/QUOTE]

And of course there've been "suspicious" cable cuts in the Baltic Sea and other European waters ....

https://www.economist.com/europe/2026/09/21/russias-grey-zon...

reply
MadrasThorn 9 minutes ago
US used to be capable of deterring aggressive action.

Remove Elon Thiel Trump Vance ASAP

reply
quickthrowman 2 hours ago
It has been conjectured that the easiest way to locate a buried fiber optic cable is to start digging.

(The actual easiest method is to install the conduit or direct burial cable with tracer wire)

reply
bluGill 40 minutes ago
Fiber was all installed long after we knew locating it was important. I suspect it all had a tracer wire installed with it. Of course sometimes the tracer wire can break.

Before fiber a lot of things were put into the ground with no thought how you would find it again.

reply
quickthrowman 23 minutes ago
Indeed, tracer wire is usually bonded to a (sacrificial) magnesium anode, but it’s a crapshoot whether the tracer wire is still intact.

I sometimes hire directional boring and excavation contractors and if there’s any doubt as to where an electrical conduit or natural gas pipe is, I opt for the hydrovac truck to minimize risk.

reply
wat10000 15 minutes ago
Always bring a length of fiber optic cable when you're in the wilderness. If you get lost or stranded, bury the cable. Within a few hours, a backhoe will stop by to dig it up, and the crew will rescue you.
reply
dboreham 2 hours ago
Local people know where they are because once upon a time a D8 drove through pulling a big orange spool of fiber conduit and passers by chatted about how another fiber was going in. Also around here the contractors seeded the disturbed ground with a wildflower mix (presumably to discourage weeds) so you can also look for straps of wildflowers.

And of course there are marker posts that say "buried fiber optic cable do not dig".

reply
pixl97 27 minutes ago
Once you get out farther from civilization these signs become less common and there are more situations where they can get removed.

Most cheap hired labor doesn't realize you shouldn't dig up the market posts and throw them in the dumpster on construction sites. Add a little bit of rain and suddenly the contractor showing up with an excavator that was told everything was properly marked, and can see markings in other places but not where they are digging, and disaster occurs.

Also soils aren't static, I've been on sites where the midpoint of a buried telecommunications cable had drifted over 8 feet from where the markers were a few hundred yards apart. Just looking at the markers and assuming a straight line isn't a safe bet. Looking at the fence rows to the left and right of the cable and you could see a matching bow in the fence rows.

reply
throwaway173738 20 minutes ago
Just call 811 and let the utility figure it out.
reply
chresko 27 minutes ago
Are we sure this wasn't Claude? Has Dario made an announcement yet?
reply
tomasphan 2 hours ago
ATC seems like a great place to have redundancy with a starlink dish on the roof.
reply
Hikikomori 12 minutes ago
Put two for redundancy
reply
LastTrain 20 minutes ago
Ah yes let’s make our entire infrastructure depend on a company controlled by a single person.
reply
cyanydeez 46 minutes ago
So of course if the FAA is going to put AI into the watchtowers, they're obviously going to use local AI and not rely on some random Amazon cloud infra in some middle eastern country.

right. of course, no ones going to cut corners on the safe use of AI and secure infrastructure in this administration.

reply
ofcrpls 2 minutes ago
[flagged]
reply
dboreham 2 hours ago
[flagged]
reply