Doesn't seem that complicated from the article though. They say that they immediately knew it was clock based due to the strict interval. With REA etc. it might become easier to proof, but I guess it's a measurement problem that only appears in large networks?
Can someone with more understanding explain why this was apparently so difficult to find out?
Oh I’m sure there is a German phrase for this.
Similar to “allzuschön um wahr zu sein”, “allzugroß” etc.
* * * * * sleep 17 && …
That way I still maintain a per-job 60-second cadence consistently (which random would not), but don’t do it right on the top of the minute.
Obviously cron jobs are not all equal in terms of load and duration, and hours are not all equal in terms of usage, but that still helps prevent trivial spikes similar to the one mentioned in the article (disclaimer: I'm not aware of a recent counterfactual experiment proving this, but that seems a fair assumption).
The funny thing is, modern electrical codes already account for this phenomenon. In order to implement Net Metering (selling your power to the grid), modern standards implement two kinds of jitter:
- Sandia Frequency Shift: a minute shift in the power frequency generated by inverters. If the grid is connected, this is effectively eliminated; if the inverter is disconnected from the grid, it detects the frequency has shifted and disconnects the inverter from the grid.
- Slip-Mode Frequency Shift: basically the same thing, but phase shift (which I don't understand).
Both of these are used for anti-islanding (don't create an island of power). When the inverter detects it's isolated and stops outputting current, it then has to check every 5 minutes, with jitter, for whether the grid is restored or not. It has to add random jitter, because otherwise all the mini battery/solar installs will come online at the same time and swamp the grid, causing another outage.
The turbines/generators set the grid frequency itself.
Having the batteries disconnect/connect would cause the generators to slow down and then speed up (or vice versa) which makes the rest of the grid frequency fluctuate as well.
It may also trigger the FCAS part of the grid that is paid to keep the frequency within the required range which is +/- 10mHz-200mHz.
There was an post a couple of days ago about a telco debrief of an interesting outage that happened. Several things went wrong like the audible bell was manually silence from being heard as a step in routine working but was forgotten to put back into normal mode once work was complete. The actual loss in power triggered the automatic switch to the batteries, but something else seemingly benign happened at the same time so that when power was restored, the batteries had been depleted to the point the charging process could not start because of something similar to this. I wish I could remember the write up, as it was really well written about the cascades of failures that occurred.
Why though? Due to increased torque?
Broadly speaking, it’s a collective momentum problem. If part of the system inertia sags, another part needs to step in to compensate for the inertia loss, otherwise you have brownouts and eventually, a blackout.
Seems like they did not find the exact manufacturer or the exact devices and were not able to change firmware then?
Given the issue is so widespread (international, affecting pretty much every grid), it's likely that multiple manufacturers use similar approaches.