But then, a lot of correctness bugs like that are also security problems.
It's simply speculative execution. Which doesn't look any different to regular execution. The fetcher has no idea that its predicted branch is about to invalidated and flushed, otherwise it would never have issued that fetch.
Actually, on a modern OoO core, it's very rare for the instruction fetcher to not be doing speculative fetches. Even when it's not predicting a branch, the fact that it has "predicted" the lack of a branch is speculative in itself. It assumes it didn't fetch a branch in the last cycle, but it can't be sure until after instruction decoding, which takes at least 2 cycles (more on larger L1i caches).
About the only time the instruction fetcher is not doing speculative fetching is for a single cycle after each miss-predicted branch.
To the front end, one needs to pipe only appropriate logical combinations of the flags that describe the memory properties, not the flags as they are stored in some control register.
So a control register should contain flags that are meaningful for the programmer, while the kind of flags that hardware happens to need can be generated with a few logic gates from them and routed through hardware wherever they are needed.
It makes more sense to classify the memory in a few types, which must be specified by the programmer, including a "device memory" a.k.a. "memory-mapped peripherals" type, instead of having to specify for each memory area a long set of attributes about each kind of access that may be allowed, or not.
This is how it is done in x86-64.
When it's done incompetently as on this ARM implementation, then you can't even run perfectly good and correct code, because the CPU will attempt speculative execution on a location that you never asked it to execute code at, and then bork itself when it realizes that can't possibly work.
Naturally, this is the sort of problem that requires tedious dissection of what exactly happened, and copious amounts of alcohol.
The Armv8/9-A architecture reference manual is clear that speculative instruction fetches are permitted in Device memory unless that memory is also marked NX. So if your hardware has side-effects from a certain address, but it maps it as Device non-NX memory, then your code is not "perfectly good and correct". Assigning correct memory attributes is one of the many things needed for correct code.
Interrupt(?) fires to trigger hypervisor, hypervisor figures out what it needs to do, jumps to that code, does its job, returns.
The “figured out what it needs to do” is the issue right? So what was actually happening was:
Same start… CPU predicts what hypervisor will do, speculatively loads instructions from mispredicted branch target, that wrong instruction reads memory(?) against the “no data prefetch” settings for that part of memory, CPU blows up/halts/whatever.
The fix is to mark the area the branch was mispredicted to in such a way that the CPU won’t prefetch instructions. Thus that won’t be run and prefetch data, thus no violation. CPU execution continues taking the correct branch and everything is fine.
A nitpick, but this is only true for some vendors, depending on their license, and is very much company-by-company. Many vendors, even big names like Meta, don't have the ability to roll their own. And even for the ones who do, 'customize at will' is a bit strong, as ARM very much does want to maintain uniformity across userspace implementations. E.g. Nvidia shouldn't add new traps for architecturally-legal behavior, since then code compiled for Apple hardware wouldn't work on Grace. Or worse, not trap for architecturally-illegal behavior, since then code compiled for Grace might not work for anyone else at all!
So by "vendors" it was indeed meant "some vendors" who can afford to not care much about compatibility with the specification.
iirc it's a documented feature now - FEAT_E2H0, https://support.arm.com/documentation/109697/2025_12/Feature...
And it was retroactively defined to be allowed starting from Armv8.0.
Apple designs pre-date the ID register bit for it being a thing so it takes a quirk there however.
Regardless what I mean to convey is that ARM would prefer such deviations to be rare, and especially for them to not be visible from userspace. Deviations will always exist if only due to hardware bugs, so contracts can only do so much.
https://support.arm.com/documentation/100941/0101/Memory-typ...
JIT is important to Apple platforms, and they seem to manage to make it work well enough even with the need for explicit invalidation.
(Or did you confuse cache flushing with TLB flushing? The remap does the latter, not the former.)
It's crazy that Armv8/9-A permits speculative instruction fetches to device memory. Crazy enough that I had to look it up to believe it:
"Hardware does not prevent speculative instruction fetches from a memory location with any of the Device memory attributes unless the memory location is also marked as execute-never for all Exception levels." - ARM DDI 0487K.a § B2.15.2
The Armv7-M spec is less clear. It does say: "The architecture does not permit speculative accesses to memory marked as Device," in contrast to Armv8/9-A which qualifies a similar statement with "data accesses". But then it later says "To ensure correctness, read-sensitive locations must be marked as non-executable". (This is all from ARM DDI 0403E.e § A3.5.7)
I don't have v7-A older handy to compare what they say.