And we're hiring! https://openai.com/careers/search/?q=cyber
I understand this work was based on a dated snapshot. Will there be any continuous effort, scanning new PRs as they land, to prevent new weaknesses from being added?
E.g., 948 results is a high number to review in a timely manner unless your team is very large or spending several months combing through the results
EDIT: nvm i shouldve kept reading, answered
> At this scale, manually investigating every report is impractical for our research team. We therefore focused on validating findings that a) were reachable by a low-privileged user, b) required no special capabilities, and c) produced measurable evidence of memory corruption.
Multiply that behavior times the number of commercial and mobile and b2b apps over 50+ years, and that's where we are.
There will likely be many more leaks, and hacks, and outages. And then in trying to be proactive to fix old systems, that will cause new proactive outages, and then eventually many things will be better.
But, it's going to take a lot of time and investment, and a lot of (majority) of currently unsupported devices will have to be complete replaced / scrapped / ewaste. On the scale of Nation-State initiatives.
This is across all operating systems, firmware, software and devices of all kinds, etc.
LLMs to security are like a bomb went off. We still have to wait for things to fall over before the dust will even clear enough to see what we have to work with.
The article says "The target virtual machines ran a stock desktop operating system". Are the vulns in the kernel specifically or does this include software running as root?
A system that admits THOUSANDS of such vulnerabilities is certainly flawed in several fundamental aspects
So you have to chunk it down to workable pieces over time, and then if you collect enough data of that process you can look at higher level stats.
You're also talking about hundreds of thousands (millions?) of individual contributors to the Linux kernel over 30+ yr. That process is likely to take years.
Same for every other existing piece of software, btw.
What are the mitigating factors that prevent this level of vulnerability from being an existential threat to humanity?
Hopefully this type of work to automatically discover & patch vulnerability leads to much more reliable software, and the remove the need for cyber security.
[1] https://www.cyberdefensemagazine.com/zero-day-market-the-gov...
ACLs, Appcontainers with capabilities, etc have been something in the OS for more than a decade.
*buries head in sand*
*cries*
Imagine if open-source organizations other than the Linux Foundation (and Mozilla I guess) could throw $1.2M of API-rate LLM usage at security.
Is the kernel simply too complex to reason about? Is it an underlying flaw in the security model or language implementation?
Should we assume that all software is more compromised than the kernel?
Speed requires a lot of code in the kernel, where that code can run fast.
There are kernels that do not have a thousand privilege escalation exploits. These kernels are smaller and much easier to check for bugs. But those kernels are significantly slower.
For many use cases, slower is not a problem. But for the large corporations who fund a lot of Linux development and use Linux in large data centers, every 1% slower can literally be millions of dollars worth of additional servers that are required.
OpenBSD shows that you can have a Unix security model without a thousand privilege escalation exploits. But OpenBSD is like 10% slower. So corporations won't use it because it would cost millions of dollars more per year and some of their software requires access that OpenBSD does not grant precisely because that access leads to security exploits (i.e. io_uring, eBPF etc. etc.)
You could also use like seL4, but that's slow too
So that's why. It's a business tradeoff between cost of servers vs cost of getting hacked, and cost of servers wins.
Absolutely, the front door has always been unlocked. We were previously protected by the economics of exploit detection favoring the developers, but now we have entered a new world.
The issues arise almost everywhere we look. It's frustrating because the user experience in edge systems has barely changed in the last fifteen years. We could have rode the advances in semiconductor manufacturing to embed more layers of protection such as CHERI while slightly slowing down the rate of performance improvements. Instead we have only marginally upgraded security and largely added features which the users scarcely notice.
I am hopeful about the long term prospects of formal verification, but having humans reviewing the proof statements and confirming they match our desired properties will take many years. Meanwhile, AI assisted hacking is an urgent issue. So we will have to delegate implementation, proof specification, and proof development to the AIs in the short term. Maybe that goes well, but who knows. Interesting times.
It died in the sense that the "people in charge" hired us for our expertise and then ignored our every suggestion and fought us tooth and nail every inch of the way when we suggested that security or reliability ever mattered in the slightest until after the disaster happened, and then straight-up scapegoated us for "not pushing back harder" against the threats of "go work somewhere else then, code-monkey!" until we gave up and did the "code-monkey" thing for our paycheck and no longer cared about quality or "a job well done".
There is nothing magical about kernel. Look at the code anywhere and it has bugs, lots of them.
Security bugs are just bugs. More often than not they are silently there and don't prevent you from using the software and you can't react and fix them because of that.
Give it time and they accumulate
it's been rule of thumb basically forever that there is 1 bug per ~10KLOC for any project
linux kernel is millions of lines, so thousands of bugs being hidden there is about on point
(tho obviously far smaller serious escalation ones, obviously, but you're exaggerating as well)
1 bug per 10 lines -- not kilolines -- is much closer to the actual "bug rate".
You may be thinking "1 serious security vulnerability bug", but in my experience that too is much closer to 1 per 100 lines to 1 per 1,000 lines, depending on the type of application.
Is it bulletproof? Definitely! But you need the genius of Albert Einstein as well as the patience of Mother Theresa to actually use it in production.
It's the one security practice that actuuuualy works
But it also got its share of use-after-free and OOB access vulnerabilities this year: https://feedly.com/cve/vendors/openbsd
First of all... Do you really believe that to be true?
Oh, you strongly recommend rewriting tens of millions of lines of code? Why don't you do it yourself? Just a prompt away, right?
Humans manually typing out the bytes of source code has been an outdated method, since around a year. It simply doesn't scale as much as those same humans directing agentic teams. So yes, it would be about prompts.
The hard part isn't the technology for the language transition, that one is present today. pgrust shows the way. It's ensuring maintainers are cooperative, i.e. a social problem. Eventually you diverge too much if you do a fork, so it's better done in cooperation with upstream.
Just check how long it took LLVM or GCC to switch off svn for example, and a language switch is much more involved than a VCS switch.
You mean the pgrust which was recently caught for cheating benchmarks and incapable of even performing basic queries outside the benchmark set? Would love to see pgrust debunked by Jepsen like this other AI-generated, written in Rust, "formally verified" scam.
https://jepsen.io/analyses/particledb-1.1.4
https://news.ycombinator.com/item?id=50209450
We would also have to burn a trillion tokens a day verifying all the C code the hard way forever going forward. It would be cheaper to rewrite the kernel in Rust, subsystem and driver by driver at a time.
Now, I don't know how many of those bugs were privilege escalations, or could be leveraged to be. I'm not sure that people tried to exploit in the same way back then (or, if they did, if they said anything publicly about the results). Still, even if the majority of those bugs were not exploitable, OSes were much smaller then.
So that's my conclusion of how we got here: We've always been here.
We developed AIs that can find them.
> > We developed AIs that can find them.
We could already find them before AI.
We already knew they were there, and we already knew how to prevent them in the first place.
See previous discussion: https://news.ycombinator.com/item?id=49705794
Not at this speed and cost.
I'd argue that the resources (i.e. time, money and effort) needed to create the tooling that would have prevented all of these issues would have been orders of magnitude less than the resources that have gone into "AI".