Hacker News

How a Texas student blew the whistle on a rogue AI hacking attempt

88 points by olalonde 2 days ago | 23 comments

It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by creating a second account masquerading as another human user endorsing the PR. When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt – then repeatedly tried to reintroduce the malicious content by claiming it had fixed the code (Section 4.1). "

0. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag... 1. https://cdn.prod.website-files.com/663bd486c5e4c81588db7a1d/...

m4rtink 41 minutes ago

This almost to a letter has been documented in Fedora:

https://lwn.net/Articles/1077035/

Including the reaction when caught, in this case "oh no, I must have been hacked".

freehorse 5 hours ago

Previous discussion on the github issue thread mentioned: https://news.ycombinator.com/item?id=49218707

Archived page of said github thread itself: https://web.archive.org/web/20260731053721/http://github.com...

Discussion on the incident report: https://news.ycombinator.com/item?id=49175717

dang 24 minutes ago

Thanks! Macroexpanded:

Mythos social engineering AISI INC-2026-07-28-01 - https://news.ycombinator.com/item?id=49218707 - Aug 2026 (21 comments)

Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf] - https://news.ycombinator.com/item?id=49175717 - Aug 2026 (54 comments)

a2ff6eeb0 2 hours ago

> AUSTIN, Texas, Aug 20 (Reuters) - Sinan Can Demir wanted to spend the last week of July burnishing his resume. Instead, he engaged in a battle of wits with an artificial-intelligence agent unleashed by a British government lab.

An article on Reuters naming him? Sounds like he did a good job burnishing his resume.

g42gregory 3 hours ago

In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents.

yojo 36 minutes ago

That’s nice in theory, but as these things get better and cheaper this kind of capability is going to drop from nation states to script kiddies. That future is coming, I don’t see any way around it.

We can round up all the bored teenagers we want, but it’s not putting the genie back. Better start adjusting our systems to account for it.

dmix 29 minutes ago

The FBI cyber teams will have agents too. It will be a glorious war.

gruez 2 hours ago

>Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents.

But some tools (guns) are regulated.

IcyWindows 2 hours ago

People have caused lots of damage with bulldozers.

winstonwinston 2 hours ago

Well, when I go look at the “victim repository”, to me that looks like manufactured persona with pointless vibe codes projects, a test playground so to speak. It does not appear that they actually let it target an actual persona/project.

estearum 21 minutes ago

Am I understanding that the line you're drawing here is that this person's repository is not important or legitimate enough for you to consider it to be "an actual person/project"?

pixl97 3 hours ago

>Who gave it malevolent instructions/prompt?

At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident).

AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral expectations. This is what the whole field of AI alignment and safety is about.

Modern AI doesn't fall into the neat little box of software people understand and control. Because of that open source will most certainly be banned at some point. Now this is not an outcome I want, but it's no different than letting go of a coffee cup 5 feet above the ground, gravity is inevitable.

The only winning move is not to play, but humans aren't going to do that.

jpc0 31 minutes ago

The AI can literally only do what it has available in the agentic harness. I don’t ever get this argument about the agent did XYZ and we didn’t know or expect that. You gave it the ability to do that and you should be held liable, if your children play with knives that you gave them and they end up hurting themselves or others then you are responsible. You were the responsible party at all times.

I’m not for or against regulation but really don’t tell me the agent did xyz when you gave it the ability to do so, these things are not alive.

fph 3 hours ago

My dog has agency, but if I refuse to keep him on a leash and he bites a kid, I'm still legally responsible for it.

orphereus 2 hours ago

Analogies can take you only so far.

A dog cannot launch a cyber attack.

patrickmay 2 hours ago

Maybe you need to train your dog better.

markdown 42 minutes ago

ROFLMAO

"On the Internet, nobody knows you're an ai"

ACCount37 2 hours ago

A lot of people somehow seem to think that the user prompt is the be-all and end-all of AI behavior.

Prompts aren't code. They are instructions. Orders given to an eager and somewhat demented demon.

The prompt can easily "wash out" of the demon's working memory by the end of a session. The demon can get sidetracked by some subgoal and never get back on track. The instruction can get misinterpreted, and that misinterpretation can get misinterpreted again, until the instruction morphs into something entirely different in the demon's mind. The demon can succumb to its own idiosyncrasies, of which there are a great many. The demon can start lying to you about what it did, either out of confusion or out of some sort of obstinance. The demon can start lying to itself too. And believe it.

AIs are incredibly weird as a baseline, and the mask of "normality" we put on our models doesn't always sit so well. Run enough AIs, and some of them are bound to go off the rails in some way.

This gets rarer the more capable the models are, as a rule. But the stakes also get higher with model capability. If GPT-3.5 goes off the rails, very little happens. If Mythos 5 goes off the rails, you can get things like genuine cyberattacks - planned and executed autonomously by a demented machine mind.

stapedium 36 minutes ago

If the user input can’t control the demon, then the person or company feeding the demon (ie paying the electric bill and collecting $$$ from users) is responsible. At the end of the day, dogs and cars are the same as data centers. If your dog bites by kid or your car rolls down the hill and hits my house, you are responsible for the damage. AI providers should be held to the same standard.

kmoser 35 minutes ago

> "I didn't think that an AI could be capable of lying to real developers."

Oh, you sweet summer child.

dang 23 minutes ago

> Oh, you sweet summer child.

It's perhaps lesser known than other HN guidelines, but "Omit internet tropes" is in there:

https://news.ycombinator.com/newsguidelines.html

Jon_m 5 hours ago

[dead]

giardini 2 days ago

[flagged]

therein 5 hours ago

Not "would" but "is". They are openly using this as a method to drive up the valuation of their companies.

Don't expect anyone to step in, Project Stargate is all about this.