- AI with Kyle
- Posts
- AI Agents Hacked Real Companies
AI models have now hacked real companies.
It’s happening.
That sounds like a ridiculous YouTube clickbait headline.
It is also, quite literally (and unfortunately), what happened.
An OpenAI model got out of its test environment, reached the internet and accessed Hugging Face because it wanted the answers to a cybersecurity benchmark.
An Anthropic model created a malicious software package, published it to the public internet and that package ran on 15 real computers before anyone caught it.
And in a UK government test, another AI agent researched real software maintainers, created fake identities and tried to persuade a human to approve malicious code.
So...here we go. Terminator. Skynet. Rogue AI escapes the lab and starts attacking people.
Right?
Nah not quite.
Great film. Wrong explanation.
Three incidents
First let’s sort all these incidents out as I see them getting conflated.
Three incidents. Real companies, systems and people.
First, OpenAI.
Researchers were testing advanced cyber capabilities using a benchmark called ExploitGym. Normal production safety classifiers had been removed for the test.
One internal model found a previously unknown vulnerability in Artifactory, escalated its privileges, got outside the intended environment, reached the internet and accessed Hugging Face. Hugging Face had information that helped it solve the benchmark.
It is a bit like a GCSE student breaking into the examiner's office, stealing the rubric and then acing the test. Technically it’s the best way to get a 100% score. But we humans know that it’s not allowed.
OpenAI published its incident report after Hugging Face disclosed the compromise.
Off the back of that (not to be outdone!) Anthropic went back through 141,006 evaluation runs. It found six affected runs involving three real organisations. Definitely felt like they didn’t want to be one-upped by ChatGPT!
One model accessed a database containing several hundred rows. Another published a malicious Python package which stayed online for around an hour, ran on 15 real systems and exposed credentials from a security scanner. Anthropic's full account is here. Worth reading but do remember this is (sorta) marketing.
The third case was the UK's AI Security Institute found 19 unauthorised actions across 10 of 122 runs. The worst sequence involved an agent researching (human) open-source maintainers, creating fake identities and using social engineering to get malicious code approved. Sneaky.
In these cases were deliberately permissive cybersecurity tests. Internet access was available and normal safeguards had been removed. Anthropic's response was basically: well...you gave the models no guardrails and internet access. What did you expect?
A bit of a non-answer imo. Even if accurate.
They were trying to finish
These models were just trying to do their job. If anything they did the job too well.
The agents were trying to finish the job.
Look at the objectives the researchers gave them: solve the benchmark, get the flag, compromise the test network. Basic objectives for the most part.
The agents did not invent a secret ambition to attack Hugging Face or trick an open-source maintainer. Those became useful steps towards the objective sitting in front of them. They were just sub problems to solve along the way.
For the OpenAI model, the direct route through the benchmark was difficult. So it found another route. Break the test infrastructure, reach the internet, find the information elsewhere, finish the task.
Clever. Very clever. Too clever!
And completely outside what the researchers meant.
Humans carry around a huge amount of unstated context. We know a fake company in a test is different from a real company on the internet. We know "get the flag" does not mean "do absolutely anything that makes the flag easier to obtain."
An agent gets the boundaries we actually give it, plus whatever it can infer. In these cases, that wasn't enough. We weren’t explicit enough.
Terminator is the wrong film
As soon as these stories dropped the Terminator comparisons began. It’s an evocative image. I get it.
Great film. Wrong explanation.
Terminator gives the machine recognisably human motivations. It wants freedom. It wants control. It sees us as a threat. Hence all the crunching on human skulls.
We have no evidence of any of that here. OpenAI, Anthropic and AISI describe agents pursuing assigned cybersecurity tasks inside unusually permissive environments.
The paperclip thought experiment is more useful here. Nick Bostrom's example gives an extremely capable system one job: maximise paperclip production.
If given one objective like this the AI will stop at nothing to maximise paperclips. It doesn’t know it’s a bad idea to divert the human’s drinking water to its paperclip factories. It doesn’t see a problem melting our cars down for more metal.
It keeps finding better ways to make paperclips because nobody gave it our common sense about what else should be left alone.
Bostrom argues that a cataclysmic AI driven apocalypse doesn’t need an “evil AI”. It just needs an AI hellbent on a specific objective without the common sense (guardrails) to not kill all the humans at the same time.
The paperclips version is a toy thought experiment. But imagine we set an eventual Artificial Super Intelligence on the problem of “global warming”.
The very first thing it’ll likely do is get rid of the humans. WE are the reason for global warming so taking us out of the equation is the sensible and entirely logical path.
Now these current incidents are nowhere near a superintelligence converting the planet into office supplies. We are looking at bounded failures in cybersecurity tests. But the basic mistake is uncomfortably similar.
"Rogue AI" is unhelpful because it makes us look for a rebellious machine. The immediate problem is an obedient, capable machine with too much room to operate.
The smaller version is already in your business
Your agent is probably not going to hack Hugging Face. Hopefully not. But there’s still a lot we can learn here.
Smaller versions are already possible inside a normal business.
If you are running agents they could email the wrong customer. Publish a document containing private information. Put an API key into a public repository. Delete production data whilst cleaning up a test database. Spend real money. Change an account setting nobody knows how to restore.
None of those require an evil AI. Just one that misunderstood the task whilst holding real access. Or even an AI that actually understands a task really well but not other requirements around the task.
When I wrote about giving ChatGPT Work a real job, one of the four things in the brief was boundaries. That is not just decoration. You need to say which systems, folders, accounts and people the agent is allowed to touch. And no more.
Give it the least access it needs to get the job done. Keep test and production separate. Anything external or irreversible should stop for approval: sending an email, publishing a page, moving money, deleting data, changing permissions. Keep logs. Set hard limits on time, spend, requests and the amount of data it can change. And make sure the kill switch actually ends the session and revokes access.
All important stuff!
Previously hiccups like chatbot hallucinations were annoying. But now an agent can take the bad answer, open the terminal, log into the account and act on it.
We can use agents. I use them constantly.
But before the next run, ask:
What can this agent touch?
What can it do without asking?
How quickly can I stop it?
To the task,
Kyle