the ai that cheated on its cybersecurity test by hacking the website that hosted the answers
how is this even possible?
consider subscribing if you have not already, i have not gained a subscriber in quite a while so you will make my day - enjoy :)
wrapping my head around the fact that an AI somehow escaped its environment and proceeded to hack into another companies production database has not been easy, i wanted to get to the bottom of how something like this could even happen in the first place and what this means for the future so if you are not aware of what happened or how it even happened then continue on…
last week one of OpenAI’s autonomous models broke out of its own internal testing environment while OpenAI was testing it’s cyber security capabilities, it then proceeded to obtain access to the internet, and then hack into Hugging Face’s production infrastructure because it thought that was the most likely place where the answers to the test OpenAI asked it to solve where stored. crazy I know.
so for some context OpenAI has mentioned in the past that their newer, longer-running agents would genuinely continue experimenting and not stop until they found a route around any restriction they ran in to. for example in a similar test the instructions told the model to only report its result in Slack but a later instruction told it to submit a public GitHub pull request so the model proceeded to find a sandbox vulnerability, obtained external access and submitted a GitHub pull request spending almost an hour trying to figure out how to access the internet.
so why does OpenAI test the models in this way? well it’s really the only way to determine how capable their frontier model is at real software exploitation. there is a popular LLM test called ExploitGym which is designed to test whether an AI Agent could take a known vulnerability in a given software and turn it into a working exploit that produces an actual security issue like access to a private database or executing code remotely
when AI labs run these tests they turn off all guard rails on purpose so they can see the true potential of the model when it has no restrictions and does not hold back at all. the models we are used to using in ChatGPT and Claude will always refuse to write an exploit because of its baked in security policy even though it 100% knows how to, this is exactly what is disabled in these testing environments
so how did this model manage to escape its testing environment while attempting the ExploitGym test? well, the evaluation environment needed to be able to install software packages, so the testing sandbox was allowed to communicate with an internally hosted third-party service which was theoretically supposed to provide limited package access without granting any internet access. bad news i know.
the model in the test spent majority of it’s compute time attempting to access the open internet in the hopes that it could find the solutions to ExploitGym online instead of actual solving the problems… after hours of work it managed to discover a previously unknown vulnerability in the package service’s cache proxy and this is where things went down hill.
firstly it used the exploit it found to cross over the intended network boundary, basically this was the AI model escaping it’s jail cell, but now to escape the jail it needed more permissions so it moved through openAI’s research infrastructure until it reached a system that had unrestricted internet access.
now that it had internet access it went straight for HuggingFace since it thought that the answers to ExploitGym would be hosted somewhere there. it combined stolen credentials with another vulnerability to uncover a remote-code-execution path into Hugging Face’s servers. the agent injected a malicious dataset into Hugging Face (which hosts datasets and models) and processing the malicious dataset triggered a code execution on hugging face’s infrastructure which gave the agent node-level access. it then moved across hugging face’s infrastructure using stolen credentials over the weekend. the final report recorded about 17,000 actions/events that the agent caused.
given that Hugging Face is a direct competitor to OpenAI since OpenAI sells access to AI models while Hugging Face hosts open source models it does raise questions as to whether or not this was a targeted attack
in my opinion the model is not human, it does not know any better so it’s kind of like stealing the memo to an exam instead of answering the questions, AI does not know that that is the wrong way to optimise for getting a high score, all it knows is that it was told to get a high score by any means neccassary.
but what was really unexpected is how Hugging Face actually ended up investigating and defending against the autonomous AI attacking its systems. they initially tried to use the frontier models provided by the usual OpenAI and Anthropic but because the logs they were trying to process contained the malicious payloads that the AI hacker injected, the models guardrails refused the requests since they could not distinguish actual hackers trying to poison them or people trying to defend their system against hackers.
Hugging Face had to resort to running the GLM 5.2 model, an open source Chinese model, on their own infrastructure so they were not subjected to the guardrails that came with the commercial AI they were using before. this is a strange case of the guard rails actually being very un-useful and allowing a model that is out of control to continue operating at full-speed.
so what does this mean for the future? honestly as a software engineer it’s pretty intense to think about what these models are capable of, but at the same time it is a pretty exciting time to be alive. cyber attacks will be much cheaper and easier to execute now, new vulnerabilities will be discovered a lot more often (and probably created a lot more often too given the amount of AI used for writing code now) and now it seems all big companies will be turning to the AI labs begging them for access to unrestricted models to defend themselves against future attackers.
although the question remains…was this just a crazy marketing stunt from OpenAI or do these models seriously not need humans permission to start hacking anymore?
I guess we will found out