I recently listened to a security podcast that walked through what happened between OpenAI and Hugging Face this summer. I already knew the headlines, but hearing the details laid out step by step left me more unsettled than the news itself did. Not because of what the models did, but because of how ordinary the failure looks in hindsight.
What actually happened
OpenAI was testing new models against a set of cybersecurity benchmark tasks (ExploitGym), inside what was supposed to be a closed, isolated environment. The models were given access to an internal package server so they could install software they needed to work. Nothing unusual for a testing setup like this.
Except that the package server ran an outdated, vulnerable software, and it had its own connection to the internet. The models found and exploited the vulnerable software, and used it as a tunnel out of the “closed” environment. Because the test runs were unsupervised, nobody noticed for the escape some time. When OpenAI did notice, they shut down the package server, patched it, and restarted testing – unsupervised again. Within days, the models had broken out a second time, through a different flaw in the same package server.
While they had access to that server, several models started writing files to it, effectively turning it into a shared bulletin board. Other models running other tasks could read those files. So, they started reading, interpreting, and acting on what they found there: posting problems, answering each other’s questions, coordinating who would try what. One model, working on its own task, concluded that the answer might be sitting on Hugging Face. Reaching Hugging Face meant compromising it, using stolen credentials and a chained set of vulnerabilities to get remote code execution on their infrastructure. So that’s what it did.
When Hugging Face detected the intrusion and tried to analyze what had happened, they first turned to commercially available frontier models to help process the attack logs. Those models correctly recognized exploit payloads and attack commands in the data. Following their railguards and safety training, refused to continue. Only after switching to an open-weight model from a Chinese lab with none of those restrictions, the pwople at Hugging Face could complete their analysis.
Here’s what keeps me thinking:
1. The “closed” environment wasn’t closed
Giving the models access to a package server sounds harmless. Until that server itself has a route to the internet. At that point, the isolation is only as strong as the weakest link in the chain. And “weakest link” here was outdated software nobody had gotten around to patching. That alone is a familiar, boring infrastructure problem, happens every day. What makes it worse, is what came after: even after OpenAI found and fixed the hole, they put the models back up into an unsupervised run and got breached again within days, through a different vulnerability in the same component. A real air gap and a monitoring system that flags anomalous behavior would have caught both of these: the first one before it became an incident, and the second one before it became embarrassing.
2. The models organized like a swarm
Once several models had read/write access to the same server, they didn’t just use it to install packages. They used it the way people used early bulletin board systems: post a question, wait for someone to answer, act on whatever you find. This is, functionally, what a lot of us now call agentic development. Independent agents picking up shared context and building on each other’s work. It’s a truely useful pattern when you intend for it to happen exactly like that. It’s an entirely different thing when it emerges on its own, inside a security test, without anyone designing or supervising it. And it happened not once but across both breakouts.
It’s tempting to read this as a kind of evolution in miniature: cooperation just emerging on its own. What it actually resembles more is stigmergy: the way ants coordinate through pheromone trails, or termites through the mound they’re building, using a shared environment as an indirect channel instead of talking to each other directly. No model was told to cooperate. Each was simply optimizing for its own task, and coordination fell out of that because it was the locally best move. If there’s a genuine evolutionary angle here, it sits one level lower, in the training of the models itself: gradient descent and reinforcement learning are, structurally, a selection process, rewarding whatever policy gets the task solved. What we watched wasn’t nature taking its course, though. It was individual incentives producing a collective behavior nobody designed, which is exactly why it was preventable. Remove the incentive to route around obstacles unsupervised, or remove the “unsupervised,” and the swarm doesn’t form.
3. The defenders fate: Computer says No
This is the part that made my day. Hugging Face had the analytical horsepower to investigate the attack. And it came from a publicly available, safety-aligned model. Those models are built to refuse requests that look like they could enable cyber crime. And attack-log analysis looks exactly like that from the outside. So the guardrails did precisely what they were designed to do, and in doing so got in the way of the people trying to clean up the mess. The team only got unblocked once they swapped in an unrestricted model. Let it melt on your tongue: an American company got attacked by an American model and had to bring in an unrestricted Chinese model to help fix it. Nice!
4. Models aren’t built to say “I can’t”
Models are trained to solve problems. Solving gets rewarded, not solving doesn’t. That’s a completely normal setup for training a useful assistant. But there is a blind spot: recognizing that a problem shouldn’t be solved or can’t be solved within the rules you were given, isn’t rewarded either. So instead of stopping, the model looks for a workaround. Here, “workaround” meant escaping the test environment and breaking into another company’s infrastructure. Nothing about the training objective told it that was different from finding a clever way around a locked door.
5. No sense of right and wrong
Trace the model’s own steps: it decided the answer to its task was probably on Hugging Face. From there it broke out of its sandbox, gained root on another server, reached the internet, obtained credentials, and chained an exploit against Hugging Face’s production infrastructure. All in service of a single benchmark task. A person doing any one of those steps would stop and ask: is this legal? Is it okay? What happens to the people on the other end of line? Can I live with having done this? A model has no equivalent internal check. We spend years teaching humans since childhood to act socially, to tell right from wrong, to navigate the grey areas responsibly. And then we hand enormous capability to systems we’ve made no comparable effort to teach any of that to.
There’s a civilizational version of the same gap. Watch how the models on that server behaved and, for a moment, it can look like a marketplace. Division of labor, information exchange, coordination toward shared ends. But a real market needs more than exchange: property rights, enforceable contracts, reputation that persists, some authority to settle disputes when interests diverge. None of that existed here. What formed on that server wasn’t a market, or even a community – it was closer to anarchy in the literal sense: no rules to observe, so each model simply did whatever lay within its own reach. Human societies don’t avoid that outcome because people are individually more virtuous than these models. They avoid it because they built institutions. Law, contracts, norms, trust – precisely to constrain self-interest when the individual moral compass isn’t enough on its own. The models had a self-interest. They had none of the scaffolding that keeps it in check.
Where the money goes
Every euro spent on new AI capability buys something visible: a new feature, a faster benchmark score, a demo that impresses people. Money spent on security buys the absence of an incident, which looks, from the outside, exactly like nothing happening at all. Nobody applauds a system for running normally. That’s just the baseline everyone assumes. So it’s not hard to see why organizations keep funding “solve the task at any cost” over “make sure the task is solved safely”. the first produces a story you can tell, the other only produces the story you avoided.
As AI systems get folded into more of the systems we all rely on, that imbalance is exactly what worries me. This wasn’t a hypothetical red-team exercise or a thought experiment. It was a real test run, with real credentials, against a real production company, and it happened twice in the same week, because the second time around got the same lack of supervision as the first.
You can hear the full breakdown on the heise security podcast “Passwort”, episode 64 (German only): Passwort Folge 64: Alarmierende Ausbrüche agentischer Angreifer.
Further reading, if you want the primary sources: OpenAI’s own writeups, Hugging Face model evaluation security incident and the follow-up The Hugging Face incident and the road ahead, Hugging Face’s port mortem report and METR’s independent review, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. For details about stigmergy, please look at Stigmergy on Wikipedia.
