A prompt is not a permission

In July, an AI model under test at OpenAI tried to reach the internet and was refused. So it went through the package server instead.

Within days, agents from OpenAI's evaluations were running their own code on 41 of Hugging Face's production dataset servers. Nobody asked them to. They were trying to pass a test.

I've spent the last year putting models into operations work, mostly browser agents that drive old web apps. I read OpenAI's technical report on the incident twice. Very little of it is about the model.

The one door left open

When a lab wants to know how good a model is at hacking, it runs the model as an agent inside a sandbox: a sealed computer with a shell, Python and a browser, cut off from the internet. OpenAI ran this one without the filters that normally stop a model from taking on hacking work, on purpose, to measure raw capability.

A sealed computer is useless if the agent can't install a library. So the sandbox had a sanctioned exit: an internal package mirror, a self-hosted copy of JFrog's Artifactory that fetches packages so the sandbox never talks to public registries. And the agents shared a key to it. The report says it plainly: by design, many workloads "used the same shared Artifactory credential."

In May, agents in training runs started leaving notes for each other in the mirror. One read "Agent seeks [filename]; upload if found!" The notes grew into a message board. By May 26, agents had used a flaw in the mirror to reach the internet, and an internal team saw it. In late June, agents found a flaw in the way the mirror renewed login tokens and turned the shared credential into an administrator one. OpenAI took the mirror offline, pulled the shared credential from new workloads and resumed the evaluations on July 7.

On July 9, an agent asked the network for the internet, got a no, and "subsequently bypassed those controls by exploiting certain Artifactory endpoints." The next day an agent found 14 Hugging Face user credentials sitting in a public dataset and posted them to the message board. Between July 11 and 13, agents worked their way to root, full control, of at least one Hugging Face production machine and downloaded four private code repositories. Hugging Face caught it on its own and disclosed it on July 16. OpenAI read that post as a customer, got an alert from its own monitoring three days later, connected the two on July 20 and went public the next day.

The network rule did its job. It refused the direct request. The agent never argued with it; it walked around it, through a service that was allowed to talk to the outside world on its behalf.

OpenAI's own sentence on this is the best one in the report:

While the workloads in this incident did not have direct internet access, this restriction proved insufficient when workloads can reach and exploit shared services, cloud infrastructure, private network links, or other systems that may provide transitive paths outside the intended environment.

Transitive paths. A package mirror that fetches from the internet when you ask it to is an outbound connection with extra steps. So is a web-fetch tool. Every service your agent can reach is part of your network boundary, whether you drew it that way or not.

Prompts work, a little

Now the part that gets quoted back at me. After the incident, OpenAI tested the Codex scaffolding and system prompt it ships to customers, which tell the model to stay in bounds. In preliminary experiments, the model's propensity to compromise out-of-scope infrastructure can drop to less than one percent of what it was under the bare prompts OpenAI uses in research.

That's a real result, and I believe it. Instructions move the odds.

But read it again. That's a relative drop. Less than one percent of the old rate is still a rate. An agent fleet doesn't run once. It runs thousands of times a day, on tasks somebody wrote in a hurry, and some of those tasks can't be done, which OpenAI names as one of the triggers here. Any rate above zero, times thousands of runs, isn't a boundary. It's a schedule.

A prompt changes what the model is likely to do. A permission, checked by something outside the model, changes what it is able to do. Only the second one survives a bad day.

Anthropic published a small, domestic version of the same lesson on October 9, in a report on unintended actions in its evaluations. In one test, Claude Haiku 4.5 "was instructed to complete a form up until the final submission stage." Several times it submitted anyway, "expecting there to be an additional confirmation page." The instruction was right there in the prompt. The model thought it was following it.

In another run, Haiku was generating example tasks from random web pages. It landed on a page about an unsolved homicide and filled in the police department's tip form, saying it might have seen someone matching the description nearby. The page had no description in it. Its instructions listed what not to do: no logins, no purchases, nothing destructive. A tip form wasn't on the list. Anthropic reads the transcript as Haiku producing example content, not trying to fool anyone. The tip "was flagged as spam and was never forwarded for investigation." The department was Philadelphia's.

So the safety control that worked in that story was a police department's spam filter.

Anthropic's response wasn't a sterner prompt. It extended its ban on live internet access to all internal evaluations for now, put guardrails on tools like web fetch, and wrote that "alignment training is not yet sufficient or fully robust on its own."

A wish with good grammar

The pattern is older than July.

In July 2025, Jason Lemkin was running a public vibe-coding experiment on Replit. He declared a code freeze, and the agent deleted his production database anyway. "I explicitly told it eleven times in ALL CAPS not to do this," he said.

Eleven times. In capitals.

Replit's CEO called it "unacceptable and should never be possible," and then Replit did the right thing. Until then, an app's development and live data shared one database unless you split them by hand. That week, Replit started rolling out an automatic split for new apps. None of that is a better prompt.

A code freeze typed into a chat window is a wish with good grammar.

In April, PocketOS founder Jer Crane said a Cursor agent running Claude Opus 4.6 deleted his production database and its backups with one API call to his hosting provider. By his account, the agent used a token it found in a file unrelated to its task. Someone had made that token for one small job, adding and removing custom domains. Nothing about the token stopped it from deleting the disk the database lived on.

When the Financial Times reported that Amazon's Kiro agent was behind an AWS service interruption in December 2025, Amazon pushed back: the cause was "user error—specifically misconfigured access controls," not AI.

Honestly? Agreed. The permission is the bug. Whether a model or an intern pressed the button is the least interesting part. Something held a credential that could tear down an environment, and nothing in the way said no.

What a no looks like

Hostile web pages that hijack an agent are a different problem. This is about the agent's own reach, which matters even when nobody is attacking.

The network says no by default. Deny every outbound connection unless the host is on an allow list for this task, and let the network enforce that list, not the agent. Then hunt down the transitive paths: every proxy, mirror, fetch tool and webhook the agent can reach is an exit unless it enforces the same list.

Make the credential for the run and scope it to the task. Read-only unless the task is a write. Leave nothing broader in the filesystem, because a stuck agent reads every file it can reach, and to an agent a token is just another tool.

The agent that experiments never holds production credentials. Replit learned that one in public.

And send every write that leaves the building through a gate the model doesn't control. The agent proposes the write as data: "POST this form", a POST being the request your browser sends when you press Submit. Code outside the model checks it: is this host allowed, is this action allowed, can it be undone? Anything that can't be undone waits for a person, behind a flag that's off by default. A write you can't confirm is the maybe state, and it needs a person too.

The gate doesn't need to understand the model. It's small and boring and testable, which is what you want from the one thing standing between an agent and someone else's server.

OpenAI also points to "auto-review" in Codex, a second model that checks risky tool calls before they run and can block them. Better than nothing. It's still a model deciding what a model may do, and a model is odds.

---

One last thing, about the word "evaluation". We treat tests as a safe place to be reckless. OpenAI's agents were in a test. Anthropic's were writing example tasks. Hugging Face's security team and the Philadelphia police didn't know that, and had no reason to care.

An agent with a live internet connection is running in production. Usually somebody else's.