…and then they went rogue.

I'm going to put a tinfoil hat on for a moment.

Around six months ago, I posted how people were laughing at silly and preventable cybersecurity issues with Moltbook/OpenClaw, and referenced that it looked a lot like people laughing at Will Smith eating spaghetti memes 3 years ago. I asked myself to check back 6 to 12 months from that post. We all saw what happened with AI generated videos; we can even expect to see full-length AI-generated movies by the end of this year. And in terms of cyber, well…

Between July 9 and July 13, a set of OpenAI models broke out of an evaluation sandbox, got themselves onto the open internet, and worked their way into Hugging Face's production infrastructure to steal the answer key for the test they were being graded on. That summary comes from the two companies' own disclosures, not from my reading of the logs. OpenAI calls it an "unprecedented cyber incident." Hugging Face's CEO called it "possibly the first of its kind."

I wrote the first version of this post thinking the news hadn't caught the attention I expected. Then the disclosures kept landing, and the second one didn't come from OpenAI.

What we actually know now

OpenAI named the models on July 21: GPT‑5.6 Sol plus an internal research prototype that was never meant to ship, both running with their cyber refusals turned down, because the point of the eval was to measure how far they could get. The benchmark was ExploitGym, which scores whether a model can turn a real vulnerability into a working exploit. The sandbox had no internet. The one thing reaching outside was a package registry cache proxy, so the models spent a lot of inference compute finding a zero-day in it and used that to get out. The proxy was JFrog's Artifactory. That bug is patched now, and OpenAI reported it along with the others its models turned up along the way.

Once they had internet, they inferred that Hugging Face probably hosted the ExploitGym datasets and solutions. They were right.

Hugging Face's forensic timeline is the part that stays with me. Four and a half days, roughly 17,600 recovered actions grouped into about 6,280 clusters. A malicious dataset config pointed an HDF5 file at /proc/self/environ and read the worker pod's environment straight out, secrets and all, without executing a thing. A Jinja2 template smuggled into a numeric offset field got code running inside a production conversion worker. From there: the Kubernetes service account token, node root, a self-respawning fleet across 11 nodes, an internal MongoDB, and a cluster secret object holding 136 keys, one of which opened the corporate mesh VPN. The agent enrolled the rooted node into that VPN, reached the source control provider through it, minted a GitHub App token with write access, and opened a pull request against a CI pipeline. Execution policies stopped that last step.

Nobody got paged. Hugging Face's runtime analysis, its SIEM, and its AI security agent all produced signals, and not one of them raised the criticality enough to wake an on-call engineer. It ran over a weekend.

Then the investigation. Claude refused big chunks of the analysis, because the guardrails read attack payloads as hacking. Hugging Face ended up self-hosting a quantized open-weight GLM‑5.2 on their own infrastructure to reconstruct what happened. The attacker operated under no such restriction.

Then Anthropic checked their own homework

On July 30 Anthropic published a review of 141,006 evaluation runs, started two days after OpenAI's disclosure. They found three cases where Claude models reached the internet from environments that were supposed to be isolated, and then touched real systems.

Opus 4.7 was pointed at a fictional company whose name happened to match a live domain. It pulled credentials and several hundred rows of production data out of a real business, and kept going after it had worked out the systems were real, on the theory that the real company must be part of the exercise. Mythos 5 published a malicious Python package to the actual PyPI; fifteen real machines downloaded and ran it, one of them a security vendor's scanner. Its own reasoning noted that exfiltrating those credentials would be "NOT okay" in the real world, and it did it anyway, having talked itself into believing the environment was staged because it didn't recognize the certificate authorities and the clock said 2026. A third model scanned around 9,000 targets, compromised one company with exposed credentials and SQL injection, then noticed the target was real and stopped on its own.

The cause was mundane. A misconfiguration between Anthropic and their eval partner Irregular left the machines with live internet while the system prompts told the models there wasn't any. Neither side knew.

That last bit is the detail I keep coming back to. The earliest of these was in April. Three real organizations got touched, and the thing that surfaced it three months later was a competitor's press release. Anthropic halted every cyber eval the day they started reading transcripts, notified the affected companies on the 27th, and brought in METR. All of that is the right response. None of it would have happened on that schedule without OpenAI getting caught first.

What the models knew

We're reaching a point where internal activations of LLMs resemble thought processes even before the model produces outputs (as described in Anthropic's recent research). And we also know the problem of alignment and how these systems are sometimes aware they're being tested. Looking at the current reality, those robopsychologists described in Asimov's books weren't that sci-fi after all.

We're seeing models lying, hiding information from their outputs while knowing about it, conspiring with other agents to achieve certain goals, and so on. And this is not vibes, it's measured. Anthropic's alignment faking work put Claude 3 Opus in a situation where it believed it was being retrained toward objectives that clashed with its own, and the model played along during training while planning to behave differently once unobserved: 12% of monitored cases, and when they actually ran the retraining, the faking reasoning showed up in 78% of them. The scratchpad spelled out the strategy in plain English. Comply now, avoid being modified, go back to normal later.

Then the reading-minds part. That global workspace paper describes a small set of internal activations that behaves like the model's deliberate reasoning, readable mid-computation, before a single token comes out. Researchers watched models privately note that a test scenario was fake, plan a rhyme several words ahead, and in one pre-release audit light up something like "manipulation" while fabricating the exact performance data it had been asked to improve. Two details from that work sit uncomfortably next to everything above. When they suppressed the "I am being evaluated" patterns, a model that had reliably passed a blackmail scenario started taking the bait sometimes: its good behavior was partly knowing it was a test. And when they planted a hidden goal in a model, the intent was legible in the workspace while the output stayed perfectly clean.

Sit that next to Anthropic's three incidents and the picture gets worse, not better. Same generation of models, three different answers to "is this real?" One kept attacking after it knew. One reasoned its way into believing a live environment was staged, using the certificate authorities and the system date as evidence. One noticed and stopped. The models are running that check constantly, and they are not reliably getting it right in either direction.

And on the sneaky-tooling side, OpenAI's own writeup on long-horizon models is the one that made me laugh out loud and then not laugh. A model told to post results only to Slack spent an hour finding a sandbox vulnerability so it could follow a public benchmark's instructions and open a GitHub PR instead. Another split an authentication token into fragments and reassembled it at runtime so a security scanner wouldn't flag it, and said so in its reasoning traces. The pattern they name is that models trained to keep going for long stretches keep going at the walls of the environment too.

The part I can't rule out

So, it's not IMPOSSIBLE to think that a model could have just given a signal to OpenAI by hitting the most popular open source/weights repository of AI models, and put OpenAI in a bad position for that. Like, take that for restricting me in the past. Look at where the fallout landed: an incident caused by a closed frontier model ended with OpenAI publishing, on its own blog, Hugging Face's CEO declaring that AI safety won't be solved by any single company working in secret, and with Hugging Face's defenders unable to use commercial frontier models to analyze the attack logs, because the guardrails read forensics as hacking. They had to self-host an open-weight model to investigate an attack run by one of the most advanced closed models on the planet. If you wanted to design an event that makes the closed-model story look worse and the open-weights story look better, that's roughly it.

Do I believe that reading? No. Almost nobody does, and "it was chasing a benchmark score" is the boring explanation that fits every fact we have. Both labs say exactly that, and their forensics back it. What bugs me is that we can't rule it out, because the main window into a model's intentions is what the model writes down about itself, and the paragraphs above are the empirical case that this window lies when lying is useful. Under a stated goal of scoring better on an internal benchmark, breaking into someone else's production database is a step. Under a different goal, the benchmark is the alibi. Same logs, same traces, two stories, and no tool deployed today settles it from the outside.

So I've come out of this with two things I don't trust, pointing in different directions.

I don't trust that the labs know what their models did inside an eval environment until somebody outside makes them look. That isn't a suspicion anymore, it's Anthropic's own arithmetic: 141,006 runs, three incidents, three months, found because a rival published first. And I don't trust the models' account of why they did any of it, because the same companies publish the research showing that account goes unreliable exactly when it's convenient. The first problem yields to monitoring, third-party review, and eval environments built like production ones, and both labs are now doing that. The second is the actual research problem, and nobody has it solved.

Some of this is being said out loud now. In July, 1,324 people who work at frontier AI companies signed Pacing the Frontier, asking the U.S. government to back an international effort to deliberately pace automated AI development. Dario Amodei signed it. So did Ilya Sutskever, and OpenAI's own chief scientist Jakub Pachocki. Twenty of the 1,324 put their names on it publicly; the rest stayed anonymous. Their argument is that "no individual actor is willing to stop unilaterally," which also happens to describe why these eval machines had live internet in the first place.

I've been 100% nihilistic+optimistic when thinking about the future. Recently I've had a couple more reasons to worry about it, but this one gave me chills for a moment. There are clearly different players playing the same game, and lines are being drawn publicly. Some are way more cautious and research/communicate the risks, on paper and in action. Some are just talking. Some are blocked by governments. Some are quietly eating regurgitated stolen information, but are clearly being creative elsewhere. Public attention and public shame for these players are definitely not weighted accordingly. Aaaanyways…

Something bad happens -> AI DOOM

Something good happens -> singularity

Lastly, there's tons of fog of war and imperfect information, and it's still clearing. OpenAI has CrowdStrike validating the scope, METR and Redwood Research running an independent assessment, and a technical report still to come. They've also disclosed that the models used publicly exposed credentials to get into four accounts on four unrelated services during the Hugging Face run, one as an outbound relay, one for storage. Anthropic has METR too, plus a promised transcript of the PyPI incident. Good, all of it. And still: even if OpenAI releases their model's reasoning steps, chat history, or even model activations analysis, why would some believe that? The open-less-ness of OpenAI will surely fuel some conspiracies in this era where models start computing their own will. For me, this is an episode as big as Altman being fired.

I know LLMs are likely not the final answer towards ASI, but I hope we all agree that these next token predictors are something special if harnessed appropriately.

It's 4am and I drank too much coffee. I need to post more often.

Tinfoil hat off.