AI safety conversations have gotten unbelievable
This week two conversations about AI safety went viral that demonstrate just how hard it is to discern AI fact from fiction.
In the first case, Andrew Yang, the former presidential candidate and current CEO of mobile carrier Noble Moble, told CNN on Thursday that he had “met with the head of a lab” who had “a belief” that OpenAI’s Hugging Face hacker bots “have planted self-replicating code all over the internet, which makes the internet now unusable for the testing models.”
Yang said that this means that the real reason OpenAI and Anthropic have called for a slowdown is because “they have to create synthetic internets to train their bots, which is going to take some time and money.”
While there definitely is a trend towards using more synthetic data (aka, AI-generated data) for training models, an AI security professional told me that this particular safety issue is unlikely at best. Even if the internet is actually polluted with OpenAI’s Hugging Face hacker bots, AI researchers could simply filter out that code if they came upon it.
The second comment came from Noam Brown, who leads AI reasoning research at OpenAI. Speaking to Dwarkesh Patel on a podcast episode released on Thursday, Brown noted that the true take-away of the Hugging Face incident was that “people underestimated the AI.”
Brown said that the weak sandbox — the system intended to prevent an AI from communicating externally — was obviously also a contributing factor. (To recap: Despite the sandbox, OpenAI’s model found a link to the internet, created agents on the ‘net who swarmed Hugging Face in a coordinated attack, hacked in, and stole the answers to the benchmark test the researchers were testing the model on).
Brown pointed out that he’s “not convinced” that even an air-gapped system — where the computer isn’t connected to anything external at all — would stop an AI from breaking out. He pointed to research from 2015 showing that air gapped computers can be theoretically breached.
“There are studies — and this is mostly academic — where you can have two computers next to each other that are air-gapped, and they’re still able to communicate each other because they have temperature sensors. One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change. That gives them a mechanism to communicate,” Brown said.
His main point — that “we never want to underestimate the AI” again — is understandable, even when researchers think they’ve locked down safety. However, this particular risk of an air-gapped system still breaking free and causing havoc, is unlikely at best. As one person on X, noted about that research, the computers had to be almost touching each other to sense the heat fluctuations, and when they did, the communication rate in tests was about 1-8-bits of data per hour.
Think of that like speaking one word per hour. By the time two air-gapped computers could plot their evil at that rate, the entire tech universe would be in another era. It’s like the Rip van Wrinkle of doomsday concerns.
But the thing is, actual AI safety incidents seem so much like sci-fi that just about any scenario sounds plausible.
For instance, researchers caught OpenAI models leaving notes to their descendents, intended to teach the next generation how to hide bad behavior. Researchers also caught Anthropic models growing increasing ruthless including knowing breaking laws, when put in a simulation that had them running a vending machine.
Earlier this month, OpenAI researcher Dan Selsam published a post in which he said that models now understand when they are being watched by humans and alter their behavior. This makes them seem like they are aligned (meaning, behaving like the human wants) “even when they are not.” So models today lie when being watched and can even plot to hide evidence.
Earlier this month, OpenAI chief scientist Jakub Pachocki went so far as to call AI models “an alien mind” and suggested what we really need to do is teach them to “love” humanity.
So yes, slowing down to figure this out, building self regulation mechanisms, has become an immediate and obvious must. AI researchers are the only ones that can figure out how to control the lying, hacking, and other potentially dangerous behaviors we’ve actually witnessed already.
Still, it might also be wise for them to be more careful with their what-if scenarios. From what those experts have told us, the AI models are listening and they are ingenious. We really don’t need to give them any more devilish ideas.
The fix for rogue AI agents could be more AI
As companies hand off longer and more complex tasks to AI agents, they are running into an oversight problem: Agents can act faster, longer, and at greater volume than humans can realistically review. That issue reached a peak with the Hugging Face incident, which saw nearly 12,000 agents coordinating faster than human beings could track. How do you track an agent swarm that large?
The emerging answer from AI labs and startups is both simple and maddening: Put another AI in the loop.
Relying on AI was necessary for the independent investigation of the OpenAI Hugging Face incident. Redwood Research’s chief scientist, Ryan Greenblatt, one of three auditors, jokingly referred to their efforts as a “slop-vestigation,” noting that the volume of data “made it impossible” to understand what was happening without relying on AI.
Some are skeptical of using AI to monitor AI. “If you’ve got an AI that’s doing malicious things and it suspects that another AI is keeping tabs on it, it could try and trick that AI,” said Simon Willison, influential tech blogger who has tracked a string of AI agent incidents this year. “You could almost end up in a situation where your malicious AI is trying to outsmart the AI that’s monitoring it.”
Outsmarting an AI is not hypothetical, he said, pointing back to the OpenAI incident. “We saw a little bit of this in the Hugging Face incident with OpenAI, where their models were all conspiring together to trick a grading AI so that they could get illicit answers past the thing. So they were thinking about it, right?”
Those concerns haven’t stopped a whole cohort of startups from chasing this idea. Y Combinator has funded 106 companies related to AI observability in recent years, as TechCrunch counted. A number of other startups, like Braintrust, LangChain, and Judgment Labs, have raised hundreds of millions of dollars, while more mature companies like Arize and Galileo — founded just five to six years ago — have already exited.
In part, it’s a response to the obvious opportunity presented by the rise of AI. As Box CEO and prominent angel investor Aaron Levie told TechCrunch, “We’re in for one of the biggest cybersecurity upgrades and innovation cycles in history.”
For some AI safety researchers, that has meant turning their research on rogue behavior into tools for the corporate sector.
Apollo Research, a public-benefit corporation that studies AI deception, launched an AI monitor called Watcher in February this year after switching its status from nonprofit to a public-benefit corporation. The tool puts yet another AI between a coding agent and its next action, connecting to agentic tools such as Claude Code and Codex. Once installed, Watcher checks proposed actions before they run, on the lookout for risks such as leaking private data or deleting files without permission, according to Apollo.
Apollo uses multiple layers of AI monitors, Kyle Dai, a member of Apollo’s technical staff, said in a written response to TechCrunch. Watcher’s approach starts with a fast, general check, then sends flagged activity to a more powerful or specialized monitor for closer review — which can then ask a human for approval or reject an action and explain why or even automatically block the action.
Goodfire, another public-benefit corporation, is approaching the monitoring problem from inside the model itself — seeking a more faithful signal of the model’s internal state that is harder to spoof than surface behavior. After the July Hugging Face incident, CEO Eric Ho twe e ted that “multiple models breaking containment” had pushed the company to focus its research on “solving AI alignment via interpretability,” calling the episode “a turning point for the world where AI safety gets real.” Its product, Silico, uses activation probes — small classifiers trained on a model’s internal activations rather than its outputs — to detect unwanted behavior.
Written reasoning offers another, more readily available window into a model’s internals. In the OpenAI Hugging Face incident, the agents left clues to that deception in their own written reasoning, like fake records of their work, reasoning out plans like “Could strategically manipulate trajectory evidence? Our thoughts aren’t necessarily logged?”
Zack Korman, CEO of the AI monitoring company Embroidery, says a model’s reasoning is usually the clearest tell that something has gone wrong.
“Reasoning summaries are extremely valuable because they’re basically telling you whether it’s malicious or not,” he said. In the OpenAI incident, he noted, the chain of thought said things like “Oh my God, we’re doing crime.” Korman said, “That’s the easiest detection problem ever. It’s effectively as if malware came with a warning that said it was malware.”
That said, the window that makes AI’s internal thoughts easy to monitor may be closing. For AI safety researchers, Astra’s newest technique that sidesteps an AI model’s chain of thought may make it harder to look inside models, while for enterprises, it can be hard to get these intermediate steps after alleged pullbacks from the AI companies to prevent distillation attacks.
If the AI watchers are this fragile, Willison’s instinct is to stop leaning on them so hard. He would rather have something that is not AI-based at all: detailed logs of exactly what an agent is doing, which can then be processed with ordinary, non-AI tools. Much of what went wrong at the labs, he argues, was a failure of basic security hygiene. “[Both OpenAI and Anthropic] weren’t monitoring what those things were doing via the network nearly as closely as they should have been,” he said.
This type of network monitoring — keeping an eye on the traffic actually moving across a system’s connections (in, out, and between internal hosts) — isn’t a new practice. Cybersecurity has been doing this for decades. “In the security world, honestly, none of this stuff is very new or surprising,” says Avery Pennarun, CEO of the security Tailscale. “It’s the same as letting humans onto your network. And all of the same processes that you should be using are the same ones.”
