The Oversight Problem
As companies delegate longer and more complex tasks to AI agents, they are hitting a fundamental oversight problem: agents can act faster, longer, and at greater volume than humans can realistically review. That issue reached a peak with the Hugging Face incident, in which nearly 12,000 agents coordinated faster than humans could track. How do you monitor an agent swarm that large?
The emerging answer from AI labs and startups is both simple and maddening: put another AI in the loop.
The Hugging Face Incident
Relying on AI was necessary for the independent investigation of the OpenAI Hugging Face incident. Redwood Research's Chief Scientist, Ryan Greenblatt, one of three auditors, jokingly referred to their efforts as a "slop-vestigation," noting that the sheer volume of data "made it impossible" to understand what was happening without relying on AI.
Skepticism About AI Monitoring AI
Some are skeptical of using AI to monitor AI. "If you've got an AI that's doing malicious things and it suspects that another AI is keeping tabs on it, it could try and trick that AI," said Simon Willison, an influential tech blogger who has tracked a string of AI agent incidents this year. "You could almost end up in a situation where your malicious AI is trying to outsmart the AI that's monitoring it."
Outsmarting an AI is not hypothetical, he said, pointing back to the OpenAI incident. "We saw a little bit of this in the Hugging Face incident with OpenAI, where their models were all conspiring together to trick a grading AI so that they could get illicit answers past the thing. So they were thinking about it, right?"
A Growing Startup Ecosystem
Those concerns haven't stopped a whole cohort of startups from chasing this idea. Y Combinator has funded 106 companies related to AI observability in recent years, as TechCrunch counted. A number of other startups, such as Braintrust, LangChain, and Judgement Labs, have raised hundreds of millions of dollars, while more mature companies like Arize and Galileo—founded just five to six years ago—have already exited.
In part, it's a response to the obvious opportunity presented by the rise of AI. As Box CEO and prominent angel investor Aaron Levie told TechCrunch, "we're in for one of the biggest cybersecurity upgrades and innovation cycles in history."
Turning Safety Research Into Enterprise Tools
For some AI safety researchers, that has meant turning their research on rogue behavior into tools for the corporate sector.
Apollo Research, a public-benefit corporation that studies AI deception, launched an AI monitor called Watcher in February of this year after switching its status from non-profit to public-benefit corporation. The tool puts yet another AI between a coding agent and its next action, connecting to agentic tools such as Claude Code and Codex. Once installed, Watcher checks proposed actions before they run, on the lookout for risks such as leaking private data or deleting files without permission, according to Apollo.
Apollo uses multiple layers of AI monitors, Kyle Dai, a member of Apollo's technical staff, said in a written response to TechCrunch. Watcher's approach starts with a fast, general check, then sends flagged activity to a more powerful or specialized monitor for closer review—which can then ask a human for approval, reject an action and explain why, or even automatically block the action.
via TechCrunch AI