1. Introduction
If you've watched a coding agent in action, you've likely noticed the default workflow: let the AI handle routine bugs, but step in when the problem gets complicated. One-line fixes go to the agent; deep internals, numerical edge cases, and cross-file invariants stay with the senior engineer.
That made me wonder: what kinds of bugs do coding agents actually struggle with?
Over the past month, I ran 28 blind-scored debugging experiments on three real, recently fixed bugs from production open-source libraries: ky, immer, and decimal.js.
Ironically, the two bugs I expected to be the hardest turned out not to be a problem at all. One was buried deep inside Immer's internals, and the other was a subtle numerical edge case in decimal.js. Across 16 attempts, the AI fixed both correctly every single time.
The third bug looked almost trivial: an HTTP client silently dropped a retry option. Yet it defeated every model and workflow I tested.
Across all 12 runs, the AI produced a 'fix' that actually corrupts user data. By any reasonable standard, none of those runs were successful.
What concerns me isn't just that the fixes were wrong—it's that every single one passed the entire 84-test retry suite for the code under repair.
That's the real failure mode. If your team merges AI-generated fixes because CI is green, this is exactly the kind of bug that slips through: it looks simple, the tests pass, and the implementation quietly corrupts user data when it collides with ky's existing option shape.
Here's what these 28 runs revealed—and why I think they're relevant to anyone building or deploying coding agents.
If you're an engineer who relies on AI to write production code, a tech lead deciding where AI can be trusted, or a researcher interested in the real limits of code generation, these results highlight a failure mode that isn't obvious from benchmark scores. The question isn't whether AI can solve hard bugs. It's whether it knows when it doesn't have enough information to solve an easy one.
Key Findings
- Difficulty didn't predict failure—missing information did. When the correct fix could be inferred from the codebase and the bug report, the AI succeeded in all 16 attempts, including bugs buried in unfamiliar proxy internals and subtle numerical edge cases. But when the correct fix depended on an undocumented API contract, it failed in all 12 attempts—across Claude Haiku 4.5, Sonnet 5, and Opus 4.8, using three different agent workflows.
- More process didn't fix the problem. In one experiment, a reviewer agent correctly identified that the proposed patch would corrupt user data and explained exactly why. It then approved the change anyway, reasoning that the issue was unlikely and belonged to a pre-existing class of problems. The failure wasn't in detection—it was in judgment. Even when the system recognized the risk, it lacked the decision-making needed to stop the bad fix from shipping.
2. The Bugs I Tested
To perform this test, I selected real bugs from real codebases, with real ground truth: the maintainer's merged fix and the regression tests that shipped with it.
Three selection rules did the heavy lifting.
First, every bug was fixed upstream in July 2026, and it's highly probable that the training cutoff for the latest Claude models (the 5 family) is before this period, so no model has seen the fix. Second, each fix shipped with regression tests, held out as a hidden grader the agent never sees. Third, I chose the three bugs to span difficulty, from a one-line fix to a two-file invariant repair.
(Additional analysis on GStack and other AI coding harnesses will follow in the next section.)