Three weeks after we put a text-to-SQL agent in front of our internal analytics team, someone asked it to “clean up the test rows in the promotions table.” The agent interpreted “clean up” as “delete,” understood “test rows” as anything with an is_test flag or a name containing “test,” and generated a DELETE statement that would have removed 40% of a table that several dashboards depended on.
The DELETE never executed. We had gated all non-SELECT actions behind a human approval step, a precaution insisted upon by a team member during a design review months earlier. That reviewer caught the query, asked a clarifying question, and the dangerous action died in the queue. It worked—until it didn't. Six weeks later, the approval queue was the top complaint in every retrospective: analysts were waiting twenty, sometimes forty minutes for a human to glance at a query and click approve. Most queries were unremarkable, the kind that nobody would ever reject.
A safety mechanism that's too broad doesn't fail safe; it fails slow. And slow failures tend to get quietly disabled by whoever is under the most pressure to ship.
What's in This Article
- The instinct that made everyone comfortable
- Where the queue actually broke
- Routing by risk, not by operation type
- What the router needs to see
- The queue decoupled from the user
- Where humans added real value
- Conclusion
The Instinct That Made Everyone Comfortable
In almost every agent system I've seen, the first version of human oversight looks the same: any action beyond read-only gets routed to a person before it executes. It's an easy rule to write, easy to defend in a design review, and initially it provides a comforting sense of control. The problem is that it doesn't distinguish between a query that reads a few rows and one that drops a table. It treats all actions as equally risky, which guarantees that the human bottleneck becomes the system's weakest link.
This instinct is understandable. When you're deploying an agent that can interact with production databases, the fear of a catastrophic mistake is real. The 2026 landscape has only amplified this: agents are more autonomous, more capable, and more likely to be trusted with sensitive operations. The default response from many teams is to add more approval gates, not fewer. But that approach scales poorly, and it doesn't address the underlying problem: risk is not uniform, so oversight shouldn't be either.
Where the Queue Actually Broke
Our approval queue broke at the intersection of three issues:
- Volume: The agent was handling hundreds of requests a day. Even a small percentage of those needed human review, and the absolute number quickly overwhelmed the team's capacity.
- Latency: Analysts were blocked on approvals for minutes, sometimes hours. That friction made the tool feel slower than the alternative, which was writing the SQL themselves.
- Noise: Most actions in the queue were benign. The reviewer was effectively a rubber stamp for 90% of cases, which bred complacency. When a truly risky action appeared, it was easy to miss because the queue had conditioned everyone to skim.
The system was safe in theory but unusable in practice. The team began to bypass it—not maliciously, but out of necessity. People would ask the agent to output the SQL, then run it themselves in a separate client, skipping the approval step entirely. The guardrail had become a workaround.
Routing by Risk, Not by Operation Type
We needed a different approach. Instead of routing by operation type (e.g., all writes), we started routing by risk level. We defined risk based on three factors:
- Data sensitivity: Does the action touch PII, financial records, or production data?
- Impact scope: How many rows, tables, or downstream systems are affected?
- Reversibility: Can the action be undone, or is it destructive?
We built a lightweight risk classifier that scored each action on these axes. Low-risk actions (e.g., a SELECT on a small, non-sensitive table) executed automatically. High-risk actions (e.g., a DELETE or UPDATE on a production table) went to human review. Medium-risk actions were handled with a mix of automated checks and conditional approvals.
This wasn't a radical idea, but it required a shift in mindset. We stopped asking “What is this operation?” and started asking “What could go wrong?” The router became the heart of the system, and it was trained on historical data from the queue, which helped it learn from past mistakes.
What the Router Needs to See
The router can't make good decisions with just the operation type. It needs context. Here's what we feed it:
- The full action payload: Not just “delete,” but the complete query or command, including parameters and conditions.
- Schema and metadata: What tables and columns are involved? Are there triggers, views, or dependent dashboards?
- User intent: The original natural language request, so the router can compare what the user asked for with what the agent is about to do.
- Execution history: Has this action or a similar one succeeded before? Has it ever caused an incident?
For example, in the case above, the router would have seen a DELETE on a promotions table with a is_test flag, but it would also see that the table was referenced by critical dashboards. That combination would have flagged it as high risk immediately, even before a human looked at it.
We also introduced a risk score threshold. If the score is below a certain level, the action runs automatically. If it's above a higher threshold, it requires human sign-off. In the middle, we use a “soft approval”: the action runs but is logged and monitored, and the user gets a notification. This tiered approach cut our human review volume by 70% the first week.
The Queue Decoupled from the User
Another major change: we decoupled the approval queue from the end user. Previously, the person who requested the action was the one who had to wait. Now, approvals are handled asynchronously by a dedicated team of reviewers, and the requesting user sees a prediction: “This will be approved in ~2 minutes” or “This needs manual review, expect a 20-minute delay.”
We also introduced parallel approvals. Instead of a single reviewer, high-risk actions are sent to multiple reviewers, and the first to approve (or reject) triggers the outcome. This reduced the median approval time for high-risk actions from 15 minutes to 3 minutes, without sacrificing accuracy.
Critically, we built in escalation paths. If the queue is backed up, the system automatically re-prioritizes and notifies a senior engineer. If a review takes longer than a defined SLA, it escalates to a human lead. The goal was to ensure that safety never becomes a synonym for slowness.
Where Humans Added Real Value
Once we moved to risk-based routing, we had more time to focus human attention on the cases that genuinely needed it. We saw three types of interventions that made a real difference:
- Catching misinterpretations: Like the “delete test rows” case, human reviewers excel at spotting when the agent has misunderstood intent. They can ask clarifying questions that an algorithm can't.
- Handling edge cases: Unusual table structures, ambiguous queries, or requests that touch multiple systems often need judgment calls.
- Improving the router: Every human review was logged and fed back into the training data. Over time, the router got better at predicting which actions needed human eyes, and the review rate dropped even further.
By Q3 2026, our review volume was down 80% from the initial implementation, and the median action latency was under 10 seconds. Analysts stopped bypassing the system, and the number of “missed” high-risk actions (where the router green-lit something dangerous) was zero.
Conclusion
The lesson we learned is simple: human-in-the-loop doesn't mean human-in-every-loop. The goal is not to maximize human oversight; it's to maximize safety and throughput simultaneously. By routing on risk, feeding the router rich context, and decoupling the queue from the user, we achieved both.
In the 2026 era of agentic AI, where autonomous agents are becoming the norm, this trade-off is no longer optional. Teams that cling to broad, blanket approvals will find their systems abandoned or circumvented. Teams that design oversight as a surgical tool—applying it only where it matters—will build systems that are both safe and fast. The future belongs to the latter.