For years, keeping a model healthy in production meant keeping it close to the model you shipped. You watch drift against a reference window. You track latency against an SLO. You check accuracy against a holdout. A number moves, you retrain.
That stopped working once models started calling tools.
The industry response was fast. Gartner expects more than 40 percent of agentic AI projects to be canceled by the end of 2027, citing escalating costs, unclear value, and inadequate risk controls. Every observability vendor now ships agent tracing.
What went unexamined is how teams actually ran the migration. Most ran it as an addition: new spans on top of the old stack; nothing came off it. The inherited signals still fire, and several now report healthy on runs that failed.
I learned this from a multi-step review pipeline I run, which fans out to parallel model reviewers and writes their verdicts into an application datastore. The first bad verdict it shipped had a fully green trace: every span succeeded, latencies normal, output wrong.
Migration by Addition: The Part Nobody Audits
What got added is real progress. OpenTelemetry's GenAI semantic conventions now define agent spans: createagent, invokeagent, execute_tool, and plan. The spec still carries a Development status, worth knowing before you standardize on it.
Langfuse, LangSmith, and other platforms have extended their traces to capture agent steps. These additions are useful, but they don't replace the underlying assumptions of MLOps monitoring. They layer on top.
When you migrate by addition, you inherit five assumptions that agents break. Here's what they are, and how to spot the damage.
Assumption 1: A Successful Span Means a Successful Step
In classical MLOps, a span that returns 200 OK is a good span. The model ran, the latency was acceptable, and the output was within the expected distribution. That logic collapses with agents.
The span is just a record of execution, not a signal of correctness. An agent can call a tool successfully, receive a valid response, and still make a wrong decision because the prompt was ambiguous or the context was insufficient. The trace will show a green path, but the outcome is incorrect.
In my pipeline, that first bad verdict came from a reviewer that had all green spans. The failure was semantic, not operational. The model saw the data, processed it, and returned a verdict that was confident and wrong.
What to do: Add semantic checks that validate the output of each step. For example, compare the agent's final output against the expected schema, or run a lightweight verifier that ensures the decision is consistent with the input data. This can be a simple rule-based check or a secondary model, but it must be part of the pipeline.
Assumption 2: Drift Is Measurable Against a Reference Window
In traditional models, you track drift by comparing the current data distribution to the training set. If the input features start to shift, you retrain. Agents don't have a static reference window.
An agent's inputs are often unstructured and dynamic: user queries, tool results, intermediate reasoning. These change with the task, not just the data. A drift alarm might fire unexpectedly, not because the model is bad, but because the agent is exploring a new domain.
Conversely, drift might be absent even when the agent is failing. The inputs might look perfectly normal, but the agent's behavior has degraded because the tool it calls has changed its API or the context window is being used differently.
What to do: For agents, shift from input drift to output drift. Track the distribution of final actions and intermediate decisions. If the agent's choices start to cluster differently or show odd patterns, that's a signal. Also, monitor the health of the tools the agent calls; if a tool's response time or error rate changes, the agent's behavior will change too.
Assumption 3: Latency Is a Proxy for Performance
Latency used to be a reliable indicator. If the model was slow, you had a problem. With agents, latency is a mix of model inference, tool calls, reasoning loops, and retries. A high-latency trace might be fine if the agent is doing more work, or a low-latency trace might hide a shortcut where the agent skipped a needed step.
In my pipeline, the first bad verdict had normal latency. The agent returned quickly, but the quickness was because it didn't fully read the context. The trace showed a fast, green path, but the result was flawed.
What to do: Benchmark latency per step type. Know the typical latency for a tool call, a model call, and a planning step. Set separate thresholds for each. Look for anomalies relative to the step, not the entire trace. A sudden drop in latency might indicate the agent is cutting corners.
Assumption 4: The Pipeline Is Linear and Predictable
MLOps assumes a linear pipeline: data in, model out. Agents are graph-like, with branches, loops, and conditional tool calls. The number of paths is huge, and the outcome depends on the choices the agent makes.
This breaks a key MLOps practice: you can't just monitor the main path. You need to trace all branches, and some branches might be rare but critical. A failure in a branch that occurs 1% of the time might go unnoticed if you only monitor the happy path.
What to do: Use trace analytics to map out common paths. Identify the branches that lead to errors or poor outcomes. Set alerts on those branches, even if they are rare. Also, consider chaos testing: inject unusual inputs or tool failures to see how the agent handles them.
Assumption 5: A Green Trace Means a Healthy System
The biggest myth is that a green trace equals a healthy system. With agents, a green trace might mean the agent did exactly what it was told, but what it was told was wrong. Or the agent might have made a suboptimal choice that still succeeded.
This is why observability alone is insufficient for AgentOps. You need evaluation, not just monitoring. The trace tells you what happened, but you need to evaluate whether what happened was correct.
What to do: Implement a feedback loop where the agent's outputs are scored. This can be automated with a rubric or a judge model, or it can be manual with human review. For critical tasks, require a second approval. The goal is to close the loop between monitoring and improvement, so that green traces are truly green.
The Path Forward: AgentOps as a Different Discipline
AgentOps isn't just an extension of MLOps; it's a separate discipline that requires its own tools and mindset. The five assumptions above are the starting point for redesigning your monitoring stack.
In 2026, the state of practice is still early. OpenTelemetry conventions are in development, and most vendors are iterating. But you don't need to wait for the perfect tool. You can start by auditing your existing traces and asking a simple question: does a green trace actually mean the run was successful?
For me, that question caught the first bad verdict. And it will catch many more if I keep asking it.