Introduction
In my organization, I've worked as a backend engineer and architect. My primary responsibility is ensuring that the services we design not only meet their functional requirements but also scale to millions of requests per minute, maintain 99.99% uptime, and remain cost-effective enough to keep OPEX in check. For most of my career, the traffic hitting those services followed a pattern I could reason about—human-driven, forecastable, and self-limiting. That has changed. Agent traffic doesn't behave that way, and the two dominant scaling models (on-demand and serverless) we've built both break under its pressure. In this article, I'll walk through the mindset shift needed when scaling for agentic traffic—and the specific patterns worth considering.
If you've been in AI engineering, you've likely watched this shift happen. The traffic hitting your endpoints and model gateways no longer resembles what traffic used to look like—or how human traffic behaved. Agentic traffic arrives in unpredictable bursts. It repeats itself and retries relentlessly. It drains scaling costs significantly more than human-shaped traffic.
I want to explore why that assumption is now broken, why the two dominant scaling models we've built inherit the problem, and what the solution should look like moving into 2026 and beyond.
The Assumption Underneath Everything
Every scaling framework we've built assumes traffic looks the way people generate it. Compare what's on the left of this table with what's now on the right:
| Dimension | Human-Driven Traffic | Agent-Driven Traffic |
| --- | --- | --- |
| Shape | Diurnal curve with forecastable peaks. Tomorrow looks like today. | No schedule. Bursts triggered by orchestration events, not clocks. The pattern itself shifts as agents and prompts change. |
| Onset Speed | Ramps over seconds to minutes; you can watch it build. | Near-instantaneous. A parallel fan-out or a tight loop reaches full rate in milliseconds—faster than reactive scaling can respond. |
| Concurrency | Independent users; the aggregate smooths out by the law of large numbers. | Correlated fan-out from a single trigger. One orchestration spawns many synchronized calls. No statistical smoothing. |
| Retries | Bounded. People give up, refresh occasionally, back off out of frustration. | Programmatic and relentless. Without an explicit retry budget, an agent turns one fault into a retry storm. |
| Latency Tolerance | Sub-second or the user abandons. | Often tolerant of seconds to minutes—reasoning runs in the background. This slack is exploitable. |
| Cost Driver | Request count roughly tracks cost. | Request count is decoupled from cost. One heavy reasoning chain can consume more compute than a thousand lightweight calls. |
| Failure Mode | Graceful degradation—users drop off. | Self-amplifying. Loops drain resources and, on serverless, bill you for every redundant call before any signal fires. |
Agentic traffic violates all seven of these assumptions simultaneously. That's why the answer isn't a better version of either model we already have—it's a fundamentally different place to put the intelligence. Let me show you what I mean by walking through how the discipline of scaling has evolved.
Generation 1: Anticipation (On-Demand Instances)
Earlier in my career, working on large-scale video streaming backends serving millions of concurrent viewers, capacity planning was a human exercise. When a major live event was about to kick off, we knew exactly when the spike was coming, roughly how steep it would be, and when it would flatten. The work was in the anticipation: pre-warming EC2 fleets days ahead, setting min/max autoscale bounds, staffing a [continued]...