Every major inference framework shipped prefill-decode disaggregation in 2026. NVIDIA built it into Dynamo, SGLang made it the default for large-scale deployments, and vLLM added a KV connector API to support it natively. The industry consensus is forming fast: split prefill and decode onto separate GPU pools, and throughput will improve.
But that consensus is wrong for most teams.
Doubleword's analysis shows that a balanced disaggregated deployment can match the throughput of a colocated setup, but at small GPU counts, rounding losses dominate. You can't allocate fractional GPUs, so the benefits of specialization are eaten by incomplete worker utilization. At that scale, the practical advantage is independent SLO tuning—not raw throughput.
A June 2025 study evaluating hundreds of thousands of design points found that disaggregation is most effective for prefill-heavy traffic patterns and larger models. For the mixed-traffic workloads that most teams actually run, queueing and inter-node KV cache transfer dominated end-to-end latency. Teams that disaggregated often just moved the bottleneck—they didn't remove it.
I encountered this firsthand while serving a mid-size classification model. TPOT spiked under bursty traffic, and the initial instinct was to separate prefill from decode. Instead, I enabled chunked prefill on the same GPU pool. TPOT stabilized. The problem was scheduling interference, and chunked prefill addressed it without adding a network hop.
What Disaggregation Solves: The Interference Problem
Prefill and decode have opposite hardware profiles. Prefill is compute-bound: it runs parallel matrix multiplications across the full input sequence, driving GPU compute utilization to 80–95%. Decode is memory-bandwidth-bound: it performs sequential KV cache reads to generate one token at a time, with compute utilization below 5% on an H100.
When both phases share a GPU, they fight over the same resources. A single large prefill request can cause decode tokens to stall, creating latency spikes. Disaggregation solves this by isolating the two phases on dedicated hardware, preventing interference and enabling independent tuning.
The Threshold: Why a Thousand GPUs Matters
So when does disaggregation actually pay off? Three conditions must hold:
- You have enough GPUs to absorb rounding losses. Splitting a pool of 16 GPUs into 8 prefill and 8 decode workers leaves no slack for traffic shifts. With 1,000 GPUs, you can allocate 200 to prefill and 800 to decode while maintaining healthy utilization buffers.
- Your traffic is prefill-heavy. If most requests are long-context inputs (e.g., document analysis or code generation), prefill dominates compute. Separating it prevents decode from being starved. For short, mixed requests, chunked prefill already smooths spikes.
- You have a network fast enough for KV cache transfer. Disaggregation requires moving KV caches from prefill to decode workers over the network. At high concurrency, this can add microseconds to milliseconds of latency—often negating gains unless you have ultra-low-latency interconnects.
Below that threshold—roughly a thousand GPUs—chunked prefill is the better default. It reduces interference, stabilizes TPOT, and keeps the system simple. Disaggregation should be a deliberate step taken when scale demands it, not a default assumption.