Google Cloud Releases llm-d 1.0: Cache-Aware Routing Across TPU and GPU Fleets
The Google team has released llm-d 1.0, the first stable version of its distributed inference stack. Announced at the PyTorch Conference North America, llm-d is an open-source, Kubernetes-native layer that pools vLLM or SGLang model servers into a single coordinated serving system. It is aimed at self-hosted agent and reinforcement learning (RL) workloads, where conventional round-robin load balancing breaks down.
At a Glance
- A full serving stack: Google cites a published benchmark on a 17,000-node, multi-region GKE deployment.
- Where it runs: Kubernetes with NVIDIA, AMD, or Intel GPUs or Google TPUs, serving models through vLLM or SGLang.
- Where the gains come from: Vendor-reported performance improvements stem from cache-aware routing, tiered KV cache offload, and prefill/decode disaggregation.
- Best case: More than 85% reduction in median time to first token (TTFT), alongside 2.4x throughput, on an Ironwood TPU coding workload.
- Remaining gap: RL time-slicing lifts accelerator duty cycle only above 70%, leaving roughly 30% idle time.
What llm-d 1.0 Is
llm-d turns vLLM or SGLang model servers into one coordinated, Kubernetes-native inference pool. By combining cache-aware routing, tiered KV cache offload, and prefill/decode disaggregation, it targets self-hosted agent and RL workloads that defeat naive round-robin load balancing. The 1.0 release marks the project's first stable milestone and arrives as enterprises increasingly demand open, portable infrastructure for distributed inference in 2026.
Why It Matters
As agentic and reinforcement learning workloads scale, inference systems must coordinate across large fleets rather than distribute requests evenly. llm-d's cache-aware routing and disaggregated prefill/decode approach address these bottlenecks for self-hosted deployments. With the 1.0 release, Google positions llm-d as a stable, production-ready layer for these demanding use cases, supported by a multi-region GKE benchmark at the 17,000-node scale.
via MarkTechPost