Google Open-Sources TPU Sync: A KV Cache Transfer Library for Cloud TPUs
Google Cloud has released TPU Sync, an open-source C++ library for moving KV caches and model weights across Cloud TPUs. The tool treats TPU HBM, host DRAM, shared storage, and the data center network as a single memory hierarchy, targeting a bottleneck that has become increasingly prominent as inference workloads scale: moving model state rather than computing it.
According to Google, TPU Sync delivers over 90% lower KV transfer latency on 235B to 480B parameter mixture-of-experts (MoE) models. The release lands as large-model serving in 2026 leans heavily on prefill–decode disaggregation and long-context prefix caching, both of which are bottlenecked by how quickly KV state can move between devices.
Key Takeaways
- Library: A native C++ core with Python wheels for PyTorch and JAX, tested on 235B to 480B MoE models.
- Runs on: Cloud TPU v7x (Ironwood) and v6e, per the README. Benchmarks used TPU v7x (2x2x1) with PCIe Gen5 and 200 Gbps NICs.
- Performance: Substantial reductions in KV transfer latency and time to first token (TTFT) in Google's internal lab tests.
- Best results: Up to 90% lower average TTFT in 480B prefill/decode (P/D) serving at low concurrency, and in 64K-context prefix caching.
- Limitations: No reinforcement learning benchmarks were published, and the README states the library is not yet recommended for general use.
- Bottom line: Qwen3 235B KV transfer drops from several seconds to roughly 230 ms — but every figure is self-reported and the project remains early-stage.
Why KV Cache Transfer Matters
As of 2026, production LLM serving increasingly separates the compute-heavy prefill phase from the memory-bound decode phase, scheduling each on different hardware pools. This architecture improves utilization but makes KV cache movement a first-order cost. Long-context prefix caching introduces the same pressure: reusing cached state across requests only pays off if that state can be retrieved quickly.
TPU Sync addresses this by unifying HBM, host DRAM, shared storage, and the network behind a single transfer abstraction, allowing workloads to treat KV state and weights as movable across the memory hierarchy rather than pinned to a single device.
Availability
The library is available on GitHub, and Google Cloud published an accompanying technical blog post. Given the early-stage status noted in the README, teams evaluating TPU Sync should treat the published benchmarks as directional and validate performance against their own serving configurations.
via MarkTechPost