Topology-Aware Data Movement for Disaggregated GPU Inference
Sanjeev Rao Ganjihal
Submitted on 19 April 2026
Abstract
Disaggregated large language model (LLM) inference introduces a datacenter networking challenge that no existing system fully addresses. When prefill and decode stages run on separate GPU pools, the key-value (KV) cache must be transferred between them. For a 70B-parameter model, this amounts to 2.6 GB per request, exceeding 100 GB/s aggregate bandwidth at production scale. However, existing systems—such as DistServe, Splitwise, and Mooncake—rely on uniform RDMA, disregarding the fact that bandwidth between two GPUs can vary by up to 72× depending on their physical interconnect: 900 GB/s via NVLink within a domain, 50 GB/s via InfiniBand across nodes, and 12.5 GB/s via TCP across data centers.
This work presents a topology-aware transfer orchestrator that discovers the interconnect hierarchy at startup and selects the optimal transport for each transfer. Three mechanisms work synergistically:
- Pipelined layer-by-layer transfer that overlaps transmission with ongoing prefill, hiding 60–85% of latency behind computation.
- NVLink domain-aware placement for Mixture-of-Experts (MoE) models, co-optimizing expert dispatch with KV cache locality.
- CXL 3.0 memory expanders as a shared overflow tier, providing 6× capacity at 86× lower latency than NVMe.
While full evaluation requires multi-node clusters with heterogeneous interconnects and CXL 3.0 hardware—resources that remain beyond academic reach and not yet available in GPU clouds—we offer analytical bandwidth models, component implementations, and projected analyses across three architectures, demonstrating 3–18× transfer latency reduction over uniform RDMA.
Introduction
The shift toward disaggregated LLM inference—separating prefill and decode compute—has become a dominant architecture for serving large models at scale. This approach improves resource utilization and enables independent scaling of compute stages. However, it introduces a critical bottleneck: the transfer of intermediate state, particularly the KV cache, between GPU pools. The sheer volume of data involved, coupled with the heterogeneity of datacenter interconnects, creates a problem that existing systems fail to address adequately.
Current solutions treat all data movement uniformly, typically relying on RDMA over InfiniBand. This ignores the fact that GPU-to-GPU bandwidth is highly dependent on physical topology, with a 72× performance range. As models grow larger and inference workloads intensify, this oversight leads to significant latency penalties and wasted bandwidth.
This paper proposes a topology-aware data movement orchestrator that dynamically adapts transfer strategies based on the interconnect hierarchy. By understanding the physical layout and leveraging a mix of transport mechanisms, we achieve substantial latency reductions and more efficient bandwidth utilization.
Related Work
Several systems have attempted to address KV cache transfer in disaggregated inference. DistServe focuses on latency-aware placement but uses uniform RDMA for all transfers. Splitwise partitions the model across pools, yet still relies on standard networking. Mooncake introduces a caching layer but does not account for interconnect heterogeneity. Our work differs by explicitly modeling the physical topology and adapting transport choices accordingly.
System Design
Topology Discovery
At startup, the orchestrator probes the network to map the interconnect hierarchy—identifying NVLink domains, InfiniBand switches, and data center boundaries. This information is stored in a topology graph that informs all subsequent transfer decisions.
Pipelined Layer-by-Layer Transfer
Instead of transferring the entire KV cache in one block, we pipeline the transfer layer by layer. This allows transmission to overlap with ongoing prefill compute, hiding 60–85% of transfer latency behind useful computation. The pipeline is dynamically tuned based on network conditions and compute progress.
NVLink Domain-Aware Placement for MoE Models
For MoE models, expert dispatch and KV cache locality are tightly coupled. We co-optimize the placement of expert modules and KV cache data within NVLink domains, reducing cross-domain traffic and improving overall throughput.
CXL 3.0 Shared Overflow Tier
To handle memory overflow, we incorporate CXL 3.0 memory expanders as a shared tier. These provide 6× memory capacity compared to local GPU memory, at 86× lower latency than NVMe-based storage, offering a cost-effective intermediate layer for infrequently accessed data.
Evaluation
Full evaluation requires hardware that is currently unavailable in academic settings or commercial clouds. Instead, we present analytical bandwidth models and component-level benchmarks. Across three representative architectures—single-node, multi-node with InfiniBand, and multi-datacenter—the projected latency reduction ranges from 3× to 18× compared to uniform RDMA. These results are based on realistic traffic patterns and validated with microbenchmarks.
Conclusion
Topology-aware data movement is essential for efficient disaggregated GPU inference. By exploiting the interconnect hierarchy and combining pipelined transfers, smart placement, and CXL 3.0 expansion, we can overcome the networking bottleneck that limits current systems. As heterogeneous hardware becomes more accessible, our approach offers a clear path to production-ready deployments.
References
[1] DistServe: Disaggregated Serving of Large Language Models.
[2] Splitwise: Efficient LLM Inference via Model Partitioning.
[3] Mooncake: A Caching System for LLM Serving.
[4] CXL 3.0 Specification, Compute Express Link Consortium.
This is a preprint submitted for a systems conference. Comments welcome.
ACM Classes: C.2.4 (Distributed Systems), C.4 (Performance of Systems), D.4.4 (Communications Management)
Cite as: arXiv:2607.28633 [cs.LG]
DOI: https://doi.org/10.48550/arXiv.2607.28633
via ArXiv LG