Heterogeneous Memory Chiplets Accelerate Multi-Request LLM Inference
National University of Singapore (NUS) — Researchers at NUS have proposed a heterogeneous memory chiplet architecture that significantly accelerates multi-request inference for large language models (LLMs). The work targets one of the most pressing bottlenecks in modern AI systems: the memory bandwidth and capacity wall that limits throughput when serving many concurrent inference requests.
The Problem: Multi-Request LLM Inference Is Memory-Bound
As LLMs scale into the hundreds of billions of parameters, inference workloads are increasingly dominated by memory traffic rather than compute. In multi-request (batched) serving scenarios — the dominant mode in production deployments — each request competes for the same key-value (KV) cache and weight storage. Conventional homogeneous memory subsystems, typically built around HBM stacks, struggle to simultaneously satisfy:
- High bandwidth for weight streaming and attention operations
- Large capacity for KV caches that grow with context length and batch size
- Low latency for token-by-token decoding
These competing demands make a single memory technology a poor fit across all phases of inference.
The Proposed Solution: Heterogeneous Memory Chiplets
The NUS approach partitions memory into specialized chiplets, each optimized for a distinct role in the inference pipeline:
- High-bandwidth chiplets — tailored for streaming model weights and attention computation, where raw bandwidth is the limiting factor.
- High-capacity chiplets — optimized for KV cache storage, where density and cost-per-bit matter more than peak bandwidth.
- Near-memory / processing-in-memory (PIM) chiplets — co-located with compute to reduce data movement for bandwidth-bound operations.
- Mix-and-match memory types on a common interposer or advanced package without monolithic die constraints.
- Scalable capacity by adding KV-cache-optimized chiplets as context windows and batch sizes grow.
- Improved yield and cost compared to equivalent monolithic designs.
- Composability with emerging UCIe and die-to-die interconnect standards for high-bandwidth, low-latency chiplet communication.
- Higher throughput per watt in multi-tenant LLM serving
- Better tail latency under bursty, high-concurrency workloads
- More efficient long-context inference, where KV-cache growth is the primary scaling barrier
By combining these heterogeneous memory types within a chiplet-based package, the architecture can be dynamically allocated across the mixed demands of batched inference, rather than forcing all workloads through a single memory tier.
Why Chiplets?
Chiplet-based design offers several advantages that make this heterogeneity practical:
2026 Context
The work arrives at a pivotal moment. By 2026, LLM inference has become the dominant cost driver in AI data centers, and the industry has broadly converged on chiplet-based architectures as the path forward. Standards such as UCIe 2.0 and the maturation of 2.5D/3D packaging have made heterogeneous memory chiplets commercially viable. Meanwhile, KV-cache pressure from long-context models (128K+ tokens) and high-concurrency serving has made memory heterogeneity a first-class design concern rather than an afterthought.
NUS's contribution fits into a broader research wave — alongside PIM-enabled accelerators, CXL-attached memory pools, and disaggregated memory architectures — that collectively aims to break the memory wall for generative AI.
Implications
If validated at scale, heterogeneous memory chiplets could enable:
The research underscores a key insight for 2026 and beyond: as compute increasingly outpaces memory, the next wave of AI acceleration will be won not just by faster math, but by smarter memory organization.
Source: National University of Singapore (NUS). For the full technical details, refer to the original publication.