Tsinghua AI Open-Sources AKV: A Cache Engineering Server for LLM Agents on SGLang
Researchers at Tsinghua University and CUHK have open-sourced AKV, a serving system for KV cache engineering in LLM agents. Rather than summarizing old conversation history, AKV-enabled agents selectively drop KV cache entries and keep reusing the rest—an approach that preserves fidelity while cutting inference costs.
Across three models and four agent benchmarks, AKV reduces average cost by 27.7% while raising average task scores by 1.7 points, compared with threshold-based summarization. The code is released under the Apache-2.0 license and builds on SGLang.
TL;DR
- What it is: A serving system plus a policy library—no model weights included. Tested on GPT-OSS-120B (5.1B active parameters), MiniMax-M2.7 (230B total, 10B active), and MiroThinker-1.7-Mini (30B total, 3B active).
- Runs on: NVIDIA GPUs via SGLang. The paper used 8× A800-80GB GPUs in BF16; the quick start serves GPT-OSS-120B on 2 GPUs with tensor parallelism of 2.
- Performance: 27.7% lower average cost and +1.7 points average score versus threshold summarization.
- Best result: GPT-OSS-120B on SWE-bench Verified—cost per task drops from 20.0 to 9.7 cents while the score rises from 40.0 to 43.2.
- Worst result: MiniMax-M2.7 on BrowseComp-Plus scores 58.9, 3.4 points below Text Engineering's 62.3.
What Is Cache Engineering, and How Does It Differ from Context Engineering?
Context engineering—the discipline of deciding what an agent sees at each step—became a dominant design pattern through 2025 as long-horizon agents pushed context windows to their limits. Cache engineering tackles the same problem one layer lower: instead of rewriting the prompt, it manages the KV cache directly, deciding which cached tokens to retain, evict, or recompute.
In practice, this means agents can discard selected KV cache entries tied to stale or low-value history while continuing to reuse the remainder. The result is a smaller effective memory footprint and cheaper prefill, without the information loss that lossy summarization introduces.
How AKV Fits into the 2026 Agent Stack
As of 2026, agentic workloads dominate inference spend, and KV cache management has become a first-class infrastructure concern alongside prefix caching and disaggregated prefill/decode. AKV slots into this landscape as a policy-driven layer on top of SGLang, giving teams a tunable knob between aggressive context truncation (cheap but lossy) and full-history retention (accurate but expensive).
Because AKV ships as a serving system rather than a fine-tuned model, it can be adopted without retraining and composed with existing SGLang optimizations. The Apache-2.0 license also makes it straightforward to integrate into commercial agent platforms.
Getting Started
AKV is available on GitHub at thu-nics/AKV, with the accompanying paper on arXiv. Teams already running SGLang can evaluate the policy library against their own agent traces to measure cost and accuracy trade-offs before committing to a deployment.
Source: Tsinghua University and CUHK research, released October 2026.
via MarkTechPost