KVBoost: Deviation-Guided Chunk-Level KV Cache Reuse for Efficient LLM Inference

Transformer-based large language models (LLMs) face significant prefill latency challenges because key-value (KV) tensors must be recomputed for each incoming request. While existing prefix-caching systems mitigate this overhead, they require prompts to share a leading contiguous prefix—a constraint that severely limits their effectiveness when relevant content appears at arbitrary positions within the prompt.

Introducing KVBoost

This paper presents KVBoost, a chunk-level KV cache reuse system designed for HuggingFace-compatible decoder models that enables cache reuse regardless of content position. By decoupling positional identity from content identity through a dual-hash keying scheme, KVBoost supports both exact and approximate cache matches, overcoming the rigid constraints of traditional prefix caching.

Technical Approach

To mitigate attention boundary errors that arise from independently cached chunks, KVBoost implements two complementary repair strategies: SelectiveRecompute, which re-encodes boundary regions, and CacheBlendRecompute, which identifies and recomputes high-deviation tokens following a probe pass. The system further integrates asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction under a fixed memory budget—making it particularly well-suited for edge and resource-constrained deployment scenarios common in 2026 production environments.

Evaluation & Results

Evaluated on the Qwen/Qwen2.5-3B model across 1,000 bug-localization samples, KVBoost achieves a 4.49x reduction in time-to-first-token (142.4 ms vs. 639.1 ms baseline) and outperforms prefix caching by 16%, while maintaining accuracy parity (99.2% vs. 99.1%). The system delivers a practical, memory-bounded inference acceleration layer that is fully compatible with RoPE-based models without requiring architectural modifications.

via ArXiv AI