VAR-Q: Tuning-Free KV Cache Quantization for Visual Autoregressive Image and Video Generation

Authors: Boxun Xu, Jiaji Lu, Yuxuan Yin, Zihu Wang, Ziyue Liu, Yu Wang, Zirui Liu, Peng Li

arXiv: 2611.05380 [cs.CV] · Submitted 12 July 2026

Subjects: Computer Vision and Pattern Recognition (cs.CV); Performance (cs.PF)

Comments: A preliminary version of this work was presented at the ICCV 2025 BiVision Workshop

Abstract

Visual autoregressive generation spans two dominant paradigms: next-scale prediction for images and video, and next-frame prediction for video. Both decode block-by-block rather than token-by-token, yielding KV caches whose statistics diverge from those of language models. At scale, KV memory exceeds the weights and limits large-scale deployment, driven by the large number of visual tokens. Our analysis of these caches reveals three findings.

First, a small set of massive-value channels carries the functional load: ablating these outlier channels collapses generation, whereas ablating a matched set of inlier channels leaves quality intact. Second, the outliers organize along two axes—feature and block—with magnitudes persistent within each cell and shifting across cells. Third, this structure arises from two independent mechanisms: projection bias and RoPE-induced scattering.

Guided by these findings, we propose VAR-Q, a tuning-free KV-cache quantization framework that groups along the same two axes. The generation loop provides block boundaries for free, removing the need for outlier detection or calibration. Paired with bit-packing kernels for runtime memory reduction, VAR-Q delivers low-bit quantization across nine models spanning both paradigms, cutting KV memory by up to 78.5% with negligible quality degradation and enabling 6x larger inference batches.

Key Contributions at a Glance

• Problem: Block-by-block decoding in visual autoregressive models produces KV caches with statistics distinct from those of language models, and their memory footprint can exceed model weights at scale.
• Insight: A small set of massive-value outlier channels is functionally critical; these outliers are structured along feature and block axes and originate from projection bias and RoPE-induced scattering.
• Method: VAR-Q, a tuning-free quantization framework that groups along the same two axes, leveraging free block boundaries from the generation loop and bit-packing kernels for runtime memory savings.
• Results: Low-bit quantization across nine models and both paradigms, up to 78.5% KV memory reduction with negligible quality loss, and 6x larger inference batches.

via ArXiv CV