Nunchux AI has released VC-Attention, a training-free low-bit attention kernel designed specifically for video Diffusion Transformers (DiTs). It addresses two bottlenecks simultaneously: value quantization error and the slow softmax stage.
Why Attention Is the Video Bottleneck
Video DiTs flatten a clip into a single sequence of spatiotemporal tokens and run full self-attention at every layer. A 5-second 720p Wan2.2-14B clip spans roughly 70K tokens. On the RTX 5090, attention accounts for more than 64% of generation time. The research team reports that attention consumes about two-thirds of every MiniMax-H3 denoising step on a single B200.
Low-bit Tensor Cores accelerate the two matrix products—QK and PV—but two obstacles remain. First, prior methods like SageAttention2 smooth queries and keys. After QK smoothing and rotation, the value term contributes 82% of output error on Wan2.2. Second, the softmax between the products still runs in FP32. On B200 and H200 hardware, that exponential and its FP8 cast become the longest pipeline stage.
V-Smooth: Fixing Value Outliers
Value outliers sit in a few tokens, and their channels shift across heads, layers, and steps. A Hadamard rotation preserves token norms, so it does not remove them—rotating V changes value error by just 0.2%.
V-Smooth takes a different route:
- Group: An online k-means clusters value tokens per batch and head. Keys and values are permuted together, so non-causal attention output is unchanged.
- Demean: Each 128-token hardware block subtracts its mean. Only the residual is quantized, using per-channel E4M3 at 8 bits or NVFP4 at 4 bits.
- Restore: The mean is added back using the row sum already kept by online softmax. No second pass or extra buffer is needed.
Averaged over 100 Wan2.2 heads, the block mean removes 8% of block... [content truncated in source]
via MarkTechPost