Clustering and Token Denoising for Faster and More Robust VLMs
Baptiste Rossigneux, Inna Kucher, Vincent Lorrain, Emmanuel Casseau
Submitted: August 19, 2026 (arXiv:2608.19285)
Abstract
Recent visual-language models (VLMs) have significantly extended the capabilities of pretrained large language models (LLMs) by integrating visual tokens alongside text. Approaches such as LLaVA have demonstrated impressive results, but the computational cost of processing up to 576–729 visual tokens poses a major challenge for edge deployment. While some token pruning techniques require retraining, many are training-free and can adapt to architecture changes seamlessly. In this work, we introduce ClustRS, a two-part, training-free algorithm for robust token pruning. The first component employs an attention-weighted clustering algorithm that selects representative tokens from each semantic cluster. The second component, Residual Shrinkage, applies a one-pass denoising step to the selected tokens. These lightweight, training-free steps make LLaVA suitable for real-world data, improving robustness across a wide range of image-noise types and intensities. Our experiments on the ScienceQA-IMG and MM-VET benchmarks show that ClustRS outperforms attention-based and diversity-based methods by up to 20% under extreme noise and token conditions—reducing tokens by 97% (down to 16 tokens) on LLaVA 1.5 7B—and achieves exceptional results on LLaVA-OneVision, matching baseline performance with fewer than one-third of the tokens under mild noise conditions. This study highlights a simple yet powerful alternative to pruning rules that rely solely on scores or diversity, paving the way for compute-efficient and noise-resilient VLM deployment in real-world settings. As of 2026, the demand for efficient VLM deployment at the edge continues to grow, with a focus on minimizing latency and energy consumption while preserving accuracy under noisy conditions. Our approach directly addresses these needs, offering a practical solution for on-device inference in applications such as autonomous systems, smart cameras, and mobile assistants.
via ArXiv CV