VisAlloc: Allocating Visual Supervision for On-Policy Self-Distillation
Overview
On-policy self-distillation (OPSD) has emerged as a promising technique for improving visual reasoning in multimodal large language models (MLLMs). However, not all training samples benefit equally from privileged visual guidance, and uniform token-level supervision can dilute visually grounded signals. In 2026, as MLLMs scale toward increasingly agentic and perception-heavy applications, allocating supervision more precisely has become a central concern.
To address this, we propose VisAlloc, a dual-level framework that combines sample-level visual filtering with token-level counterfactual reweighting.
Method
Sample-Level Visual Filtering
VisAlloc employs a fixed paired-teacher filter that retains only those samples answered correctly from an evidence-centered positive crop but incorrectly from a spatially displaced negative crop. This ensures that supervision is applied to samples where visual grounding demonstrably matters.
Token-Level Counterfactual Reweighting
At the token level, two EMA teachers score the same student-generated response, and their absolute log-probability differences serve as a measure of view sensitivity. Additional supervision is directed toward tokens admitted by either sensitivity or teacher-distribution disagreement, subject to shared frozen-prior, template-bigram, and student–teacher divergence gates.
An inverse-frequency redistribution preserves the total additional weight within each micro-batch, while the positive teacher remains the sole distillation target. Inference requires only the full-image student, adding no overhead at deployment.
Results
Across eight benchmarks, our 4B and 9B models achieve macro-averages of 81.44 and 84.92, outperforming Vision-OPD reproductions by 0.97 and 0.52 points, respectively.
Availability
Code is available on GitHub: https://github.com/Kaelyn01/VisAlloc
Paper Details
- arXiv: 2611.25069 [cs.CV]
- Submitted: 18 November 2026
- Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
- DOI: https://doi.org/10.48550/arXiv.2611.25069
- Authors: Haoran Xu, Tian Chen, Yuhe Wu, Yanbo Zhao, Junchi Liao, Hongyu Wang, Jiaze Li, Wentao Wang, Yanfeng Li, Rui Qian
via ArXiv CV
