Counting Events, Not Seeing Them: Diagnosing and Repairing the Readout Failure in Video LLMs
arXiv:2611.09949 [cs.CV] | Submitted 21 October 2026 | Dongwu Qian
Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Machine Learning (cs.LG)
Abstract
Video LLMs fail conspicuously at counting how many times an object crosses the screen—a failure commonly attributed to missing object permanence. Using construction-grounded synthetic testbeds with independently validated labels, we show this attribution is incomplete: the count signal is present but unread. Linear probes decode per-window presence at 93%, while global count decoding collapses to a duration confound. Concurrent work reports the same representation–output dissociation for repetition counting; we show that in exit-and-reappear event counting this failure is repairable. Two readout-directed repairs—episode-localization supervision and a discrete event-symbol side channel—raise same-object counting from 46.7% to 65.0% (oracle 66.7%) on a chance-33% benchmark across 7B–30B systems. Transfer to an independent factory costs nothing, and an audit of the measurement apparatus exposes four data-integrity failures that had obscured the boundary.
A real-video audit (REALCOUNT, n=18, blind-audited) shows the boundary persists—best 55.6% at chance 33.3%—and symbol transfer fails zero-shot. A 1-epoch in-domain adaptation bench is negative (six levers at 28–50%), but a 3-epoch schedule shows this is budget-limited for the symbol channel: with a 3-epoch schedule and sequence-disjoint oracle timelines, the reader reaches 94.4% on REALCOUNT, and with the deployed detector's timelines a 2-epoch run reaches 72.2%. The residual gap is detector evidence and the reader's match to detector noise, not reader capacity: a frozen-feature boundary probe finds the signal weak (AUC 0.62 over a 0.51 permutation control) and non-transferring across clips (0.77 within-clip vs. 0.62 across windows).
Key Findings
Diagnosing the Failure
- Video LLMs struggle to count objects crossing the screen, conventionally blamed on missing object permanence.
- This attribution is incomplete: the count signal exists in the model's representations but is not correctly read out.
- Linear probing recovers per-window object presence at 93% accuracy, yet global count decoding collapses to a duration confound.
- The pattern mirrors the representation–output dissociation reported in concurrent repetition-counting work.
Repairing the Failure
Two readout-directed interventions were tested across 7B–30B systems on a chance-33% benchmark:
| Method | Same-Object Counting Accuracy |
|---|---|
| Baseline | 46.7% |
| Episode-localization supervision + event-symbol side channel | 65.0% |
| Oracle ceiling | 66.7% |
- Transfer to an independent factory incurs no cost.
- An audit of the measurement apparatus uncovered four data-integrity failures that had obscured the performance boundary.
Real-Video Validation (REALCOUNT)
- Blind-audited set (n=18): the boundary persists, with the best result at 55.6% against a 33.3% chance baseline.
- Zero-shot symbol transfer fails.
- A 1-epoch in-domain adaptation bench is negative (six levers at 28–50%).
Budget and Capacity Analysis
- The symbol channel is budget-limited, not capacity-limited:
- 3-epoch schedule + sequence-disjoint oracle timelines: 94.4% on REALCOUNT.
- 2-epoch run with deployed detector timelines: 72.2%.
- The residual gap stems from detector evidence and the reader's match to detector noise—not reader capacity.
- A frozen-feature boundary probe finds the signal weak (AUC 0.62 vs. a 0.51 permutation control) and non-transferring across clips (0.77 within-clip vs. 0.62 across windows).
Metadata
- Comments: Preprint v1 (2026-10-21). 11 pages, 4 figures, 6 tables. A corrected-prompt retrain of the REALCOUNT adaptation arms is in progress; v2 will update those results. Comments welcome.
- Cite as: arXiv:2611.09949 [cs.CV]
- DOI: https://doi.org/10.48550/arXiv.2611.09949
via ArXiv CV