VidEngram: Agentic Long-Term Video Understanding via Structured Hierarchical Memory
Authors: Zinuo Cheng, Yueqian Lin, Zhongxi Wang, Hai "Helen" Li, Yiran Chen
Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Multimedia (cs.MM)
arXiv: 2611.18578 [cs.CV] — Submitted 28 Jul 2026
Comments: 10 pages, 4 figures, 4 tables. Demo video and code are linked in the paper
The Problem: Re-Encoding Video for Every Query
Long-video question answering is wasteful and lossy when a multimodal model must re-encode the same video for every query. Context limits force sparse sampling or truncation, and no reusable representation remains for later questions. This cost compounds in 2026-era deployments, where long-form video—lectures, surveillance archives, meeting recordings, and livestreams—is increasingly the dominant input modality, yet inference budgets remain tightly constrained.
The Approach: Build Memory Once, Query Many Times
We present VidEngram, an agentic video QA system that builds a persistent hierarchical memory once at ingest and answers later questions from that memory. The pipeline works as follows:
- Joint multimodal captioning. VidEngram captions video and audio jointly with Qwen2.5-Omni.
- Hierarchical consolidation. Segment memories are consolidated into episode and entity layers, forming a structured, reusable representation.
- Agentic planning and retrieval. A text-only agentic planner, equipped with specialized retrieval tools, produces grounded answers from the stored memory.
Because perception is paid once, subsequent queries rely only on low-cost text-only planning—a design aligned with the broader 2026 shift toward amortized inference and persistent agent memory.
Inspectable by Design
The web interface makes this process transparent: users can watch memory formation, inspect retrieval traces, verify cited evidence in the video player, and explore entity relationships.
Results
On the full OmniVideoBench, VidEngram with a GPT-5-mini planner outperforms a strong single-pass Qwen2.5-Omni baseline (37.0% vs. 32.3%). Gains are concentrated on longer videos, where single-pass coverage becomes sparse—precisely the regime in which reusable memory matters most.
Availability
The paper includes a demo video and code links. Full text and metadata are available via arXiv:2611.18578 [cs.CV].
via ArXiv CV
