VidEngram: Agentic Long-Term Video Understanding via Structured

VidEngram: Agentic Long-Term Video Understanding via Structured Hierarchical Memory


Authors: Zinuo Cheng, Yueqian Lin, Zhongxi Wang, Hai "Helen" Li, Yiran Chen

Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Multimedia (cs.MM)

arXiv: 2611.18578 [cs.CV] — Submitted 28 Jul 2026

Comments: 10 pages, 4 figures, 4 tables. Demo video and code are linked in the paper


The Problem: Re-Encoding Video for Every Query


Long-video question answering is wasteful and lossy when a multimodal model must re-encode the same video for every query. Context limits force sparse sampling or truncation, and no reusable representation remains for later questions. This cost compounds in 2026-era deployments, where long-form video—lectures, surveillance archives, meeting recordings, and livestreams—is increasingly the dominant input modality, yet inference budgets remain tightly constrained.


The Approach: Build Memory Once, Query Many Times


We present VidEngram, an agentic video QA system that builds a persistent hierarchical memory once at ingest and answers later questions from that memory. The pipeline works as follows:


  1. Joint multimodal captioning. VidEngram captions video and audio jointly with Qwen2.5-Omni.
  2. Hierarchical consolidation. Segment memories are consolidated into episode and entity layers, forming a structured, reusable representation.
  3. Agentic planning and retrieval. A text-only agentic planner, equipped with specialized retrieval tools, produces grounded answers from the stored memory.

  4. Because perception is paid once, subsequent queries rely only on low-cost text-only planning—a design aligned with the broader 2026 shift toward amortized inference and persistent agent memory.


    Inspectable by Design


    The web interface makes this process transparent: users can watch memory formation, inspect retrieval traces, verify cited evidence in the video player, and explore entity relationships.


    Results


    On the full OmniVideoBench, VidEngram with a GPT-5-mini planner outperforms a strong single-pass Qwen2.5-Omni baseline (37.0% vs. 32.3%). Gains are concentrated on longer videos, where single-pass coverage becomes sparse—precisely the regime in which reusable memory matters most.


    Availability


    The paper includes a demo video and code links. Full text and metadata are available via arXiv:2611.18578 [cs.CV].


    via ArXiv CV

Related