TraCE: Trajectory-Aware Concepts for Explainable Video Understanding
Authors: Wooil Lee, Junpyo Hong, Jongseo Lee, Tuomas Oikarinen, Tsui-Wei Weng, Jinwoo Choi
arXiv: 2611.07009 [cs.CV] — Submitted 4 Nov 2026
Venue: Accepted to NeurIPS 2026
Project page: https://leewooil.github.io/TraCEprojectpage/
Overview
Video understanding models often rely on motion, yet their explanations can obscure which motion sources actually support a prediction. The authors identify motion-source under-specification as a key limitation of existing video explainable AI (XAI): region-centric explanations can be source-ambiguous, while motion vocabularies restricted to single-person pose sequences are source-incomplete.
To address this, they propose TraCE (Trajectory-Aware Concepts for Explainable Video Understanding), a concept-based framework that discovers source-flexible motion concepts by clustering optical-flow trajectories from training videos. These concepts capture dynamic evidence from people, objects, and interactions, while object and scene concepts account for static context.
Method
The resulting typed trajectory/object/scene interface supports:
- Class prediction for action recognition
- Question-conditioned concept matching for video question answering (VQA)
Evaluation
TraCE was evaluated across five random seeds on six benchmarks: NTU RGB+D Mutual, Chiral SSv2, UCF-101, KTH, TOMATO, and STAR.
Relative to non-interpretable baselines, TraCE maintains competitive task performance:
- Action recognition: average accuracy drop of 1.0 points
- VQA: nominal gain of 0.4 points
Motion perturbation and concept-removal tests provide complementary evidence of directional semantic alignment and decision faithfulness, while sample-level interventions illustrate targeted model debugging.
Significance
As video XAI matures in 2026, TraCE's contribution is notable for closing the gap between interpretability and performance: the framework demonstrates that source-aware, trajectory-grounded concept explanations can be achieved without sacrificing downstream task accuracy. Its typed interface design also unifies two previously separate tasks—action recognition and video question answering—under a single interpretable concept vocabulary.
via ArXiv CV