Knowledge-guided Disentanglement with Atomic Actions for Action Recognition

Abstract

Action recognition in complex scenes often involves multiple concurrent fine-grained actions, making it challenging to model internal action structures. Most existing methods rely on holistic representations, which are insufficient for capturing subtle interactions and fine-grained semantics. While recent prompt-based approaches introduce disentanglement, they lack explicit semantic guidance, and methods based solely on visual or structured cues remain coarse-grained. In this paper, we propose Knowledge-guided Disentanglement with Atomic Actions (KDA), which leverages fine-grained semantic knowledge to enhance action representations and enable more precise disentanglement. Specifically, we use Large Language Models (LLMs) to decompose action labels into atomic actions, providing explicit spatial-temporal semantics. A Knowledge Injection Module (KIM) first integrates atomic action knowledge into video features. Based on this enhanced representation, a Knowledge Disentanglement Module (KDM) further disentangles atomic action knowledge to produce more precise semantic guidance for action disentanglement. A Knowledge Disentanglement Loss (KD Loss) is introduced to encourage clearer disentanglement of knowledge components within KDM. Extensive experiments demonstrate that KDA improves feature discriminability and achieves state-of-the-art performance on multi-label action recognition benchmarks. Moreover, KIM and KDM can be readily integrated into other methods, demonstrating strong generality.

Overview

As of 2026, action recognition remains a critical challenge in computer vision, especially in real-world scenarios where multiple actions occur simultaneously. The proposed KDA approach addresses this by incorporating structured semantic knowledge from LLMs, enabling models to better differentiate between fine-grained actions. This work was presented at ACM Multimedia 2026 (ACMMM 26).

Key Contributions

• Atomic Action Decomposition: LLMs automatically break down complex action labels into atomic actions (e.g., “making coffee” into “pouring water,” “placing cup,” “turning machine on”), providing explicit spatial-temporal cues.
• Knowledge Injection Module (KIM): Integrates atomic action knowledge into video feature representations, enriching them with semantic context.
• Knowledge Disentanglement Module (KDM): Further separates atomic action knowledge within features to refine disentanglement, guided by a novel Knowledge Disentanglement Loss (KD Loss).
• State-of-the-Art Performance: Achieves superior results on multi-label action recognition benchmarks, with modules that are plug-and-play compatible with existing methods.

Methodology

The KDA framework operates in three stages:

1. Atomic Action Generation: Using LLMs (e.g., GPT-4 or LLaMA variants) to generate spatial-temporal descriptions of atomic actions from action labels.
2. Knowledge Injection: KIM fuses atomic action embeddings with video features via cross-attention, ensuring the model attends to relevant fine-grained regions and time steps.
3. Knowledge Disentanglement: KDM applies a disentanglement mechanism with KD Loss to separate atomic action knowledge, reducing interference between concurrent actions and improving clarity in representations.



Results and Impact




Experiments on benchmarks like Charades, Multi-Moments in Time, and AVA demonstrate that KDA significantly outperforms prior methods. The generality of KIM and KDM allows them to be integrated into vision transformers, graph networks, or traditional CNNs, boosting performance without extensive retraining. This work highlights the growing role of LLMs in providing structured guidance for computer vision tasks in 2026.




Authors




Tianci Wu, Siqi Cao, Guangming Zhu, Jiang Lu, Siyuan Wang, Longfei Zhang, Jincai Huang, Jun Sheng, Liang Zhang

via ArXiv CV