Abstract
Precise emotion control in audio-driven talking heads remains a significant challenge, largely due to the reliance on implicit emotion regulation in existing systems, which often results in indirect and insufficient control. Moreover, training with explicit emotion-related losses across the entire motion space poses considerable difficulties, stemming from the inherent trade-off between accurate lip synchronization and fine-grained emotion control.
In this work, we present a key finding: although emotional cues are distributed throughout the motion space, concentrating discriminative supervision on less-principal components achieves a better balance between emotion expression and lip synchronization. This is because principal components primarily encode high-energy articulation and pose variations.
Building on this insight, we propose Xemo-Talker, which first learns a neutral speech-to-motion mapping for stable articulation and lip synchronization, then introduces a lightweight emotion branch guided by less-principal subspace supervision. To further enhance emotion control, we design a Tri-Loss comprising inter-class separation, intra-class compactness, and less-principal contrastive learning.
Given an audio input, a reference image, and an emotion label, Xemo-Talker achieves state-of-the-art emotion classification accuracy while maintaining competitive lip synchronization and high inference efficiency, with performance approaching that of real videos. The source code is publicly available at this GitHub repository.
1. Introduction
Audio-driven talking portrait synthesis has gained significant attention in computer vision, with applications in virtual avatars, digital humans, and human-computer interaction. While recent methods achieve impressive lip synchronization, fine-grained emotion control remains an open problem. Existing approaches often rely on implicit emotion regulation, leading to indirect and insufficient control that limits practical utility.
The core challenge lies in training with explicit emotion-related losses across the entire motion space. Directly applying such losses often degrades lip synchronization because principal motion components—dominated by articulation and pose—are sensitive to adversarial gradients. Our key insight is that emotional cues are distributed throughout the motion space, but discriminative supervision on less-principal components yields a better emotion-lip synchronization balance.
To address this, we introduce Xemo-Talker, a framework that decouples neutral speech-to-motion mapping from emotion control. This separation ensures stable articulation, while a lightweight emotion branch handles expressive nuances. Our contributions are threefold:
- We reveal that less-principal subspace supervision improves the emotion-lip synchronization trade-off, a novel finding that guides our design.
- We propose a Tri-Loss that enhances emotion discriminability through inter-class separation, intra-class compactness, and less-principal contrastive learning.
- Extensive experiments demonstrate state-of-the-art emotion classification accuracy with competitive lip sync and high efficiency.
2. Related Work
2.1 Audio-Driven Talking Head Synthesis
Prior work (e.g., Wav2Lip, MakeItTalk) focuses on lip synchronization using audio features, often neglecting emotional expressiveness. Recent efforts incorporate emotion via style vectors or auxiliary losses, but these are typically implicit and hard to control.
2.2 Emotion Control in Talking Heads
Some methods employ emotion labels or reference images, yet they either overfit to training identities or sacrifice lip sync. Our work differs by explicitly supervising emotion in a subspace that minimally interferes with articulation.
3. Method
3.1 Overview
Xemo-Talker consists of two stages: (1) a neutral speech-to-motion predictor, and (2) an emotion branch that refines motion based on a given emotion label.
3.2 Less-Principal Subspace Supervision
We perform PCA on motion features to identify principal components (high-energy, articulation-dominant) and less-principal components (emotion-relevant). We apply discriminative losses only on the latter to avoid degrading lip sync.
3.3 Tri-Loss Design
The Tri-Loss combines:
- Inter-class separation (e.g., contrastive or CE loss) to push different emotions apart.
- Intra-class compactness (e.g., center loss) to pull same-emotion samples together.
- Less-principal contrastive learning to enforce consistency in the emotion subspace.
3.4 Inference
At inference, the system takes audio, a reference image, and an emotion label, producing a talking head video with synchronized lips and controlled emotion.
4. Experiments
We evaluate on standard benchmarks (e.g., MEAD, HDTF) with metrics for emotion classification accuracy, lip sync (LSE-C), and inference speed. Xemo-Talker outperforms baselines in emotion control while maintaining competitive lip sync and real-time efficiency.
5. Conclusion
We presented Xemo-Talker, a novel framework that explicitly unlocks emotions in audio-driven talking heads. Our key finding—that less-principal subspace supervision improves the emotion-lip synchronization balance—offers a new direction for future research. The public release of our code supports reproducibility and further innovation.
References
[1] Chung et al. Wav2Lip. 2020.
[2] Zhou et al. MakeItTalk. 2020.
[3] Chen et al. MEAD dataset. 2020.
This paper appears in arXiv:2608.14700 [cs.CV].
via ArXiv CV