Human-Centric Intelligence in the Era of Foundation Models: A Survey

Human-Centric Intelligence in the Era of Foundation Models: A Survey

Authors: Yang Chen, Tianqi Wang, Xiaorui Jiang, Yilei Man, Yihua Shao, Mengyuan Liu, Zhi Chen, Xiaofeng Cao, Qibin Zhao, Chi Harold Liu, Albert Y. Zomaya, Nicu Sebe, Jingren Zhou, Dacheng Tao, Song Guo, Jingcai Guo

Submitted: August 18, 2026

Abstract

Human-centric intelligence is undergoing a transformation in the era of foundation models, driven by an increasing emphasis on scale, transferability, and general-purpose modeling. Despite this momentum, the field has yet to fully integrate with foundation models to achieve the level of progress witnessed in other domains. Moreover, recent advancements across the landscape remain fragmented across tasks, modalities, and research communities, obscuring the intrinsic conceptual and methodological connections. To bridge these divides and rethink human-centric intelligence in the foundation-model era, this paper introduces a full-spectrum human context taxonomy that integrates six interconnected levels—treating humans as observable subjects through visual appearance and spatial geometry, as dynamic actors through kinematic dynamics and interaction modeling, and as situated agents through world simulation and embodied agency. We then present the methodological foundations of the field, covering human-centric data families, computational architecture paradigms, and representative training and inference optimization strategies. Subsequently, we systematically review representative methods across these levels and organize the associated datasets, benchmarks, and evaluation metrics. We further discuss open challenges and promising research directions toward human-centric intelligence that is scalable, trustworthy, physically grounded, and deployable, aiming to provide a coherent framework and practical reference for advancing the field. Finally, we offer a systematically organized and continuously updated collection of human-centric AI literature and resources on our project page.

1. Introduction

The rapid advancement of foundation models—large-scale pre-trained architectures capable of generalizing across tasks—has reshaped the landscape of artificial intelligence. In fields such as natural language processing and computer vision, these models have demonstrated remarkable capabilities in transfer learning, few-shot adaptation, and multi-modal reasoning. However, human-centric intelligence, which focuses on understanding and modeling humans in their physical and social contexts, has not yet fully leveraged these advances. This gap stems from the inherent complexity and diversity of human-centric tasks, which span appearance recognition, motion analysis, interaction understanding, and embodiment in real or simulated environments.

As of 2026, the proliferation of foundation models has opened new avenues for human-centric intelligence, yet the field remains fragmented. Existing research is scattered across subcommunities—some focusing on static attributes like pose or appearance, others on dynamic behaviors such as gestures or social interactions, and still others on higher-level agency in simulated worlds. This fragmentation hinders the cross-pollination of ideas and the development of unified frameworks.

To address this, we propose a comprehensive taxonomy that organizes human-centric intelligence into six interconnected levels. This taxonomy serves as a lens to structure our survey, enabling us to (1) identify methodological commonalities across seemingly disparate tasks, (2) trace the evolution of techniques from traditional approaches to foundation-model-based methods, and (3) highlight emerging directions that integrate physical grounding and embodied agency.

The remainder of this paper is organized as follows. Section 2 introduces the full-spectrum human context taxonomy. Section 3 details methodological foundations, including data, architectures, and optimization strategies. Section 4 provides a systematic review of representative methods across the taxonomy levels. Section 5 compiles datasets, benchmarks, and evaluation metrics. Section 6 discusses open challenges and future research directions. Section 7 concludes the survey.

2. A Full-Spectrum Human Context Taxonomy

To bridge the divides in human-centric intelligence, we propose a taxonomy that views humans through three dual lenses:

• Observable Subjects: At the most basic level, humans are described by their visual appearance (e.g., body shape, clothing, facial features) and spatial geometry (e.g., 3D body pose, location in space). This level grounds all higher-level understanding.
• Dynamic Actors: Humans are not static entities; they move, act, and interact. This level encompasses kinematic dynamics (e.g., gait, gesture, motion patterns) and interaction modeling (e.g., human-object, human-human, and human-environment interactions).
• Situated Agents: Beyond immediate perception, humans are situated in rich environments and can act purposefully. This level includes world simulation (e.g., virtual environments that model physical and social rules) and embodied agency (e.g., agents that perceive, reason, and act in physical or simulated worlds).

These six levels are not mutually exclusive; they form a hierarchy of increasing complexity and abstraction, from low-level perception to high-level cognition and action. This taxonomy allows us to classify existing methods, identify gaps, and guide future research.

3. Methodological Foundations

A unified understanding of human-centric intelligence requires a solid methodological base. We identify three key pillars:

3.1 Human-Centric Data Families

Data is the cornerstone of modern AI. Human-centric data spans a wide spectrum, including:

• 2D and 3D visual data: images, videos, depth maps, and point clouds capturing human appearance and geometry.
• Motion capture data: precise kinematic recordings from marker-based or markerless systems.
• Interaction data: recordings of human-object and human-human interactions, often with semantic annotations.
• Simulated data: synthetically generated environments and agents, crucial for training embodied models.
• Multimodal data: combinations of visual, textual, audio, and sensor data, enabling richer representations.

In 2026, datasets have grown in scale and diversity, but challenges remain in data collection ethics, annotation quality, and domain generalization.

3.2 Computational Architecture Paradigms

Architectures for human-centric intelligence have evolved from hand-crafted features and shallow models to deep neural networks, and now to foundation models. Current paradigms include:

• Transformers and variants: self-attention mechanisms that capture long-range dependencies, essential for modeling spatial and temporal relationships.
• Graph neural networks: useful for modeling structured relationships, such as human skeletons or object graphs.
• Diffusion models: emerging for generative tasks, such as realistic human motion synthesis.
• Multimodal architectures: designed to fuse heterogeneous inputs, aligning visual, textual, and sensor data.

Given the compute demands of foundation models, efficient architectures and model compression techniques are also critical for real-world deployment.

3.3 Training and Inference Optimization

To fully exploit foundation models in human-centric tasks, specific strategies are employed:

• Pre-training and fine-tuning: leveraging large-scale pre-trained models and adapting them to human-centric downstream tasks.
• Prompting and in-context learning: using language or visual prompts to guide model behavior without extensive fine-tuning.
• Few-shot and zero-shot learning: enabling generalization to novel tasks with minimal labeled data.
• Knowledge distillation and model compression: transferring knowledge from large teacher models to compact student models, facilitating edge deployment.

4. Systematic Review of Methods

We organize representative methods according to our taxonomy levels:

• Visual Appearance and Spatial Geometry: Methods for 2D and 3D human pose estimation, body shape recovery, and cross-view alignment. Recent works leverage foundation models for improved generalization across domains.
• Kinematic Dynamics and Interaction Modeling: Approaches for action recognition, motion prediction, and social interaction analysis. Foundation models are used to capture complex temporal dependencies and contextual cues.
• World Simulation and Embodied Agency: Techniques for generating realistic virtual environments (e.g., neural rendering) and developing embodied agents that interact with these worlds. Simulation-to-real transfer is a key focus.

For each level, we highlight how foundation models have been integrated, the benefits gained, and the remaining limitations.

5. Datasets, Benchmarks, and Evaluation Metrics

We provide a curated list of existing datasets and benchmarks corresponding to each taxonomy level, along with discussion of common evaluation metrics (e.g., accuracy, F1 score, mean per-joint position error, success rate in embodied tasks). We also highlight gaps, such as lack of diverse, ethically sourced data.

6. Open Challenges and Future Directions

Despite progress, several challenges remain:

• Scalability: Developing methods that can handle the complexity of human-centric data at scale.
• Trustworthiness: Ensuring fairness, privacy, and robustness in human-centric AI systems, given the sensitive nature of human data.
• Physical Grounding: Connecting models to real-world physics to improve accuracy and safety in physical interactions.
• Deployability: Enabling real-time, low-latency inference on resource-constrained devices.

Future research should focus on unified frameworks that integrate multiple levels, continual learning for adapting to evolving human contexts, and collaborative systems that combine human and machine intelligence.

7. Conclusion

This survey provides a comprehensive and structured overview of human-centric intelligence in the foundation-model era. By proposing a full-spectrum taxonomy, we aim to unify fragmented research efforts and facilitate future progress. We hope this work serves as a practical reference for researchers and practitioners, and we invite the community to engage with our continuously updated project page.

References and Resources

For a complete list of references and a continuously updated collection of literature and resources, please visit our project page (URL provided on the paper's arXiv page).

via ArXiv CV