Modeling Human Behavior in the Prisoner's Dilemma with Large Language Models
Authors: Raina Gao, Tristan Raffo
Subjects: Computation and Language (cs.CL); Computer Science and Game Theory (cs.GT)
Cite as: arXiv:2610.23860 [cs.CL]
Submitted: 10 August 2026
DOI: https://doi.org/10.48550/arXiv.2610.23860
Abstract
The Prisoner's Dilemma has long served as a foundational framework for studying human decision-making and cooperative outcomes across social scenarios, yet human behavior consistently deviates from the predictions of fully rational economic theory. Previous work has examined human and LLM outcomes in the Prisoner's Dilemma, but these studies have each lacked at least one key feature: a memory component for LLM agents, accessible open-source models, or comparison against empirical human data.
This project investigates whether open-source LLMs—TII Falcon 7B Instruct, Google Gemma 7B, Meta Llama 3.1 8B Instruct, Microsoft Phi Mini MoE Instruct, and Qwen 2.5 7B Instruct—can model human cooperation patterns in repeated Prisoner's Dilemma games. It also examines how an LLM's knowledge that its opponent will be shuffled affects its decisions.
Using a standardized prompting framework, we simulated interactions in two trial types:
- Fixed-partner setting: both agents retain memory across rounds.
- Shuffled-partner setting: a memory-enabled agent, informed that its opponent changes each round, interacts with a memoryless agent.
For each trial, we measured cooperation rates over 100 rounds and in 10-round increments, and compared them to human data from Montero-Porras et al. (2022).
Key Findings
Under both frameworks, all LLMs differed from the human cooperation rate. Open-source LLMs generally over-cooperated relative to humans—particularly in the shuffled framework, where human cooperation averaged 28.5% while several models exceeded 80%. Qwen-2.5-7B-Instruct showed deterministic tendencies, whereas Phi-mini-MoE-Instruct displayed greater variability and briefly approximated human rates.
These findings suggest that LLMs have potential as behavioral simulation agents and provide a benchmark across two interaction environments, although open-source models are not yet reliable substitutes for human participants.
2026 Context
As open-source instruction-tuned models have matured through 2025–2026, researchers increasingly probe their capacity for social and economic reasoning. This study contributes a dual-environment benchmark for cooperative behavior, highlighting a persistent "over-cooperation" gap that remains relevant as LLMs are proposed as stand-ins for human participants in behavioral and game-theoretic experiments.
Metadata
- Primary subject: Computation and Language (cs.CL)
- Cross-list: Computer Science and Game Theory (cs.GT)
- Version: arXiv:2610.23860v1 [cs.CL]
- License/DOI: https://doi.org/10.48550/arXiv.2610.23860
via ArXiv CL+LG