Overview
Large language model (LLM) agents can solve complex interactive tasks, yet reasoning and tool calls take time. In continuously changing environments, an otherwise correct action may arrive too late. This paper introduces RealtimeBench, a benchmark of 180 tasks across six game types, designed to evaluate agents while games continue running during inference.
The Real-Time Gap
Paired evaluations reveal a striking reversal: GPT-6.1-sol outperforms the faster Jev when games are paused, but falls behind when games keep running. In other words, raw capability does not translate into real-time effectiveness—deliberation latency becomes a first-class obstacle in dynamic settings.
Two-Timescale Offline Recursive Self-Improvement (TORSI)
To close this gap, the authors propose Two-Timescale Offline Recursive Self-Improvement (TORSI), a method that separates deliberation from action across timescales:
- Slow teacher: Between attempts, the teacher studies failures and tests corrections in paused games.
- Reusable instructions: The teacher writes instructions distilled from those corrections.
- Fast student: During gameplay, the student acts without teacher calls, following the learned instructions.
- Validated adoption: Validation scores determine which updates are adopted.
Results
Two rounds of TORSI raise Jev's average test score from 13.51 to 24.19 on a 0–100 progress scale, improving performance across all six game types. Notably, the resulting instructions also improve average scores for three other models without retuning—suggesting the learned guidance generalizes beyond the student it was written for.
Toward Real-Time Intelligence
These findings point toward real-time intelligence: enabling powerful LLM agents to remain effective in a continuously changing world by organizing deliberation, learning, and action across timescales.
Paper Details
- Title: The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock
- Authors: Ningxin Pan, Hanyu Li, Yeqing Shen, Ning Ding, Yehui Tang, Wentao Zhang
- Subjects: Artificial Intelligence (cs.AI)
- Submitted: 14 Oct 2026
- Cite as: arXiv:2610.20665 [cs.AI]
- DOI: https://doi.org/10.48550/arXiv.2610.20665
via ArXiv AI