Overview
Tool-using large language model (LLM) agents often incur significant wall-clock time not only from model inference but also from serial action–observation turns. Each tool call, environment transition, and observation delays subsequent decisions, creating bottlenecks in real-time applications.
To address this, we introduce Speculative Macro Commit (SMC), a runtime mechanism designed for two-tier agent systems. In this architecture, a large authoritative actor model generates the official trajectory, while a faster speculative drafter model continuously predicts and executes future action chains on an isolated environment snapshot. SMC mines recurring multi-action patterns from training traces and stores them in a macro library, which is then used to match against the drafter's predicted action chains at runtime.
When the actor's next tool call aligns with the first drafted action, SMC commits the pre-executed draft steps—including their observations—to the official trajectory. This approach enables efficient reuse of multi-step predictions, minimizing redundant computations.
Experimental Results
We evaluated SMC using Qwen3.5-27B INT4 as the authoritative actor model and Qwen3.5-4B as the speculative drafter model. Key findings include:
- On the τ²-Bench Telecom subset, SMC matches the sequential agent's overall accuracy while reducing latency by 10.23% compared to the Speculative Actions (SA) baseline and 18.59% over sequential execution.
- On AppWorld, SMC reduces wall-clock time by 7.7% over the SA baseline and 44.9% over sequential execution, albeit with a slight reduction in task completion.
Significance
SMC offers a practical method for leveraging multi-step speculative execution to reduce agent latency beyond single-step speculative actions. By integrating a macro library and snapshot-based pre-execution, SMC enhances efficiency without compromising accuracy in most scenarios.
Availability
Our code is publicly available at this GitHub repository.
Accepted at MLSP2026.
Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
Cite as: arXiv:2609.03236 [cs.AI] (or arXiv:2609.03236v1 [cs.AI] for this version)
DOI: https://doi.org/10.48550/arXiv.2609.03236
via ArXiv AI