Overview
Emphatic temporal-difference learning (ETD) stabilizes the expected off-policy TD update and alters its projection geometry, but neither property alone determines the behavior of constant-stepsize sampled dynamics. In this work, we show that these two guarantees can diverge in practice, and we propose a repair mechanism that restores stability under constant stepsizes. As constant-stepsize training remains the default in modern deep reinforcement learning pipelines in 2026, understanding this gap has direct practical relevance.
The Stability Gap
We construct an ergodic two-state counterexample in which the ETD mean map contracts while the sampled product exhibits a positive top Lyapunov exponent. This demonstrates that contraction of the mean map does not guarantee stability of the sampled dynamics. A regenerative-cycle analysis separates this sign behavior from the infinite variance of the follow-on trace, clarifying which quantity drives the instability.
Regularized Emphatic TD (RETD)
We introduce regularized emphatic TD (RETD), a normalized first-order post-shock repair that:
- Leaves the trace and importance ratios unchanged
- Stores the emphatic TD signal in a leaky scalar state
- Releases a delayed correction
RETD's raw equilibrium is an affine shift of the ETD equilibrium. Single- and two-regularization readouts recover the ETD fixed point exactly.
Theoretical Results
We prove almost-sure convergence for harmonic diminishing stepsizes. We also establish a conditional constant-stepsize moment-contraction result derived from a Markovian random-product bound. RETD has certified negative exponents on the two-state construction and on one Baird point, whereas the positive Baird ETD sign remains numerical.
Experimental Validation
Paired 10,000-run experiments validate:
- Both separations of stability behavior
- Fixed-point recovery
- A nonmonotone stability region
- Task dependence of the observed effects
Scope and Limitations
RETD changes post-shock dynamics. It does not reduce the shared follow-on-trace variance. This distinction is important for practitioners who may expect variance reduction from regularization-based repairs; RETD targets a different failure mode.
Relevance in 2026
As off-policy reinforcement learning continues to underpin large-scale decision-making systems—from agentic workflows to robotics—theoretical guarantees under realistic constant-stepsize regimes remain a central concern. This work contributes a rigorous separation between mean-map contraction and sampled stability, and offers a constructively verified repair with provable convergence properties.
Citation
arXiv:2609.19170 [cs.AI], submitted 14 Sep 2026. DOI: https://doi.org/10.48550/arXiv.2609.19170
Authors: Xingguo Chen, Zhaohui Wu, Jinguo Ye, Chao Li, Shangdong Yang, Guang Yang, Skylar Liang, Wenhao Wang
via ArXiv AI