Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets
Mark Shapiro
Submitted on 31 July 2026
Abstract
A dense, pretrained language model can be retrofitted with recurrent depth, enabling it to learn an iterative latent transition that persists even after outcome-only annealing. We demonstrate this by splitting Qwen2.5-0.5B-Instruct into three components: a Prelude, a weight-tied Recurrent Block, and a Coda, with an identity-preserving one-loop path and a re-entry bridge for subsequent loops. At loop 1, the retrofit performs comparably to its base model on a preregistered ARC (Abstraction and Reasoning Corpus) battery.
Our study yields three primary findings:
- Reusable Mechanism, Not Memorization: The retrofit functions as a reusable procedure rather than a terminal-answer lookup, and installs at two parameter budgets: 6M trained parameters over frozen base weights, and 180M for a full-block retrofit. With intermediate-step supervision, the model computes one task step per loop and maintains performance when only final answers are graded. The adapter matched the full-block approach overall (83.8% vs. 84.0%), led through depth 11, and trailed beyond that point. Verbal fine-tuning achieved 79–86% on controlled verbal renderings (zero-shot transfer was minimal), and adapter verbal training initiated from the installed mechanism outperformed matched fresh training by 18.6 points, including on a held-out test set.
- Extrapolation Beyond Supervised Depth: The operation extrapolates to roughly 1.5 times its supervised depth, maintaining 70% accuracy through depth 18.
- Superiority Over Scratchpad Models: A same-size scratchpad-trained model matched the recurrent model within its learned horizon but collapsed beyond it. The recurrent model won overall (84% vs. 72%), retained 53% vs. 2.5% beyond depth 10, and answered 7.6 times faster. This demonstrates that an iterative transformer can perform deeper reasoning in latent space more efficiently than comparable or larger models fine-tuned on the same task.
A second task—running the rule in reverse—exposed critical limitations: the inverse was learnable in isolation, but no continuation acquired it while preserving the installed mechanism and general capability, indicating a catastrophic-interference boundary. Learned depth selection remains an open problem.
via ArXiv CL+LG