Computer Science > Computation and Language
arXiv:2610.12545 (cs.CL)
Submitted on 8 October 2026
Title
AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits
Authors
Shu Ishida, Aliasghar Khani, Tianyu Zhang, Cong Luong, James Seale Smith, Adam Gaier
Abstract
Autoregressive decoding is inherently less suited to non-causally ordered data—such as cross-references or future dependencies—because its outputs cannot be revised post hoc, unlike iterative human reasoning. Diffusion LLMs partially mitigate this limitation, but they still cannot flexibly insert or remove arbitrary numbers of tokens from a pre-generated context.
We introduce AutoIndexer, a generative model equipped with a novel training and decoding mechanism that simulates a Chain of Edits, allowing the model to flexibly modify its prior outputs in hindsight via token insertions, substitutions, and deletions. The model learns to predict marker tokens for opening and closing edits, simultaneously moving a cursor to positions within the existing token sequence—analogous to operating a text editor.
Training sequences are obtained by perturbing ground-truth labels with inverse chained edit operations, and the model learns to edit by undoing those inverses. This perturbation-based training makes AutoIndexer robust to erroneous context and adaptable to editing tasks such as code correction. We demonstrate that our method exhibits post-generation context-editing capabilities that most autoregressive and diffusion models lack, without compromising performance on standard LLM benchmarks.
The code is available at https://github.com/AutodeskAILab/autoindexer.
Subjects
- Computation and Language (cs.CL)
- Artificial Intelligence (cs.AI)
- Machine Learning (cs.LG)
Cite as
arXiv:2610.12545 [cs.CL]
DOI
https://doi.org/10.48550/arXiv.2610.12545
via ArXiv CL+LG