Overview
A new arXiv paper (arXiv:2610.13992, submitted 26 September 2026) examines a foundational but underexplored question in vision model interpretability: how does the choice of positional encoding shape the internal representations of a Vision Transformer (ViT)?
Authors: Mahmoud Mannes, Nikhil Maturi, Parva Mehta, Jiwon Jeong, and Aravind Singaram Kannappan.
Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG).
Cite as: arXiv:2610.13992 [cs.CV] — https://doi.org/10.48550/arXiv.2610.13992
The Problem
Positional embeddings encode the spatial location of each image patch in a Vision Transformer, allowing the model to distinguish between different spatial arrangements of visual content. Despite their importance, little was known about how the choice of embedding shapes a ViT's internal representation.
Key Findings
The paper shows that Absolute Positional Embeddings (APE) and Rotary Positional Embeddings (RoPE) encode spatial position through fundamentally different internal mechanisms:
- APE stores position in explicit features. Sparse autoencoders recover causal, row- and column-specific features in APE models.
- RoPE builds position inside attention — and rebuilds it when withheld. Interventions on RoPE reveal distinct rotary components that separately encode row and column information.
Causal interventions that selectively disable RoPE's row or column rotations demonstrate that decodable row and column information can be independently removed and, once rotations are restored, partly reconstructed — in part by axis-specialized attention heads.
Why It Matters
Both mechanisms are causally important for classification performance:
- Permuting APE's positional features across patches reduces top-1 accuracy by 24–31 points.
- Ablating the rotations of RoPE's primary rebuilding head reduces accuracy more than ablating those of any other head.
These results show that the choice of positional encoding fundamentally changes how spatial information is organized and processed inside Vision Transformers.
Broader Implications
More broadly, the work demonstrates how mechanistic analysis can reveal how models construct, reconstruct, and maintain structured representations — providing a basis for targeted interventions in vision models. As Vision Transformers continue to scale in 2026 across multimodal and high-resolution perception tasks, understanding the internal geometry of positional encodings offers a practical lever for debugging, editing, and improving model behavior.
Abstract (Original)
Positional embeddings encode the spatial position of each image patch in a Vision Transformer (ViT), allowing the model to distinguish different spatial arrangements of visual content. However, we know little about how the choice of embedding shapes a vision transformer's internal representation. In this work, we show that Absolute Positional Embeddings (APE) and Rotary Positional Embeddings (RoPE) encode spatial position through different internal mechanisms: APE stores position in explicit features, while RoPE builds it inside attention and rebuilds it when it is withheld. Specifically, in APE models, sparse autoencoders recover causal row- and column-specific features, whereas interventions on RoPE reveal distinct rotary components that separately encode row and column information. Additionally, causal interventions selectively disabling RoPE's row or column rotations show that decodable row and column information can be independently removed and, once rotations are restored, partly reconstructed, in part by axis-specialized attention heads. Both mechanisms are causally important for classification: permuting APE's positional features across patches reduces top-1 accuracy by 24–31 points, and ablating the rotations of RoPE's primary rebuilding head reduces it more than ablating those of any other head. Our findings show that the choice of positional encoding fundamentally changes how spatial information is organized and processed inside Vision Transformers. More broadly, our work demonstrates how mechanistic analysis can reveal how models construct, reconstruct, and maintain structured representations, providing a basis for targeted interventions in vision models.
via ArXiv CV