Depth-Aware Activation Steering via Attention Residuals
A training-free pipeline for deeper, more reliable LLM behavior modulation
Shilong Zhao (Independent Researcher) β Submitted 27 Oct 2026 β arXiv:2611.21655 [cs.CV]
Overview
Activation steering is a simple, training-free way to modulate LLM behavior, but existing methods typically inject vectors at a single layer and lack mechanisms for depth-aware propagation or internal verification. This work combines two recent ideasβAttention Residuals (AttnRes), which provide learnable cross-layer pathways, and Jacobian-lens (J-space) concept readoutβinto a lightweight, depth-aware steering pipeline.
Key Findings
On two small open models (Qwen3-0.6B and Qwen3.5-0.8B), distributing a persona vector according to AttnRes-style depth weights yields stronger internal alignment (deeper-layer concept cosine and rank, p < 0.0001) and stronger behavioral shifts than single-layer injection.
Sign-Specificity and Localization
- The effect is sign-specific: reverse vectors invert all six metrics, while random directions degrade alignment (p < 0.0001).
- A short LoRA fine-tune erases the imprint.
- Injection rewires attention strictly downstream of the injection point (upstream JS = 0.00000).
Three Practical Findings
- Layer-wise geometric decomposition of cosine readouts. At the injection layer, 123% of the increase is arithmetic, while 5β11 layers downstream, 100% is dynamical.
- Intensity window. Effects are monotone over 0.3β0.7, after which generation collapses (617 β 306 β 25 chars).
- Metric discipline. Keyword-rule metrics fail LLM-judge validation (kappa β€ 0.09), and two rule-based conclusions were consequently retracted.
- Training the depth weights on a steering objective (200 steps, frozen backbone) raises final alignment 10Γ on held-out prompts.
- Cross-persona vectors retain 96β99% of the effect: specificity holds against random, not semantic, directions.
Further Results
Contribution
The paper contributes a practical, low-overhead depth-aware steering method paired with a careful evaluation protocol. Code and data are released.
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.21655 [cs.CV] β DOI: https://doi.org/10.48550/arXiv.2611.21655
via ArXiv CV
