Depth-Aware Activation Steering via Attention Residuals: A

Depth-Aware Activation Steering via Attention Residuals


A training-free pipeline for deeper, more reliable LLM behavior modulation


Shilong Zhao (Independent Researcher) β€” Submitted 27 Oct 2026 β€” arXiv:2611.21655 [cs.CV]




Overview


Activation steering is a simple, training-free way to modulate LLM behavior, but existing methods typically inject vectors at a single layer and lack mechanisms for depth-aware propagation or internal verification. This work combines two recent ideasβ€”Attention Residuals (AttnRes), which provide learnable cross-layer pathways, and Jacobian-lens (J-space) concept readoutβ€”into a lightweight, depth-aware steering pipeline.


Key Findings


On two small open models (Qwen3-0.6B and Qwen3.5-0.8B), distributing a persona vector according to AttnRes-style depth weights yields stronger internal alignment (deeper-layer concept cosine and rank, p < 0.0001) and stronger behavioral shifts than single-layer injection.


Sign-Specificity and Localization


  • The effect is sign-specific: reverse vectors invert all six metrics, while random directions degrade alignment (p < 0.0001).
  • A short LoRA fine-tune erases the imprint.
  • Injection rewires attention strictly downstream of the injection point (upstream JS = 0.00000).

Three Practical Findings


  1. Layer-wise geometric decomposition of cosine readouts. At the injection layer, 123% of the increase is arithmetic, while 5–11 layers downstream, 100% is dynamical.
  2. Intensity window. Effects are monotone over 0.3–0.7, after which generation collapses (617 β†’ 306 β†’ 25 chars).
  3. Metric discipline. Keyword-rule metrics fail LLM-judge validation (kappa ≀ 0.09), and two rule-based conclusions were consequently retracted.

  4. Further Results


    • Training the depth weights on a steering objective (200 steps, frozen backbone) raises final alignment 10Γ— on held-out prompts.
    • Cross-persona vectors retain 96–99% of the effect: specificity holds against random, not semantic, directions.

    Contribution


    The paper contributes a practical, low-overhead depth-aware steering method paired with a careful evaluation protocol. Code and data are released.




    Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

    Cite as: arXiv:2611.21655 [cs.CV] β€” DOI: https://doi.org/10.48550/arXiv.2611.21655

    via ArXiv CV

Related