Overview
Automated interpretability scores an explanation of a sparse-autoencoder (SAE) feature by asking a model to predict, token by token, where the feature fires on evaluation examples, or to judge whether marked tokens fit it. The core finding of this work is that the evaluation data built by standard pipelines can often be scored without reading the explanation at all.
Key Findings
Position and Mark Cues in Pipeline Data
On Neuronpedia's stored fuzzing runs for 500 random GPT-2 small features, 91.3% of activating examples are cropped with the peak on the centre token. A trivial rule that answers "yes" whenever the centre token is marked achieves a balanced accuracy of 0.979, compared with 0.838 for the stored LLM scorers (recomputed from the stored per-item verdicts, on marks rebuilt from the pipeline's code). A rule reading only the mark count reaches 0.969.
The cue changes form across pipelines: current delphi always marks the same slot in its fuzzing negatives, and full-context runs mark negatives with more tokens.
Sequence-Order Cues in Simulation Benchmarks
OpenAI's simulation benchmark lists each neuron's top-activating sequences first. A rule that predicts activity on the first five sequences scores 0.227 in pooled correlation, against 0.160 for GPT-4's published simulator (post hoc).
Methodology and Contributions
- Closed forms are given for what such explanation-free rules can score.
- The authors show that one scorer's simulation scores are higher on stored snippets than on natural windows, but do not demonstrate that the ordering is changed.
- They build windows that remove the position cue, derive (but do not build) negatives that remove the mark cues, and separate the sequence cue by decomposition.
Natural-Text Evaluation
On natural-text windows cut at random offsets, the best fixed-offset prior shared by all features scores only 0.015.
On these same windows:
- A typed decision model (Jev) reaches a pooled correlation of 0.331 over 49 features, against 0.013 with shuffled explanations.
- An open 8B simulator with the original read-out reads the explanation there, scoring 0.194 against 0.019.
Recommendation
Scores should be reported alongside the best explanation-free rule on the same items.
Metadata
- arXiv: 2610.37676 [cs.CL]
- Submitted: 10 October 2026
- Author: Xupeng Chen
- Subject: Computation and Language (cs.CL)
- DOI: https://doi.org/10.48550/arXiv.2610.37676
via ArXiv CL+LG