Tracing Knowledge and Reasoning in LLMs: A New Framework for Characterizing Model-Internal Processing
arXiv:2611.17461 (cs.CL) | Submitted 12 November 2026 | Accepted to Transactions of the Association for Computational Linguistics (TACL)
Authors: Yang Wang, Xiao Wang, Yiqi Liu, Chenghao Xiao, Eleni Vasilaki, Goran Nenadic, Chenghua Lin
The Problem with Task-Level Evaluation
Contemporary evaluations of Large Language Models (LLMs) typically report performance at the dataset or task level. Such scores reveal what a model achieves, but offer little insight into the internal processing that produces those results. In an era where benchmark saturation and leaderboard-driven model selection are increasingly common, this gap leaves practitioners without a principled way to understand why a model succeeds or fails.
A Task-Centric Framework
To address this, the authors propose a task-centric framework for characterizing the model-internal processing elicited by a benchmark. The framework combines:
- Attention-based signals — capturing where the model allocates focus across input tokens.
- Activation-based signals — reflecting internal feature representations during inference.
- Representation-based measures — quantifying how information is encoded and transformed across layers.
Together, these signals characterize benchmarks along a continuum from retrieval-dominant to reasoning-dominant tasks, and they surface failure modes that task-level scores conceal.
Experimental Scope
The study is extensive: the authors evaluate 16 LLMs across 24 diverse benchmarks, followed by targeted internal-state analyses on representative models and tasks. The results reveal systematic differences in processing between knowledge-centric (retrieval-dominant) and reasoning-centric tasks—distinctions that aggregate accuracy scores cannot capture.
Why It Matters
This work reframes evaluation: rather than asking only what score a model achieves, it asks what processing patterns are associated with that score. This shift has practical implications for:
- More targeted benchmark selection — matching benchmarks to the cognitive demands of a given application.
- More informed model choice — selecting models based on processing characteristics, not just headline numbers.
- Greater interpretability — exposing failure modes hidden beneath aggregate metrics.
As LLM evaluation in 2026 moves beyond leaderboard rankings toward deeper diagnostic understanding, frameworks like this one offer a path to more transparent and trustworthy model assessment.
Availability
The code is open-sourced at github.com/extraordinarylab/trace-kr.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2611.17461 [cs.CL]
DOI: 10.48550/arXiv.2611.17461
via ArXiv CL+LG
