Tracing Knowledge and Reasoning in LLMs: A New Framework for

Tracing Knowledge and Reasoning in LLMs: A New Framework for Characterizing Model-Internal Processing


arXiv:2611.17461 (cs.CL) | Submitted 12 November 2026 | Accepted to Transactions of the Association for Computational Linguistics (TACL)


Authors: Yang Wang, Xiao Wang, Yiqi Liu, Chenghao Xiao, Eleni Vasilaki, Goran Nenadic, Chenghua Lin


The Problem with Task-Level Evaluation


Contemporary evaluations of Large Language Models (LLMs) typically report performance at the dataset or task level. Such scores reveal what a model achieves, but offer little insight into the internal processing that produces those results. In an era where benchmark saturation and leaderboard-driven model selection are increasingly common, this gap leaves practitioners without a principled way to understand why a model succeeds or fails.


A Task-Centric Framework


To address this, the authors propose a task-centric framework for characterizing the model-internal processing elicited by a benchmark. The framework combines:


  • Attention-based signals — capturing where the model allocates focus across input tokens.
  • Activation-based signals — reflecting internal feature representations during inference.
  • Representation-based measures — quantifying how information is encoded and transformed across layers.

Together, these signals characterize benchmarks along a continuum from retrieval-dominant to reasoning-dominant tasks, and they surface failure modes that task-level scores conceal.


Experimental Scope


The study is extensive: the authors evaluate 16 LLMs across 24 diverse benchmarks, followed by targeted internal-state analyses on representative models and tasks. The results reveal systematic differences in processing between knowledge-centric (retrieval-dominant) and reasoning-centric tasks—distinctions that aggregate accuracy scores cannot capture.


Why It Matters


This work reframes evaluation: rather than asking only what score a model achieves, it asks what processing patterns are associated with that score. This shift has practical implications for:


  • More targeted benchmark selection — matching benchmarks to the cognitive demands of a given application.
  • More informed model choice — selecting models based on processing characteristics, not just headline numbers.
  • Greater interpretability — exposing failure modes hidden beneath aggregate metrics.

As LLM evaluation in 2026 moves beyond leaderboard rankings toward deeper diagnostic understanding, frameworks like this one offer a path to more transparent and trustworthy model assessment.


Availability


The code is open-sourced at github.com/extraordinarylab/trace-kr.




Subjects: Computation and Language (cs.CL)

Cite as: arXiv:2611.17461 [cs.CL]

DOI: 10.48550/arXiv.2611.17461

via ArXiv CL+LG

Related