NEBench R1: A Reasoning Benchmark for Evaluating Nuclear Domain Competence in Large Language Models
Authors: Liyan Wang, Longze Li, Mohamed Hassan, Isshu Lee, Dajie Sun, Ruimin Feng, Aleksandar Vakanski
Subject: Computer Science > Artificial Intelligence (cs.AI)
arXiv: 2611.21682 [cs.AI] | Submitted 15 Nov 2026
Overview
Large language models (LLMs) are increasingly being explored for knowledge-intensive and safety-critical applications in nuclear engineering. However, existing nuclear-domain benchmarks offer limited content coverage or focus primarily on answer accuracy without systematically evaluating the quality of the underlying reasoning. As LLMs move deeper into regulated, high-consequence domains in 2026, the gap between a correct answer and a technically sound justification has become a central concern for deployment and assurance.
The NEBench R1 Benchmark
This study introduces NEBench R1, a reasoning-oriented benchmark comprising 7,546 unique multiple-choice questions extracted from U.S. Nuclear Regulatory Commission (NRC) reactor operator licensing examinations covering 18 nuclear facilities. Each question is classified according to the NRC Knowledge and Abilities catalogs and accompanied by an official explanation that serves as a reference for reasoning evaluation.
Evaluation and Findings
Six LLMs were evaluated, with overall accuracy ranging from 34.0% to 73.9% across three distinct performance tiers. To assess reasoning quality, an LLM-based judge evaluated correctly answered responses using a multidimensional rubric.
Key findings include:
- Strong comprehension, weaker justification. Responses generally demonstrated strong problem comprehension but scored lower in factual and procedural accuracy and distractor analysis—indicating that correct answers were not necessarily supported by technically sound justifications.
- Clarity is a weak signal. Response clarity was only weakly indicative of technical quality.
- Reasoning length ≠ better performance. Longer reasoning traces were associated with lower accuracy in both frontier reasoning models, suggesting that reasoning length may reflect question difficulty rather than improved problem-solving performance.
- Token consumption as a reliability signal. Reasoning-token consumption may therefore provide an additional signal for identifying potentially unreliable responses.
Significance
NEBench R1 establishes a benchmark for evaluating both answer accuracy and reasoning quality, providing a foundation for assessing the domain competence and reliability of LLMs in nuclear engineering. By pairing official NRC explanations with rubric-based judging, the benchmark moves evaluation beyond surface-level correctness toward the verifiable reasoning that safety-critical deployment demands.
Cite as: arXiv:2611.21682 [cs.AI]
via ArXiv AI
