Can LLMs Really Understand Item Difficulty Levels? Implications for Automated Item Generation Using LLMs
Abstract
Estimating item difficulty is essential in both formative assessments and large-scale, high-stakes summative evaluations. This study examines how large language models (LLMs) predict item difficulty levels using items from a large-scale Reading and Writing test. We evaluate various prompting strategies and parameter settings across multiple LLMs, comparing their performance with encoder-only language models and feature-based supervised machine learning approaches. Zero-shot GPT-4.1 with a temperature of 0 achieved the highest accuracy among LLMs, reaching a quadratic weighted kappa (QWK) of 0.578. However, this remains below ConvBERT (QWK = 0.625), which outperformed the best feature-based supervised model. Further analysis reveals that all LLMs struggled to label hard items; notably, the advanced GPT-5.4 tends to underestimate item difficulty levels. Dimension reduction of embeddings shows that item embeddings from different difficulty levels are mixed together, suggesting that semantic information alone is insufficient for reliable difficulty prediction. These findings indicate that if LLMs cannot genuinely discern item difficulty—and instead treat most items as easy as their capabilities grow—caution is warranted when using LLMs to generate items with targeted difficulty levels.
1. Introduction
Item difficulty estimation is a cornerstone of educational measurement, influencing test design, scoring, and interpretation. Traditionally, this relies on human expert judgment or statistical models based on examinee responses. However, the rapid advancement of large language models (LLMs) has opened new possibilities for automating item generation and difficulty prediction. By 2026, LLMs such as GPT-4.1 and GPT-5.4 have become integrated into educational technology platforms, raising critical questions: Can these models truly understand item difficulty, or do they merely approximate it? This study systematically investigates LLM capabilities in predicting item difficulty levels, comparing them with established methods and exploring implications for automated item generation.
2. Methods
We used a large-scale Reading and Writing test dataset, applying a range of LLMs with diverse prompting strategies (zero-shot, few-shot) and parameter settings (temperature variations). For comparison, we included encoder-only models (e.g., ConvBERT) and feature-based supervised machine learning models (e.g., logistic regression, random forests). Difficulty prediction accuracy was measured using quadratic weighted kappa (QWK).
3. Results
3.1 LLM Performance
Zero-shot GPT-4.1 with a temperature of 0 achieved the highest QWK (0.578), surpassing other LLMs. However, this was lower than ConvBERT's QWK of 0.625, which also outperformed the best feature-based model.
3.2 Difficulty-Level Analysis
All LLMs showed weak performance in labeling hard items. GPT-5.4, the most advanced model, systematically underestimated difficulty, tending to classify items as easier than they actually were. This suggests a bias likely stemming from the model's extensive training data and advanced reasoning, which may make it more capable of solving items and thus perceive them as easier.
3.3 Embedding Analysis
Dimension reduction (e.g., t-SNE) of item embeddings revealed that representations from different difficulty levels overlap considerably, indicating that surface semantics are insufficient for predicting difficulty. This aligns with the observed LLM limitations and underscores the need for integrating non-semantic features (e.g., linguistic complexity, response processes).
4. Discussion
Our findings have significant implications for automated item generation (AIG). While LLMs can produce plausible items, their inability to reliably gauge difficulty poses risks: generated items may be misaligned with intended difficulty levels, compromising assessment validity. As LLM capabilities increase, the tendency to underestimate difficulty could lead to unintended test inflation (i.e., items appearing easier than intended). This is particularly concerning in high-stakes contexts. We recommend that AIG systems incorporate explicit difficulty calibration mechanisms, such as combining LLM outputs with established statistical models or human review.
5. Conclusion
LLMs, despite their advanced language understanding, do not yet match the accuracy of specialized models like ConvBERT in predicting item difficulty. The semantic information embedded in items is insufficient for reliable prediction, and current LLMs exhibit a systematic bias toward underestimating difficulty, especially for harder items. As we move toward greater automation in educational assessment, caution is necessary. Future research should explore hybrid approaches that leverage LLM strengths while addressing their limitations in difficulty estimation.
References
- (List of relevant references would appear here, formatted per publication style.)
Paper: arXiv:2607.28634 [cs.CL]
Submitted on 17 May 2026
Authors: Xinyi Wang, Hong Jiao, Ming Li, Sydney Peters, Hanna Choi, Tianyi Zhou, Qingshu Xu
via ArXiv CL