Artificiality's First Exam: An LLM-Referenced C-Test for Supraintelligence

arXiv:2610.14005 [cs.AI] | Submitted 27 Sep 2026

Author: Jose Hernandez-Orallo

Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)

Comments: Preprint, under review. 63 pages. Code and data available at https://github.com/jorallo/ctest-one

Abstract

Under the accelerated pace of AI development in 2026, any benchmark whose difficulty is calibrated with humans or current AI systems becomes circular, or saturates soon. How can we then evaluate machines that may become more capable than the current machines and the people who write the tests?

One answer hinges on inverse problems—inductive inference questions for which difficulty depends on the complexity of the underlying mechanism, given the observations and prior knowledge. Levin's Kt, a computable variant of Kolmogorov complexity, puts a theoretical upper bound on that difficulty, balancing the simplest representation encoding for the mechanism and the logarithm of the time steps required to run that mechanism.

Previous tests derived from Levin's Kt, such as the original C-test, are based on minimalistic Universal Turing Machines (UTM) capable of very limited representations and no previous knowledge. Reasoning large language models (LLMs) are UTMs that capture a compendium of human culture as knowledge, using prompts as natural language programs.

This paper presents the first test based on Levin's Kt with LLMs as representational machines. Fresh test items can be produced at high difficulties, estimated on a ratio scale measured in bits, before any human or frontier model can solve them. We formalise the measure, devise a lightweight generation and certification pipeline, and explore the effect of the power of the reference machine and the gap between examiner and examinee. We produce a first battery of certified items administered to twelve LLMs. The resulting agent characteristic curves decrease with difficulty, down to the uncharted high-difficulty region—the supra-intelligence window of items we can certify, yet beyond generations of frontier intelligence.

Key Contributions

• First LLM-referenced C-test: A novel benchmark grounded in Levin's Kt complexity, using LLMs as the representational reference machines.
• Ratio-scale difficulty in bits: Difficulty is measured on an absolute ratio scale, enabling principled extrapolation beyond current capabilities.
• High-difficulty item generation: Fresh test items can be produced at difficulty levels no current human or frontier model can solve.
• Certification pipeline: A lightweight generation and certification pipeline ensures items are verifiable before administration.
• Empirical battery: A first set of certified items administered to twelve LLMs, yielding agent characteristic curves that decline with difficulty.
• Supra-intelligence window: Identification of a region of certified items that lie beyond current generations of frontier intelligence—a window into evaluating machines more capable than those writing the tests.

Why This Matters in 2026

As frontier models continue to saturate conventional benchmarks calibrated against human performance, the field needs evaluation methods that remain valid as machines surpass their creators. This work reframes the problem through algorithmic information theory, using LLMs themselves as the reference machines that define difficulty—an approach that sidesteps the circularity of human-calibrated benchmarks and opens a path toward evaluating truly superintelligent systems.

Citation

@article{hernandez-orallo2026artificiality,
  title={Artificiality's First Exam: An LLM-Referenced C-test for Supraintelligence},
  author={Hernandez-Orallo, Jose},
  journal={arXiv preprint arXiv:2610.14005},
  year={2026}
}

DOI: https://doi.org/10.48550/arXiv.2610.14005

via ArXiv AI