More Rules, Longer Lists? Benchmarking Self-Evolving Agents on a Hidden-Rule Curriculum

Authors: Chenjie Qiu, Boyan Xu, Shaobin Shi, Bingfeng Chen, Ruichu Cai, Dongsheng Luo

Submitted: 29 October 2026

Subjects: Artificial Intelligence (cs.AI)

arXiv: 2610.40758v1

Comments: 13 pages, 8 figures, 11 tables

Abstract

Self-evolving agents rewrite a persistent state from feedback, but it remains largely untested whether this capability allows them to learn a rule that was never explicitly provided. An agent can also succeed by recording each correction as an exception, causing its list to grow with every new rule. To investigate this, we introduce CLR-Bench, a Curriculum of Latent Rules benchmark.

Each chain adds one hidden rule per level on top of the earlier ones. A rule operates on an opaque label that represents a named grouping published on the Web—such as the founding members of an airline alliance. The task provides one example value; practice feedback reveals one additional member and one non-member; and the test checks unseen members and similar non-members.

Probes certify that:

• the base model knows the members of each grouping and does not guess it without feedback;
• the evidence identifies the grouping once labels are read as named groupings.

An automated pipeline builds chains from a new data domain, and we release 20 chains over 10 domains. We evaluate six recent self-evolution methods.

With the certifying model, these methods barely learn. The best gains at most 5 points over a frozen agent and excludes only 6–12% of unseen members from the count—fewer than the 20% reached by simply keeping the raw practice history. A fixed lesson that states the named-grouping convention lifts the same methods by 14.5 to 16.5 points. Their states hold the revealed member in 75–98% of cases and almost never the class name; when practice exposes a member the list misses, they add that member to the list.

With the stronger GPT-6-Sol, ACE and Dynamic Cheatsheet learn from feedback alone. They reach 78% accuracy and solve 59–60% of the single cells, against 53% and 8% with the certifying model, and they surpass the raw history. CLR-Bench therefore measures how well an agent learns—and this ability differs across models.

Key Contributions

• CLR-Bench: A new benchmark evaluating whether self-evolving agents can learn hidden rules through a curriculum of latent rules.
• Certifying probes: Evidence that the base model possesses the required knowledge but cannot guess it without feedback.
• Automated pipeline: Generates chains from new data domains; 20 chains across 10 domains are released.
• Evaluation of six self-evolution methods: Reveals that most methods barely learn with the certifying model, while stronger models such as GPT-6-Sol enable ACE and Dynamic Cheatsheet to learn from feedback alone.
• Insight: The ability to learn—not merely to accumulate exceptions—varies significantly across models, and CLR-Bench captures this distinction.

via ArXiv AI