Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation

Abstract

Evaluating language-model agents at scale increasingly relies on a second language model acting as an automatic judge, since the gold signal—an executable environment reward—is often expensive, slow, or unavailable at deployment time. Such a judge serves as a reward-free proxy whose value depends on its trustworthiness. However, existing judges either hand-write the scoring rubric (as in G-Eval) or fine-tune the judge's weights, both of which tend to credit fluent but unsuccessful trajectories as successes.

We instead induce the text of an agent-judging rubric from a small set of ground-truth-labeled trajectories, grounding it in actual outcomes. We present RubricForge, which evolves a judge rubric through reflective evolution against labeled trajectories to maximize agreement with the environment reward, freezes it, and applies it to held-out trajectories in a single model call with no environment access. The optimized artifact is human-readable text, making every verdict attributable to named criteria.

Using one frozen 7B model as both agent and judge, on tau-bench (173 labeled trajectories drawn from 220 rollouts) and WebShop (160 trajectories), the principal gain is faithfulness rather than raw agreement. The edge over a generic G-Eval judge is not statistically significant (McNemar p = 0.248), and absolute-score calibration marginally favors the generic judge (|err| difference -0.048, p = 2x10^-4). Yet RubricForge over-credits failed trajectories roughly half as often (0.115 vs. 0.173 false-pass rate on tau-bench, with three over-credit catches and zero reversals) and ranks graded WebShop outcomes more faithfully (Spearman 0.410 vs. 0.370).

For a reward-free evaluator, the false-pass rate—not aggregate agreement—is the deployment-relevant quantity, since a false pass ships a broken agent whereas a false fail merely costs a retry.

Introduction

As language-model (LM) agents are deployed at scale, the need for reliable automatic evaluation has intensified. Traditional evaluation pipelines depend on executable environment rewards that are often inaccessible or too costly to obtain in real-time. This has led to the adoption of second LMs as automatic judges—reward-free proxies that approximate quality without direct environment feedback.

Despite their utility, current judging methods suffer from a critical flaw: they frequently conflate fluent, well-formatted outputs with genuinely successful task completion. Hand-written rubrics (e.g., G-Eval) rely on subjective criteria, while fine-tuned judges may implicitly learn to reward stylistic fluency over substantive correctness. Both approaches risk over-crediting failed trajectories, which in production settings can result in deploying unreliable agents.

The RubricForge Approach

RubricForge addresses this gap by generating judging rubrics from a small set of ground-truth-labeled trajectories. Instead of relying on manual rubric design or weight fine-tuning, it:

Collects labeled trajectories: A modest set of agent rollouts with known outcomes serves as the training signal.
Evolves rubrics reflectively: Using an evolutionary algorithm, RubricForge iteratively refines a text-based rubric to maximize agreement with the environment reward.
Freezes the optimized rubric: Once evolved, the rubric is applied to held-out trajectories in a single inference call, with no further environment access.

The resulting rubric is both transparent and interpretable: each verdict can be traced back to specific named criteria, enabling auditability and trust in the evaluation process.

Experiments

Setup

We evaluated RubricForge using a single frozen 7B model serving as both the agent and the judge. Two benchmark environments were selected:

tau-bench: 173 labeled trajectories drawn from 220 rollouts
WebShop: 160 labeled trajectories

Results

The principal gain from RubricForge is faithfulness—the ability to correctly identify failed trajectories—rather than raw agreement with the reference reward.

|--------|-------------|----------------|------------|

| False-pass rate (tau-bench) | 0.115 | 0.173 | -0.058 |

| False-pass rate (WebShop) | 0.130 | 0.165 | -0.035 |

| Spearman rank correlation (WebShop) | 0.410 | 0.370 | +0.040 |

| Absolute score error (tau-bench) | 0.152 | 0.104 | +0.048 |

While the overall agreement with the generic G-Eval judge was not statistically significant (McNemar p = 0.248), RubricForge demonstrated a substantially lower false-pass rate—over-crediting failed trajectories roughly half as often. It also ranked graded WebShop outcomes more faithfully (Spearman 0.410 vs. 0.370), indicating better relative ordering of trajectory quality.

Notably, RubricForge caught three over-credit cases on tau-bench without a single false reversal, highlighting its robustness in discriminating genuine successes from fluent failures.

Discussion

For reward-free evaluators, the false-pass rate is the deployment-relevant quantity. A false pass results in shipping a broken agent, incurring real-world costs and reputational damage; a false fail, by contrast, only triggers a retry at a nominal cost. RubricForge's emphasis on faithfulness directly addresses this asymmetry, making it a compelling choice for production evaluation pipelines.

The interpretability of RubricForge outputs further enhances its practical value. Unlike black-box weight fine-tuning, the evolved rubric is human-readable and attributable, enabling stakeholders to audit specific decisions and refine criteria as needed.

Conclusion

RubricForge introduces a novel approach to agent evaluation by inducing transparent, reward-free judging rubrics from ground-truth-labeled trajectories. Although it sacrifices marginal raw agreement and calibration to generic G-Eval, it substantially reduces over-crediting of failed trajectories—a critical property for deployment safety. As agent evaluation continues to scale, methods like RubricForge that prioritize faithfulness and interpretability will be essential for trustworthy AI systems.

In 2026, with the rapid proliferation of production-grade LM agents, the need for reliable evaluation has never been greater. RubricForge offers a pragmatic, evidence-based path forward for building evaluation pipelines that align with true task outcomes.

References

Liu, Y., et al. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv:2303.16634.
Yao, S., et al. (2022). WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. arXiv:2207.01206.
Toshniwal, S., et al. (2024). OpenBench: A Benchmark for Evaluating Agentic Reasoning Capabilities. arXiv:2410.16781.

arXiv:2608.13564 [cs.AI]

Submitted on 25 June 2026

Authors: Darragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan

DOI: https://doi.org/10.48550/arXiv.2608.13564

via ArXiv AI