Plugin Evals Land in Claude Code
Anthropic has published a new plugin evals workflow for Claude Code. The claude plugin eval command runs a plugin against realistic prompts, grades what Claude produced, and compares the result with a run where the plugin is not loaded. It answers three questions plugin developers could not previously measure: does the skill trigger, does it survive an edit or a new model, and does it beat a bare model.
Deployable: Yes. It runs on Claude Code v2.1.269 or later against any directory with a plugin.json or .claude-plugin/plugin.json manifest, or a skills-directory plugin. Every eval run and judge grader is a real model call billed to your plan or API account.
What a Case Looks Like
An eval suite lives in an evals/ directory inside the plugin. Each case is a subdirectory holding a prompt.md and a graders/ folder. The prompt body goes to Claude exactly as written, and @path mentions are not expanded. Frontmatter on prompt.md can set max_turns (default 10), timeout_seconds (default 300), model, tags, and allowed_tools.
Graders are markdown files whose frontmatter sets a type, an optional weight, and an optional arm. There are six types. Four cost nothing because they are computed from the transcript and the files on disk: regex, tool_used, tool_order, and file_exists. Two call a judge model and add to the bill: llm, which scores the reply against prose criteria you write, and baseline, which compares it against a reference answer.
claude plugin eval init reads the plugin, asks what a good result looks like, proposes cases and graders, tries them, and writes the files. In CI, --bare <name> writes a blank template instead.
The Number That Matters Is Ξ
The headline metric is not the absolute score, but the delta (Ξ) between the plugin-enabled arm and the no-plugin baseline. A high score on its own can simply mean the prompt was easy for Claude. Only the delta tells you whether the plugin actually added valueβand whether it cleared the CI gate for skills.
via MarkTechPost