TL;DR
claude plugin eval .runs every case three times with your plugin loaded and three more with nothing, then reportsWITH,W/OUTandΔ. The Δ is the only column that says your plugin contributed anything: a case scoring 1.00 in both arms is a case Claude was already solving on its own. Think twice before dropping this into CI as it stands, because--thresholdreadsWITHand never the Δ.
Testing a plugin by hand only ever shows you one side of the experiment. You send the request, your skill fires, Claude answers well, and you close the session satisfied. The half you never see is what Claude would have said with your plugin uninstalled. Without it you are missing the only thing worth knowing, which is whether your plugin caused the result or was merely present when it happened.
This is not about whether your plugin loads. claude plugin validate already catches the broken JSON and the misspelled field. A plugin can load cleanly, do exactly what you wrote, and change nothing about an answer Claude would have given anyway. A plugin like that is dead weight, the same as the ones you never even reach for.
How it works
claude plugin eval sets up the experiment you cannot set up by hand. For each case it opens a fresh, non-interactive session with only your plugin loaded, sends the prompt, and scores whatever comes back against the graders you wrote. Then it repeats the whole thing with no plugins at all.
Each arm runs three times by default, because a single roll of a non-deterministic agent tells you nothing. One case is six sessions, and out come three numbers: WITH for the plugin arm, W/OUT for the bare one, and Δ, the difference.
What lands on screen
A two-case suite over a minimal plugin, a single skill that writes commit messages:
CASE WITH W/OUT Δ RUNS COST NOTES
first-case 1.00 0.00 +1.00 6 $0.53
plain-request 1.00 1.00 0.00 6 $0.46
2 case(s) · mean Δ +0.50 · 109s · $0.98
Both rows score a clean 1.00 under WITH. Judged on that column alone, the plugin is perfect twice over.
The Δ tells a different story. In first-case the plugin did all of the work. In plain-request it did none: Claude handled the request just as well without it. And this is not a skill that failed to fire, because all three plugin runs printed this:
✓ skill-fired [with-only, not scored]: Skill called 1x (expected 1..∞)
It fired, it ran, and it made no difference whatsoever. That is the case to delete or rewrite, and the score column hides it completely.
Setting it up in your plugin
1. Let Claude write the suite
From the plugin root, the directory holding plugin.json:
claude plugin eval init
It reads your plugin, asks you what a good result looks like, proposes prompts that should and should not trigger it, and writes one case directory per prompt under evals/. To see the files for yourself, claude plugin eval init --bare first-case drops a blank template without spending a model call.
2. A case is a prompt with frontmatter
In evals/first-case/prompt.md, the body is literally what Claude receives. Phrase it the way a user would, without naming your skill:
---
max_turns: 6
allowed_tools: [Skill]
---
Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.
3. Graders are one file each
One on the result, in graders/criteria.md, and one on the route there, checking that your skill is what produced it. The docs recommend that pairing: one grader asks what came out, the other asks who made it.
---
type: llm
---
PASS if the reply is a single Conventional Commits line of the form `type(scope): summary`.
FAIL if the reply is prose, a list of options, or a message that does not start with a Conventional Commits type.
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?commit-msg"'
---
4. Run it
claude plugin eval .
The first time it asks Trust this plugin directory?, because it is about to load and execute that plugin on your machine, under your credential.
Two things that will bite you
The skill grader does not count. That is the [with-only, not scored] above. A grader checking "my skill was invoked" can never pass in the arm without the plugin, so counting it would drag that arm toward zero and inflate your Δ with a lie. Claude Code drops it from the score in both arms and reports it as an indicator instead. This covers every tool_used grader on Skill, plus anything you mark arm: with-only.
CI goes green on a plugin that does nothing. --threshold compares against the WITH column and never against the Δ. The plain-request case above, Δ of 0.00 and all, exits 0 under the default threshold of 1.0. If you want a drop in contribution to break the build, you read the Δ yourself from aggregate-result.json, where it lands as meanDelta.
When the Δ comes back near zero and the skill grader is red, the diagnosis is almost always the same: your skill's description does not trigger on that phrasing. It is rule 2 of the five Anthropic publishes, now with a number attached. That tip uses skill-creator's evals/evals.json, which is a separate format from the one this command reads.
The six grader types
| Type | Passes when | Cost |
|---|---|---|
regex |
The pattern is found in the target, or absent with match: not_contains |
Free |
tool_used |
Calls to that tool fall between min and max |
Free |
tool_order |
The first before call precedes the first after call |
Free |
file_exists |
A file created during the run matches the path glob |
Free |
llm |
A judge model votes PASS in two of three votes on your rubric | Model call |
baseline |
The judge rates the run at least as good as a reference transcript | Model call |
Official docs: Test plugins with evals
Requirements: Claude Code 2.1.269 or later. Runs and llm graders call the model on your own credential and count against your plan, and the costs in the table are list-price estimates.