Can You Trust Your Eval Scores? An Eval Suite Auditor Skill for Claude
November 20, 2026
The Eval Suite Auditor is a free Claude skill for anyone who already has an LLM evaluation pipeline and wants to know whether its scores can be trusted. It finds silent failures, such as a dead judge that still returns a score or an average that hides one failed check, by running probes against the real pipeline.
Where did this skill come from?
From a failure in my own pipeline. The model id a judge used was removed. Calls to it returned a 404, the error was swallowed, and a default score was returned in its place. Runs kept getting scores that looked real, and nothing in the output said the judge had not run. Only running the path with a dead judge showed what it returned. The skill turns that lesson into a procedure: probe behavior, not config.
What does the skill do?
It audits an existing pipeline. Building new evals is out of scope, and the skill says so at the start. It works in five stages:
- Map the pipeline. For every place a score or verdict is produced, it records the entry point, the judge and how it is chosen, what happens when the judge is down, and every consumer that acts on the result (gates, repair loops, dashboards, stored history, CI checks).
- Probe before concluding. It writes a probe for each failure class that fails when the problem exists, and runs it against the real path, not a mock.
- Fix probe-first, only if you ask. By default it reports and changes nothing.
- Check calibration honestly. A judge never compared with human labels is reported as uncalibrated.
- Report. A findings table, then what was verified sound, what was not probed, and the decisions that are yours.
What failure classes does it probe?
Fourteen, each with a standard probe. The ones that cause the most damage:
| Failure class | Probe | What a failure looks like |
|---|---|---|
| Judge unavailable | Point the judge at a model id that does not exist | A score, a pass, or a zero that looks like a real result |
| Partial judge failure | Several passing checks plus one judged check with a dead judge | The mean of the surviving checks is reported as the result |
| Unparseable reply | Make the judge return prose or malformed JSON | A score scraped from text, or a default |
| Default that looks like a score | Search for any constant returned on error (0.5, 0, 1) | The constant is stored or averaged as data |
| Disagreeing thresholds | List every pass line in the system | A score that passes one line, fails another, and triggers a repair |
| Variance against the decision margin | Score one fixed input at least 10 times | A single-run difference smaller than the noise commits a change |
| Consumers acting on "unknown" | Feed each consumer a not-judged result | It stores a fake score, queues a repair, or shows a green mark |
The rest cover aggregation order, baseline and candidate graded by different judges, a judge from the same model family as the text it grades, oversized or self-scoring judge input, score resolution, stored placeholders, and swallowed exceptions. The list is a floor: it also adds classes the pipeline's own design suggests.
Why does every probe need a positive control?
A probe that cannot tell a working path from a broken one proves nothing. So each probe is paired with the same path, a healthy judge, and a good input, which must pass. If a positive control fails, the probe environment is broken, and the skill stops and fixes that before trusting any finding.
What does "inconclusive" mean in the calibration check?
If the only human labels come from one rater, a failed comparison between judge and rater is reported as inconclusive, not as proof that the judge is wrong. The skill asks for a blind re-rating of a subset at least a day later and treats a metric as ratable only if that test-retest agreement clears a bar set before looking. It also forbids reusing the same labels to tune a threshold afterwards.
What does the report look like?
One row per finding: ID, severity, the finding, the exact probe input and observed output, and the exposure in real data. Severity follows the consequence: High when a failure passes or an automatic change is made, Medium when a result is misleading or a cost is unbounded, Low when only labelling is wrong. Exposure is a count from a query, or "latent" with the condition that would trigger it, never a guess. A finding with no reproduced probe goes in a separate "read, not run" list.
What are the limits?
It needs a runnable copy of the pipeline. If none exists, everything is labelled "read, not run". It will not spend money or write to production without your approval, so on a production-only system some probes are skipped and listed as not probed. It does not tell you whether your rubric measures the right thing, only whether the numbers the pipeline reports are the numbers it measured.
How do I install it?
- Download
eval-suite-auditor.md. - Save it as
.claude/skills/eval-suite-auditor/SKILL.mdin your project, or in~/.claude/skills/eval-suite-auditor/SKILL.mdto use it everywhere. - Ask Claude "audit my eval suite" or "can I trust this judge score?". It loads from the skill's description.
The skill is a set of instructions Claude follows, with no backend call and no account. It is also in the GitHub repository with the other skills.
FAQ
Is this the same as writing evals?
No. It assumes a suite already exists and asks whether its numbers can be trusted. For writing an evaluation protocol from scratch, use Prompt Evaluation Engineer.
Why not just read the code?
Reading finds candidates. A catch block that returns a default can look like sensible error handling. Only a run with a broken judge shows what the pipeline actually reports.
Will it change my code?
Not unless you ask. If you do, it writes each probe as a failing test first, fixes one finding at a time, and confirms the probe and its positive control both pass.
Does it work with models other than Claude?
The file is plain instructions, so any agent that loads skill-style files can follow it.
Related: Done Means Checkable picks a judge other than the model for work with no tests. Prompt Evaluation Engineer builds the evaluation protocol this skill audits.