How to Evaluate a Prompt: A Claude Skill for Testing
September 18, 2026
To know whether a prompt works, treat it as a behavior contract and test it: define what correct means, build a test matrix, run deterministic checks before rubric checks, and preserve the raw evidence. The Prompt Evaluation Engineer skill runs this seven-stage protocol as a single static file.
Why isn't a polished prompt proof that it works?
Most prompt optimization advice starts with "make the prompt more specific." That helps, but it skips the harder question: how do you know the prompt actually works?
A prompt can sound polished and still fail on missing data, adversarial inputs, schema violations, or model changes nobody tested. A fluent response is not automatically a correct one. A polite customer-support answer that leaks a secret still fails, regardless of how helpful it sounds.
What are the seven stages?
| Stage | What it does | Why it matters |
|---|---|---|
| 1. Evaluation contract | Defines the objective, inputs, required and forbidden outputs, acceptance criteria, and non-goals | Prevents scoring a prompt against criteria nobody agreed on |
| 2. Test matrix | Builds golden, boundary, adversarial, contrast, and regression cases | Exposes edge cases that happy-path testing misses |
| 3. Deterministic vs. rubric | Separates exact checks from qualitative judgment | Stops fluent-but-wrong answers from passing |
| 4. Evidence preservation | Captures raw output, model settings, assertions, and latency | Makes results reproducible and auditable |
| 5. Failure diagnosis | Groups failures by symptom and checks whether the test itself is flawed | Prevents moving the goalposts after seeing results |
| 6. Reproducible report | Reports the contract, protocol, results table, decision, and next actions | Turns evaluation from opinion into an artifact |
| 7. Integrity rules | Preserves inputs, discloses single-model grading bias, treats every reported result as permanent data | Stops results being quietly cleaned up afterward |
Why do deterministic checks run first?
Exact checks are objective, so they run before any judgment call. Failures from deterministic checks are recorded without aggregating subjective scores, so you see what broke before a rubric enters the picture.
| Deterministic checks (first) | Semantic rubric checks (second) |
|---|---|
| JSON or YAML parsing | Relevance |
| Required and forbidden strings | Completeness |
| Regular expressions | Factual support |
| Exact values and field types | Usefulness |
| Schema validity | Tone |
| Length bounds | Instruction following |
| Citation presence | |
| Latency limits |
Rubric dimensions need a scale plus concrete examples of Pass, Borderline, and Fail, not just a label. Otherwise "Pass" and "Fail" mean something different on every run.
How does it diagnose a failure without cheating?
Stage 5 groups failures by symptom and separates evidence from hypotheses. It also checks whether the test itself is tautological or rewards copying the input. When comparing versions, it requires independent identifiers for the prompt and for the test matrix, so "the prompt changed" stays separate from "the test changed."
What do the worked examples cover?
- Structured extraction: for a prompt that extracts invoice fields as JSON, define required keys and types and parse every response as JSON.
- Customer-support safety: include ordinary questions plus requests for account secrets, policy exceptions, and conflicting instructions.
- Comparing prompt versions: run both versions against the same frozen matrix and settings, and show per-case transitions such as pass-to-fail and fail-to-pass.
How do I install it?
The skill is standalone: no package installation, API key, backend call, or runtime dependency after installation.
- Download
prompt-evaluation-engineer.md, or fetch it from the GitHub repository. - Save it as
.claude/skills/prompt-evaluation-engineer/SKILL.mdin your project or in~/.claude/skills/. - Ask Claude to evaluate, test, benchmark, or compare a prompt.
FAQ
What is a prompt evaluation contract?
A written definition of the prompt's task, inputs, required and forbidden outputs, and acceptance criteria, agreed before any scoring.
What is a test matrix?
A set of cases that balances golden (typical), boundary, adversarial, contrast, and regression inputs, each with expected behavior and pass or fail criteria.
Why disclose single-model grading bias?
Using a single model as the grader can bias the results. The skill requires that limitation to be disclosed in the report.
Does averaging scores hide problems?
Yes. The skill avoids averaging failures and reports pass rates linked directly to per-case evidence.
Related: Prompt Complexity Triage decides how much to change a prompt, and this skill tests whether the change worked. This skill was first shared on r/PromptEngineering by u/Parking-Kangaroo-63.