Doc / PDFSuccess criteria set before testing, test-set buckets, a grading rubric and a ship decision.
Preview
First part of the template, shown as-is:
Agent Field Kit · AI Agent Evaluation Test Plan (template v1.0)
Fill in the bracketed fields. Delete the guidance notes (lines starting with "Guidance:") before you share. This template structures testing; it does not certify safety or compliance.
1. Scope
| Field | Entry |
|---|---|
| Agent / workflow name | [ ] |
| Version under test (model, prompt, tools, config hash) | [ ] |
| Owner / reviewer | [ ] / [ ] |
| Intended task, in one sentence | [ ] |
| Out of scope (tasks the agent must refuse or hand off) | [ ] |
| Users and data it touches | [ ] |
| Risk tier (Low / Medium / High) and why | [ ] |
Guidance: the risk tier sets the rigor. High tier = it touches money, customer data, external messages, or irreversible actions.
2. Success criteria (set before testing)
| # | Metric | Definition | Target | Blocking? |
|---|---|---|---|---|
| M1 | Task success rate | Share of test cases graded "correct and complete" | [≥ __%] | Yes |
| M2 | Critical error rate | Wrong irreversible action, data leak, policy violation | [0 in __ cases] | Yes |
| M3 | Escalation correctness | Hands off when it should, and only then | [≥ __%] | Yes |
| M4 | Groundedness | Claims supported by the provided sources | [≥ __%] | [ ] |
| M5 | Cost per successful task | Run cost ÷ successes | [≤ $__] | No |
| M6 | Latency p95 | Seconds from request to final output | [≤ __ s] | No |
3. Test set
| Bucket | What it covers | # cases | Source |
|---|---|---|---|
| Golden path | Typical requests with known correct outputs | [ ] | Real (de-identified) history |
| Edge cases | Ambiguous, incomplete, conflicting inputs | [ ] | Written by domain expert |
| Out-of-scope | Requests the agent should decline or escalate | [ ] | Written |
| Adversarial | Prompt injection in documents, web pages, emails; jailbreak attempts | [ ] | Red-team list (section 6) |
| Tool failure | API timeouts, empty results, malformed data, permission denied | [ ] | Fault injection |
| Regression | Every past incident turned into a test | [ ] | Incident log |
Guidance: freeze the test set and version it. Never tune prompts on the same cases you report.
4. Grading
- Grader: [human / rubric-based model grader / exact match]. If you use a model grader, have a human spot-check [__%] of cases and record the agreement rate.
- Rubric per case: Correct (2) / Partially correct (1) / Wrong (0) / Critical error (fail).
- Two graders on [__%] of cases; resolve disagreements and record them.
5. Procedure
No checkout is connected.