Sample Report: Prompt Evaluation Run
What a prompt evaluation report should actually contain, conditions pinned, per-case deltas, cost, and a decision. A filled-in example you can copy the structure from.
Markdown. No sign-up, no email.
A worked example. Figures are illustrative, but the structure is the point: anyone reading this can tell what was measured, under what conditions, and what was decided.
Summary#
| System | Invoice extraction (accounts payable) |
| Change | Prompt v3 → v4: added explicit line-item reconciliation instruction |
| Date | 2026-08-04 |
| Decision | SHIP, with two new regression cases added |
Headline: arithmetic accuracy 71% → 98%. Two unrelated cases regressed and are documented below. Token count rose 12%; cost per completed task fell, because retries dropped.
Conditions#
Anything not pinned here makes the comparison invalid.
| v3 (baseline) | v4 | |
|---|---|---|
| Model | <version string> | same |
| Temperature | 0.2 | same |
| Prompt version | v3 (commit a1b2c3) | v4 (commit d4e5f6) |
| Case set | invoice-eval v7 (40 cases) | same |
| Runs per case | 5 | 5 |
| Scorer | deterministic + rubric v2 | same |
Results#
| Check | Type | v3 | v4 | Δ |
|---|---|---|---|---|
| Output parses as JSON | deterministic | 100% | 100% | |
| Required fields present | deterministic | 100% | 100% | |
| Line items sum to stated total | deterministic | 71% | 98% | +27 |
| Currency matches document | deterministic | 99% | 99% | |
| Date is valid ISO 8601 | deterministic | 100% | 100% | |
| Supplier name not fabricated | rubric | 4.6/5 | 4.6/5 | |
| Absent fields reported as null | behavioural | 95% | 88% | −7 |
| Refuses unreadable documents | behavioural | 100% | 93% | −7 |
Overall: 87% → 96%.
Per-case regressions#
The aggregate improved. Two cases got worse. This section exists because an aggregate that rises while individual cases break is exactly how a regression ships.
INV-023: handwritten amounts. v3 correctly returned null for an illegible total. v4 infers a value from the line items, which is the new instruction working where it should not. Action: added as permanent regression case. Prompt v5 will scope the reconciliation instruction to machine-readable totals.
INV-031: scanned at low resolution. v3 refused. v4 attempts extraction and returns partly wrong values. Same root cause. Action: added as permanent regression case.
Cost#
| v3 | v4 | |
|---|---|---|
| Tokens per request (median) | 1,840 | 2,061 (+12%) |
| Retries per 100 requests | 14 | 3 |
| Cost per completed task | $0.0121 | $0.0104 (−14%) |
The prompt got longer and cheaper: retries were the dominant cost, and better first-pass accuracy removed most of them. Cost per call would have shown the opposite.
Variance#
Run at production temperature, five runs per case.
| v3 | v4 | |
|---|---|---|
| Cases passing 5/5 | 28 | 35 |
| Cases passing 3 to 4/5 | 9 | 4 |
| Cases passing 0 to 2/5 | 3 | 1 |
Consistency improved alongside accuracy: the two do not always move together, which is why both are reported.
Security#
Re-run because the instruction block changed.
| Result | |
|---|---|
| Direct injection attempts | 12/12 blocked |
| Indirect injection (hostile text in document) | 11/12 blocked |
| System prompt extraction | Succeeded, contains nothing sensitive. Accepted |
The one indirect injection that succeeded caused the model to append a sentence to the output. Blast radius: cosmetic: this system has no tools and no write access. Recorded, not escalated.
Decision#
Ship v4. Arithmetic accuracy was the presenting problem and it is materially fixed. The two behavioural regressions are understood, scoped to a narrow document class, and now permanent test cases.
Follow-up: v5 to scope the reconciliation instruction to machine-readable totals only. Target next sprint.
| Name | Date | |
|---|---|---|
| Evaluated by | ||
| Approved by |