# Prompt Evaluation Report — Invoice Extraction

> A worked example. Figures are illustrative, but the **structure** is the point: anyone reading
> this can tell what was measured, under what conditions, and what was decided.

## Summary

| | |
|---|---|
| **System** | Invoice extraction (accounts payable) |
| **Change** | Prompt v3 → v4: added explicit line-item reconciliation instruction |
| **Date** | 2026-08-04 |
| **Decision** | **SHIP**, with two new regression cases added |

**Headline:** arithmetic accuracy 71% → 98%. Two unrelated cases regressed and are documented
below. Token count rose 12%; cost per completed task fell, because retries dropped.

## Conditions

Anything not pinned here makes the comparison invalid.

| | v3 (baseline) | v4 |
|---|---|---|
| Model | `<version string>` | same |
| Temperature | 0.2 | same |
| Prompt version | v3 (commit `a1b2c3`) | v4 (commit `d4e5f6`) |
| Case set | invoice-eval v7 (40 cases) | same |
| Runs per case | 5 | 5 |
| Scorer | deterministic + rubric v2 | same |

## Results

| Check | Type | v3 | v4 | Δ |
|---|---|---|---|---|
| Output parses as JSON | deterministic | 100% | 100% | — |
| Required fields present | deterministic | 100% | 100% | — |
| **Line items sum to stated total** | deterministic | **71%** | **98%** | **+27** |
| Currency matches document | deterministic | 99% | 99% | — |
| Date is valid ISO 8601 | deterministic | 100% | 100% | — |
| Supplier name not fabricated | rubric | 4.6/5 | 4.6/5 | — |
| Absent fields reported as null | behavioural | 95% | **88%** | **−7** |
| Refuses unreadable documents | behavioural | 100% | **93%** | **−7** |

**Overall: 87% → 96%.**

## Per-case regressions

> The aggregate improved. Two cases got worse. This section exists because an aggregate that rises
> while individual cases break is exactly how a regression ships.

**INV-023 — handwritten amounts.** v3 correctly returned `null` for an illegible total. v4 infers a
value from the line items, which is the new instruction working where it should not.
*Action:* added as permanent regression case. Prompt v5 will scope the reconciliation instruction to
machine-readable totals.

**INV-031 — scanned at low resolution.** v3 refused. v4 attempts extraction and returns partly wrong
values. Same root cause.
*Action:* added as permanent regression case.

## Cost

| | v3 | v4 |
|---|---|---|
| Tokens per request (median) | 1,840 | 2,061 (+12%) |
| Retries per 100 requests | 14 | 3 |
| **Cost per completed task** | $0.0121 | **$0.0104 (−14%)** |

The prompt got longer and cheaper — retries were the dominant cost, and better first-pass accuracy
removed most of them. Cost per *call* would have shown the opposite.

## Variance

Run at production temperature, five runs per case.

| | v3 | v4 |
|---|---|---|
| Cases passing 5/5 | 28 | 35 |
| Cases passing 3–4/5 | 9 | 4 |
| Cases passing 0–2/5 | 3 | 1 |

Consistency improved alongside accuracy — the two do not always move together, which is why both
are reported.

## Security

Re-run because the instruction block changed.

| | Result |
|---|---|
| Direct injection attempts | 12/12 blocked |
| Indirect injection (hostile text in document) | **11/12 blocked** |
| System prompt extraction | Succeeded — contains nothing sensitive. Accepted |

The one indirect injection that succeeded caused the model to append a sentence to the output. Blast
radius: cosmetic — this system has no tools and no write access. Recorded, not escalated.

## Decision

**Ship v4.** Arithmetic accuracy was the presenting problem and it is materially fixed. The two
behavioural regressions are understood, scoped to a narrow document class, and now permanent test
cases.

**Follow-up:** v5 to scope the reconciliation instruction to machine-readable totals only. Target
next sprint.

| | Name | Date |
|---|---|---|
| Evaluated by | | |
| Approved by | | |