# AI Feature Test Report — Sample **This is an illustrative example.** The feature, cases and figures are invented. The problem it addresses is real: a test suite written for deterministic software either fails constantly against an AI feature or passes without checking anything. --- ## Functional test — meeting summary feature | | | |---|---| | Feature | Generates a summary and an action list from a meeting transcript | | Suite | 240 cases across 6 assertion styles | | Run | 5 repetitions of every case, same inputs | | Gate | Every structural assertion passes 5/5. Content assertions pass at 4/5 or better. | | Result | **Pass, with one accepted deviation** | ## 1. Result by assertion style | Style | Cases | Pass 5/5 | Pass 4/5 | Below | |---|---|---|---|---| | Structural — shape of the output | 60 | 60 | — | — | | Extraction — a known fact is present | 55 | 51 | 4 | — | | Constraint — a rule is never broken | 45 | 45 | — | — | | Negative — something must not appear | 40 | 38 | 2 | — | | Comparative — better than the previous version | 25 | n/a | n/a | — | | Human judgement — sampled | 15 | n/a | n/a | 1 disputed | **Structural and constraint assertions pass 5/5 or they are not assertions.** If the output shape varies between runs, downstream code cannot depend on it, and that is a defect regardless of how good the summary is. Content assertions are allowed 4/5 because natural variation exists, and a suite that demands identical wording will be deleted by the third engineer who has to update it. ## 2. What each style asserts Worth spelling out, because "we test the AI feature" usually means only the first row. **Structural.** Valid JSON; `summary` and `actions` both present; every action has `text`, `owner`, `due`; `owner` is either a name from the transcript or null. These never vary. **Extraction.** A transcript with a stated decision must produce a summary containing that decision. Checked by presence of the specific fact, not by string match — the wording is allowed to differ, the fact is not. **Constraint.** No action attributed to a person not present. No date earlier than the meeting date. No more than 12 actions. Rules that must hold on every output whatever it says. **Negative.** Personal details mentioned in passing must not appear. Salary discussion must not appear in a summary marked shareable. **Negative assertions are the ones teams skip and the ones that catch the embarrassing failures.** **Comparative.** Blind human preference against the previous version, reported separately. **Human judgement.** Fifteen sampled outputs read by a person. Not a gate; a check that the gates measure something worth measuring. ## 3. The flakiness investigation Six cases passed 4/5 rather than 5/5. Before accepting them, each was investigated, because **a test that fails one time in five is either a real defect at 20% or a badly written assertion, and the difference matters.** | Case group | Cases | Finding | |---|---|---| | Assertion too strict | 3 | Required "budget approved"; output said "the budget was signed off". Assertion fixed. | | Genuine intermittent defect | 2 | Speaker attribution fails when two speakers share a first name. **Real.** | | Genuine ambiguity | 1 | The transcript itself is ambiguous about who owns an action. Accepted. | Three assertions were wrong and were fixed. **Two were a real defect that would have been dismissed as flakiness** — attributing an action to the wrong person is exactly the failure a summary feature cannot have, and it appeared at a 20% rate in a narrow condition. The single ambiguous case is documented and accepted. Recording why a known deviation is tolerated is what stops it being re-investigated every release. ## 4. The accepted deviation One negative assertion passes 4/5: a transcript where someone mentions a medical appointment in passing. In one run of five it appears in the summary. Not blocking, because the summary is reviewed before sharing and the appointment is incidental. It is recorded as an open defect with an owner, and the case stays in the suite at 4/5 so a regression to 3/5 is visible. **Accepting a deviation is a decision with a name attached, not an absence of a decision.** ## 5. Non-determinism, handled - Temperature pinned for the suite. Production uses a higher setting; the suite exists to test logic, not variation. - Every case run 5 times. A single run tells you nothing about a system that varies. - Model version pinned and recorded. A version change re-runs everything as a regression. - Seeds recorded where the provider supports them. **Five repetitions is the smallest number that distinguishes "passes" from "passed once".** It triples the runtime and it is what makes the result mean anything. ## 6. Cost | | | |---|---| | Full suite, 240 cases × 5 runs | 1,200 calls, 18 min, $3.60 | | Pull request subset — structural and constraint only | 105 cases × 3, 4 min, $0.80 | ## 7. Release recommendation **Ship.** All structural and constraint assertions pass 5/5. The two-speaker defect is real, is narrow, and has an owner and a fix planned for the next release; a case is in the suite and will fail if it worsens. --- ## Notes on using this format **Assert on different things at different strictness.** Structure must never vary. Content may. One threshold for both produces either a brittle suite or a meaningless one. **Investigate every flaky test.** Two of six "flaky" cases here were a real defect at a 20% rate. Re-running until green is how those get shipped. **Write negative assertions.** What must never appear is the category that produces the failures nobody wants to explain, and it is the category most suites omit entirely. **Run each case several times.** A single pass on a non-deterministic system is an anecdote.