# Test Automation Health Report — Sample

**This is an illustrative example.** The suite and figures are invented. The subject is the
one most teams never review: not whether the software is good, but whether the tests are.

---

## Test automation health — Q3 2026

| | |
|---|---|
| Suite | 2,840 automated tests across unit, integration, end-to-end and AI behaviour |
| Runtime | 41 minutes on the pipeline |
| Period reviewed | April – July 2026 |
| Method | Test run history, defect records, coverage against changed code |

## 1. The measure that matters

Not coverage. **Of the defects that reached production, how many could a test have caught?**

| | Defects | Share |
|---|---|---|
| Reached production | 34 | |
| A test existed and passed anyway | 6 | 18% |
| No test existed for this behaviour | 21 | 62% |
| Not testable at this level | 7 | 20% |

**Sixty-two per cent of production defects had no test.** Coverage stands at 78%, and the two
numbers do not contradict each other: coverage measures which lines ran, not which behaviours
were asserted.

The 6 defects where a test existed and passed are the more serious category. Three had
assertions so loose they could not fail; two asserted on a mock that no longer resembled the
real service; one asserted the code's behaviour rather than the requirement, so it was updated
alongside the bug.

## 2. Where the runtime goes

| Layer | Tests | Runtime | Defects caught |
|---|---|---|---|
| Unit | 2,190 | 3 min | 41 |
| Integration | 460 | 9 min | 78 |
| End-to-end | 120 | 22 min | 19 |
| AI behaviour | 70 | 7 min | 24 |

**End-to-end is 54% of runtime and catches 12% of defects.** It is also where every flaky test
lives. Integration is the best value in the suite by a wide margin and is the smallest layer
after AI behaviour.

## 3. Flaky tests, and what they cost

| | |
|---|---|
| Tests that failed then passed on re-run, no code change | 47 |
| Share of the suite | 1.7% |
| Pipeline runs affected in the period | 31% |
| Engineer time spent re-running and investigating | ~26 hours |

**Roughly one pipeline run in three was affected by a test that was not testing anything
stable.** The real cost is not the 26 hours; it is that "re-run it" has become the reflex.

That reflex is expensive. Two of the six production defects in section 1 were caught by a test
that failed legitimately, was assumed flaky, and was re-run until it passed. **A suite with
flaky tests does not merely waste time — it trains people to ignore failures.**

Investigating all 47:

| | Count | Action |
|---|---|---|
| Timing assumption — waits for a fixed duration | 19 | Wait for the condition |
| Shared state between tests | 11 | Isolate |
| Dependence on real external services | 8 | Contract test instead |
| Non-deterministic AI output, single run | 6 | Repeat and threshold |
| Genuinely intermittent product defect | **3** | **Fixed — they were real** |

## 4. The quarter that can be deleted

| | Tests | Reason |
|---|---|---|
| Assert on implementation, not behaviour | 310 | Fail on every refactor, catch nothing |
| Duplicate coverage of the same path | 240 | Same assertion at three layers |
| Test framework behaviour, not our code | 95 | Left from a template |
| Permanently skipped, over 6 months | 61 | Nobody remembers why |
| **Total** | **706 (25%)** | |

Deleting them removes roughly 9 minutes of runtime and no coverage of any behaviour. The 310
implementation-coupled tests are worse than neutral: they are the reason two refactors were
abandoned this quarter, having been scoped without accounting for the test rewrite.

## 5. The AI behaviour layer

Newest layer, best value per test in the suite: 70 tests, 24 defects caught.

| | |
|---|---|
| Cases run per pipeline execution | 70 × 3 repetitions |
| Cost per full run | $1.90 |
| Model version pinned | Yes |
| Catches that no other layer could | 24 |

Its problem is coverage, not quality. Seventy cases across eleven AI features means several
features have three cases, and 9 of the 21 untested production defects were in AI features.

## 6. Recommendations

1. **Delete the 706 tests.** No behaviour loses coverage; 9 minutes and a recurring refactor
   tax go away.
2. **Fix all 47 flaky tests or delete them.** Not next quarter. The reflex they create is what
   let two real defects through.
3. **Move end-to-end coverage down a layer.** Keep about 30 true journeys; convert the rest to
   integration. Most of the 22 minutes and most of the flakiness are here.
4. **Triple the AI behaviour layer**, prioritising the features behind the 9 untested defects.
   Best return available at roughly $6 a run.
5. **Report "defects a test could have caught" every quarter.** It is the only number in this
   report that describes whether the suite is working.
6. **Stop reporting coverage as a target.** It is 78% and it did not predict any of this.

## 7. Projected

| | Now | After |
|---|---|---|
| Tests | 2,840 | 2,340 |
| Runtime | 41 min | 24 min |
| Flaky | 47 | 0 |
| Pipeline runs affected by flakiness | 31% | — |
| AI behaviour cases | 70 | 210 |
| Cost per run | $1.90 | $6.10 |

---

## Notes on using this format

**Audit the suite, not the software.** Nobody owns test quality by default, and a suite decays
in ways that make it slower and less useful at the same time.

**Count defects a test could have caught.** Coverage was 78% while 62% of production defects
had no test. One number describes the suite; the other describes what it does.

**Treat flakiness as a correctness problem.** Three of 47 flaky tests were real intermittent
defects, and two production escapes came from a legitimate failure dismissed as noise.

**Deleting tests is a legitimate result.** A quarter of this suite cost time, blocked
refactoring, and asserted nothing about behaviour.