Test Automation Health Report — Sample
A worked example of auditing a test suite rather than the software — what the suite actually catches, the cost of the tests nobody trusts, and the quarter of it that could be deleted with no loss.
Markdown. No sign-up, no email.
This is an illustrative example. The suite and figures are invented. The subject is the one most teams never review: not whether the software is good, but whether the tests are.
Test automation health — Q3 2026#
| Suite | 2,840 automated tests across unit, integration, end-to-end and AI behaviour |
| Runtime | 41 minutes on the pipeline |
| Period reviewed | April – July 2026 |
| Method | Test run history, defect records, coverage against changed code |
1. The measure that matters#
Not coverage. Of the defects that reached production, how many could a test have caught?
| Defects | Share | |
|---|---|---|
| Reached production | 34 | |
| A test existed and passed anyway | 6 | 18% |
| No test existed for this behaviour | 21 | 62% |
| Not testable at this level | 7 | 20% |
Sixty-two per cent of production defects had no test. Coverage stands at 78%, and the two numbers do not contradict each other: coverage measures which lines ran, not which behaviours were asserted.
The 6 defects where a test existed and passed are the more serious category. Three had assertions so loose they could not fail; two asserted on a mock that no longer resembled the real service; one asserted the code's behaviour rather than the requirement, so it was updated alongside the bug.
2. Where the runtime goes#
| Layer | Tests | Runtime | Defects caught |
|---|---|---|---|
| Unit | 2,190 | 3 min | 41 |
| Integration | 460 | 9 min | 78 |
| End-to-end | 120 | 22 min | 19 |
| AI behaviour | 70 | 7 min | 24 |
End-to-end is 54% of runtime and catches 12% of defects. It is also where every flaky test lives. Integration is the best value in the suite by a wide margin and is the smallest layer after AI behaviour.
3. Flaky tests, and what they cost#
| Tests that failed then passed on re-run, no code change | 47 |
| Share of the suite | 1.7% |
| Pipeline runs affected in the period | 31% |
| Engineer time spent re-running and investigating | ~26 hours |
Roughly one pipeline run in three was affected by a test that was not testing anything stable. The real cost is not the 26 hours; it is that "re-run it" has become the reflex.
That reflex is expensive. Two of the six production defects in section 1 were caught by a test that failed legitimately, was assumed flaky, and was re-run until it passed. A suite with flaky tests does not merely waste time — it trains people to ignore failures.
Investigating all 47:
| Count | Action | |
|---|---|---|
| Timing assumption — waits for a fixed duration | 19 | Wait for the condition |
| Shared state between tests | 11 | Isolate |
| Dependence on real external services | 8 | Contract test instead |
| Non-deterministic AI output, single run | 6 | Repeat and threshold |
| Genuinely intermittent product defect | 3 | Fixed — they were real |
4. The quarter that can be deleted#
| Tests | Reason | |
|---|---|---|
| Assert on implementation, not behaviour | 310 | Fail on every refactor, catch nothing |
| Duplicate coverage of the same path | 240 | Same assertion at three layers |
| Test framework behaviour, not our code | 95 | Left from a template |
| Permanently skipped, over 6 months | 61 | Nobody remembers why |
| Total | 706 (25%) |
Deleting them removes roughly 9 minutes of runtime and no coverage of any behaviour. The 310 implementation-coupled tests are worse than neutral: they are the reason two refactors were abandoned this quarter, having been scoped without accounting for the test rewrite.
5. The AI behaviour layer#
Newest layer, best value per test in the suite: 70 tests, 24 defects caught.
| Cases run per pipeline execution | 70 × 3 repetitions |
| Cost per full run | $1.90 |
| Model version pinned | Yes |
| Catches that no other layer could | 24 |
Its problem is coverage, not quality. Seventy cases across eleven AI features means several features have three cases, and 9 of the 21 untested production defects were in AI features.
6. Recommendations#
- Delete the 706 tests. No behaviour loses coverage; 9 minutes and a recurring refactor tax go away.
- Fix all 47 flaky tests or delete them. Not next quarter. The reflex they create is what let two real defects through.
- Move end-to-end coverage down a layer. Keep about 30 true journeys; convert the rest to integration. Most of the 22 minutes and most of the flakiness are here.
- Triple the AI behaviour layer, prioritising the features behind the 9 untested defects. Best return available at roughly $6 a run.
- Report "defects a test could have caught" every quarter. It is the only number in this report that describes whether the suite is working.
- Stop reporting coverage as a target. It is 78% and it did not predict any of this.
7. Projected#
| Now | After | |
|---|---|---|
| Tests | 2,840 | 2,340 |
| Runtime | 41 min | 24 min |
| Flaky | 47 | 0 |
| Pipeline runs affected by flakiness | 31% | — |
| AI behaviour cases | 70 | 210 |
| Cost per run | $1.90 | $6.10 |
Notes on using this format#
Audit the suite, not the software. Nobody owns test quality by default, and a suite decays in ways that make it slower and less useful at the same time.
Count defects a test could have caught. Coverage was 78% while 62% of production defects had no test. One number describes the suite; the other describes what it does.
Treat flakiness as a correctness problem. Three of 47 flaky tests were real intermittent defects, and two production escapes came from a legitimate failure dismissed as noise.
Deleting tests is a legitimate result. A quarter of this suite cost time, blocked refactoring, and asserted nothing about behaviour.