Test Automation Health Report: Sample
A worked example of auditing a test suite rather than the software, what the suite actually catches, the cost of the tests nobody trusts, and the quarter of it that could be deleted with no loss.
Markdown. No sign-up, no email.
This is an illustrative example. The suite and figures are invented. The subject is the one most teams never review: not whether the software is good, but whether the tests are.
Test automation health: Q3 2026#
| Suite | 2,840 automated tests across unit, integration, end-to-end and AI behaviour |
| Runtime | 41 minutes on the pipeline |
| Period reviewed | April, July 2026 |
| Method | Test run history, defect records, coverage against changed code |
1. The measure that matters#
Not coverage. Of the defects that reached production, how many could a test have caught?
| Defects | Share | |
|---|---|---|
| Reached production | 34 | |
| A test existed and passed anyway | 6 | 18% |
| No test existed for this behaviour | 21 | 62% |
| Not testable at this level | 7 | 20% |
Sixty-two per cent of production defects had no test. Coverage stands at 78%, and the two numbers do not contradict each other: coverage measures which lines ran, not which behaviours were asserted.
The 6 defects where a test existed and passed are the more serious category. Three had assertions so loose they could not fail; two asserted on a mock that no longer resembled the real service; one asserted the code's behaviour rather than the requirement, so it was updated alongside the bug.
2. Where the runtime goes#
| Layer | Tests | Runtime | Defects caught |
|---|---|---|---|
| Unit | 2,190 | 3 min | 41 |
| Integration | 460 | 9 min | 78 |
| End-to-end | 120 | 22 min | 19 |
| AI behaviour | 70 | 7 min | 24 |
End-to-end is 54% of runtime and catches 12% of defects. It is also where every flaky test lives. Integration is the best value in the suite by a wide margin and is the smallest layer after AI behaviour.
3. Flaky tests, and what they cost#
| Tests that failed then passed on re-run, no code change | 47 |
| Share of the suite | 1.7% |
| Pipeline runs affected in the period | 31% |
| Engineer time spent re-running and investigating | ~26 hours |
Roughly one pipeline run in three was affected by a test that was not testing anything stable. The real cost is not the 26 hours; it is that "re-run it" has become the reflex.
That reflex is expensive. Two of the six production defects in section 1 were caught by a test that failed legitimately, was assumed flaky, and was re-run until it passed. A suite with flaky tests does not merely waste time, it trains people to ignore failures.
Investigating all 47:
| Count | Action | |
|---|---|---|
| Timing assumption, waits for a fixed duration | 19 | Wait for the condition |
| Shared state between tests | 11 | Isolate |
| Dependence on real external services | 8 | Contract test instead |
| Non-deterministic AI output, single run | 6 | Repeat and threshold |
| Genuinely intermittent product defect | 3 | Fixed, they were real |
4. The quarter that can be deleted#
| Tests | Reason | |
|---|---|---|
| Assert on implementation, not behaviour | 310 | Fail on every refactor, catch nothing |
| Duplicate coverage of the same path | 240 | Same assertion at three layers |
| Test framework behaviour, not our code | 95 | Left from a template |
| Permanently skipped, over 6 months | 61 | Nobody remembers why |
| Total | 706 (25%) |
Deleting them removes roughly 9 minutes of runtime and no coverage of any behaviour. The 310 implementation-coupled tests are worse than neutral: they are the reason two refactors were abandoned this quarter, having been scoped without accounting for the test rewrite.
5. The AI behaviour layer#
Newest layer, best value per test in the suite: 70 tests, 24 defects caught.
| Cases run per pipeline execution | 70 × 3 repetitions |
| Cost per full run | $1.90 |
| Model version pinned | Yes |
| Catches that no other layer could | 24 |
Its problem is coverage, not quality. Seventy cases across eleven AI features means several features have three cases, and 9 of the 21 untested production defects were in AI features.
6. Recommendations#
- Delete the 706 tests. No behaviour loses coverage; 9 minutes and a recurring refactor tax go away.
- Fix all 47 flaky tests or delete them. Not next quarter. The reflex they create is what let two real defects through.
- Move end-to-end coverage down a layer. Keep about 30 true journeys; convert the rest to integration. Most of the 22 minutes and most of the flakiness are here.
- Triple the AI behaviour layer, prioritising the features behind the 9 untested defects. Best return available at roughly $6 a run.
- Report "defects a test could have caught" every quarter. It is the only number in this report that describes whether the suite is working.
- Stop reporting coverage as a target. It is 78% and it did not predict any of this.
7. Projected#
| Now | After | |
|---|---|---|
| Tests | 2,840 | 2,340 |
| Runtime | 41 min | 24 min |
| Flaky | 47 | 0 |
| Pipeline runs affected by flakiness | 31% | |
| AI behaviour cases | 70 | 210 |
| Cost per run | $1.90 | $6.10 |
Notes on using this format#
Audit the suite, not the software. Nobody owns test quality by default, and a suite decays in ways that make it slower and less useful at the same time.
Count defects a test could have caught. Coverage was 78% while 62% of production defects had no test. One number describes the suite; the other describes what it does.
Treat flakiness as a correctness problem. Three of 47 flaky tests were real intermittent defects, and two production escapes came from a legitimate failure dismissed as noise.
Deleting tests is a legitimate result. A quarter of this suite cost time, blocked refactoring, and asserted nothing about behaviour.