Worked Example — The 52-Minute Pipeline Nobody Waited For
A worked example of a test suite that had become an obstacle — how a slow, flaky pipeline trained a team to bypass it, and what deleting two thirds of it achieved.
This is an illustrative example. The team, suite and figures are invented. The end state — a suite so slow and unreliable that people route around it — is reached gradually and is rarely noticed until something escapes.
The situation#
A team of fourteen. The pipeline took 52 minutes and failed on about a third of runs for reasons unrelated to the change being tested.
The behaviours that had grown around it were entirely rational:
- Merge, then go to lunch; check the result afterwards
- On a red build, re-run first and investigate only if it fails twice
- For urgent fixes, use the documented emergency bypass
- Batch several changes into one merge to pay the 52 minutes once
The last one is the most damaging and the least obviously wrong. It was standard practice.
What made it slow#
| Layer | Tests | Runtime | Defects caught, 6 months |
|---|---|---|---|
| Unit | 1,900 | 4 min | 38 |
| Integration | 280 | 7 min | 71 |
| End-to-end (browser) | 410 | 38 min | 14 |
| AI behaviour | 30 | 3 min | 19 |
End-to-end was 73% of the runtime and caught 12% of the defects. It was also the source of every flaky failure — 44 of the 47 identified.
The 410 browser tests had accumulated the way they usually do: each time a defect escaped, a browser test was added, because that is where the defect had been seen. Nobody ever asked whether it belonged at that level, and nobody removed one.
What it cost#
Measured over one month before any changes were made.
| Pipeline runs | 620 |
| Runs failing for unrelated reasons | 198 (32%) |
| Engineer hours re-running and investigating | 41 |
| Emergency bypasses used | 23 |
| Defects that reached production via a bypass | 4 |
| Median changes per merge | 6 |
The suite's own unreliability was routing changes around it four times a month. Every bypass was authorised, logged and justified. The process was working as designed and the design assumed the pipeline was worth waiting for.
The audit#
Two days. Every end-to-end test was classified by what it actually asserted.
| Tests | Verdict | |
|---|---|---|
| A real user journey, worth a browser | 34 | Keep |
| Business logic reachable at the integration layer | 186 | Move down |
| Duplicate of another test | 88 | Delete |
| Asserts on markup, not behaviour | 61 | Delete |
| Permanently skipped over 6 months | 41 | Delete |
Three hundred and thirty-five browser tests either belonged elsewhere or asserted nothing.
The 61 markup tests were the clearest case: they checked class names and element structure. They failed on every styling change, they had never caught a defect, and they were the reason two design changes had been abandoned as too expensive.
The rebuild#
Four weeks, alongside normal work.
Delete 190 tests. Duplicates, markup assertions and long-skipped tests. No behaviour lost coverage.
Move 186 down to integration. Faster, more reliable, and closer to the logic they assert. This was most of the four weeks.
Keep 34 browser tests. True end-to-end journeys — sign-up, purchase, refund, export.
Fix all 47 flaky tests. Of these, 19 waited a fixed duration instead of for a condition, 11 shared state, 8 called real external services, 6 ran a non-deterministic AI feature once. Three were real intermittent product defects and were fixed as bugs.
Triple the AI behaviour layer, from 30 to 95 cases. Best defect-per-test ratio in the suite and the smallest layer, which is a combination that only survives because nobody was looking.
The result#
| Before | After | |
|---|---|---|
| Tests | 2,620 | 2,395 |
| Runtime | 52 min | 14 min |
| Runs failing for unrelated reasons | 32% | 1% |
| Engineer hours per month on re-runs | 41 | 2 |
| Emergency bypasses per month | 23 | 1 |
| Median changes per merge | 6 | 1 |
| Defects reaching production per month | 7 | 3 |
| Cost per run | $0.60 | $2.40 |
Two secondary effects mattered more than the runtime.
Merges became single changes. Nobody batches to avoid a 14-minute wait. When something does break, the cause is one change rather than six, and it is found in minutes.
Red builds are believed again. At 1% unrelated failure, a red build means something. The "re-run it first" reflex disappeared within a fortnight without anyone being asked to stop.
What was learned#
A slow, flaky suite is worse than a smaller reliable one. It does not fail safe. It trains people to bypass it, and the bypasses are where defects escape.
Tests accumulate at the wrong level. Each of the 410 browser tests was added for a good reason, at the level where the defect had been seen rather than the level where the logic lives.
Deleting tests is a legitimate engineering result. One hundred and ninety went, coverage of behaviour was unchanged, and two shelved design changes became affordable.
Flakiness is a correctness problem. Three of the 47 were real defects hiding in the noise, and the noise let four more through the bypass route.