DevOps Worked Example — Eleven Days to Four Hours

This is an illustrative example. The team, timeline and figures are invented. What is worth copying is the diagnosis: almost none of the eleven days was work.

The situation#

A platform team of nine at a logistics company. Merge to production took a median of 11 days. The team had asked for three more engineers twice, and had been refused twice, which everyone found frustrating for different reasons.

Leadership believed the team was slow. The team believed it was understaffed. Neither had looked at where the eleven days went.

The measurement#

One person spent a day going through the deployment log and the ticket history for the last 40 changes, timestamping each transition. No tooling was bought.

Under two hours of the eleven days was work. Everything else was queuing.

This reframed the staffing argument entirely. Three more engineers would have produced more changes entering the same queues, which makes queues longer, not shorter. The team was not slow. It was waiting.

The trap they nearly fell into#

The first plan was to attack the largest number: four days waiting for the test environment. The proposal was a second shared environment, which would have cost roughly £40,000 a year and, by the team's own estimate, halved the wait to two days.

It was the wrong move for a reason that only became clear when the queues were considered together. Two shared environments have the same failure as one — they are shared. The wait was caused by contention, and contention returns as soon as the team is a little busier. It would have bought a year.

The alternative was ephemeral environments created from code, per change, destroyed after. More work up front — about three weeks — and it removes the queue rather than shortening it.

The order the changes had to happen in#

This is the part that mattered most, and it was not obvious at the start.

The release window and the approval board both existed because rollback was not trusted. Nobody said so directly; the stated reasons were process and governance. But when asked what would happen if a Thursday release went wrong, the answer was a shrug and an estimate of "most of a day". A three-day wait and a 1.5-day approval are entirely rational responses to a deployment you cannot undo.

So the sequence had to start somewhere other than the biggest number.

Week 1 — make rollback real. The previous artefact was already retained; nothing had ever verified that deploying it worked. It did not: a database migration two releases earlier had dropped a column that the previous version still selected. This is the finding that justified the whole exercise. Every rollback since that migration would have failed, and nobody knew.

Backward-compatible migrations became a rule — add columns before using them, remove them a release later — and rollback was tested on an ordinary Tuesday. Then again a fortnight later.

Weeks 2–4 — ephemeral environments. Created from code per change, destroyed on merge. Four days of waiting became nine minutes of provisioning.

Week 5 — retire the release window. With rollback tested twice and environments no longer contended, the Thursday window had no remaining justification. It was removed for changes that passed post-deploy verification.

Week 6 — narrow the approval board. From every change to changes touching payment handling or personal data — about 10% of them. The board had been approving 40 changes a month, none of which it had the context to assess, and the two that mattered were getting the same fifteen minutes as the other thirty-eight.

Weeks 7–9 — automate the regression pass. Deliberately last. Automating a manual pass is the most work and the least urgent, and doing it first would have delivered a fast test into a queue that still took nine days.

The result#

The failure rate fell while deployment frequency rose sevenfold. That surprises people, and the mechanism is simple: an 11-day pipeline batches changes. Each release contained a fortnight of work from several people, and when one broke, the first job was working out which of eleven changes did it. Frequent deployment made every release small enough that the answer was obvious.

What was learned#

Split lead time into work and wait before doing anything else. It took one person one day and it turned "we need more people" into "we have four queues", which have different remedies and much lower costs.

Find out what a control is protecting against. The release window and the approval board looked like bureaucracy. They were compensating for an untested rollback, and removing them first would have been reckless rather than bold.

Fix rollback before speed. It was the cheapest change of the nine weeks — half a day — and it is what made every subsequent change safe to make. It also uncovered a rollback that had been silently broken for months.

Do not add capacity to a queue. Three more engineers would have made the measured numbers worse, and the team would have been blamed for it.

Back to DevOps