Kubernetes Worked Example — The Outage That Happened Every Tuesday
A worked example of an intermittent outage nobody could reproduce — why the monitoring never saw it, what a controlled node drain revealed in an hour, and the three-line fix.
This is an illustrative example. The company, cluster and figures are invented. The failure pattern — an outage that monitoring cannot see because it is asking the wrong question — is common enough to be worth walking through.
The situation#
An e-commerce company. Roughly once a week, always between 02:00 and 04:00, checkout would fail for somewhere between 30 and 90 seconds. Not enough to page anyone. Enough that overnight orders showed a dip, and enough that the support queue had a recurring "payment page hung" complaint nobody could reproduce.
The team had been looking at it for two months. Every investigation ended the same way: every dashboard was green for the entire window. Pods running, nodes healthy, CPU normal, no restarts, no errors in the application log.
What the monitoring was actually watching#
The cluster monitoring answered three questions: are the nodes up, are the pods running, is resource usage within bounds. All three were yes throughout every incident.
None of them is the question a customer cares about, which is can I complete a checkout right now. There was no monitoring on that at all, and the gap is why two months of investigation produced nothing.
The only real evidence was the load balancer's own metrics, which showed connection failures during the windows — failures to backends the cluster was reporting as healthy.
The hour that solved it#
Rather than continue investigating at 03:00, the team reproduced the conditions at 14:00: they drained a node manually, on purpose, while watching a checkout.
It failed immediately, in the same way, and everything became visible at once.
Node patching was automated and staggered, running in the small hours. Draining a node is exactly what patching does. The team had been treating the incidents as random because they happened at night, when in fact they were the most predictable event in the cluster.
Three findings from that hour:
Checkout had three replicas and no pod disruption budget. All three were scheduled on the node being drained, and all three were evicted simultaneously. The service was fully unavailable until replacements started — 41 seconds in the controlled test.
Two services had no readiness probe. Their replacement pods entered the Service endpoints the moment the container started, several seconds before the application could accept connections. Traffic arrived and was refused, which is what the load balancer had been recording.
One deployment's liveness probe checked the database. During the disruption that dependency was briefly slow, so every pod of that service failed its liveness check at the same moment and was killed. What should have been a brief slowdown became a restart of the whole service.
The fix#
Three changes, none of them large.
Pod disruption budgets on every multi-replica workload, minAvailable: 1. This alone would have prevented the outage: the drain would have waited for a replacement to be ready before evicting the last replica.
Readiness probes on the two services that lacked them, checking that the application can serve rather than that the process exists.
The liveness probe stopped checking the database. Liveness now checks the process only. Dependency health moved to readiness, where the consequence is removal from traffic instead of a kill.
Total change: about three lines of YAML per affected workload, and half a day including testing.
What was added afterwards#
Two things, because the fix was not the whole lesson.
Service-level monitoring. A synthetic checkout runs every 30 seconds against production and alerts on failure. It answers the question the customer asks. The existing node and pod monitoring was kept — it is useful — but it is no longer the only thing watched.
A monthly controlled drain. A node is drained during working hours, on purpose, with someone watching. It takes an hour. Two further problems have been found this way since: a stateful set whose volume was bound to a single availability zone, and a deployment whose resource requests were so large no other node could accept it, leaving it Pending for minutes during every update.
The result#
| Before | After | |
|---|---|---|
| Checkout outages per month | 4–5 | 0 |
| Longest observed outage | 90 sec | — |
| Time to detect, had it recurred | Never detected | 30 sec |
| Effort to fix | Half a day | |
| Investigation time already spent | 2 months |
What was learned#
Monitor the business function, not just the infrastructure. Every dashboard was green during every outage, and every dashboard was correct. They were answering a question nobody was asking.
Reproduce the conditions rather than waiting for the event. Two months of investigating at 03:00 produced nothing. One deliberate drain at 14:00 produced three findings and the fix.
A disruption that happens on a schedule is not random. The nightly timing was the largest clue available, and it was read as an obstacle to investigation rather than as evidence.
Probes are the most consequential three lines in a manifest. Two of the three findings were probes — one missing, one asking the wrong question — and both had been left at defaults that looked harmless.