Cluster Readiness Review — Sample
A worked example of reviewing a Kubernetes cluster before it carries production traffic — probes, resource requests, disruption budgets, and the findings that only appear when a node is drained.
Markdown. No sign-up, no email.
This is an illustrative example. The cluster, workloads and figures are invented to show the shape of a review that finds what breaks before customers do. Copy the structure; run the checks against your own cluster.
The review is organised by what happens during an ordinary disruption — a node drained for patching, a rolling update, a pod evicted — because that is when a cluster that looked healthy stops being healthy, and it happens on a schedule rather than by bad luck.
Cluster readiness — production candidate#
| Cluster | 3 control plane, 9 worker nodes |
| Workloads | 41 deployments, 6 stateful sets |
| Reviewed | 21–24 July 2026 |
| Method | Manifest review, live inspection, one controlled node drain |
| Verdict | Not ready. 3 blocking findings, all fixable within a week. |
1. Blocking findings#
B1 — 17 deployments have no readiness probe#
What it means. A pod is added to Service endpoints as soon as its container starts, before the application can serve. Every rolling update sends traffic to processes that are not listening yet.
Observed. During the controlled drain, 2 of the 17 returned connection errors for 8–14 seconds while replacements started. No alert fired, because from the cluster's point of view nothing was wrong.
Fix. A readiness probe on each, checking that the application can serve — not that the process exists.
B2 — 6 liveness probes point at dependencies#
What it means. These probes return unhealthy when the database is unreachable. The container is then killed and restarted, which does not fix a database, and the restart loop begins.
Why it is blocking. It converts a slow dependency into a cluster-wide restart storm, and it does so on every pod simultaneously, because they all see the same dependency. The resulting incident looks like a cluster failure and is not.
Fix. Liveness checks the process only. Dependency health belongs in readiness, where the consequence is removal from traffic rather than a kill.
B3 — No pod disruption budgets#
Observed during the drain. Draining a single node took all 3 replicas of the checkout service down at once. The service was fully unavailable for 41 seconds. This was a planned drain on a Tuesday afternoon; it would be identical during automated patching at 03:00, with nobody watching.
Fix. A disruption budget on every multi-replica workload. minAvailable of one is enough to prevent the simultaneous case and costs nothing.
2. Resource requests#
| Deployments | Note | |
|---|---|---|
| No requests set at all | 12 | Scheduler cannot place them sensibly; first to be evicted |
| Requests copied from an example | 19 | Actual usage 5–20% of requested |
| Requests based on measurement | 10 | — |
| Limits far above requests | 22 | Node overcommitted under simultaneous load |
Aggregate requested CPU is 3.4 times measured peak usage. The cluster appears to need 9 nodes and is doing the work of about 4. This is not a cost finding alone: over-requesting causes pods to sit in Pending during rolling updates because there is no room for the new one before the old one goes.
Fix. Set requests from measured usage — the data is in the metrics already retained.
3. What the drain revealed#
The controlled drain was the most valuable hour of the review, and none of what follows was visible from the manifests.
| Observation | Consequence |
|---|---|
| Checkout lost all replicas simultaneously | 41 seconds fully unavailable (B3) |
| Two services returned errors while starting | 8–14 seconds of failed requests (B1) |
| One stateful set took 6 minutes to reschedule | Its volume is zone-bound; only 2 nodes qualify |
| A pod entered Pending for 3 minutes | Over-large requests, no node had room |
| No alert fired for any of the above | Monitoring watches nodes and pods, not the service |
The last row is the one to sit with. Every failure above was invisible to the monitoring in place, because the monitoring answers "are the pods running" rather than "is the business function working". Both are useful. Only one of them notices a 41-second outage.
4. Configuration and access#
| Item | Status |
|---|---|
| Namespaces separate environments | ✅ Yes |
| Secrets from a secret store, not manifests | ✅ Yes |
Images by digest, not latest | ⚠️ 8 by tag, 3 by latest |
| Network policy — default deny | ❌ No. Any pod can reach any pod. |
| Role bindings | ⚠️ 4 service accounts with cluster-admin |
| Audit logging | ✅ Enabled, shipped off-cluster |
latest is worth naming: those 3 deployments will pull a different image after any restart, which means the version running is not the version anyone approved, and the difference appears at a moment nobody chose.
5. What is already right#
- Control plane is genuinely highly available, verified by failing one node.
- Manifests are in version control and applied by the pipeline. No manual
kubectl applyinto production — the discipline that makes reconciliation predictable rather than mysterious. - Node patching is automated and staggered.
- Stateful set backups are tested; a restore was performed in June.
- Namespace resource quotas prevent one workload starving the cluster.
6. Sequence to production#
| Week | Action |
|---|---|
| 1 | Readiness probes on the 17 · fix the 6 liveness probes · disruption budgets on all multi-replica workloads |
| 1 | Repeat the controlled drain. The gate is a drain with no customer-visible impact. |
| 2 | Requests from measured usage · pin images by digest · remove the 3 latest tags |
| 3 | Default-deny network policy, then allow what breaks · reduce the 4 cluster-admin bindings |
| 3 | Service-level monitoring and alerting, not node-level |
| 4 | Address the zone-bound volume on the slow stateful set |
The gate is not the checklist. It is the repeated drain: if a node can be taken out during working hours and no customer notices, the cluster is ready for the patching window that will do the same thing at 03:00 without asking.
Notes on using this format#
Drain a node as part of the review. It took an hour and produced five findings that no amount of manifest reading would have surfaced. Everything else in this document is inspection; this is evidence.
Separate blocking from important. Twelve deployments with no resource requests is a real problem and it is not a reason to hold a launch. Three findings that cause customer-visible outages during routine maintenance are.
Make the exit gate an observable event, not a completed list. A checklist can be signed off with the outage still present. A clean drain cannot.