# Cluster Readiness Review — Sample

**This is an illustrative example.** The cluster, workloads and figures are invented to show
the shape of a review that finds what breaks before customers do. Copy the structure; run the
checks against your own cluster.

The review is organised by **what happens during an ordinary disruption** — a node drained for
patching, a rolling update, a pod evicted — because that is when a cluster that looked healthy
stops being healthy, and it happens on a schedule rather than by bad luck.

---

## Cluster readiness — production candidate

| | |
|---|---|
| Cluster | 3 control plane, 9 worker nodes |
| Workloads | 41 deployments, 6 stateful sets |
| Reviewed | 21–24 July 2026 |
| Method | Manifest review, live inspection, one controlled node drain |
| Verdict | **Not ready.** 3 blocking findings, all fixable within a week. |

## 1. Blocking findings

### B1 — 17 deployments have no readiness probe

**What it means.** A pod is added to Service endpoints as soon as its container starts,
before the application can serve. Every rolling update sends traffic to processes that are
not listening yet.

**Observed.** During the controlled drain, 2 of the 17 returned connection errors for 8–14
seconds while replacements started. No alert fired, because from the cluster's point of view
nothing was wrong.

**Fix.** A readiness probe on each, checking that the application can serve — not that the
process exists.

### B2 — 6 liveness probes point at dependencies

**What it means.** These probes return unhealthy when the database is unreachable. The
container is then killed and restarted, which does not fix a database, and the restart loop
begins.

**Why it is blocking.** It converts a slow dependency into a cluster-wide restart storm, and
it does so on every pod simultaneously, because they all see the same dependency. The
resulting incident looks like a cluster failure and is not.

**Fix.** Liveness checks the process only. Dependency health belongs in readiness, where the
consequence is removal from traffic rather than a kill.

### B3 — No pod disruption budgets

**Observed during the drain.** Draining a single node took all 3 replicas of the checkout
service down at once. The service was fully unavailable for 41 seconds. This was a planned
drain on a Tuesday afternoon; it would be identical during automated patching at 03:00, with
nobody watching.

**Fix.** A disruption budget on every multi-replica workload. `minAvailable` of one is enough
to prevent the simultaneous case and costs nothing.

## 2. Resource requests

| | Deployments | Note |
|---|---|---|
| No requests set at all | 12 | Scheduler cannot place them sensibly; first to be evicted |
| Requests copied from an example | 19 | Actual usage 5–20% of requested |
| Requests based on measurement | 10 | — |
| Limits far above requests | 22 | Node overcommitted under simultaneous load |

Aggregate requested CPU is **3.4 times measured peak usage**. The cluster appears to need 9
nodes and is doing the work of about 4. This is not a cost finding alone: over-requesting
causes pods to sit in Pending during rolling updates because there is no room for the new one
before the old one goes.

**Fix.** Set requests from measured usage — the data is in the metrics already retained.

## 3. What the drain revealed

The controlled drain was the most valuable hour of the review, and none of what follows was
visible from the manifests.

| Observation | Consequence |
|---|---|
| Checkout lost all replicas simultaneously | 41 seconds fully unavailable (B3) |
| Two services returned errors while starting | 8–14 seconds of failed requests (B1) |
| One stateful set took 6 minutes to reschedule | Its volume is zone-bound; only 2 nodes qualify |
| A pod entered Pending for 3 minutes | Over-large requests, no node had room |
| No alert fired for any of the above | Monitoring watches nodes and pods, not the service |

**The last row is the one to sit with.** Every failure above was invisible to the monitoring
in place, because the monitoring answers "are the pods running" rather than "is the business
function working". Both are useful. Only one of them notices a 41-second outage.

## 4. Configuration and access

| Item | Status |
|---|---|
| Namespaces separate environments | ✅ Yes |
| Secrets from a secret store, not manifests | ✅ Yes |
| Images by digest, not `latest` | ⚠️ 8 by tag, 3 by `latest` |
| Network policy — default deny | ❌ No. Any pod can reach any pod. |
| Role bindings | ⚠️ 4 service accounts with cluster-admin |
| Audit logging | ✅ Enabled, shipped off-cluster |

`latest` is worth naming: those 3 deployments will pull a different image after any restart,
which means the version running is not the version anyone approved, and the difference appears
at a moment nobody chose.

## 5. What is already right

- Control plane is genuinely highly available, verified by failing one node.
- Manifests are in version control and applied by the pipeline. **No manual `kubectl apply`
  into production** — the discipline that makes reconciliation predictable rather than
  mysterious.
- Node patching is automated and staggered.
- Stateful set backups are tested; a restore was performed in June.
- Namespace resource quotas prevent one workload starving the cluster.

## 6. Sequence to production

| Week | Action |
|---|---|
| 1 | Readiness probes on the 17 · fix the 6 liveness probes · disruption budgets on all multi-replica workloads |
| 1 | Repeat the controlled drain. **The gate is a drain with no customer-visible impact.** |
| 2 | Requests from measured usage · pin images by digest · remove the 3 `latest` tags |
| 3 | Default-deny network policy, then allow what breaks · reduce the 4 cluster-admin bindings |
| 3 | Service-level monitoring and alerting, not node-level |
| 4 | Address the zone-bound volume on the slow stateful set |

**The gate is not the checklist.** It is the repeated drain: if a node can be taken out during
working hours and no customer notices, the cluster is ready for the patching window that will
do the same thing at 03:00 without asking.

---

## Notes on using this format

**Drain a node as part of the review.** It took an hour and produced five findings that no
amount of manifest reading would have surfaced. Everything else in this document is
inspection; this is evidence.

**Separate blocking from important.** Twelve deployments with no resource requests is a real
problem and it is not a reason to hold a launch. Three findings that cause customer-visible
outages during routine maintenance are.

**Make the exit gate an observable event**, not a completed list. A checklist can be signed
off with the outage still present. A clean drain cannot.