DevOps: SOPs
Six procedures covering alert hygiene, incident response, the remediation allowlist, capacity review, cost review and the expiry calendar.
Markdown. No sign-up, no email.
SOP 1: Alert hygiene#
Run: weekly, thirty minutes.
An alert channel nobody trusts is worse than no channel, because the real alert arrives in it and dies there unread.
- Every alert that fired: was it acted on?
- Anything that fired and was ignored twice gets fixed or deleted. There is no third option.
- Anything that has never fired gets tested. An untested alert is a hope.
- Target under five alerts per action taken.
Deleting a useless alert is progress, not risk. Teams keep them out of superstition, and the cost is that every alert becomes background noise.
SOP 2: Incident response#
Run: on any detected fault.
- Restore first, diagnose second. They are different jobs and doing them in the wrong order extends the outage while someone reads logs.
- Declare severity early. Downgrading is free; discovering later it was SEV1 is not.
- One person is incident lead. The root-cause agent gives candidates; the lead decides which to pursue.
- Freeze automated remediation on the affected system for the duration. An agent making changes during an incident makes the timeline unreadable, and the timeline is how the cause gets found.
- Postmortem within 48 hours, blameless, with one question at its centre: what would have detected this sooner?
SOP 3: Remediation allowlist review#
Run: quarterly, and on any new automated action.
Adding an action requires all four:
| Requirement | Test |
|---|---|
| Bounded | It cannot affect anything outside its stated target |
| Reversible | Undone in minutes, and the undo has been run |
| Observable | Logged with its trigger, visible to a person afterwards |
| Rate limited | Stops and escalates after two attempts in an hour |
Removing an action requires only that nobody can name a time it helped.
A remediation firing repeatedly is masking a fault. The stop-after-two rule exists because the alternative is an agent restarting a service every four minutes for a weekend while the underlying memory leak goes unexamined.
SOP 4: Capacity review#
Run: monthly.
For every resource that can run out, a dated projection: disk, memory, connection pools, API quotas, IP ranges, licence seats.
The output is a date, not a percentage. "Disk full in 11 days" gets acted on. "Disk at 78%" gets scrolled past.
SOP 5: Cost review#
Run: monthly, with Finance.
- Spend against usage, per service. Diverging lines are the finding.
- Anything provisioned and unused for 30 days: justify or remove.
- Anything oversized against its actual peak: resize.
- Reserved and committed spend against actual: over-committing is as wasteful as over-provisioning.
SOP 6: Expiry calendar#
Run: monthly, 90 days ahead.
Certificates, domains, API keys, licences, signing keys, service credentials.
Expiry is the single most predictable outage available and remains one of the most common, because it is nobody's job until the morning it is everybody's. A 90-day list with an owner per row removes the entire category.
Escalation#
| Situation | Goes to |
|---|---|
| SEV1, customer data or revenue affected | Incident lead, immediately, all hands |
| An automated remediation fired more than twice on the same target | Stop it. Investigate the fault it is hiding |
| A capacity projection under 14 days | Same day, with a spend decision if needed |
| Cost rising while load is flat | Finance and the CTO |
| Anything touching security posture | Security. Never remediated automatically |