DevOps: Workflows
The observe, detect, diagnose, act loop, where automated remediation is allowed to close it, and the handoffs with Engineering, Security, Finance and Support.
Markdown. No sign-up, no email.
Two loops run here. A fast one that mostly closes without a person, and a slow one that must not.
What arrives#
| From | What | Becomes |
|---|---|---|
| Every system | Metrics, logs, traces | The baseline, and departures from it |
| Engineering | Deploys and configuration changes | The correlation set for root cause |
| Security | Threat and posture findings | Work, never automated remediation |
| Finance | Budget | The ceiling on automated scaling |
| Support | Customer reports | An incident, and a detection gap to explain |
A customer report is always also a detection failure. Fix the fault, then ask why monitoring did not see it first. That second question is what moves the self-detection rate.
What leaves#
| To | What |
|---|---|
| Engineering | Root-cause candidates with evidence, and the change most likely responsible |
| Security | Anomalies with a security shape, immediately |
| Finance | Cost per unit of load, unused resources, projected spend |
| The business | Incident communications, from a person |
| Knowledge Management | Postmortems and what would have detected it sooner |
The two loops#
The fast loop is deliberately narrow. Widening it is a quarterly decision with four tests to pass, never a convenience taken during an incident.
Handoff contracts#
With Engineering. They own the change; operations owns the environment. The root-cause agent hands over candidates with evidence rather than a verdict, because a confident wrong cause sends a team down a two-hour path.
With Security. Security findings are never automatically remediated. Changing a firewall rule or an access policy in response to an anomaly is exactly how an attacker gets a system to lock out its own operators.
With Finance. They set the scaling ceiling; operations may scale freely beneath it and never above it. Automated spending without a ceiling is the single most expensive thing an agent can be given.
To Support. Support hears about degradation from operations, not from the customer who is already typing.
Cadence#
| Continuous | Observation, allowlisted remediation |
| Daily | Overnight anomalies, automated actions and whether each held |
| Weekly | Alert hygiene, repeat incidents |
| Monthly | Capacity projections, cost review, 90-day expiry list |
| Quarterly | Allowlist review, incident trend analysis |
The failure this design is built against#
An environment covered in dashboards where the first person to notice a problem is still a customer, and an automation layer that quietly restarts a failing service for a fortnight while the fault that causes it is never examined.
Both look healthy on a status page. The self-detection rate catches the first, and the stop-after-two rule catches the second.