DevOps: AI Agents
The move from DevOps to AIOps, the ten areas AI watches continuously, and the hard limits on automated remediation.
Markdown. No sign-up, no email.
Traditional operations is a discovery problem. Something breaks, a person notices, a person investigates, a person fixes. Every step waits for attention, and attention is the scarcest thing in the building at 3am.
AIOps changes the first two steps and deliberately leaves the last one guarded.
The ten areas watched continuously#
| Area | What the watching actually catches |
|---|---|
| Server monitoring | Resource exhaustion before it becomes an outage |
| Application monitoring | Latency and error-rate movement, per endpoint |
| Infrastructure monitoring | The layer people forget until DNS or a certificate expires |
| Capacity planning | The date a resource runs out, projected from the trend |
| Incident detection | A problem, before a customer reports it |
| Root-cause analysis | Which change or dependency caused this, not just that it broke |
| Predictive maintenance | Disks, certificates, keys and licences, before expiry |
| Cloud optimisation | Resources provisioned and unused |
| Cost optimisation | Spend trend against usage trend, which are not the same line |
| Automated remediation | The narrow set of fixes safe to apply without a person |
The four agents#
Observation agent#
Watches metrics, logs and traces, and reports deviation from normal rather than threshold breaches. Thresholds catch the failures you predicted; a service quietly serving twice as many errors as usual while staying under every limit is the one that surprises you.
Reach: telemetry, read-only. Output: anomalies, ranked, with the baseline they departed from.
Root-cause agent#
Correlates a symptom with what changed: deploys, configuration, dependency versions, traffic composition. Most incidents have a change behind them, and the delay in finding it is most of the outage.
Reach: telemetry, deploy history, configuration history. Read-only. Output: ranked candidate causes with the evidence for each, never a single confident answer.
Capacity agent#
Projects when a resource runs out at the current trend, and what it will cost to not run out.
Reach: usage history and billing. Output: dated projections. "Disk full in 11 days" is actionable; "disk at 78%" is not.
Cost agent#
Tracks spend against usage. The interesting finding is almost never the total, it is the resource that has been running for three months and serving nothing.
Reach: billing and inventory, read-only. Output: unused resources, oversized instances, and spend rising faster than load.
Automated remediation, and its limits#
This is the part that must be bounded, because an agent with production write access and a bad hypothesis can turn a degraded service into an outage faster than any human could.
Allowed without a person, on a named allowlist, rate-limited, always logged:
| Action | Why it is safe |
|---|---|
| Restart a stateless service that failed its health check | Bounded, reversible, well understood |
| Clear a known-safe temporary directory above a threshold | The alternative is disk exhaustion |
| Scale out within a pre-approved ceiling | Costs money, not data |
| Rotate a log file, recycle a stuck worker | Routine housekeeping |
| Fail over to a standby that is already healthy | The path is designed and tested |
Never without a person:
- Anything that deletes data, or could
- Schema or migration operations
- Scaling beyond the approved ceiling, which is a spending decision
- Modifying security groups, firewall rules or access policy
- Rolling back a release. That is an accountability call
- Any action on a system already in a declared incident, where an automated change makes the timeline unreadable
Three hard rules on the allowlist. Every automated action is logged with its trigger. Any action repeating more than twice in an hour stops and escalates, because a remediation firing in a loop is masking the real fault. And the allowlist is reviewed quarterly, because it grows by exception and nobody remembers to shrink it.
What stays with a person#
- Declaring an incident, and its severity
- Rolling back a release
- Any change to access, network or security posture
- Approving spend, including scaling past the ceiling
- Closing an incident and writing what actually happened