Every infrastructure team is being asked the same question right now: where does AI actually fit into operations? The honest answer is narrower than the vendor keynotes suggest and wider than the skeptics assume. This post describes the patterns that survived contact with production — what agents do daily on real infrastructure, what stayed human, and how the boundary is decided. No benchmarks, no demos. Just what runs.
What agents actually do
Three patterns carry most of the value in production. None of them are exotic, and that is the point — the value is in the boring volume.
Log triage and alert clustering. A monitoring stack produces thousands of events a night; a human can act on a dozen. An agent that clusters related alerts, suppresses the flapping duplicates, and assembles the surviving incidents into a ranked queue changes the morning shift entirely. The pattern that works: the agent never pages anyone. It prepares the queue; the on-call engineer starts the day at incident twelve instead of event ten thousand.
Dependency and vulnerability scanning. govulncheck, SBOM diffs, and VEX triage run on every build. The agent’s job is not to find CVEs — the scanners do that. The agent’s job is to read the advisory, check whether the affected code path is actually reachable in this deployment, and draft the verdict: exploitable here, not exploitable here, or needs a human look. A two-hour advisory review becomes a two-minute confirmation.
Draft runbooks and RCA first-pass. After an incident, someone has to write the timeline and the follow-up. An agent that assembles the timeline from logs, alerts, and deploys — and drafts a first-pass root-cause hypothesis list — turns a two-hour documentation chore into a thirty-minute review. The engineer corrects the draft; the correction is where their judgment lives.
Where the boundary sits
The boundary is not a technical line — it is a policy, and it should be written down. The version that works in practice:
- Agents prepare, humans decide. Every agent output that reaches production — a config change, a rollback, a notification to a customer — passes a human gate. The gate is cheap because the agent did the reading.
- Agents own reversible, low-blast-radius actions. Scaling a non-critical service, restarting a worker, opening a PR. If the action can be undone by the same agent that did it, it is a candidate.
- Humans own irreversible and ambiguous calls. Incident command, production schema changes, anything customer-facing, anything where the cost of being wrong is not a revert.
The failure mode to design against is not the agent being wrong — it is the agent being confidently wrong and the human rubber-stamping it. The mitigation is structural, not motivational: require the agent to show its evidence (the log lines, the advisory text, the diff) next to every recommendation, so approval is an informed decision rather than a click.
What did not work
Two patterns failed in production and are worth naming, because they are the ones the hype sells.
Fully autonomous remediation. Letting agents close the loop — detect, decide, fix, verify — works in demos with synthetic faults. In production, the fault graph is messier: the “obvious” fix for the alert is sometimes the wrong fix for the incident, and an agent that executes faster than a human can intervene turns a small mistake into an outage. The working compromise is one-step autonomy: the agent executes a single reversible step and stops for confirmation.
Agents writing production code unreviewed. Generated IaC and scripts look correct and fail in specific, plausible ways — wrong region, missing retry, a security group opened a bit too wide. The pattern that works is agent-as-draftsman: the agent writes, the pipeline gates (tests, linting, vulnerability scans), the human reviews the diff. The review is faster because the draft exists, not because it is trusted.
The compounding part
The real gain is not the hours saved on any single task. It is that an agent-augmented operation records everything it does — every triage decision, every drafted runbook, every reviewed diff — and that record becomes searchable context for the next incident. Operations stop depending on one engineer’s memory of that one night three months ago.
That is also the standard to hold any AI-operations proposal to: not “what does it automate?” but “what does it remember, and who can query it?” The first answer makes a demo. The second makes an operation.