Incident Response for Autonomous Fleets: The On-Call Agent
When an autonomous fleet fails, someone must respond — and the responder may itself be an agent. Incident response for autonomous systems (S7.10) established who pulls the plug; the operations series turns that into a process: detection, triage, containment, recovery, and post-mortem. The on-call agent is the pattern that keeps the process running at machine speed.
Detection before panic
The fleet’s watchdogs (OP-3) and health checks (OP-8) detect incidents before humans do: a supervisor subtree goes dark, a spend meter crosses its gate (OP-4), a progress heartbeat stalls. Detection produces an alert with context — which component, which metric, which blast radius — not just “something is wrong.” The alert is the incident’s opening statement, and it should arrive with the log tail attached.
The on-call agent
The on-call agent is a defined role, not a general-purpose AI: it has a runbook (OP-1), a bounded toolset, and an escalation path. On alert, it triages: severity, scope, and whether the runbook covers it. For known incidents, it executes the documented response. For novel incidents, it contains (isolate the failing subtree, stop the bleeding spend) and escalates — to the human on call or to the incident channel. The on-call agent is the operational translation of the narrow-gate discipline (S7.1): it can act, but its actions are gated by policy.
Contain, recover, learn
Containment is the priority — the blast radius must shrink before the root cause is found. Recovery follows the runbook or the post-mortem’s first-draft fix. Learning is the post-mortem (OP-10): every incident produces a timeline, a root cause, and a runbook update so the next occurrence is a known incident. The incident response loop is how the fleet gets more reliable over time — each failure buys a runbook entry and a watchdog improvement. An autonomous fleet that cannot respond to its own incidents is not autonomous; it is unsupervised.
Grounded in the OP operations series, the S7.10 incident-response article, the S7.11 recovery runbook, the OP-3 watchdog article, and the OP-10 post-mortem article. Fifth article in the Round D operations track.
