Watchdogs That Watch the Watchers: Layered Liveness
Every autonomous system needs a watchdog — something that notices when a worker stops working. But a watchdog is itself a process, and a process can die. The layered liveness pattern answers the recursion: each layer watches the layer below, and the top of the stack watches itself through external probes. The fleet is only alive if its watchdogs are watched.
The layers
Layer one is the worker: each agent heartbeats its liveness to its supervisor (S1.3, S1.5). Layer two is the supervisor: it restarts crashed workers and reports their health upward. Layer three is the fleet orchestrator: it tracks supervisors and escalates when a whole subtree goes dark. Layer four is the external probe: something outside the stack — a scheduler, a health-check endpoint, a human with a pager — confirms that the stack as a whole is reachable. Each layer has one job: notice when the layer below is lying by omission.
The lie of the silent process
The classic failure is the silent process: the worker is technically running but no longer making progress — hung, deadlocked, or looping on the same error. Heartbeats catch the dead process but miss the zombie. The fix is progress-based liveness: the worker must report not just “alive” but “making progress,” and the watchdog must know the expected progress rate. The health-check pattern (OP-8) is the operational form of this — probing for work done, not just process presence.
Watchdogs as cost and as value
Layered liveness costs: each layer is code to write, metrics to store, and noise to filter. The value is the failure budget it protects — a fleet that fails fast fails cheap, and a fleet that fails silently fails expensively. The watchdog stack is the operations equivalent of the trust chain (S7.3): every layer verifies the one below it, and the whole stack is verified from outside. Watchdogs that watch the watchers are how the fleet survives its own autonomy.
Grounded in the OP operations series, the S1 supervision-tree work, the S7.3 zero-trust comms article, and the S12 health-check series. Third article in the Round D operations track.
