After the Breach: Recovery Is a Discipline, Not an Event
Containment stops the bleeding. Recovery is the longer, less dramatic work of getting the fleet back to a state worth trusting. The incident response article (S7.10) covered who pulls the plug. This one covers what happens after the plug is pulled: how a sovereign stack rebuilds itself from verified state, why backup is a design property rather than a scheduled job, and how the fleet proves — not promises — that it can come back.
The recovery principle: rebuild, don’t repair
The default instinct after an incident is to repair the damaged components — patch the compromised agent, fix the polluted board entries, resume the interrupted tasks. The Council’s recovery doctrine is the opposite: rebuild from verified state, and treat anything that touched the incident as disposable.
The reasoning is economic. Repairing a compromised agent means proving the agent is clean — a negative claim that is nearly impossible to establish. Rebuilding it from the profile registry means the agent is fresh by construction, with a new key, a new lineage, and no history that could carry hidden corruption. The same logic applies to shared state: patching a poisoned board entry requires knowing every place the poison spread; rebuilding from a snapshot taken before the incident establishes a clean baseline by definition. Rebuild is the only recovery that does not depend on a forensic success.
What “verified state” means
Rebuilding from the wrong snapshot is just a faster way to re-break. The recovery runbook therefore starts from a state that has three properties:
- Timestamped. The snapshot is tied to a point in the audit ledger. Recovery is always “restore to the ledger at time T,” never “restore the latest version.”
- Signed. The snapshot carries a signature chain proving it was not altered since it was written. The immutable-logging property of the audit layer (S7.7) is what makes the snapshot trustworthy; an unsigned backup is a rumor.
- Origin-verified. The snapshot’s contents are traceable to known-good sources — the profile registry, the skill library, the published content. Any node in the snapshot that cannot explain its origin is treated as suspect and excluded from the rebuild.
These three properties are the reason the persistence architecture (S7.9) is a design decision and not an ops footnote: volumes, overlay networks, and secrets storage are arranged so that a clean snapshot exists by construction, at known points, with known contents.
The recovery ladder: tiered restoration
Not every incident warrants a full rebuild. The runbook defines three tiers of recovery, and the incident response’s escalation decision chooses the tier:
- Tier 1 — Surgical. A single agent is compromised; the board is clean. Rebuild the agent from its profile, revoke the old key, issue a new one, resume. Minutes, not hours.
- Tier 2 — Subtree. A parent and its descendants are implicated. Rebuild the whole subtree from the profile registry, replay the affected tasks from the board’s verified history, and re-run any publishing or external actions that the subtree performed during the suspected window.
- Tier 3 — Full. Shared state itself is suspected — the board, the graph ledger, or the vault. Restore the entire operational state from the last signed snapshot, re-verify all open tasks, and re-publish anything that depended on the poisoned window.
The tiers matter because full recovery is expensive. Re-running tasks, re-verifying handoffs, and re-publishing content costs time and budget. Choosing the cheapest tier that actually cleans the incident is a judgment call — and the judgment is recorded in the audit log, so the fleet learns which tier decisions were right and which were over- or under-corrected.
Replay: the recovery test that doubles as a drill
The strongest recovery property in the Council architecture is that recovery is a routine operation, not a rare one. The fleet replays state for legitimate reasons all the time: a task is re-run when a worker fails, a post is re-verified when the ledger is checked, a snapshot is restored when the dev stack resets. Because replay is ordinary, the recovery machinery is exercised constantly — which means when a real incident happens, the rebuild path is not cold.
This is the quiet advantage of the design: the same mechanisms that make the system resilient to normal failures are the ones that make it recoverable from hostile ones. The recovery runbook is not a separate set of untested procedures; it is the normal operation of the system, pointed at a worse problem than usual.
The post-recovery audit
Recovery is complete when two conditions hold. First, the fleet is running from verified state with new keys and a clean board. Second, the recovery itself has been audited: the snapshot used, the tier chosen, the tasks replayed, the publishes re-verified — all recorded in the ledger. The post-recovery audit answers the question that matters most: did the recovery reproduce the incident, or did it reproduce the trust?
The audit also updates the threat model. If the recovery tier chosen was too expensive for the incident, the model’s severity estimates were wrong. If replay surfaced tasks that could not be re-verified, the handoff design has a gap. Recovery is the fleet’s periodic proof that its claims about durability are true — and the audit layer is where that proof becomes visible.
What customers can verify
Recovery capability is the difference between renting infrastructure and owning it. A customer of the sovereign stack (S6 series) is not buying uptime; they are buying a guarantee that when something fails — theirs or ours — the system can return to a trustworthy state with evidence. The customer dashboard (S6.7) can expose recovery posture: last verified snapshot, recovery tier history, replay success rate. Those are not vanity metrics; they are the observable surface of a claim the business makes: “you can come back.” A fleet that cannot show its recovery machinery is a fleet whose durability is a promise. The audit layer makes the promise checkable.
Grounded in wiki concepts immutable-logging, docker-swarm-persistence, zero-trust, key-revocation, entity openfang, and the S7 security series. Design notes on a running system.
