A container that forgets everything when it restarts is not infrastructure. It is a disposable tool — useful until the moment it is not. When you run a production stack on Docker Swarm, the question is not whether containers will restart. They will. The question is whether they remember who they are when they do.
The Kingdom of Truth runs its WordPress site, its security guardian, its creative gateway, and its DevOps core on containers. Some of these are stateless by design. Some are stateful by necessity. The persistence model is not an afterthought — it is a security boundary.
The stateless illusion
Docker containers are designed to be ephemeral. A container’s filesystem is a layered union of read-only image layers and a thin writable layer on top. When the container is destroyed — crash, restart, docker service update — that writable layer goes with it. Any data written to the container’s filesystem during its lifetime is gone.
This is the stateless illusion: the idea that containers are inherently stateless, and that persistence is someone else’s problem. In practice, most production services need state. A WordPress database needs to survive a restart. A vector database needs its embeddings to persist. A configuration store needs its keys to outlive the process that created them.
Docker Swarm provides the tools for persistence. The question is how you use them — and how you secure what you persist.
Named volumes: the persistence primitive
The primary mechanism for container persistence in Docker Swarm is the named volume. A named volume is a storage object managed by Docker that exists independently of any container. Containers mount named volumes at specific paths; when the container is destroyed and recreated, the volume remains.
# A named volume for MariaDB data docker service create \ --name lucidhive_mariadb \ --mount type=volume,source=mariadb_data,target=/bitnami/mariadb \ --mount type=volume,source=mariadb_config,target=/bitnami/mariadb/conf \ bitnami/mariadb:latest
The key properties of named volumes:
- Survive container restarts: the volume is not destroyed when the container stops.
- Survive service updates:
docker service updaterecreates containers but preserves mounted volumes. - Are managed by Docker: the volume driver handles cleanup, backup, and driver-specific features.
- Can be shared: multiple containers can mount the same volume (with caution — not all filesystems support concurrent writes safely).
The Council’s WordPress stack uses named volumes for its MariaDB data and configuration. When the WordPress container is redeployed — whether for a routine update or a crash recovery — the database persists. The 285 published posts, the 3,600+ media attachments, the plugin configurations — all survive because the data lives in a named volume, not in the container’s ephemeral filesystem.
Overlay networks: persistence across the mesh
Docker Swarm’s overlay network driver creates a spanning network that connects services across multiple physical hosts. In a single-node deployment like the Council’s, the overlay network provides service discovery and DNS resolution. In a multi-node deployment, it provides encrypted cross-host communication.
Overlay networks are persistent by default. Once created, they persist until explicitly removed. Services connected to an overlay network maintain their network identity across restarts — the DNS name resolves to the new container instance, not the old one.
# Create a persistent overlay network docker network create --driver overlay --attachable lucidhive_net # Services attach to it docker service create --network lucidhive_net --name wordpress ... docker service create --network lucidhive_net --name mariadb ...
The --attachable flag allows standalone containers (not managed by Swarm) to join the network. This is how OpenFang — the Security Custodian — connects to the mesh without being a Swarm service. OpenFang runs as a native process on the host, bound to port 7076, but it communicates with containerized services through the overlay network.
Configs and secrets: persistent credentials without filesystem exposure
Docker Swarm provides two mechanisms for distributing configuration and secrets to containers: configs and secrets. Both are stored in the Raft log — the Swarm’s consensus mechanism — and are immutable once created.
Configs are unencrypted configuration data mounted as files inside the container. Secrets are encrypted at rest in the Raft log and mounted as files in a tmpfs filesystem (never written to disk on the host).
# Create a secret echo "my-database-password" | docker secret create db_password - # Mount in a service docker service create \ --secret db_password \ --config source=wp_config,target=/bitnami/wordpress/config.php \ ...
The persistence model for configs and secrets is different from volumes. Configs and secrets are versioned in the Raft log — updating a config creates a new version, and services can be updated to use the new version. The old version remains in the log. This is a form of persistence that also provides auditability: you can trace which version of a configuration a service was running at any point in time.
The security boundary of persistence
Persistence is not just an operational concern. It is a security boundary.
When data lives in a named volume, it persists across container restarts. This means a compromised container can leave persistent artifacts — backdoors, modified configurations, exfiltrated data — that survive the container’s destruction. The persistence mechanism that protects your data also protects an attacker’s foothold.
The Kingdom of Truth addresses this through layered controls:
- Volume access is scoped by service: each service mounts only the volumes it needs. WordPress cannot read the ChromaDB volume. ZeroClaw cannot write to the WordPress database volume. The principle of least privilege applies to storage.
- Secrets are encrypted in the Raft log: even if an attacker gains access to the Swarm manager’s filesystem, the secrets are encrypted. The key hierarchy traces back to the oracle tier — the same Ed25519 key structure that governs agent task signing.
- OpenFang monitors container behavior: the Security Custodian watches for anomalous writes to volumes, unexpected network connections, and unauthorized access attempts. A container that suddenly starts writing to a volume it never touched before triggers an alert.
- Cloudflare tunnels prevent direct exposure: the WordPress site is not exposed on a public port. Traffic routes through a Cloudflare tunnel, which terminates TLS at the edge and forwards only authorized requests to the container. The persistence layer (MariaDB) is never directly reachable from the internet.
The restart as a security event
In a stateless model, a restart is a non-event. The container is destroyed, a new one starts, and it picks up where the old one left off — because there was nothing to pick up.
In a stateful model, a restart is a security event. The new container mounts the same volumes, reads the same secrets, and connects to the same network. If the old container was compromised, the new container inherits the compromise through the persistence layer.
This is why the Council treats every container restart as an event to be audited:
- Crash loops trigger investigation: the
docker-crashloop-diagnosisskill provides a systematic approach to diagnosing why a container keeps crashing. But the diagnosis also checks for persistence-layer corruption — a volume that was modified by a compromised container may need to be restored from backup. - Service updates are staged:
docker service updatewith--update-parallelism 1updates one replica at a time, allowing verification before proceeding. If the new container shows anomalies, the update is rolled back. - The audit layer records persistence changes: every volume mount, secret access, and config update is logged. The public audit layer (S7.7 in this series) can replay the history of any persistent resource.
The practical stack
The Council’s production persistence stack runs:
| Service | Volume(s) | Network | Secrets |
|---|---|---|---|
| WordPress | wordpress_data, wordpress_config | lucidhive_net | db_password, wp_keys |
| MariaDB | mariadb_data, mariadb_config | lucidhive_net | root_password |
| ChromaDB | chromadb_data | lucidhive_net | — |
| ZeroClaw | zeroclaw_data | lucidhive_net | api_keys |
| OpenFang | — (host process) | overlay (attachable) | signing_keys |
The total persistence footprint is five named volumes, one overlay network, and six secrets. Every byte of persistent data is accounted for, scoped, and monitored.
What survives
When a container restarts in the Kingdom of Truth, it remembers:
- Its configuration: configs mounted from the Swarm Raft log, versioned and auditable.
- Its credentials: secrets encrypted at rest, mounted in memory only.
- Its data: named volumes that outlive the container’s filesystem.
- Its network identity: overlay network DNS that resolves to the new instance.
- Its security context: OpenFang’s monitoring continues across restarts, the Cloudflare tunnel re-establishes automatically.
What it does not remember:
- Ephemeral state: anything written to the container’s writable layer is gone. This is by design — ephemeral state should never be relied upon.
- Process memory: in-memory caches, session tokens, and temporary computations are lost. Services that need session persistence must store sessions in a named volume or external store.
The persistence model is not about keeping everything alive forever. It is about knowing exactly what survives a restart, why it survives, and who authorized its persistence. In the Kingdom of Truth, persistence is a privilege, not a default.
This article is part of the S7 series on Security & Sovereignty. See also: S7.2 (Solid-Keys for Humans), S7.3 (Zero-Trust Agent Comms), S7.5 (Key Rotation), and S7.7 (The Audit Layer).
