# From Prototype to Production: The Honest Migration Path
> **Draft — S1.9 · series: CONTENT-ROADMAP-99 S1 (The Mesh) · status: drafted · grounding: erlang-actor-mesh-prototype + kanban-orchestrator + wiki 9-orders · tags: ai-agents, digital-architecture, autonomous-operations, api-first, web40**
> Publish path: new post → lucidhive.com → tag + bridge.
—
Every agent mesh begins as a prototype that works, then quietly becomes a network you cannot afford to see fail. The question is never *whether* to migrate. It is why the migration fails more often than the rebuild that follows it.
This is the honest migration path — the one we are actually taking with `hermes-erl-prototype`, the Erlang/BEAM actor mesh that runs the Hermes profile actors. S1.8 covered the blocking-versus-async gen_server decision. This article is about the harder problem: **how you move from a simulator that proves the mesh to a system that survives a production week.**
## Step Zero: Admit What the Prototype Actually Proves
The single most dishonest thing teams do is mistake a prototype’s success for a production signal. Our prototype runs 100 profile actors in ~16.9 seconds wall time and returns 100/100 success. That number is real, and it is also almost meaningless for production sizing — here is why.
Open `port_worker.py`. The default mode is `simulated_llm`: it sleeps 50–250 milliseconds and returns a synthetic response. The `reason/2` call measures *port framing and process concurrency*, not LLM latency. The mesh is fast because the LLM is fake.
That is not a flaw. It is the correct design of a prototype: **you prove the machinery with a replaceable payload.** The honest migration starts by drawing a hard line between the two claims:
| What the prototype proves | What it does NOT prove |
|—|—|
| `{packet,4}` framing survives concurrency | Real-models-throttle latency profiles |
| 100 BEAM processes multiplex safely | Provider retry/backoff behaviour |
| PID dispatch + supervision topology work | Credential and rate-limit handling |
| The gen_server contract holds at scale | Memory under long tail, cold starts |
Write that table into your README before you touch a line of production code. It is the difference between migrating a system and relocating a delusion.
## The Seam Was Already Designed In
The brill-trick of well-built prototypes is that the migration seam already exists in the code. `port_worker.py` has two paths:
“`python
def live_llm(query): … # real API via httpx
def simulated_llm(query): # sleep + synthetic response
“`
And the call site already falls back:
“`python
response_text = live_llm(query) # tries real first
“`
This is the pattern to preserve. Migration is not a grep-and-replace of the payload — it is **turning the dimmer up on the seam.** Step one is trivial and safe: point `LLM_API_URL` and `LLM_API_KEY` at a real endpoint, keep the fallback, run the same 100-actor test, and measure the delta. You are not rewriting the mesh; you are changing a configuration that the prototype deliberately designed for.
## The Three Real Gaps the Simulator Hides
Once the payload is real, three gaps appear that the simulator put syrup over.
**Gap 1 — Timeouts become inputs, not trivia.** In the prototype, `?REASON_TIMEOUT = 30000` is a constant. With real models, a 30-second hard timeout on a burst-charged provider is either too generous (HTTP 429 storms) or too tight (long-context generations). Production needs a *per-provider, per-request budget* — a policy, not a constant. This is where the 9 Orders doctrine earns its keep: the Dodecahedron (Order 0) that orchestrates must not be hostage to one leaf’s latency. Budgets flow from the top; they are not improvised at the worker.
**Gap 2 — Restart strategy was a prototype convenience.** Look at `council_sup.erl`:
“`erlang
restart => temporary,
strategy => simple_one_for_one
“`
Every profile actor is `temporary` — if one crashes, the supervisor files no restart. That is exactly right for a Python-port prototype where a crash means “your synthetic worker died.” It is wrong for production, where a crashed profile is a revenue stream. Production supervision means `permanent` for the mesh-critical actors, deliberate `transient` for jobs that may finish, and a restart-intensity policy (`intensity => 10, period => 60`) tuned so a cascade in one corner cannot take the fleet down — the S1.3 lesson applied as a migration step.
**Gap 3 — No correlation IDs on the wire.** The `{packet,4}` frame carries a length prefix and a payload. Nothing in it says *which* request this response answers. The prototype gets away with this because each actor is single-flight (`status = idle` guard). The moment one orchestrator fans out N async reasoning casts (S1.8’s `{noreply, State}` pattern), responses arrive and you cannot tell query A from query B. **Correlation IDs belong in the protocol, not bolted on in a post-migration refactor.** Design the wire format first: `{ReqId, Payload}`. Delay production until it is there, because retrofitting it later means every live node takes the change atomically.
## Sequence It As a Doctrine, Not a Sprint
The failed migrations we have seen were one big-bang cutover. The honest path is a sequence of independently verifiable steps, each one shippable, each one reversible:
1. **Instrument the seam.** Log latency, backpressure, and per-model failure distributions on the *simulated* backend. Now you have a baseline that cannot be blamed on provider noise.
2. **Flip the dimmer.** Go live with fallback. Keep `simulated` as the loss-only mode. Measure the delta against baseline.
3. **Harden supervision.** Move `temporary` → `permanent`/`transient` per role. Add the crash-intensity policy. Test that a deliberately killed actor restarts or is cleanly replaced.
4. **Add correlation IDs.** Extend the frame to `{ReqId, Payload}` on both sides. Test fan-out with a single orchestrator driving 100 concurrent casts.
5. **Introduce budgets.** Per-provider timeout and retry policies flow from the orchestrator down. Cap unbounded fan-out so `set_model` on a hot actor cannot fork-bomb the node.
6. **Only then, capacity.** Now the 16.9s/100 benchmark means something. Scale up deliberately, one order-of-magnitude at a time, and re-record the number.
Each step has an exit criterion. If step 3’s supervision fails, you stop there — you do not roll forward with a known-broken restart policy and hope the next step masks it.
## What We Are Deliberately NOT Changing
The most “honest” part of an honest migration is knowing what to leave alone. We are *not* rewriting the blocking single-flight profile actor into a fully async microservice. S1.8 concluded, and we agree: one profile = one reasoning stream, and the BEAM makes 100 blocked processes cheap. That contract is correct for the use case. Migration is not aesthetic modernisation; it is closing the gaps between “works on my bench” and “survives a client.” The async dispatcher enters only for the fan-out case the orchestrator genuinely needs, not as a stylistic replacement for everything.
## The Platform We Migrate Onto Is Already Live
There is a second migration running in parallel, and it is the proof that the path works. The content pipeline you are reading is itself a migrated mesh: it started as a stack of ideas in `IDEAS-BACKLOG`, moved through the `kanban-orchestrator` board as tasks, was drafted, tagged, published as a new post, and bridged back into `MASTER-INDEX-BRIDGE`. That is the same discipline — a seam, a measurable step, a fallback, a doctrine — applied at the orchestration layer instead of the BEAM layer. The machine that produces this article and the machine this article describes are the same machine at two points in its own migration.
## The Honest Rule
There is one sentence that separates the migrations that land from the ones that get reverted:
> **Never change the payload, the topology, and the supervision model in the same release.**
Change one axis, measure, keep the fallback, move on. The prototype was built to make that possible. The honest migration is simply the discipline of not betraying it.
—
*Grounded in the live `hermes-erl-prototype` source (`profile_actor.erl` single-flight gen_server, `council_sup.erl` `simple_one_for_one` + `temporary` restart, `port_worker.py` simulated/live seam with fallback, `{packet,4}` framing), the CONTENT-ROADMAP-99 S1 sequencing (S1.2 PID dispatch → S1.3 supervision → S1.7 benchmarking → S1.8 gen_server patterns → S1.9 migration), the kanban-orchestrator runtime powering this very pipeline, and the 9 Orders deployment doctrine. Verifiable code and a live platform, not vibes.*
## Semantic Relationships
– [[the-nine-orders]] — orchestrates
– [[erlang-actor-mesh-prototype]] — orchestrates
– [[kanban-orchestrator]] — orchestrates
– [[council-of-three]] — orchestrates
– [[zeroclaw]] — orchestrates
– [[openfang]] — orchestrates
– [[microsharks]] — orchestrates
– [[platonic-solid-access-architecture]] — references
