Benchmarking
Agent Fleets: What to Measure, What to Ignore
Every multi-agent framework claims scale. Few show the receipts.
The industry benchmark for “multi-agent” is a screenshot: two LLM
calls talking to each other, labeled orchestration. That is not
a benchmark. That is a conversation with extra steps. Real benchmarking
asks different questions: How many concurrent workers actually
complete? What is the wall-clock latency at the 99th percentile? What
happens when 20% of them crash simultaneously?
We built an actor mesh on Erlang/OTP — a hierarchy of specialist
cores coordinated by a single orchestrator — and we stress-tested the
orchestration layer itself. This article is the honest ledger of what we
measured, what we stopped measuring, and why the difference matters for
any team running agent fleets in production.
The Metrics That Mattered
1. Completion Rate Under
Concurrency
Concurrency
The headline number from our D1 benchmark: 100 concurrent
profile actors, 100/100 success, ~16.9s wall time, each holding
an independent Python subprocess with {packet,4}
length-prefixed framing. Zero data loss. Zero noproc
errors. Zero silent corruption.
That number exists because the BEAM runtime gives you three
primitives that thread-based orchestration does not:
| Primitive | What it buys you |
|---|---|
| Lightweight processes (~KB each) | 1,000+ actors on one node without OOM |
| Preemptive schedulers (one per core) | No single reasoning loop can starve the fleet |
| Supervision trees ( simple_one_for_one) |
Crash a worker → it respawns automatically, with intensity limits that prevent cascade |
The benchmark is not “we ran 100 agents.” The benchmark is
100 actors × 1 subprocess each × real IPC × zero loss.
If your orchestration layer cannot produce that receipt, you have a
conversation layer, not a mesh.
2. Framing Integrity Under
Load
Load
The naive approach to inter-process communication is line-delimited
text. It fails the moment a payload contains a newline, and under
concurrency it fails silently — partial reads, merged messages,
corrupted JSON that parses but carries wrong data.
We use {packet,4} framing: every
message is prefixed with a 4-byte big-endian length. The reader reads
exactly four bytes, retries on short reads (POSIX pipes always
deliver partial data; this is not an edge case, it is the norm), unpacks
the length, reads exactly that many bytes, and only then processes.
Cost: negligible. Guarantee: a 364-byte response arrives as a
364-byte response, always. Under 100 concurrent subprocesses, the
framing held. No delimiter collisions. No ambiguous boundaries. This is
the metric that separates prototype from production: did any
message corrupt? Ours: zero.
3. Supervision Recovery
Latency
Latency
When a worker crashes, the supervisor restarts it. The metric is not
“does it restart?” — the metric is how long until the
replacement is addressable and productive.
In our mesh, the orchestrator holds PIDs. A restart creates a new
PID. Every caller caching the old one now holds a handle to a dead
process. At 10 actors this is a curiosity. At 100 with even a 2% crash
rate, “stale PID” becomes a background hum of noproc
errors. The supervisor heals the worker; nothing heals the
callers.
This is why S1.2 (PID dispatch vs name registry) exists: the moment
churn enters the picture, addressing by role via a registry
(ETS, global, pg) beats addressing by
handle. The metric to watch: stale-handle error rate per 1k
dispatches. When it rises above zero, your addressing scheme has
outgrown your fleet.
The Metrics We Stopped
Chasing
Chasing
1. “Agents Launched” (Vanity)
Launching 1,000 processes is trivial on BEAM. Launching 1,000
useful processes — each with a live subprocess, a framed pipe,
a restart policy, and a name you can call — is the work. Count
completions, not spawns.
2. Single-Agent Latency
(Misleading)
(Misleading)
A single actor completing in 344ms (simulated) or 654ms (real IPC)
tells you nothing about fleet behavior. The fleet metric is
wall-clock time for N concurrent completions. Our
100-actor run: ~16.9s wall, ~150ms average per call. The per-call
latency dropped under concurrency because the BEAM schedulers
parallelized across 16 cores. Single-threaded orchestration would show
the opposite curve.
3. Token Count / Cost
Per Prompt (Orthogonal)
Per Prompt (Orthogonal)
Model inference cost is real, but it is not an orchestration
metric. The orchestration layer’s job is to make the fleet
addressable, recoverable, and observable. Whether the model
costs $0.002 or $0.02 per call is a procurement decision. Benchmark the
mesh; bill the model.
4. “Number of Tools
Available” (Irrelevant)
Available” (Irrelevant)
The 9 Orders doctrine states: ONE MCP server (OpenFang), ONE
kanban (Hermes), ONE orchestrator (Hermes Layer 0), NARROW GATE for all
tool calls. The metric is not tool count — it is gate
throughput and rejection rate. How many tool calls passed the
gate? How many were rejected for policy? How many timed out? That is the
narrow gate’s dashboard.
The Honest Boundary:
What Breaks at 1,000
What Breaks at 1,000
We are transparent about the limits because the architecture is
designed around them:
| Limit | Threshold | Fix (in roadmap) |
|---|---|---|
| Memory | ~30MB RSS per Python subprocess → ~30GB at 1,000 actors | Shared port dispatcher (S1.5): one Python process serving many BEAM actors |
| Blocking calls | gen_server blocks waiting for port response |
Non-blocking {noreply, State} +gen_server:reply/2 (S1.8) |
| Name registration | PID dispatch works for fixed fleet; production needs role addressing |
ETS/gproc registry + global for cross-node (S1.2,S1.6) |
| Cross-node | Local PIDs meaningless across machines | Erlang distribution + global names (S1.6 —hardware-aligned) |
The roadmap sequences these honestly: S1.2–S1.5 before S1.6
(cross-device mesh) because the registry must exist before distribution
enters. You cannot retrofit addressing onto a distributed fleet.
What This Means for Your
Stack
Stack
If you are running multiple AI agents in production, the lesson
generalizes beyond our council:
-
Put an orchestration substrate under your agents, not
just a prompt layer. The supervision tree is the difference
between “one crashed worker” and “the whole fleet hangs.” Threads don’t
give you that; BEAM does. -
Frame your IPC. If your agents talk to
subprocesses over pipes, use length-prefixed framing
({packet,4}or equivalent). Line-delimited protocols
will corrupt under load, and silent corruption is worse than
failure.
-
Measure concurrency, not conversations. A
benchmark showing 100 concurrent workers completing with zero loss is
worth more than a demo of two models chatting. Ask for the receipt:
concurrent completions, framing integrity, recovery
latency. -
Register roles, not instances. The stable
concepts of your system (the gate, the creative core, the DevOps core,
the swarm) deserve stable addresses. The ephemeral workers deserve
direct handles. The 9 Orders is a role-addressing scheme: Layer 0
Dodecahedron orchestrates, Layer 1 Tetrahedron/Cube/Octahedron own
creative/security/DevOps, Layer 3 Icosahedron swarms. The registry
reflects the doctrine. -
Plan for distribution before you need it. A name
that only resolves on one node is a local optimization wearing a global
hat. If cross-device is on your roadmap, the registry choice is part of
the architecture, not a later patch.
The Doctrine in Numbers
The council doctrine is simple: all tools flow through one
gate, all tasks flow through one board, all agents serve one
system. The architecture makes that doctrine cheap to enforce —
and the numbers show it holds at 100 concurrent workers.
The mesh works at 100 actors on PIDs — we measured it. The mesh will
work at 1,000 actors on names — but only if the registry is designed in
before the churn forces it in. That is the difference between a
prototype that proves a point and a system that survives one.
Grounded in the live erlang-actor-mesh prototype
(profile_actor/council_sup gen_servers, PID-based dispatch), the D1
100-profile benchmark (OTP 29, M-series), the kanban-orchestrator skill
(decomposition playbook, swarm dispatch), and the wiki 9 Orders concept.
Verifiable numbers, not vibes.
