--- title: "Benchmarking Agent Fleets: What to Measure, What to Ignore" id: "12428" type: "post" slug: "benchmarking-agent-fleets-what-to-measure-what-to-ignore" published_at: "2026-08-04T20:13:22+00:00" modified_at: "2026-08-04T20:13:22+00:00" url: "https://lucidhive.com/benchmarking-agent-fleets-what-to-measure-what-to-ignore/" markdown_url: "https://lucidhive.com/benchmarking-agent-fleets-what-to-measure-what-to-ignore.md" excerpt: "Benchmarking Agent Fleets: What to Measure, What to Ignore Every multi-agent framework claims scale. Few show the receipts. The industry benchmark for “multi-agent” is a screenshot: two LLM calls talking to each other, labeled orchestration. That is not a benchmark...." taxonomy_category: - "Digital Architecture" taxonomy_post_tag: - "actors" - "ai agents" - "autonomous operations" - "benchmarking" - "erlang" - "mesh" - "orchestration" --- SHARE [https://www.facebook.com/sharer.php?u=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F](https://www.facebook.com/sharer.php?u=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F) [https://twitter.com/intent/tweet?text=Benchmarking+Agent+Fleets%3A+What+to+Measure%2C+What+to+Ignore&url=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F&via=](https://twitter.com/intent/tweet?text=Benchmarking+Agent+Fleets%3A+What+to+Measure%2C+What+to+Ignore&url=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F&via=) [#](#) [javascript:if(window.print)window.print()](javascript:if(window.print)window.print()) [#](#) # Benchmarking Agent Fleets: What to Measure, What to Ignore Every multi-agent framework claims scale. Few show the receipts. Contents [The Metrics That Mattered](#the-metrics-that-mattered) [1. Completion Rate Under Concurrency](#1-completion-rate-underconcurrency) [2. Framing Integrity Under Load](#2-framing-integrity-underload) [3. Supervision Recovery Latency](#3-supervision-recoverylatency) [The Metrics We Stopped Chasing](#the-metrics-we-stoppedchasing) [1. “Agents Launched” (Vanity)](#1-agents-launched-vanity) [2. Single-Agent Latency (Misleading)](#2-singleagent-latencymisleading) [3. Token Count / Cost Per Prompt (Orthogonal)](#3-token-count-costper-prompt-orthogonal) [4. “Number of Tools Available” (Irrelevant)](#4-number-of-toolsavailable-irrelevant) [The Honest Boundary: What Breaks at 1,000](#the-honest-boundarywhat-breaks-at-1000) [What This Means for Your Stack](#what-this-means-for-yourstack) [The Doctrine in Numbers](#the-doctrine-in-numbers) - Advertisement - The industry benchmark for “multi-agent” is a screenshot: two LLM calls talking to each other, labeled *orchestration*. That is not a benchmark. That is a conversation with extra steps. Real benchmarking asks different questions: **How many concurrent workers actually complete? What is the wall-clock latency at the 99th percentile? What happens when 20% of them crash simultaneously?** We built an actor mesh on Erlang/OTP — a hierarchy of specialist cores coordinated by a single orchestrator — and we stress-tested the orchestration layer itself. This article is the honest ledger of what we measured, what we stopped measuring, and why the difference matters for any team running agent fleets in production. - Advertisement - ## The Metrics That Mattered ### ### 1. Completion Rate Under Concurrency The headline number from our D1 benchmark: **100 concurrent profile actors, 100/100 success, ~16.9s wall time**, each holding an independent Python subprocess with `{packet,4}` length-prefixed framing. Zero data loss. Zero `noproc` errors. Zero silent corruption. That number exists because the BEAM runtime gives you three primitives that thread-based orchestration does not: | Primitive | What it buys you | | --- | --- | | Lightweight processes (~KB each) | 1,000+ actors on one node without OOM | | Preemptive schedulers (one per core) | No single reasoning loop can starve the fleet | | Supervision trees (simple_one_for_one) | Crash a worker → it respawns automatically, with intensity limits that prevent cascade | The benchmark is not “we ran 100 agents.” The benchmark is **100 actors × 1 subprocess each × real IPC × zero loss**. If your orchestration layer cannot produce that receipt, you have a conversation layer, not a mesh. - Advertisement - ### ### 2. Framing Integrity Under Load The naive approach to inter-process communication is line-delimited text. It fails the moment a payload contains a newline, and under concurrency it fails silently — partial reads, merged messages, corrupted JSON that parses but carries wrong data. We use **`{packet,4}` framing**: every message is prefixed with a 4-byte big-endian length. The reader reads exactly four bytes, retries on short reads (POSIX pipes *always* deliver partial data; this is not an edge case, it is the norm), unpacks the length, reads exactly that many bytes, and only then processes. Cost: negligible. Guarantee: a 364-byte response arrives as a 364-byte response, always. Under 100 concurrent subprocesses, the framing held. No delimiter collisions. No ambiguous boundaries. This is the metric that separates prototype from production: **did any message corrupt?** Ours: zero. - Advertisement - ### ### 3. Supervision Recovery Latency When a worker crashes, the supervisor restarts it. The metric is not “does it restart?” — the metric is **how long until the replacement is addressable and productive.** In our mesh, the orchestrator holds PIDs. A restart creates a new PID. Every caller caching the old one now holds a handle to a dead process. At 10 actors this is a curiosity. At 100 with even a 2% crash rate, “stale PID” becomes a background hum of `noproc` errors. The supervisor heals the worker; *nothing heals the callers*. This is why S1.2 (PID dispatch vs name registry) exists: the moment churn enters the picture, **addressing by role via a registry (ETS, `global`, `pg`) beats addressing by handle.** The metric to watch: *stale-handle error rate per 1k dispatches.* When it rises above zero, your addressing scheme has outgrown your fleet. - Advertisement - ## ## The Metrics We Stopped Chasing ### 1. “Agents Launched” (Vanity) Launching 1,000 processes is trivial on BEAM. Launching 1,000 *useful* processes — each with a live subprocess, a framed pipe, a restart policy, and a name you can call — is the work. **Count completions, not spawns.** ### ### 2. Single-Agent Latency (Misleading) A single actor completing in 344ms (simulated) or 654ms (real IPC) tells you nothing about fleet behavior. The fleet metric is **wall-clock time for N concurrent completions**. Our 100-actor run: ~16.9s wall, ~150ms average per call. The per-call latency *dropped* under concurrency because the BEAM schedulers parallelized across 16 cores. Single-threaded orchestration would show the opposite curve. ### ### 3. Token Count / Cost Per Prompt (Orthogonal) Model inference cost is real, but it is not an *orchestration* metric. The orchestration layer’s job is to make the fleet *addressable, recoverable, and observable*. Whether the model costs $0.002 or $0.02 per call is a procurement decision. Benchmark the mesh; bill the model. - Advertisement - ### ### 4. “Number of Tools Available” (Irrelevant) The 9 Orders doctrine states: **ONE MCP server (OpenFang), ONE kanban (Hermes), ONE orchestrator (Hermes Layer 0), NARROW GATE for all tool calls.** The metric is not tool count — it is **gate throughput and rejection rate**. How many tool calls passed the gate? How many were rejected for policy? How many timed out? That is the narrow gate’s dashboard. ## ## The Honest Boundary: What Breaks at 1,000 We are transparent about the limits because the architecture is designed *around* them: | Limit | Threshold | Fix (in roadmap) | | --- | --- | --- | | Memory | ~30MB RSS per Python subprocess → ~30GB at 1,000 actors | Shared port dispatcher (S1.5): one Python process serving many BEAM actors | | Blocking calls | gen_server blocks waiting for port response | Non-blocking {noreply, State} + gen_server:reply/2 (S1.8) | | Name registration | PID dispatch works for fixed fleet; production needs role addressing | ETS/gproc registry + global for cross-node (S1.2, S1.6) | | Cross-node | Local PIDs meaningless across machines | Erlang distribution + global names (S1.6 — hardware-aligned) | The roadmap sequences these honestly: S1.2–S1.5 *before* S1.6 (cross-device mesh) because the registry must exist before distribution enters. You cannot retrofit addressing onto a distributed fleet. - Advertisement - ## ## What This Means for Your Stack If you are running multiple AI agents in production, the lesson generalizes beyond our council: 1. **Put an orchestration substrate under your agents, not just a prompt layer.** The supervision tree is the difference between “one crashed worker” and “the whole fleet hangs.” Threads don’t give you that; BEAM does. 2. **Frame your IPC.** If your agents talk to subprocesses over pipes, use length-prefixed framing (`{packet,4}` or equivalent). Line-delimited protocols *will* corrupt under load, and silent corruption is worse than failure. - Advertisement - 3. **Measure concurrency, not conversations.** A benchmark showing 100 concurrent workers completing with zero loss is worth more than a demo of two models chatting. Ask for the receipt: *concurrent completions, framing integrity, recovery latency.* 4. **Register roles, not instances.** The stable concepts of your system (the gate, the creative core, the DevOps core, the swarm) deserve stable addresses. The ephemeral workers deserve direct handles. The 9 Orders is a role-addressing scheme: Layer 0 Dodecahedron orchestrates, Layer 1 Tetrahedron/Cube/Octahedron own creative/security/DevOps, Layer 3 Icosahedron swarms. The registry reflects the doctrine. 5. **Plan for distribution before you need it.** A name that only resolves on one node is a local optimization wearing a global hat. If cross-device is on your roadmap, the registry choice is part of the architecture, not a later patch. - Advertisement - ## The Doctrine in Numbers The council doctrine is simple: **all tools flow through one gate, all tasks flow through one board, all agents serve one system.** The architecture makes that doctrine cheap to enforce — and the numbers show it holds at 100 concurrent workers. The mesh works at 100 actors on PIDs — we measured it. The mesh will work at 1,000 actors on names — but only if the registry is designed in *before* the churn forces it in. That is the difference between a prototype that proves a point and a system that survives one. *Grounded in the live erlang-actor-mesh prototype (profile_actor/council_sup gen_servers, PID-based dispatch), the D1 100-profile benchmark (OTP 29, M-series), the kanban-orchestrator skill (decomposition playbook, swarm dispatch), and the wiki 9 Orders concept. Verifiable numbers, not vibes.* - Advertisement - - Advertisement - TAGGED:[actors](https://lucidhive.com/tag/actors/) [ai agents](https://lucidhive.com/tag/ai-agents/) [autonomous operations](https://lucidhive.com/tag/autonomous-operations/) [benchmarking](https://lucidhive.com/tag/benchmarking/) [Digital Architecture](https://lucidhive.com/tag/digital-architecture/) [erlang](https://lucidhive.com/tag/erlang/) [mesh](https://lucidhive.com/tag/mesh/) [orchestration](https://lucidhive.com/tag/orchestration/) Share This Article [Facebook](https://www.facebook.com/sharer.php?u=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F) [https://twitter.com/intent/tweet?text=Benchmarking+Agent+Fleets%3A+What+to+Measure%2C+What+to+Ignore&url=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F&via=](https://twitter.com/intent/tweet?text=Benchmarking+Agent+Fleets%3A+What+to+Measure%2C+What+to+Ignore&url=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F&via=) [Copy Link](#) [Print](javascript:if(window.print)window.print()) [#](#) Share [https://www.facebook.com/sharer.php?u=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F](https://www.facebook.com/sharer.php?u=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F) [https://twitter.com/intent/tweet?text=Benchmarking+Agent+Fleets%3A+What+to+Measure%2C+What+to+Ignore&url=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F&via=](https://twitter.com/intent/tweet?text=Benchmarking+Agent+Fleets%3A+What+to+Measure%2C+What+to+Ignore&url=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F&via=) [#](#) [javascript:if(window.print)window.print()](javascript:if(window.print)window.print()) [#](#) 00votes Article Rating Subscribe [Login](https://lucidhive.com/wp-login.php?redirect_to=https%3A%2F%2Flucidhive.com%2Fbenchmarking-agent-fleets-what-to-measure-what-to-ignore%2F) 0 Comments OldestNewestMost Voted