How a Council of AI Agents Orchestrates 100 Concurrent Workers Without Chaos
Every month, a new “multi-agent framework” launches. Every month, the same
screenshot: a chat window where one model talks to another model. That is not
multi-agent orchestration. That is a conversation with extra steps.
Real orchestration is a systems problem. It asks: how many workers can you
actually run concurrently, with what latency, and what happens when fifty of
them decide they need an answer at the same instant? We built a council of AI
agents — a hierarchy of specialist cores coordinated by a single orchestrator —
and we stress-tested the orchestration layer itself. This is what the numbers
look like when you stop talking about agents and start measuring them.
The claim everyone makes
Most teams describe “scale” as: we ran a prompt, it produced output, we called
it an agent. Under load, those systems fall apart in three predictable ways:
1. Process explosion — one thread per agent, and the OS scheduler becomes
the bottleneck long before the model does.
2. Port/framing corruption — subprocesses that talk to each other with
naive line-based protocols drop data under concurrency, and the corruption
is silent.
3. Supervision collapse — when a worker dies, nothing restarts it, and the
parent waits forever on a child that no longer exists.
We designed around all three from the start, using the Erlang/BEAM runtime as
the orchestration substrate and a length-prefixed binary framing protocol for
inter-process communication. Then we measured it.
Why BEAM, not threads
The orchestration layer runs on Erlang/OTP, which is not a fashionable choice
for AI work — and that is precisely why it works. The BEAM virtual machine is
built for exactly this shape of problem:
- **Processes, not threads.** Erlang processes are lightweight (a few KB),
isolated, and garbage-collected independently. A single node supports over
one million of them. Your “thousand agents” question isn’t a question on
BEAM; it’s a Tuesday.
- **Schedulers match cores.** On an M-series Mac, OTP runs one scheduler per
core (16 on the Max). The runtime does preemptive scheduling — no worker
can starve the others, no matter how long its reasoning loop runs.
- **Supervision trees.** Every worker is a child of a supervisor with a
restart policy. `simple_one_for_one` means a dying worker is respawned
automatically, with intensity limits that prevent crash-loop cascades. The
“who restarts it?” question has a structural answer, not a hope.
Each profile actor is a `gen_server` — a stateful process that owns an
identity (id, name, model), a status (`idle | reasoning | responding |
error`), and a dedicated subprocess port for its work. Dispatch is by PID, not
by name: the orchestrator holds the handles and talks to workers directly.
The wire protocol that doesn’t lose bytes
The workers are Python subprocesses — one per actor — because that is where
the actual model calls and tool work happen. Erlang and Python do not share
memory, so they talk over pipes. The naive approach is line-delimited text;
it fails the moment a payload contains a newline, and under concurrency it
fails silently.
We use `{packet,4}` framing: every message is prefixed with a 4-byte
big-endian length. The reader reads exactly four bytes, retries on short
reads (POSIX pipes deliver partial data; this is not an edge case, it is the
norm), unpacks the length, reads that many bytes, and only then processes.
No delimiters to collide with. No ambiguous boundaries. The cost is
negligible; the guarantee is that a 364-byte response arrives as a 364-byte
response, always.
The benchmark
Phase 1 (simulated reasoning, no subprocesses): 1 profile → ~344ms lifecycle;
10 profiles → ~2.7s; 100 profiles → ~29.6s, zero contention.
Phase 2 (real Python subprocess IPC, `{packet,4}` framing): 1 profile →
~654ms; 10 profiles → ~2.2s; 100 profiles → ~16.9s, ~150ms average per
call, 100/100 success, zero data loss.
Read that second result again: one hundred BEAM processes, each holding an
independent Python subprocess, all reasoning concurrently, all answered,
nothing corrupted. The OS scheduler handles the subprocess load; the BEAM
handles the orchestration; the framing protocol handles the data integrity.
That is what “the mesh works” looks like in numbers instead of adjectives.
What breaks at 1000
We are honest about the limits, because the whole point is to design around
them:
- **Memory.** Each Python subprocess costs roughly 30MB RSS. At 1000 actors
that is ~30GB before the model calls. The fix is a shared port dispatcher —
one Python process serving many BEAM actors — instead of per-actor
subprocesses.
- **Blocking calls.** The current `gen_server` blocks waiting for the port
response. A non-blocking design returns `{noreply, State}` and replies via
`gen_server:reply/2` when the port delivers. For high-throughput reasoning,
that is the next architectural step.
- **Name registration.** PID-based dispatch is perfect for a prototype; a
production mesh wants a name→PID registry (ETS or gproc) so actors can be
addressed by role, not by handle.
What this means for your stack
The pattern generalizes beyond our council. If you are running multiple AI
agents in production:
- **Put an orchestration substrate under your agents, not just a prompt
layer.** The supervisor tree is the difference between “one crashed worker”
and “the whole fleet hangs.”
- **Frame your IPC.** If your agents talk to subprocesses over pipes, use
length-prefixed framing. Line-delimited protocols will corrupt under load,
and silent corruption is worse than failure.
- **Measure concurrency, not conversations.** A benchmark that shows 100
concurrent workers completing with zero loss is worth more than a demo of
two models chatting.
The council doctrine is simple: all tools flow through one gate, all tasks
flow through one board, all agents serve one system. The architecture makes
that doctrine cheap to enforce — and the numbers show it holds at 100
concurrent workers.
*This article is grounded in a live internal benchmark (100-profile actor
mesh, OTP 29, M-series). Verifiable numbers, not vibes.*
