Beyond One Host: What Erlang Distribution Actually Gives a Cross-Device Agent Mesh
A single machine can orchestrate a hundred agents. We measured it:
100 profile actors, 100/100 success, ~16.9 seconds of wall time over
real Python-port IPC, zero contention, on one Mac. That is a real
result, and it is the trap.
The next question is not “how do we push a hundred agents harder on
one box.” It is: what happens when the agents live on different
machines? The moment your fleet is split across a Mac mini, a
Raspberry Pi, and a retired laptop, every assumption the single-node
prototype quietly made — that a PID means something, that a crash looks
like a certain thing, that “the mesh” has a single heartbeat — stops
being true.
This article is the honest map of that crossing: what Erlang
distribution gives you beyond one host, what it costs, and what the
five-device hardware round we are actually running will prove.
The single-node ceiling,
precisely
precisely
Our prototype (profile_actor.erl +
council_sup.erl) is deliberately simple. Each profile actor
is a gen_server; the supervisor starts it, hands back a
PID, and the caller holds the handle. Inside one node that is the right
shape — which is exactly what the Phase 2 benchmark proved at 100
profiles, each with an independent Python subprocess on a
{packet,4} port, with no data loss.
The ceiling is not CPU, RAM, or the GIL — those are beatable on one
box. It is simpler and more stubborn: one machine is one fault
domain. Every process shares a single power source, disk, and
network fate. A mesh built to survive agent crashes is worthless if it
cannot survive a host crash — and it cannot move work
elsewhere, because there is no elsewhere. The reason to cross hosts is
not throughput but survivability and co-location:
putting agents near the sensors and radios they drive, letting no single
box own the fleet’s fate.
What “beyond one
host” actually means in Erlang
host” actually means in Erlang
Erlang is one of the only runtimes where distribution is not an
add-on but a first-class feature of the language itself. “Beyond one
host” is not HTTP polling between services or a message bus you bolt on.
It is a set of built-in primitives:
- EPMD — the Erlang Port Mapper Daemon — the phone
book that lets nodes find each other’s ports. It is the discovery layer
and the first thing that fails when firewalls interfere. - Cookies.
-setcookie lucidmeshis the
shared secret; two nodes with different cookies cannot talk. It is the
cluster’s lightweight auth boundary. net_adm:ping/1— the health probe; it
both establishes a connection and reports its state.
nodes()then lists every reachable
node.rpc:call/4— execute any function on
any node as if it were local: the transparent remote-dispatch
primitive.globaland
pg— cluster-wide name registry and
process groups. These make a name resolve from any node — the
cross-node fix for the PID problem covered in S1.2.mon_node— monitor nodes so callers
learn immediately when a host disappears, instead of hanging on a dead
connection.
Notice what is not in that list: no custom protocol, no REST
layer, no configuration service. The runtime ships the mesh. That is the
whole argument for building this on BEAM.
The hardware round:
tailnet as the fabric
tailnet as the fabric
Our test plan is not a thought experiment — it is a five-device
hardware round with a named sequence: start with 2 devices, add the 3rd
to prove healing, add the 5th for the full mesh. The cross-device fabric
rides on Tailscale: the devices join one tailnet, each
gets a fixed 100.x.x.x IP, and Erlang’s
net_adm:ping discovers by hostname through tailnet DNS — as
if finding a sibling on a LAN. No SIMs, no serviced routers, no public
IPs exposed.
# On each device: same cookie, unique name
erl -name council@$(hostname).tailnet -setcookie lucidmesh
# On any node, probe the mesh:
net_adm:ping('[email protected]').
nodes(). % => [[email protected], ...]Phase 1 — the 2-device ping is the crux and needs
zero custom code: just erl + Tailscale + cookie.
If node discovery works over the tailnet, the foundation is proven; if
it does not, no amount of agent code fixes it.
Phases 2–3 — profile actors, then 3-device healing
make it a real mesh: spawn a profile actor remotely, take one
device offline, and watch surviving nodes detect the disconnect and
supervision heal on return. Yanking a device is the single most
important test the single-node prototype could never run — on one node
there is no one to notice.
The failure model changes
completely
completely
This is the honest part, and the part most distributed-systems
writing skips. Crossing hosts does not just add capability; it
replaces your failure model:
- A process crash and a node crash are different
events. On one node, the supervisor owns restart. Across nodes,
the supervisor is local to its node — a peer can only observe
that your node went away;mon_nodetells them, and the
cluster topology decides what survives. - PID locality (the S1.2 hangover). A PID is only
meaningful on the node that created it. “Send this to the reasoner” must
resolve through a cluster-wide name
(global/pg), not a node-local handle. - Partial failure is the default. Across machines,
“some of the mesh answered, one hung, one never got the message” is the
normal state; orchestration must treat partial as a first-class
outcome. - Network partitions lie. A node that stops answering
may be down, or may just be unreachable. Distribution gives you
detection (node_down), not certainty. The design must
assume ambiguity.
None of this is a reason not to do it. All of it is a reason to test
the healing path by actually yanking a machine, which is
precisely the hardware round’s protocol.
The 9 Orders at the fleet
scale
scale
The doctrine maps cleanly once you think of the orchestrator as the
cluster rather than a single process: the Dodecahedron
orchestrates all dispatch, the Icosahedron swarms ephemeral sub-tasks,
the Octagon remembers, the Nonagon reviews. On a multi-host mesh each is
a named role that survives node churn, not a box that
can die. Distribution asks only that the names (openfang,
openclaw, zeroclaw) resolve cluster-wide —
flipping the 9 Orders from a deploy diagram into an addressing scheme
for a real fleet.
The honest status and the
takeaway
takeaway
Ninety percent of the pieces are ready. The BEAM is installed, the
actor code is written and benchmarked at 100 profiles on one node, the
Python port bridge is proven with {packet,4} framing, the
tailnet is within reach. What is not done — and cannot be
hand-waved — is the part that only hardware can prove: two devices
discovering each other, three healing from a yank, five forming a
connected mesh.
If you are orchestrating AI agents that will ever touch more than one
machine, borrow the sequence rather than the scale:
- Start with two, not five. Distribution bugs are
cheapest to catch at the first pair. - Share a cookie, not a hope. Cookie auth and a
common tailnet are the entire bootstrap; get them right first. - Test healing by pulling the plug. The mesh means
nothing until a gone host is detected, its work redistributed, and its
return absorbed. - Address roles, not machines. If your names only
resolve on one node, you have not crossed the boundary — you have merely
started another node.
A single machine can orchestrate a hundred agents — that is the
ceiling that made the prototype possible. Crossing it, one real device
at a time, with distribution built in and healing proved by pulling the
plug on real hardware, is what turns a benchmark into a mesh that
survives a host going dark.
