Welcome to AIEdTalks’ Newsletter!
In today's edition:
Your Multi-Agent System Is a Distributed System
Let’s dive in.
Sponsored by
Pass the Kubernetes exam on your first attempt.
RBAC. Networking. Storage. KodeKloud breaks these complex Kubernetes concepts down into hands-on practice, until you are ready to ace your exam.
Provision live clusters, drain nodes, debug failing pods, fix networking, set up storage. Every concept is explained in plain language before you type a command.
The curriculum is built backwards from the exam objectives and the work you will do on the job. Nothing extra. The scenarios you practice are the ones the exams test, and the ones you hit in production.
Challenges, hands-on labs, and crash courses teach beginners to think like a Kubernetes engineer from day one. Advanced labs go deeper into security, scaling, and complex troubleshooting. Every lab runs on real infrastructure with automated validation.
You're not memorizing answers. You're solving real Kubernetes problems. Starting from zero or sharpening what you have, this is certification prep that sticks.
Today’s Edition
AI TOPIC
Your Multi-Agent System Is a Distributed System
Welcome back. If you have built microservices, queues, or any system with more than one moving part, you already have most of the skills to build multi-agent AI. This issue covers the part you do not have yet. It also maps the patterns you know onto the agents you are now wiring together.
This is a longer, design-focused issue. By the end, you should be able to look at a multi-agent design and name the pattern each part needs, and the order to build them in.
Here is what is inside:
Why a multi-agent system is a distributed system
The one thing about failure your instincts get wrong
A table that maps each distributed-systems idea to its agent version
Eight patterns to build with, each with the code shape
A build order, from Stage 0 to Stage 4, with a rule for when to move up
When the right number of agents is one
AI SYSTEMS: You've built this before
Your instincts about coordination are right. Your instincts about failure are wrong.
In What Is Loop Engineering? we looked at a single agent as a while-loop. Now put several agents together. What you get is not new. It is a distributed system, and it behaves like one.
Start with the failure that catches everyone.
A team wired four agents into a research pipeline. It passed testing. In production, two of them, an analyzer and a checker, started handing work back and forth. Nothing told them to stop. The pipeline ran for days and spent tens of thousands of dollars before anyone noticed. The dashboards were green the whole time. Nothing threw an error. The agents were "working."
That one failure holds the whole issue. Your coordination instincts would have built this pipeline fine. Your failure instincts are why nobody caught it.
Part 1 — It really is a distributed system
Remove the model and look at the shape. You have independent parts. They talk by passing messages. They fail on their own. They need coordinating. They hold their own state. They have to be traced across many steps. That is a distributed system. You have debugged this shape for years.
It is not even a new idea. An agent is an independent thing with its own state that sends messages and can create more of itself. That is the actor model, which Carl Hewitt described in 1973, in an AI paper. We are rebuilding it.
The orchestrator-and-workers design that Anthropic uses for its research agents is also old. A query planner splits work across shard workers. A kernel schedules threads. A tech lead breaks a sprint into tickets. Every framework you have heard of, LangGraph, AutoGen, CrewAI, the OpenAI Agents SDK, is an old distributed-systems idea with a new name.
So most of what you know carries over:
Message passing between parts
Partial failure as the normal case
Orchestration and coordination
Distributed state and consistency tradeoffs
Latency budgets
Tracing across steps
If that were the whole story, you would be done. It is not.
Part 2 — The new failure model
Here is what is different. This is the core of the issue.
A server is deterministic. Same input, same output. When it breaks, it breaks loudly. It crashes, times out, or throws an error. Your monitoring and alerts assume this.
An agent is stochastic. Same input, different output. When it fails, it usually fails quietly. It does not crash. It returns a clean, confident, wrong answer and keeps going. No error. No red light. The wrong answer flows to the next agent, which treats it as fact.
That is the trap. Your instinct says green means healthy. For an agent, green means "still running." It does not mean "correct." The runaway above was not failing. It was doing what it was told, over and over.
Distributed systems has a famous list of false assumptions, called the fallacies of distributed computing (Deutsch and Gosling). The first one is "the network is reliable." The agent era adds one more: "the model is reliable."
Here is the full set of instincts to re-check, side by side:
This is not a soft claim. A UC Berkeley study (MAST, Cemri et al., 2025) hand-labeled more than 1,600 traces across seven multi-agent frameworks. It found 14 distinct failure modes. About 42% came from specification and design, not from the model. Their conclusion is worth keeping: a better base model alone will not fix these. They are systems problems.
Part 3 — The translation table
Most of what you need already has a name. Here is the map from the idea you know to where it lands in a multi-agent system.
Distributed-systems idea | Agent version |
|---|---|
Actor model (Hewitt, 1973) | Each agent is an actor. Private state, async messages, can create more |
"Let it crash" and supervisor trees (Erlang) | A supervisor agent restarts or reroutes stuck sub-agents, with a restart cap |
Timeouts, retries, backoff with jitter | Wrap every model and tool call. Jitter avoids synchronized retries |
Circuit breaker (Nygard) | Trip on runaway loops and cost spikes (see the earlier issue) |
Bulkheads | Isolate agent pools so one failure cannot sink the system |
Idempotency keys, at-least-once delivery | Put keys on side-effecting tools. Assume any action can fire twice |
Saga and compensating transactions (1987) | Multi-step work that half-finishes needs an undo step, not a database rollback |
Dead-letter queue | Send tasks the agents cannot finish to a human review queue |
Backpressure, flow control | Cap fan-out so parallel agents do not blow up cost |
Quorum, voting (Paxos, Raft) | Run several samples, take the majority answer, to fight randomness |
Distributed tracing (OpenTelemetry) | One span per model call and per tool call across the whole run |
CALM theorem | Split work so sub-tasks are independent. Coordination is the cost to avoid |
Part 4 — Eight patterns to build with
The fixes are things you already know. You do not need new theory. You need to apply the old theory on purpose. For each pattern: where it comes from, how it changes for agents, and the code shape.
1. Message passing, not shared state (actor model). Agents should talk by passing messages. They should not read and write one shared blob of state. AutoGen's newer runtime and Ray already model agents this way. The agent twist: pass the full context, not a one-line instruction. If you pass too little, the receiving agent fills the gap with a guess.
2. A hard ceiling, enforced in code. This is the most important line you will write. Cap the steps, the dollars, and the wall-clock time. Put the cap in your code, outside the agent. You cannot ask a runaway process to stop itself.
class RunBudget:
def __init__(self, max_steps=25, max_usd=5.0, max_seconds=300):
self.max_steps, self.max_usd, self.max_seconds = max_steps, max_usd, max_seconds
self.steps, self.usd, self.start = 0, 0.0, time.monotonic()
def charge(self, step_usd):
self.steps += 1
self.usd += step_usd
if self.steps > self.max_steps: raise StopRun("step cap")
if self.usd > self.max_usd: raise StopRun("cost cap")
if time.monotonic() - self.start > self.max_seconds: raise StopRun("time cap")
# call budget.charge(step_cost) at the top of every loop turn, in your code, not the model's3. Timeouts, retries, and backoff, with a check. This is standard practice against the "network is reliable" fallacy. The agent twist: a retried model call may return a different answer. So retry with a check, not blind hope. If a tool failed, hand the error back to the model as an observation. Let it react instead of silently re-running.
4. Idempotency keys on every side-effecting tool. Assume at-least-once delivery. Any tool that charges a card, sends an email, or writes a row can fire more than once. Give the caller a key and skip the work if you have seen that key before.
def send_email(to, body, idempotency_key):
if store.seen(idempotency_key): # already did this exact action
return store.result(idempotency_key)
result = email_api.send(to, body)
store.save(idempotency_key, result)
return result5. Supervisor trees ("let it crash"). This comes from Erlang. A supervisor agent watches the workers. It restarts or reroutes the ones that fail or stall. Add a restart cap, so a restart loop does not become the next runaway. Anthropic's lead-agent and sub-agent design is a supervisor tree with a different name.
6. The saga pattern for multi-step work. This comes from Garcia-Molina and Salem (1987). A long transaction becomes a series of steps, each with an undo, because there is no atomic rollback across services. In a multi-agent system, a workflow that books, then emails, then writes must be able to walk backward when a later step fails. Each undo must be idempotent too.
def run_saga(steps): # steps = [(do, undo), ...]
done = []
try:
for do, undo in steps:
do(); done.append(undo)
except Exception:
for undo in reversed(done): # undo in reverse order
undo()
raise7. Quorum and voting to fight randomness. Classic consensus assumes reliable nodes agreeing on an order. You cannot get that from non-deterministic agents. But you can borrow the idea. Run the same step several times and take the majority answer. This is called self-consistency. The original paper (Wang et al., 2022) measured real gains over a single try: +17.9 points on GSM8K, +11.0 on SVAMP, +12.2 on AQuA. Two limits: it only works where answers can be compared, and it multiplies token cost. Use it on the few high-stakes decisions, not on everything.
8. Distributed tracing (OpenTelemetry). Record one span per model call and one per tool call, across the whole run. OpenTelemetry now ships GenAI conventions (gen_ai.request.model, gen_ai.usage.input_tokens, tool-execution spans), so you do not have to invent your own. Anthropic has said that adding full tracing was what let them find why agents failed and fix it. Do this before you add a second agent. Otherwise you are debugging a distributed system with print statements.
Circuit breakers, bulkheads, and backpressure finish the list. Circuit breakers got their own deep dive in Your Multi-Agent System Needs a Circuit Breaker, so this issue keeps them short.
Part 5 — A build order
Do not build all of this at once. Build it in the order that removes the most risk per hour of work. Each stage has a rule for when to move up.
Stage 0 — Default to one agent. If the task is write-heavy or tightly linked, like writing one document or one code change, do not build a swarm. Spend your effort on context first. Move up only when you can show the task splits into independent, mostly-read parallel threads, and the answer is worth several times the token cost.
Stage 1 — Make the single agent production-grade. In order: tracing (one span per call), timeouts and retries with backoff, durable execution so a crash resumes instead of restarts (Temporal, LangGraph checkpointers, or Ray), and the hard ceiling from pattern 2. Move up when a single step's context regularly runs into the tens of thousands of tokens. That is a capacity problem more agents can help with.
Stage 2 — Add a supervisor before you add peers. Prefer orchestration, where a lead agent directs workers, over choreography, where agents hand off to each other freely. Add supervisor restarts with caps. Make every side-effecting tool idempotent. Assume at-least-once.
Stage 3 — Fan out, with limits. Only now add parallel sub-agents. Put backpressure on the fan-out and bulkheads between pools. Do not let sub-agents create their own sub-agents unless you have budgeted for it. Enforce that in your code, not in a prompt. Add the circuit breaker on cost-rate and loop detection.
Stage 4 — Handle randomness and partial failure. Add voting on the few decisions that justify the cost. Wrap multi-step, side-effecting workflows in sagas with idempotent undos. Send tasks the agents cannot finish to a dead-letter queue for a human. Verify outputs with a separate checker, like grepping citations against sources. Do not let an agent grade its own work.
The rule that should change your architecture: Anthropic found that on one research eval, token usage alone explained about 80% of the performance difference between runs. Model choice explained about 5%. So when a single agent stalls, the first question is "is it out of context?", not "is the prompt wrong?" More agents add context capacity. A prompt tweak cannot beat the model's context limit.
Two ways this hides
One: it works with two agents and breaks with five. Small systems do not loop or fan out enough to hurt. The cost and coordination failures show up at the scale you did not test.
Two: the bill is the alert. No crash, no page. The first signal is the invoice, or a customer spotting a wrong answer. By then it is a postmortem, not a warning.
Find out if you have this problem tonight
Check one — find the ceiling. Open your multi-agent code. Point to the line that caps total steps, and the line that caps dollars or time per run. If those limits live inside a prompt ("please do not loop"), you do not have a ceiling. Move them into code, outside the agent.
Check two — replay a failure. Pick a run that went wrong. Can you see a trace with one span per model call and per tool call, with token counts attached? If you cannot reconstruct what each agent did and what it cost, add tracing before you add another agent.
When one agent is the right answer
Sometimes the right number of agents is one. The clearest case for this came from Cognition's Don't Build Multi-Agents (Walden Yan). His two rules: share the full context, not just single messages. And remember that every action carries a hidden decision, so parallel agents making conflicting hidden decisions produce a broken result. His example: two sub-agents build a game. One renders a Super-Mario-style background. The other builds a bird that does not match. Neither saw the other's context.
Multi-agent works for wide, parallel, read-heavy work like research and search, where the threads are independent and the answer is worth the cost. For tightly-linked, write-heavy work, keep it single-threaded with good context. Anthropic's own multi-agent system beat its single-agent version by about 90% on an internal research eval. It also spent about 15 times the tokens to do it. Reach for more agents when the job is worth that trade, and not before.
Closing
You did not leave distributed systems behind when you started building agents. You came right back to them. The patterns still work. The one instinct to drop is the one that says a quiet system is a healthy one.
Design for unreliable agents on an unreliable network, or let the invoice prove they were. There is no third option.
Rate today's Newsletter
Sources and honesty notes: actor model, Hewitt, Bishop and Steiger (1973). Fallacies of distributed computing, Deutsch and Gosling (1990s). Saga, Garcia-Molina and Salem (1987). Circuit breaker and bulkhead, Nygard, "Release It!". CALM theorem, Hellerstein and Alvaro. Self-consistency gains, Wang et al. (2022). The multi-agent figures (about 90% eval gain, about 15x tokens, about 80% of variance from token usage) are from Anthropic's own multi-agent research post. They are self-reported, on an internal eval. The 14 failure modes are from the MAST study (Cemri et al., UC Berkeley, 2025). The eleven-day runaway is a single-source postmortem with an unverified dollar figure. The pattern is widely reported. One caveat to flag: "voting" borrows the idea of quorum consensus, not its guarantees. Agents are randomly wrong, not adversarial, so classic Byzantine math does not directly apply. Framework details move fast, so treat framework names as current-as-of-writing and the patterns as durable.
👋 That’s All Folks!
Before you go, just a few public service announcements:
See you soon,
AIEdTalks’ Newsletter Team
