Welcome to AIEdTalks’ Newsletter!
This is Part 4 of 4 in “The Agent Loop, From Scratch.”
In today's edition:
Break #1: retries multiply
Break #2: the model's retry is your duplicate write
Four tests that run without an API key
The loop readiness checklist, and when to stop hand-rolling
Let’s dive in.
After reading, please rate today’s edition.
Rate today's Newsletter
Blu Dot surpasses 2,000% ROAS with self-serve CTV ads
Blu Dot used Roku Ads Manager to drive incredible results for its furniture sales event. Its strategy hinged on custom audiences and retargeting, where intent was strongest.
“Roku has been a top performer,” said Blu Dot’s Claire Folkestad. “We have seen…CPMs lower than any other CTV partner we've worked with.”
Today’s Edition
The loop you'd let page you at 3 a.m.
Part 4 of 4. Two failures I've watched happen, and the checklist that prevents both.
Three weeks in, we have tools, a loop, and a budget. It works. That's not the same as shippable.
This is Part 4 of 4 in “The Agent Loop, From Scratch.” We're building one production-shaped agent loop, one piece per week, in raw Python with no framework.
Today, the two failures that cost real money, and how to test for them.
Break #1: Retries multiply
I've seen a “flaky model” incident where the model was fine and the retry math was the problem.
Count the layers. The SDK retries connection errors, 408, 409, 429 and 5xx — twice by default. Its default timeout is ten minutes, and it retries timeouts too. Now wrap that in your own @retry(3) decorator, because that's what you do with flaky dependencies. Now run it inside an 8-step loop.
One user request, 72 upstream attempts, against a key that was rate-limited in the first place. You built a load generator pointed at your own quota.
The fix: one retry budget, in one place. Set an explicit timeout, let the SDK do the retrying, and when it gives up — degrade. That's the model_unavailable status from Part 2. Don't catch it and try again.
client = anthropic.Anthropic(timeout=60.0, max_retries=2) # one place, one policyBreak #2: The model's retry is your duplicate write
This one is subtler and it reaches customers.
A tool that creates something — a ticket, a refund, an email — times out on your side after the downstream call already succeeded. You correctly return is_error. The model sensibly tries again. Now the customer has two tickets.
You already know this problem. It's at-least-once delivery. Agents have it too, and the retry is coming from a non-deterministic client that will happily vary its arguments between attempts.
The fix: idempotency on a business key. Not on the model's call ID — it generates a fresh one every retry. On something that identifies the real-world action:
def create_ticket(order_id: str, summary: str) -> dict:
# One open ticket per order, however many times the model calls.
if order_id not in TICKETS:
TICKETS[order_id] = {"ticket_id": f"T-{order_id}", "order_id": order_id, "summary": summary}
return TICKETS[order_id]Rule of thumb: if a tool writes, it needs an idempotency key before it ships.
Test it without an API key
The loop is ordinary code. Test it like ordinary code. A fake client returning scripted responses makes control flow deterministic and free.
# tests/test_loop.py
import itertools
from types import SimpleNamespace as NS
from agent.loop import run_agent
from agent.budget import Budget
from agent import tools
def usage(i=500, o=50):
return NS(input_tokens=i, output_tokens=o,
cache_creation_input_tokens=0, cache_read_input_tokens=0)
def tool_use(name, args, id="toolu_1"):
return NS(type="tool_use", id=id, name=name, input=args)
def text(t): return NS(type="text", text=t)
def resp(stop, *blocks): return NS(stop_reason=stop, content=list(blocks), usage=usage())
class FakeClient:
def __init__(self, responses):
self._responses = iter(responses)
self.messages = self # so client.messages.create(...) works
def create(self, **kwargs):
return next(self._responses)
def test_happy_path():
fake = FakeClient([
resp("tool_use", tool_use("get_order_status", {"order_id": "A-1001"})),
resp("end_turn", text("Shipped via UPS, ETA Oct 6.")),
])
r = run_agent("Where is A-1001?", client=fake)
assert r.status == "done" and "UPS" in r.text
def test_bad_arguments_become_is_error_not_a_crash():
fake = FakeClient([
resp("tool_use", tool_use("get_order_status", {"id": "A-1001"})), # wrong arg name
resp("end_turn", text("Sorry, I hit an error.")),
])
r = run_agent("Where is A-1001?", client=fake)
assert r.messages[2]["content"][0]["is_error"] is True
def test_runaway_loop_is_capped():
forever = (resp("tool_use", tool_use("get_order_status", {"order_id": "A-1002"}))
for _ in itertools.count())
r = run_agent("loop forever", client=FakeClient(forever),
budget=Budget(model="claude-sonnet-5-5", max_steps=3))
assert r.status == "max_steps" and r.budget.steps == 3
def test_side_effecting_tool_is_idempotent():
tools.TICKETS.clear()
a = tools.create_ticket("A-1002", "late")
b = tools.create_ticket("A-1002", "still late!!")
assert a == b and len(tools.TICKETS) == 1pytest -q — four tests, no network, no spend.
These test control flow. Whether the agent gives good answers is a different question, and it belongs in an eval suite, not here. Engineers moving into AI conflate the two constantly: control-flow bugs are deterministic and testable; quality is statistical and needs evals. Don't try to solve one with the other.
The loop readiness checklist
Save this. Every box should be checked before an agent loop ships:
Step cap and cost cap, both enforced every iteration
Explicit model timeout (not the 10-minute default) and one retry budget
Per-tool timeouts, and timeouts on the I/O clients underneath
Tool arguments validated; failures returned as is_error with fix-it guidance
Every tool_use block answered; tool-result message contains only tool_result blocks
Tool output truncated or summarised to a fixed size
Side-effecting tools idempotent on a business key
Every stop_reason mapped to a typed status
Per-step log line: step, stop reason, tokens in/out, cumulative USD
Offline tests: happy path, bad args, runaway loop, idempotency
Model ID and price table pinned and dated
When to stop hand-rolling
Own the loop until you understand it. Then move on purpose:
To learn, debug, or keep tight control of cost and flow — your own loop (this series)
Less boilerplate, same provider — Anthropic's Tool Runner (beta)
Pause/resume, human approval, long-running state — a durable-execution or graph framework
Multi-agent fan-out — sub-agents as tools inside this loop first, framework after
The point was never that frameworks are bad. It's that you should be able to read their source and recognise what you built.
That's the series
Four weeks, one repo, about 200 lines. If you followed along, you now own an agent loop you could defend in a design review — and more usefully, you can read any framework's loop and know exactly what it's doing to your bill and your blast radius.
Your turn
Hit reply: which box on that checklist is unchecked on your system right now? That's the most honest question in the series, and the answers will shape what I write next.
If you prefer watching to reading, subscribe to the AIEdTalks YouTube channel. https://www.youtube.com/@AIEdTalks
AI is easy to demo. Hard to ship.
Sources
SDK retry/timeout defaults and the API behaviour come from Anthropic's docs. The 72-attempt arithmetic, the idempotency-on-a-business-key rule, the test design, the readiness checklist, and both failure stories are my own, written generically from production experience.
Who's writing this?
I'm a research engineer with 18+ years building production systems, 35+ patents, and 17+ publications. I work on infrastructure for multi-agent systems. AIEdTalks is my field notes: real problems, what I tried, and what I learned.
Views are my own.
👋 Before you go
💬 Hit reply. I read every reply.
▶️ Watch it instead. This week's breakdown is on YouTube. Subscribe to the channel for more.
📨 Know an engineer who'd find this useful? Share your referral link and earn rewards.
💡 Topic ideas or sponsorships: Reply to this email.
Until next time,
AIEdTalks team.
P.S. AI is easy to demo. Hard to ship. That's what this newsletter is about.
