Welcome to AIEdTalks’ Newsletter!

In today's edition:

Let’s dive in.

In November 2025, a team shipped a market-research pipeline: four LangChain agents coordinating over the A2A protocol. It passed testing. In production, two of them — an Analyzer and a Verifier — started ping-ponging: the Analyzer generated content, the Verifier asked for more, the Analyzer obliged. Week 1 was $127. Week 2, $891. Week 3, $6,240. Week 4, $18,400. The loop ran for roughly 264 hours — eleven straight days — before the billing dashboard surfaced a number big enough to stop it. Final bill: about $47,000. One caveat up front: that number comes from a single first-person write-up by engineer Teja Kusireddy, not a company post-mortem, so treat the exact figure as an anecdote. The pattern, though, is real and reproducible.

Here is the part that should worry you. The team had observability. Every API call returned 200. Every span was green. CPU and memory stayed flat, because LLM calls are I/O-bound. Their dashboards showed a healthy system the entire time. The post-mortem found two root causes: no per-agent budget cap, and no mechanism to terminate the session before the next API call. They had monitoring. They did not have the one signal that mattered — chain depth and per-agent cost — so they were blind in exactly the place agents fail.

You know distributed tracing. You can read a flame graph, chase a p99 spike, find the slow span. Almost none of that instinct will save you here, because the failure modes are different in kind, not degree.

Traditional Application Performance Monitoring (APM) answers one question: is it working? It was built for a deterministic contract — same request, same code path, same response — where "healthy" means 200s and low latency. Agents break every assumption in that sentence. They are non-deterministic: same prompt, different path every run. They fail silently: a confident, grammatically perfect, completely wrong answer returns HTTP 200, so your error rate reads 0% while your accuracy is 40%. And their cost is decoupled from request count — one "slow" request can burn ten times the token budget of a normal one. Agent observability has to answer a different question: why did it behave that way?

The numbers make the mismatch concrete. Anthropic, in its writeup of the multi-agent Research system behind Claude, reported that agents "typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats." It also found that on the BrowseComp evaluation, token usage by itself explains 80% of the performance variance. So the single variable most predictive of agent quality — tokens consumed — is the one your APM never recorded, because it counts requests, not tokens.

The failure is silence, not a stack trace. This is the single biggest operational difference, and it's the one your reflexes get wrong. Traditional software crashes: you get a 500, a stack trace, a Sentry alert. Agents return a confident wrong answer with a 200 and no error. Anthropic's team describes it exactly: users reported agents "not finding obvious information," but the team couldn't see why — were the agents using bad search queries? Choosing poor sources? Hitting tool failures? Adding full production tracing of decision patterns is what let them diagnose failures systematically. Hamel Husain makes the same point from the practitioner side: the real work is error analysis. His guidance is direct — manually review 20–50 LLM outputs whenever you make significant changes — because the failure mode is a quietly degraded retrieval step that returns a clean 200 for thousands of interactions. You cannot alert on an exception that never fires.

Loops and cost are first-class failure modes, and infrastructure metrics are blind to them. The $47K loop is the canonical case, but the shape is everywhere: an agent calls a broken tool that returns empty results, and because its prompt says "retry until you get data," it calls that tool 400 times in five minutes — every call a valid 200, latency normal, CPU flat. The signal that catches this isn't in any APM tool. It's chain depth (how many steps a run traverses) and per-agent, per-run token cost. One documented case caught two looping agents by alerting when chain depth exceeded 8 — within three minutes, versus eleven days. The deeper trap is that cost grows quadratically: a naive loop re-sends the entire history every step, so a 20-step run at 1,000 tokens/step bills roughly 210,000 input tokens, not 20,000. And there's a quality cliff inside that context growth. Chroma's July 2025 study tested 18 frontier models and found every one degrades as input length grows, well before the window is full. A long-running agent literally gets dumber as it fills its own context, and no latency or error metric will tell you.

Days 1–2: Get traces flowing. Pick one backend. On LangChain/LangGraph, LangSmith is one environment variable and understands the primitives natively. Want open-source and self-hosting? Langfuse is MIT-licensed and OpenTelemetry-native. The strategic move regardless of vendor: instrument with OpenTelemetry via OpenInference or OpenLLMetry, not a proprietary SDK, so your instrumentation lives in your app and you can re-point it later. The exit cost from a closed SDK is a multi-week re-instrumentation; the exit cost from OpenTelemetry is a config toggle. Auto-instrumentation gets you most of the way in an afternoon, and the overhead is under one percent because OTel batches asynchronously and your LLM calls already take seconds.

Days 3–4: Type your spans and attach the four attributes that matter — session_id, agent name, prompt version, and per-span token count with cost. Without prompt version, you can't trace a behavior change to its cause, because prompts drift silently. Without per-agent cost, you are the $47K team. The span types to distinguish: generation (an LLM call, with token counts), tool (a function or API call), retriever (a vector or DB lookup), agent (an orchestration decision), handoff (with from_agent and to_agent), and guardrail. These map directly to OpenTelemetry GenAI semantic conventions and are auto-detected by both Langfuse and the OpenAI Agents SDK.

Day 5: Build two things. First, the dashboard: cost per completed task (not per request — a cheap agent that fails twice and escalates is expensive), chain-depth distribution, and consecutive tool-failure count. Second, alerts that match agent failure shapes: page on chain depth greater than 8, on three or more identical tool calls in a single run (the exact signal the $47K team lacked), and on a hard per-session cost ceiling. Critically, an alert is not enforcement. The $47K team had a budget alert; it fired two days too late. Pair every alert with a circuit breaker that actually halts the next call. For storage, sample tail-based: keep 100% of traces with an error or above a cost threshold, sample the rest at 5–10%. That keeps the bill sane without throwing away the trace you'll want at 2 AM.

Two checks, five minutes.

Check 1: Open your current observability tool and try to answer "what did my agent's most expensive run last week cost, and which sub-agent and which tool spent it?" If you can't break the number down by agent and tool, you have request-level monitoring, not agent observability — and you can't see a cost spiral forming.

Check 2: Grep your last 1,000 production runs for the maximum number of steps or tool calls in any single run. If your max is wildly above your median, or if you have no max-iteration cap and no per-run cost ceiling wired to an actual kill switch, you are one ambiguous tool description away from the $47K story.

If both come back clean, your traces are tight and your brakes are wired. If they don't, you just found the incident before it found you.

In a microservice, you read source code to understand behavior. In an agent, the trace is the source code — the runtime decisions happen inside the model, and without typed spans, per-agent cost, and chain-depth tracking, you are debugging by reading the model's mind.

You already know distributed tracing. The discipline is the same. The failure surface is different. Point your tracing at the agent decisions, not just the infrastructure — and wire the alerts to brakes, not dashboards.

Instrument the decision chain, or keep flying on a green dashboard that cannot see the failure. There is no third option.

Before you go, just a few public service announcements:

See you soon,

AIEdTalks’ Newsletter Team