Welcome to AIEdTalks' Newsletter!
In today's edition:
What LLM observability is — and why your APM isn't enough
What OpenTelemetry (OTel) is, in one page
The metrics that matter — per step, not averages
LangSmith vs Langfuse vs Arize, compared
Let's dive in.
In partnership with
Your employees are connecting AI to everything. Now what?
ChatGPT and Claude don't just answer questions anymore. Employees are connecting them directly to Notion, Linear, Jira, and the rest of your stack. The AI can read, write, and take actions on company data. Most IT and security teams have no visibility into any of it.
Harmonic Security Connectors changes that. It sits inline with every AI-to-app connection, so you see each call, control what data moves, and block destructive actions before they happen. Employees notice nothing different.
See what's actually running across your business in a live demo.
Today’s Edition
AI SYSTEMS
LLM Observability, From Scratch
Here's a scene you'll recognize. You built an AI feature. It worked great in testing. You shipped it. Now a user says it gave a wrong answer — but when you try the same question, it works fine. You check your logs. Nothing looks broken. No error. No stack trace. The feature is "up." And you have no idea what went wrong.
That gap — the thing running fine while being wrong — is what LLM observability fixes. This primer assumes you can write code and read logs, and nothing at all about AI observability. We'll build it up from zero. By the end you'll be able to: read an agent trace, understand what to instrument, know which metrics to track, and choose between LangSmith, Langfuse, and Arize.
1. Why your usual debugging stops working
Normal code is predictable. Same input, same output. When it breaks, it breaks loudly — an exception, a 500, a failed test. You add a log line, reproduce the bug, and fix it. AI features break three of those assumptions:
Same input, different output. Ask a model the same question twice and you can get two different answers. You often can't reproduce the bug on demand.
It's not one call, it's many. A single question can trigger a chain of hidden steps.
It fails quietly. The model can return a smooth, confident, wrong answer — and your code sees a normal, successful response.
So "check the logs and reproduce it" stops working. You need to see everything the AI did, step by step, after the fact. That is what observability means here.
2. What actually happens when an AI agent runs
Say a user asks: "What changed in our Q3 refund policy?" Behind that one question, your code might: (1) send the question to the model; (2) the model replies "search the policy docs"; (3) your code runs the search; (4) your code sends the results back; (5) the model writes the answer. That's two model calls and one tool call for one question. Real agents do this five, ten, or twenty times. So if the answer is wrong, which step went wrong? You can't tell from the final answer. You need to see each step.
3. The two words you need: trace and span
Just two words. A span is one step (one model call, one tool call). A trace is all the steps for one request, stitched together in order. Think of an itemized receipt: each line item is a span; the whole receipt is the trace. Each span records what went in, what came out, how long it took, and how many tokens it used. Here is a trace for the refund question:
TRACE "What changed in our Q3 refund policy?" 2.4s $0.011
+- span search policy docs 120ms
+- span model call (decide) 380 in / 90 out 0.6s
+- span look up policy record 40ms
+- span model call (write answer) 1,200 in / 210 out 1.5s
`- result "Refunds now allow 30 days..." grounded? yes
The whole skill of debugging AI features is this: open the tree and read each step. You immediately see which step was slow, which cost the most, and which produced the wrong thing.
4. Tokens and cost (a quick detour)
A token is a chunk of text — roughly three-quarters of a word. You pay per token, both for what you send (input) and what you get back (output). Two things surprise people: agents re-send the whole conversation on every step, so a 10-step agent pays for a growing pile of text ten times; and a stuck agent that loops keeps paying on every loop. That's why "how many tokens did each step use" is a metric you actually watch.
5. How you get these traces: you "instrument"
To get spans, you instrument your code — add a little recording code around each AI call. Good news: you almost never write this by hand; auto-instrumentation libraries do it for you. If you're curious what one model call looks like recorded by hand:
# You usually let an auto-library do this. Shown only to demystify it.
with tracer.start_as_current_span("model call") as span:
span.set_attribute("model", "gpt-4o") # which model
reply = call_the_model(messages) # the actual call
span.set_attribute("input_tokens", reply.usage.prompt_tokens)
span.set_attribute("output_tokens", reply.usage.completion_tokens)One standard worth knowing: OpenTelemetry (OTel)
OpenTelemetry, or OTel, is an open, industry-standard way to record traces. It isn't AI-specific. It matters for one reason: it stops you getting locked into one vendor. If you record in the OTel standard, any observability tool can read your traces, and you can switch tools later without re-instrumenting. Two common auto-libraries that emit this standard are OpenLLMetry and OpenInference. One caveat: the AI-specific part of OTel is still young and changing in 2026 — use it, but pin your library version so a rename doesn't surprise you.
6. What to look at (the metrics)
Once you have traces, four simple questions tell you almost everything:
Is it slow? Measure how long each step takes. Don't look at the average — look at the slowest 1% ("p99"). Like a coffee line: the average wait is 2 minutes, but 1 in 100 people wait 20, and those are the ones who complain.
Is it expensive? Watch tokens in and out, and cost per request. Tag each request with its agent and task so you can see where the money goes.
Is it failing? Watch errors, how often tools fail, and how often the agent loops. A run costing 50x the normal amount is almost always stuck in a loop.
Is it right? The new one. A step can succeed and still be wrong. To catch that you use evals — small tests that score whether an answer was good. The simplest start is a thumbs up / thumbs down button.
7. The tools (you don't build this yourself)
A tool records and displays the traces for you. The three you'll hear most:
LangSmith | Langfuse | Arize Phoenix | |
|---|---|---|---|
In plain terms | Made by LangChain; best if you use LangChain | Open source; run it yourself for free | Open-standard first; works with anything |
Cost (verify) | Free to start; paid per seat + usage | Free self-hosted; cloud from ~$29/mo | Free to self-host; paid tier for extras |
Pick it if... | You already build on LangChain / LangGraph | You want open source and to host it yourself | You use many frameworks or want no lock-in |
Other names you'll run into: Datadog LLM Observability (if your team already uses Datadog), Braintrust (focused on evals), and Weights & Biases Weave. You don't need to learn them all today.
8. Do this first (your day-one path)
Pick one AI feature.
Add an auto-instrumentation library so it starts producing traces.
Run one request and look at the trace — confirm you can see each step with its tokens and how long it took.
That's it for day one. You can now see your agent.
Then, when you're ready: add tags to slice cost, build a slow-step (p99) and cost dashboard, and add a thumbs up/down to start judging quality.
9. Key terms
Observability — answering "what happened and why" after the fact.
Trace — all the steps for one request, in order.
Span — one step (a model call or a tool call).
Token — a chunk of text (~3/4 of a word); you pay per token, in and out.
p95 / p99 — the wait time the slowest 5% / 1% of requests see.
Instrument — add small recording code so each step produces a span.
OpenTelemetry (OTel) — the open standard for recording traces; keeps you vendor-neutral.
Eval — a small test that scores whether an answer was actually good.
Groundedness — whether an answer sticks to the real source documents.
The one idea to remember
You can't fix what you can't see. A normal log tells you the request finished. A trace tells you what the AI did at every step — the only way to catch an answer that was confident, successful, and wrong. Start by making one agent visible. Everything else builds on that.
Notes on the numbers: the common stat that ~89% of teams have observability but only ~52% run evals is from LangChain's 2026 survey (self-reported). The AI-specific parts of OpenTelemetry are still changing in 2026, so pin your library versions. Tool pricing and licenses change often — re-check the official pages before you commit (for example, Arize Phoenix is "source-available," which is not quite the same as fully open source).
Rate today's Newsletter
👋 That’s All Folks!
Before you go, just a few public service announcements:
See you soon,
AIEdTalks’ Newsletter Team
