Welcome to AIEdTalks’ Newsletter!
In today's edition:
LangGraph’s Agent observability with Langfuse
Let’s dive in.
Today’s Edition
AI TOPIC
You can't debug what you can't see: LangGraph observability with Langfuse
Welcome to this week's issue. This one is a teaching piece — no war story, just the shortest path from "my agent is a black box" to "I can see every LLM call, every tool call, and every token I spent." In the next few minutes you'll:
Spin up self-hosted Langfuse in about 10 minutes with Docker Compose
Wire it into a LangGraph agent with one callback
Read your first trace and understand what each node is telling you
Know exactly what to harden next — cost, evals, alerts, PII, sampling
Agent observability
Last week we talked about circuit breakers for multi-agent systems. Circuit breakers protect you when something goes wrong. Observability tells you what went wrong. If you're shipping any LangGraph agent to real users without traces, you are debugging by print statement — and print statements will not show you a runaway loop until the bill arrives.
Why agent observability is not just APM
If you've done distributed tracing before — OpenTelemetry, Jaeger, Datadog APM (Application performance monitoring)— the mental model is the same but the shape of the trace is different. A normal request has maybe 20 spans: HTTP handler, three DB queries, a cache lookup, one downstream API call.
A LangGraph agent turn produces:
One span per graph node execution
One span per LLM call (with the full prompt and completion)
One span per tool invocation (with input and output)
One span for retrieval if you're doing RAG
Loops. Sometimes many.
A single turn can produce 30+ spans. Without a UI that groups them by trace and renders the tree, you're grep-ing structured logs trying to spot the moment things went sideways. That's the job Langfuse does.
Step 1 — Spin up Langfuse
Langfuse v3 is the current stable line. It ships as a Docker Compose stack:
bash
git clone https://github.com/langfuse/langfuse.git
cd langfuse
docker compose up -dFive services come up. Know what each does — you will debug at least one of them:
langfuse-web — the Next.js UI on
localhost:3000langfuse-worker — background jobs (ingestion, evals, retention)
postgres — application metadata (users, projects, API keys)
clickhouse — the trace store. Every span lives here. Columnar, fast for the queries the UI runs.
redis — job queue and cache
minio — S3-compatible blob store for large payloads
Give the dev box at least 4 GB RAM. ClickHouse is the hungry one.
Open localhost:3000, create an account (this becomes local admin), create an org, create a project, then go to Settings → API Keys and grab two keys:
LANGFUSE_PUBLIC_KEY— starts withpk-lf-...LANGFUSE_SECRET_KEY— starts withsk-lf-...
Step 2 — Instrument a LangGraph agent
Install the SDKs:
bash
pip install langfuse langgraph langchain-openaiSet three env vars:
bash
export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_HOST=http://localhost:3000Now the agent — a minimal ReAct graph, one tool, one LLM node, one loop:
python
from langfuse.langchain import CallbackHandler
from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
def get_weather(city: str) -> str:
return f"It's 24°C and sunny in {city}."
agent = create_react_agent(
model=ChatOpenAI(model="gpt-4o-mini"),
tools=[get_weather],
)
handler = CallbackHandler()
result = agent.invoke(
{"messages": [("user", "What's the weather in Bangalore?")]},
config={
"callbacks": [handler],
"metadata": {
"langfuse_user_id": "[email protected]",
"langfuse_session_id": "demo-session-1",
"langfuse_tags": ["dev", "weather-agent"],
},
},
)That is the entire integration. One callback, one config block. The three metadata keys — user id, session id, tags — are what unlock filtering later. Put them in from day one; retrofitting them across a million traces is not fun.
Step 3 — Read the trace
Refresh the Langfuse UI. Tracing → Traces. Click the newest trace. You'll see:
A trace tree on the left — root call, then the LangGraph node, then the LLM call, then the tool call, then the next LLM call
Latency on every node — where the time went
Token counts and cost on every LLM call — how much this turn cost you
Input and output on every node — exact prompt, exact response, exact tool args
This is the moment observability clicks. You can see, without a single log line, that the LLM called get_weather with "Bangalore", got the response, and made a second LLM call to phrase the answer. If it had looped ten times, you'd see ten LLM calls stacked in the tree. That's how you catch runaway loops before your CFO does.
From tutorial to production
The first trace is the easy 20%. Here's what "production-ready" adds, in the order to tackle it:
Cost tracking — Langfuse auto-computes cost for known models. For custom or self-hosted models, add pricing in Settings → Models. Then filter traces by cost to find your top spenders.
Evals — Attach scores to traces. Start with thumbs-up/down through the SDK's
scoremethod, then graduate to LLM-as-judge evaluators in the UI. Use Datasets to run offline evals against a fixed input set before every release.Alerts — Native Monitors exist on Cloud and Enterprise self-hosted. Community self-hosted: point Prometheus/Grafana at ClickHouse, alert on error rate and p95 latency per project.
PII masking — The SDK exposes
maskhooks. Redact before ingestion, not after. Log the pattern that fired, never the raw value.Sampling — At high volume, configure SDK sample rates. Sample 100% of errors, 5–10% of successes.
Do these one per week, and by the end of the month you have real observability, not a demo of it.
Find out if you have this problem tonight
Two checks:
Ask your team: "How would we know if the agent looped 15 times for one user last Tuesday at 3 pm?" If the answer involves grepping logs, you have this problem.
Look at your last agent incident. Count the LLM and tool calls between trigger and failure. If it's more than five, you needed a trace tree.
Instrument, or debug in the dark. There is no third option.
👋 That’s All Folks!
Before you go, just a few public service announcements:
See you soon,
AIEdTalks’ Newsletter Team
