Welcome to AIEdTalks' Newsletter!
In today's edition:
Why you can't test an AI like normal code
Error analysis: read your outputs first
LLM-as-judge, done right (and how to grade the grader)
Turning evals into a CI gate
Let's dive in.
Sponsored by
What would you delegate if you had a whole team?
With Skydive, you can build a team of AI agents to take work off your plate.
Create agents for customer support, sales, marketing, engineering, ops, and more. Give each one a role, connect the tools they need, and hand off the work.
Your agents can work on their own or together, sharing context and handing off tasks to get bigger jobs done.
Start with one agent. Build a whole team around you.
Today’s Edition
AI SYSTEMS
Evals, From Scratch
Here's the problem that breaks every new AI developer. You build an AI feature, try a few questions, the answers look great, and you ship it. A week later you change one line of the prompt to fix something and, without knowing it, you break three other things. No test fails. No error appears. You find out when a customer complains.
Normal tests can't catch this, because the thing you're testing gives a different answer every time. That's what this primer is about: how to test a system that isn't predictable. The practice is called evals (short for evaluations). As the practitioner Hamel Husain puts it, teams that fail at AI "almost always share a common root cause: a failure to create robust evaluation systems." This primer assumes you can write code and have written a normal unit test, and nothing about AI. We'll build it from zero.
1. Why your usual tests stop working
A normal unit test says: given input X, assert the output equals Y. It works because the code is deterministic — same input, same output, forever. An AI model is non-deterministic: ask it the same question twice and you can get two different answers. So there's no single Y to assert against. Worse, the model can be confidently wrong — a fluent answer that reads fine but is incorrect, which your code sees as a normal success.
One surprise worth knowing early: even at "temperature 0" (the model's most repeatable setting) you still won't get identical outputs every time — partly because of how these models run on shared servers under changing load. So you can't rely on "run it once and check." You have to think in terms of how often it's right, across many examples.
2. The one-sentence mental model
Evals are unit tests for a system that's probabilistic instead of predictable. But there's a twist that trips up everyone. With normal tests you know the right answer up front. With AI, you often don't know what "good" even means until you've read a pile of real outputs. The researcher Shreya Shankar found this is fundamental: "it is impossible to completely determine evaluation criteria prior to human judging of LLM outputs." In plain terms: you discover the tests by reading real outputs first. That's why this starts not with code, but with reading.
3. Step one: error analysis (just read your outputs)
This is the most important step, and it needs no tools. Open a spreadsheet and read your AI's actual outputs, one by one. The method, borrowed from how researchers analyze interviews:
Collect real examples — the full record of real requests: the question, what the AI did, its final answer. (No data yet? Make up realistic ones.)
Write notes ("open coding"). For each bad output, write one plain sentence about what went wrong. Don't categorize yet — just journal. Note the first thing that broke, since an early mistake usually causes the later ones.
Group the notes ("axial coding"). Cluster them into named failure types — "wrong tone," "made up a policy," "ignored the question." That grouped list is your failure taxonomy.
Count them. Tally how often each type happens and sort by frequency. Now you know what to fix first, backed by numbers instead of vibes.
How many to read? Start with at least 100. You can slow down when about 20 in a row stop revealing anything new ("saturation"). And this isn't a detour from the real work — experienced teams spend 60–80% of their time on this loop. Reading your data is the work. One warning: don't hand the reading to the AI at the start — you know a complaint about "lag" is a latency bug, not a vague "performance" note.
4. The three kinds of evals
Turn each top failure into an automatic check. Three kinds, cheapest first:
Code-based checks (assertions). Plain code with a clear pass/fail: a regex, a JSON-schema check, "must not contain X." Free, instant, reliable. The rule: if you can check it with code, always check it with code.
LLM-as-judge. For things code can't measure — tone, "did it answer the question?" — use another AI to grade against a rubric. Powerful, but it costs money per check and can be wrong, so it must be validated first.
Human review. A person judging outputs. This is your ground truth — the answer key you calibrate everything against. Less over time, but never zero.
The healthy mix: lots of cheap code checks, a few validated LLM judges for subjective stuff, and a small steady amount of human review.
5. LLM-as-judge, done right
Using an AI to grade an AI works only if you do three things. Make it answer yes/no, not 1-to-5 — nobody can say what separates a 3 from a 4, so scores drift. Write a clear rubric with examples. And grade the grader: the step beginners skip.
Take ~20 examples a human already labeled, run your judge on them, and check two numbers: of the outputs that should pass, how many did the judge pass (the catch-good rate)? Of the outputs that should fail, how many did it fail (the catch-bad rate)? Why both? If your AI is already good 99% of the time, a lazy judge that always says "PASS" is 99% accurate and catches zero problems — a smoke detector that never beeps. Trust a judge only when it is good at both. (The formal, chance-corrected version of this agreement is a number called Cohen's kappa; you don't need the math today, just the idea that raw agreement flatters a lazy judge.)
One newer idea: for agents that take many steps, judge the whole trajectory (which tools it called, in what order), not just the final answer. A refund bot can produce a fine-looking reply while having issued the refund before checking identity. One 2024 study reported an agent-style judge agreeing with humans about 90% of the time versus about 70% for a plain judge — but that was on a single coding benchmark, so treat it as promising, not a universal law.
6. Step three: turn evals into a CI gate
Now make it automatic — this is what stops the silent-regression nightmare from the opening.
Build a small labeled dataset (around 100 examples) from the real failures you found, plus known edge cases.
Run your evals in CI on every change to the prompt, model, or code — like a normal test suite.
Block the merge if the score drops below your bar, or more than a few points below the last passing run (so a quiet slip still trips the alarm).
Two AI-specific rules. Gate on the average across the whole dataset, not one run — a single output is noisy, so run each example a few times (say 3–5) and average. And favor the cheap code checks in CI, keeping expensive judges for a smaller set, so the pipeline stays fast. Don't chase a perfect score, either: as Hamel notes, "if you're passing 100% of your evals, you're likely not challenging your system enough."
7. A worked example, start to finish
Say you build a support agent that answers customer questions.
Collect 50–100 real replies.
Error analysis → a taxonomy: wrong tone, made up a policy, ignored the question, leaked a raw internal ID.
One code check for the objective failure — never leak an internal ID:
import re
assert not re.search(r"CUST-\d{6}", reply)One LLM-judge for the subjective one — "Does this reply actually answer the customer's question? PASS or FAIL," with two or three example gradings in the prompt.
Grade the judge against ~20 human-labeled replies; trust it only if it catches both good and bad.
Put both in CI: run the ~100-example set on every change, average 3–5 runs each, and fail the build if the pass-rate drops.
That's a complete eval system. It started with reading, not code.
8. Common beginner mistakes
Only "vibe-checking" — trying a few prompts by hand and calling it tested.
Jumping to a 1–5 "quality score" — feels rigorous, means nothing. Start from real failure types.
Trusting an LLM judge you never validated. Always grade the grader.
Too many metrics. A few meaningful checks beat twenty vanity numbers.
Grabbing generic off-the-shelf metrics. Eugene Yan warns these "barely correlate with application-specific performance."
Judging only the final answer. For agents, check the steps too.
Chasing 100%. Usually means your tests are too easy.
Key terms
Eval — an automatic test that checks one specific thing about an AI output.
Non-deterministic — same input can give different outputs.
Error analysis — reading real outputs and sorting what went wrong, before writing any check.
Failure taxonomy — your named, counted list of failure types.
Code-based check / assertion — a plain pass/fail check in code.
LLM-as-judge — using another AI to grade subjective qualities against a rubric.
Ground truth — the human-labeled correct answer everything is measured against.
Catch-good / catch-bad rate — how often the judge correctly passes good outputs and fails bad ones (check these instead of raw accuracy).
CI gate — an eval wired into your build that blocks a merge when the score drops.
Trajectory — the steps and tool calls an agent takes to reach its answer.
The one idea to remember
You can't test an AI the way you test normal code, because it isn't predictable. So you do the opposite order: read real outputs first, let them tell you what "good" means, turn those lessons into checks, and run the checks on every change. Reading your data isn't the boring part before the real work. It is the work.
Notes on the numbers: the "read at least 100 / spend 60–80% of your time" guidance and the eval method come from Hamel Husain and Shreya Shankar's widely-used evals teaching. The ~90% vs ~70% judge-agreement figures are from a single 2024 code-generation benchmark (Zhuge et al., "Agent-as-a-Judge") and are illustrative, not universal. "Criteria drift" is from Shankar et al., "Who Validates the Validators?" (2024). Even at temperature 0, outputs aren't perfectly reproducible — one cause is changing server batch load (Thinking Machines Lab, 2025) — which is why you gate on averages, not a single run.
Stop Paying for 6 Tools. One AI Does It All.
Most e-commerce sellers juggle 6–8 tools and pay hundreds monthly to keep operations running. StoreClaw replaces the stack with one autonomous AI engine that monitors competitors, optimizes listings, automates marketing, and tracks profit 24/7. Connect your store and let AI handle the work — no prompts, no complex setup, no credit card required.
👋 That’s All Folks!
Before you go, just a few public service announcements:
See you soon,
AIEdTalks’ Newsletter Team
