Welcome to AIEdTalks’ Newsletter!
In today's edition:
What LLM-as-a-judge is, and the variants you'll actually use
Why it works, with the human-agreement numbers behind it
A step-by-step build in Python, including how to validate your judge
Five failure modes, their fixes, and seven other eval methods worth knowing
Let’s dive in.
I put real effort into every issue. A quick rating helps me improve the next one.
Rate today's Newsletter
Who's writing this?
I'm a research engineer with 18+ years building production systems, 35+ patents, and 17+ publications. I work on various aspects of multi-agent systems. AIEdTalks is my field notes: real problems, what I tried, and what I learned.
In partnership with
The agentic era needs a different CRM. That’s Attio.
Teams like Parallel, Turbopuffer, and Wordsmith are already setting the pace on Attio. Get an always-on revenue engine, with agents and workflows that build pipeline, chase every buying signal, and move deals forward with your team. Whether you're working in your browser, inbox, or favorite agent, connect to your customer data in real-time through Attio's web app, MCP, API, and SDK.
Today’s Edition
AI EVALS
LLM-as-a-Judge, Without the Hand-Waving
Last issue, I unpacked Andrew Ng's AI Engineering Skills Map and his “evals first” argument. He treats a disciplined evals and error-analysis loop as the trait that separates great AI builders, and it rests on three eval types: code-based checks, LLM-as-a-judge, and human review.
The natural follow-up question is what LLM-as-a-judge actually is, and how you build one that isn't just a vibe check in a lab coat. That's today.
What it is
LLM-as-a-judge means using a strong language model to score or compare another AI system's outputs against criteria you define. Instead of a human rating “was this answer helpful?”, you give the model the input, the output, and a rubric, and it returns a verdict.
The pattern was named and measured by Zheng et al. in the 2023 paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. It introduced two now-standard tools: MT-Bench, 80 multi-turn questions across writing, reasoning, math, coding, and more, and Chatbot Arena, crowdsourced head-to-head battles that feed an Elo leaderboard.
The variants you'll actually use:
Pointwise scoring. The judge sees one output plus a rubric and returns a score or pass/fail. It's the most versatile option, and some criteria, like faithfulness to a source, only make sense this way.
Pairwise comparison. The judge picks the better of two outputs. It's more stable and suits A/B tests and model selection, but it's more exposed to position bias.
Reference-based vs. reference-free. Reference-based compares against a gold answer. Reference-free is the default for open-ended generation, where no single correct answer exists.
Binary vs. 1–5 scales. Binary pass/fail with a written critique is what most practitioners now recommend.
G-Eval (Liu et al., 2023). The model first writes its own evaluation steps, then fills in a scoring form. It beat earlier automatic metrics on correlation with human ratings for summarization, and it's built into tools like DeepEval.
Critique-then-score. The judge explains its reasoning first and gives the verdict second.
Why use it
Code can't check tone, relevance, faithfulness, helpfulness, or instruction-following. Humans can, but not at scale. LLM judges sit in between: human-like judgment at close to code speed.
The headline evidence comes from the MT-Bench paper: strong judges like GPT-4 matched human preferences over 80% of the time, which is about the same rate at which humans agree with each other.
The economics are why it caught on. A 2025 medRxiv study comparing judges with human raters found its most expensive model judge cost about $0.12 per response, against $9.17 for human evaluation, roughly 75× cheaper. Treat that as one well-documented data point rather than a universal constant, but the order of magnitude isn't in dispute.
The caveat: that 80% figure is for preference agreement on open-ended chat. It does not mean the judge can verify math, hard facts, or reasoning it can't do itself.
How to use it, step by step
Most practitioner guidance here comes from Hamel Husain, who has helped 30+ companies build judges, and Eugene Yan's survey of LLM evaluators. The common thread: the real value is being forced to look at your data. The judge is almost a side effect.
1. Start from error analysis on real traces. Read actual outputs before writing any rubric. Shreya Shankar's research calls this criteria drift: grading outputs is how people discover what their criteria actually are, so you can't fully define them upfront.
2. One narrow criterion per judge. Not eight dimensions on a dashboard. If nobody can say what separates a 3 from a 4, the scale is measuring noise.
3. Write an explicit rubric with examples. It should be clear enough that a new hire could apply it.
4. Prefer pass/fail plus a written critique. Binary forces a clear “this passed because…”. The critique captures nuance and doubles as a few-shot example later.
5. Ask for reasoning before the verdict, and return structured JSON.
6. Validate against a human-labeled set. Treat the judge as a classifier and measure true positive and true negative rates plus Cohen's kappa, not raw agreement. If only 5% of outputs fail, a judge that always says PASS scores 95% agreement while catching nothing. Keep a held-out test set that never goes into the prompt.
7. Iterate until it converges with your domain expert. On Honeycomb's query assistant, Hamel reports it took three iterations to exceed 90% agreement with their expert.
8. Run it in CI and on sampled production traffic. Version your prompts and pin temperature to 0. LangSmith, DeepEval, promptfoo, and Braintrust can all wire judges into test gates.
A minimal binary judge that returns JSON:
import json
from openai import OpenAI
client = OpenAI()
JUDGE_PROMPT = """You are evaluating whether a support reply resolves the user's request.
Return ONLY JSON: {{"reasoning": "...", "verdict": "PASS" or "FAIL"}}
PASS = directly addresses the user's primary need.
FAIL = wrong, off-topic, or ignores the request.
Write the reasoning FIRST, then the verdict.
USER: {user}
REPLY: {reply}"""
def judge(user, reply, model="gpt-4o"):
resp = client.chat.completions.create(
model=model,
temperature=0,
response_format={"type": "json_object"},
messages=[{"role": "user",
"content": JUDGE_PROMPT.format(user=user, reply=reply)}],
)
return json.loads(resp.choices[0].message.content)And measuring how well it agrees with humans:
from sklearn.metrics import cohen_kappa_score, precision_score, recall_score
human = [1, 0, 1, 1, 0, 1, 0, 0] # 1 = PASS, human labels
judged = [1, 0, 1, 0, 0, 1, 1, 0] # judge verdicts
print("kappa:", cohen_kappa_score(human, judged))
print("precision:", precision_score(human, judged))
print("recall:", recall_score(human, judged))If kappa is weak (below about 0.6), fix the rubric or the examples before you trust the judge on new data.
Pros and cons
Where it wins:
Speed: minutes instead of days.
Cost: a few dollars per run instead of hours of human time.
Coverage: tone, relevance, faithfulness, and helpfulness, which code can't check.
Scale: it can run on every pull request and on sampled production traffic.
Explainability: it returns a critique you can read and audit.
Where it breaks:
Correctness: it can't reliably verify math, logic, or facts it can't check itself.
Bias: position, verbosity, and self-preference biases are all measured and real.
Stability: the same input can flip between runs, and scores drift silently when the judge model is updated.
Who judges the judge: without human-labeled validation, you're trusting an unvalidated proxy.
Failure modes and fixes
Position bias. Judges favor an answer slot regardless of quality. In MT-Bench, GPT-4 favored the first answer in 30% of cases, and weaker judges were far worse. Fix: run both orders and keep only verdicts that agree, or call it a tie.
Verbosity bias. Longer, padded answers tend to score higher. The size and even direction vary by model family. Fix: add length controls, and test your specific judge rather than assuming.
Self-preference. Models favor their own outputs. In MT-Bench, GPT-4 gave itself about a 10% higher win rate. Panickssery et al. (2024) found that the better a model recognizes its own writing, the stronger this bias. Fix: use a judge from a different model family than the one generating outputs.
Prompt sensitivity and run-to-run inconsistency. Small wording changes shift verdicts. Fix: version prompts, freeze a test set, set temperature to 0, and aggregate multiple samples.
It can't verify what it can't compute. Fix: send math, logic, and hard facts to code assertions or retrieval instead.
Juries beat a single judge. Verga et al. (2024) tested a panel of three smaller models from different families. It tracked human judgments better than a single GPT-4 judge, at more than seven times lower cost, and showed less self-bias.
Other methods worth learning
1. Code-based assertions. Exact labels, required fields, JSON schema, regex. Cheapest, most reliable, and they never drift. This is your first line of defense.
2. Reference metrics (exact match, BLEU/ROUGE, embedding similarity, BERTScore). Fast and repeatable, but BLEU and ROUGE correlate weakly with human judgment on open-ended text. Good for benchmark continuity, weak for factuality.
3. Human review. The ground truth everything else is validated against. Route ambiguous and high-stakes cases here.
4. Pairwise arenas and Elo. The right tool for ranking models against each other. Chatbot Arena is the canonical example.
5. RAG metrics (RAGAS). Faithfulness, context precision and recall, and answer relevance. They tell you whether a failure came from retrieval (wrong context) or generation (unsupported answer), which is the most useful split for debugging RAG.
6. Agent trajectory evals. Check tool-call correctness, task completion, and the full trace. Use exact trajectory matching when you know the expected tool sequence, and a judge when reasonable variation is fine.
7. Fine-tuned small judges (e.g., Prometheus). Open-source, cheap, and version-stable. Useful when cost, model versioning, or privacy rule out a proprietary judge.
Which method for which problem
Structured output (labels, JSON, code) → code-based assertion
Open-ended quality (tone, helpfulness, faithfulness) → LLM judge, pass/fail plus critique
You have gold answers → reference-based judge or exact match
Comparing two models or prompts → pairwise judge, with positions swapped
RAG pipeline → RAGAS-style metrics plus a faithfulness judge
Agents → trajectory, tool-call, and task-completion evals
High-stakes or ambiguous → human review
Noisy or self-scoring judge → a jury of models from different families
All of this comes back to one rule: validate your judge against humans, or you're just automating a guess.
An LLM judge doesn't replace looking at your data. It's the excuse that finally makes you do it.
AI is easy to demo. Hard to ship.
Sources
👋 Before you go
💬 Hit reply. Tell me how you evaluate your AI outputs today: code checks, an LLM judge, humans, or vibes. I read every reply.
▶️ Prefer video? Subscribe to the YouTube channel for more.
📨 Know an engineer who'd find this useful? Share your referral link and earn rewards.
💡 Topic ideas or sponsorships: DM me on X.
Until next time,
AIEdTalks team.
P.S. AI is easy to demo. Hard to ship. That's what this newsletter is about.
