Welcome to AIEdTalks’ Newsletter!

In today's edition:

Let’s dive in.

After reading, please rate today’s edition. It motivates me to write better.

I'm a research engineer with 18+ years building production systems, 35+ patents, and 17+ publications. I work on infrastructure for multi-agent systems. AIEdTalks is my field notes: real problems, what I tried, and what I learned.

Views are my own.

For its first CTV campaign, Jennifer Aniston’s DTC haircare brand LolaVie had a few non-negotiables. The campaign had to be simple. It had to demonstrate measurable impact. And it had to be full-funnel.

LolaVie used Roku Ads Manager to test and optimize creatives — reaching millions of potential customers at all stages of their purchase journeys. Roku Ads Manager helped the brand convey LolaVie’s playful voice while helping drive omnichannel sales across both ecommerce and retail touchpoints.

The campaign included an Action Ad overlay that let viewers shop directly from their TVs by clicking OK on their Roku remote. This guided them to the website to buy LolaVie products.

Discover how Roku Ads Manager helped LolaVie drive big sales and customer growth with self-serve TV ads.

The DTC beauty category is crowded. To break through, Jennifer Aniston’s brand LolaVie, worked with Roku Ads Manager to easily set up, test, and optimize CTV ad creatives. The campaign helped drive a big lift in sales and customer growth, helping LolaVie break through in the crowded beauty category.

Learn more.

Cost isn't killing your agents. Quality is. And most teams are watching it instead of measuring it.

LangChain surveyed more than 1,300 professionals building AI agents for its State of Agent Engineering report. I read the whole thing.

One line stopped me: 32% named quality as their top barrier to production. Cost concerns dropped from last year.

I've been building production systems for over 18 years. Every launch I've seen stall was blamed on something visible: the bill, the latency, the infra. The real reason was almost always quieter. The system didn't do the right thing often enough, and nobody could prove it either way.

This report is the first time I've seen that written down with numbers behind it.

So this issue has two parts. First, what changed and what didn't. Second, the rule I'd follow if I were shipping an agent today.

The report breaks quality into accuracy, relevance, consistency, and tone. None of these throw errors. They just make users stop trusting the system.

That's not new. What's new is that cost stopped being a convenient excuse.

The second-biggest barrier splits by company size. Startups cite latency. Enterprises above 2,000 employees cite security and compliance.

Same pattern I've seen for two decades. Small teams fight speed. Large teams fight risk. Agents didn't change that.

1. Agents crossed into production.

57% of respondents now have agents in production. The “should we build agents” phase is over. The “why is ours unreliable” phase has started.

2. Everyone can see. Half can judge.

Nearly 89% have observability for their agents. Only about 52% run offline evals, and just 37% score live traffic.

That gap is the whole story. A trace tells you what the agent did. An eval tells you whether it should have.

Teams bought the dashboard and skipped the grading. Now they can watch every failure in high resolution without knowing which ones matter.

3. Humans are still the grader.

Human review (59.8%) is still the most common evaluation method, ahead of LLM-as-judge (53.3%).

Translation: most agents in production are graded by people reading outputs, which doesn't scale with the traffic.

So if the problem is quality, and most teams can see it but not measure it, the obvious question is: what do you build first?

This isn't LangChain's recommendation. It's mine, built from their data and from what I've watched break.

Not synthetic cases. Real ones, from users, testers, or your own logs. Write down what the agent did and what it should have done.

Start with code-based checks where you can: format, schema, required fields. Use LLM-as-judge only for what rules can't express. If you want the build-it-yourself version, I covered it in Evals, From Scratch.

If the eval score drops, the change doesn't ship. This is the step most teams skip, and it's the one that turns evals from a report into a control.

Run your eval on a sample of production traffic automatically. This is how you catch the failures your 20 cases didn't predict.

Wire it to your eval cases, not to vanity spans. Every panel should answer one question: which failures are growing?

Most teams start with observability because it's easy to install and looks like progress. I'd put it last.

Observability without evals shows you everything and tells you nothing. Evals first give your traces a meaning. Then the dashboard becomes a tool, not a screensaver.

Hit reply and tell me: what's the one eval you wish you'd written before shipping? I read every reply, and the best answers may shape a future issue.

If you prefer watching to reading, subscribe to the AIEdTalks YouTube channel. https://www.youtube.com/@AIEdTalks

AI is easy to demo. Hard to ship.

Charts are AIEdTalks visuals built from LangChain's published figures. The “no dashboard until an eval fails” rule is my own recommendation, built on their findings. It is not part of their report.

💬 Hit reply. I read every reply.

▶️ Watch it instead. This week's breakdown is on YouTube. Subscribe to the channel for more.

📨 Know an engineer who'd find this useful? Share your referral link and earn rewards.

💡 Topic ideas or sponsorships: Reply to this email.

Until next time,

AIEdTalks team.

P.S. AI is easy to demo. Hard to ship. That's what this newsletter is about.