Welcome to AIEdTalks’ Newsletter!

In today's edition:

Let’s dive in.

After reading, please rate today’s edition.

Are you ready? As revealed in our holiday playbook, Roku Ads Manager lets you easily set up, measure and optimize your self-serve holiday ad campaign. You can even create a shoppable ad unit that lets streamers buy directly from their remote.

Set up your campaign today.

Four questions, and the answer falls out.

Frontier models take a million tokens now, and a reliable argument has formed around that. One side says retrieval was always a workaround for small windows and the workaround is over. The other says the window doesn't help if the model can't use it properly.

Both are arguing about capability. Capability is not what decides this, and if you pick a side on that basis you'll get the wrong answer for your project.

Here's the decision, with the arithmetic.

Can a model find a fact buried in a million tokens? Broadly, yes. Single-needle retrieval scores sit around 90% and above at full window length on current frontier models. That problem is close to solved.

What hasn't been solved is using many scattered facts at once. Multi-needle and multi-hop scores fall off well before the window fills — published figures show a frontier model around 93% on an eight-needle test at 128K dropping to roughly 78% at a million, and the general finding across benchmarks is that the reliably usable window is somewhere around half to two-thirds of the advertised one for work that has to connect several pieces.

Worth knowing. But notice it doesn't decide anything, because retrieval has the mirror-image weakness: if your answer needs eight facts and your retriever returns the five chunks it liked best, you've lost three of them before the model saw anything.

Neither approach is reliably good at synthesising across a large scattered corpus. So stop choosing on capability and choose on the things that actually differ.

This is where it's decided, and the numbers are not close to each other.

Take a 500,000-token corpus and a mid-tier model at $2 per million input tokens. Retrieval pulls five chunks, call it 6,000 tokens with the query and system prompt.

Eighty-three times more expensive, naively. At ten thousand queries a day that is ten thousand dollars against a hundred and twenty.

But look at the third row, because that's the one that matters and it's the one the “RAG is dead” argument is actually standing on, usually without saying so.

It wasn't the bigger window. It was prompt caching.

A cached context read costs a tenth of a fresh one. That single factor takes stuffing a corpus from eighty-three times the price of retrieval down to about four times, and at smaller corpus sizes it flips the comparison entirely:

Below roughly 120,000 tokens, putting the whole corpus in the prompt costs you less than retrieving from it.

That's about 90,000 words. An internal handbook, a product spec set, a year of meeting notes, a moderately long book. If your corpus is that size, a retrieval pipeline is a vector store, a chunker, an embedding model, a reranker and four new silent failure modes, in exchange for a worse answer and a higher bill.

Don't build it. Put the documents in the prompt and turn caching on.

The cached number is doing all the work above, so you need to know when you don't get it.

Your traffic has to be dense. Caches expire — five minutes on the short tier, an hour on the long one. Writing the cache costs more than a normal read, so a cache that's never hit is worse than no cache at all. The break-even is low — on a 500K corpus, about 1.3 queries inside the window — but “low” isn't “zero”. A corpus queried twice a day pays the write premium every time and never collects. Bursty, clustered traffic is ideal. Sparse traffic is the worst case.

Your corpus has to hold still. A cache is a snapshot of an exact prefix. Change one document and you've invalidated the whole thing and you're paying the write again. Retrieval handles this far better: change a document, re-embed that chunk, done. Nothing else moves.

So “how often does it change” isn't a footnote. Change rate is what decides whether the cheap number or the expensive number applies to you.

Citations.

With retrieval you know which chunks you sent, so the source of an answer is a lookup. You get provenance as a side effect of the architecture.

With a stuffed context, the model has everything and you have to ask it which part it used. It will tell you. It will sometimes be wrong, and in a way that's hard to detect, because a confident citation to the wrong section of a document you did supply looks exactly like a correct one.

If you work anywhere that has to show its sources — legal, medical, finance, compliance, anything audited — this is not a preference. It may settle the question on its own, whatever the cost table says.

Run these in order and stop when one gives you a clear answer.

1. How big is the corpus?
Under ~120K tokens → stuff it, cache it, skip retrieval. Over a million → retrieve. In between, keep going.

2. How often does it change?
Changing hourly → retrieve; your cache will never amortise. Stable for days or weeks → stuffing stays cheap.

3. How many queries, how close together?
High volume in bursts → caching works, stuffing is viable. Low volume spread thin → you pay the write premium repeatedly, and retrieval wins by more than the table suggests.

4. Do you have to cite sources?
Yes, and it has to be auditable → retrieve, whatever the other three said.

Most projects get a clear answer by question two.

For a small, stable corpus: no retrieval. Documents in the prompt, caching on, and the two hours you saved spent on your evals.

For a large or fast-moving one: retrieval, and generously — if the model handles 200K comfortably, return twenty chunks rather than five. The old habit of sending three chunks was a response to 4K windows and it is no longer a constraint you have. Over-retrieving is now cheap, and it fixes the multi-hop weakness that pure retrieval has, by giving the model enough material to connect things.

That middle path is where most real systems land, and it's why “RAG is dead” is an unhelpful frame. Retrieval didn't die. It stopped being a compression trick and became a cost-and-freshness control, which is a smaller job, and one it does well.

Retrieval is no longer about what fits. It's about what you're willing to pay for, and how often it changes.

The capability gap that justified RAG has mostly closed. What hasn't closed is the price gap, and the price gap has a number on it: somewhere around 120,000 tokens, with a cached context, the two approaches cost the same. Below that line, retrieval is a pipeline you're building for no reason. Above it, it pays for itself many times over.

Work out which side of that line you're on before you write any code. It takes ten minutes and it's the only part of this argument that's actually about you.

Hit reply: how big is your corpus, and did you build retrieval before or after you checked whether you needed it? I read every reply.

AI is easy to demo. Hard to ship.

Notes on the facts: the cost arithmetic uses one vendor's published list prices at a mid-tier model, and the shape holds across providers but the exact crossover point won't — recompute it with your own model's numbers, it's a two-line calculation. The 120,000-token figure assumes a cache hit; without caching the crossover is roughly 6,000 tokens, which is to say “never”. The benchmark figures are collected from one aggregator page that mixes several different tests, so treat the direction as sound and the specific percentages as indicative. Prices and context limits both move faster than this newsletter publishes.

I'm a research engineer with 18+ years building production systems, 35+ patents, and 17+ publications. I work on infrastructure for multi-agent systems. AIEdTalks is my field notes: real problems, what I tried, and what I learned.

Views are my own.

💬 Hit reply. I read every reply.

▶️ Watch it instead. This week's breakdown is on YouTube. Subscribe to the channel for more.

📨 Know an engineer who'd find this useful? Share your referral link and earn rewards.

💡 Topic ideas or sponsorships: Reply to this email.

Until next time,

AIEdTalks team.

P.S. AI is easy to demo. Hard to ship. That's what this newsletter is about.