Welcome to AIEdTalks’ Newsletter!
In today's edition:
Prompt Caching: Stop Paying for Tokens the Model Already Read
Let’s dive in.
Today’s Edition
AI TOPIC
Prompt Caching: Stop Paying for Tokens the Model Already Read
A question from a real cost review:
Your agent has a 20,000-token system prompt. A session runs 50 turns. Your prompts haven't changed in weeks. Cached tokens cost a tenth of fresh ones.
So why does the bill look like nothing is cached?
For most teams, nothing is. One security company audited their agent and found a 7% cache hit rate on a workload that should have been above 80%.
Here's why that happens, and how to check your own tonight.
Where the money goes
An agent never makes one LLM call. One task fires a whole sequence. Plan. Call a tool. Read the result. Decide the next step.
Every one of those calls re-sends everything before it. The system prompt. The tool definitions. The project context the model already read three turns ago.
So take our example. 20,000 tokens, 50 turns. That's 1 million tokens of the exact same text, read and billed over and over.
After turn one, none of it produces anything new.
What is prompt caching?
Prompt caching is the provider's fix for this. The model reads your prompt once, saves what it worked out, and reuses that work when a later request starts with the same content.
You still send the whole prompt every time. The provider just recognises the part it has seen before and skips re-reading it.
Those recognised tokens are then billed at a fraction of the normal price. On Anthropic and current OpenAI models, that fraction is about 10%.
It exists because of the pattern above. Agents replay the same history on every step, and most of that history never changes. Providers were re-reading identical text millions of times a day, and charging you for it.
So caching buys you two things:
Cost. The unchanging part of your prompt drops to roughly a tenth of its price from turn two onward.
Speed. Reading the prompt is the wait before the first word appears. Skip that, and long prompts start answering much faster.
You already do this elsewhere. You don't recompute an expensive query on every request. You store the result and serve the copy. Providers now do the same for prompts.
But there's a catch, and it's the reason caching quietly fails for so many teams. The provider isn't storing your text. It's storing the work it did on that text.
To see why that matters, look at what the model actually does with a prompt.
Why any of this works
Every request runs in two phases.
Reading. The model takes in your whole prompt at once. For every token it computes three vectors — a Query, a Key, and a Value — and compares each token against all the others. This is the expensive part, and it's what you're waiting on before the first word comes back.
Writing. The model then produces its answer one token at a time, mostly looking up what it already worked out. Cheap by comparison.
Now the useful detail. A token's Key and Value depend only on the tokens before it. Nothing that comes later can change them.
So if your prompt always starts with the same 20,000 tokens, those tokens always produce the same Keys and Values. The provider computes them once, stores them, and files them under a hash of the text.
Next request that starts the same way? The hash matches, the stored work loads from memory, and the reading step is skipped.
That's the whole feature. Everything after this is pricing and discipline.
Two halves of every prompt
Look at any agent request and you'll find two parts that behave completely differently.
The part that never changes. System instructions, tool definitions, reference docs. Same on turn 50 as on turn one. This is what gets cached.
The part that grows. User messages, model replies, tool output. New every turn. Always billed fresh.
The entire discipline is one sentence: keep the unchanging part identical and at the top, and let only the growing part pile up at the bottom.
What it costs
On Anthropic, reading from cache costs 0.1x the normal input price. Writing to cache costs 1.25x — a one-time 25% premium to store the work.
Do the math and the premium pays for itself after a single cache hit. Everything after that is profit.
On OpenAI it's automatic above 1,024 tokens. No setup, and currently a 90% discount on current models.
Speed follows the same curve. Providers report up to 85% faster first-token times on long prompts. A warm cache doesn't just cost less. It feels faster, because the model is genuinely reading less.
Why it breaks
Now the counterintuitive part.
The match has to be exact. The provider hashes your text, so "the same prompt" means character for character the same. Change one thing near the top, and everything below it gets re-read at full price.
There's no error. No warning. Just a normal-looking charge on your bill.
Things that have quietly killed caches in production:
A timestamp in the system prompt. One open-source agent put
datetime.now()inside a 4,000-token system prompt. Every request produced a different hash, so the hit rate sat at zero, permanently, at roughly 10x the necessary cost.JSON key order. Some languages don't guarantee the order of keys when serialising. Your tool definitions come out slightly different between requests, the hash changes, the cache misses. Anthropic's docs call this one out by name.
Adding or removing a tool mid-session. Tool definitions sit right at the front, so touching them invalidates everything after. One measured case dropped a session from 99% to 17%.
Switching models mid-session. Each model has its own cache. Move to a cheaper model halfway through and it starts cold, rebuilding your entire context at full price. Over a long session, the cheap model with a cold cache can cost more than the expensive model with a warm one.
That last one is exactly why our routing issue said to route once per task, pin the model, and only re-route when the task changes.
Three rules cover almost all of it:
Never edit the top of your prompt. Need to pass new state? Add it to the newest message instead. Timestamps, working memory, request IDs all belong at the bottom.
Load your tools once and leave them alone. Sort your JSON keys while you're there.
One model per session. If a subtask needs a cheaper model, run it as a separate sub-agent with a short handoff, rather than switching mid-conversation.
Teams that take this seriously treat it as an operational metric. ProjectDiscovery moved one changing block out of their system prompt and into an appended message. Their hit rate went from 7% to 84%, and their bill dropped 59%. Anthropic's Claude Code team runs alerts on cache hit rate and treats a drop as an incident.
Find out if you have this problem tonight
Check 1 — look at one response. Your API already reports this. On Anthropic:
u = resp.usage
hit_rate = u.cache_read_input_tokens / max(
1, u.cache_read_input_tokens + u.cache_creation_input_tokens + u.input_tokens
)
print(f"cache hit rate: {hit_rate:.0%}")A stable system prompt over 1,024 tokens and a hit rate near zero means something at the top of your prompt is changing. Under 30% means go find it. Teams that tune for this run above 84%.
Check 2 — put a number on it. Work out what the misses actually cost you:
redundant_usd = (
static_prefix_tokens * turns_per_session * sessions_per_month
* (base_input_rate - cache_read_rate) / 1_000_000
)If that comes back as pocket change, leave it alone. Not every optimisation deserves your evening. If it's real money, the fix is usually one line: move the changing thing out of the top of the prompt, and sort your keys.
Then watch hit rate the way you watch uptime.
Pay for your prefix once, or pay for it every turn. There is no third option.
Checked your hit rate?
Reply with the number — the lowest one gets a diagnosis in a future issue.
👋 That’s All Folks!
Before you go, just a few public service announcements:
See you soon,
AIEdTalks’ Newsletter Team
