Welcome to AIEdTalks’ Newsletter!

In today's edition:

Let’s dive in.

After reading, please rate today’s edition.

When you type a prompt, you summarize. When you speak one, you explain. Wispr Flow captures your full reasoning (constraints, edge cases, examples, tone) and turns it into clean, structured text you paste into ChatGPT, Claude, or any AI tool. The difference shows up immediately. More context in, fewer follow-ups out.

Used by teams at Microsoft, Amazon, and Notion. Try Wispr Flow free on Mac, Windows, iPhone, and Android.

Start flowing free

Last issue you got a model running. This issue it has to serve other people.

Last time we got an open model running on your laptop and I argued that cost was the wrong reason to do it. This is the other half: what happens when the thing has to answer somebody other than you.

The short version is that Ollama is tuned for one user and a server is tuned for many, and almost every mistake in this issue comes from not noticing that the job changed.

Here's a benchmark of Llama 3.3 70B in FP8, on a single H100, under vLLM. The only thing changing is how many requests are in flight at once.

Same GPU. Same model. Roughly sixteen times the output, and with the server flags tuned it goes to 2,380.

This is continuous batching doing its job. The GPU is enormously parallel and a single request leaves most of it idle — the work is bounded by memory bandwidth, not arithmetic, so while one request waits on weights streaming from memory, there's room to serve dozens more essentially for free.

Which brings us to the trap.

If you benchmark at one request at a time, you will measure 122 tokens a second and plan your capacity from it. You'll conclude you need sixteen GPUs for a load that one would carry. I have watched this exact mistake get made with a straight face in a planning meeting, by people who are good at their jobs, because the benchmark was run the obvious way.

Load-test at your real concurrency, or don't bother load-testing.

Note also what the table says about latency: the per-user speed drops as the batch fills. At one request the user gets all 122 tokens a second to themselves. At 128 they get about fifteen. Throughput and latency are the same dial pointed in opposite directions, and you have to decide which one you're selling.

vLLM serves the same OpenAI-compatible interface Ollama did, which is the whole reason I made you write against that interface last issue.

Your application code doesn't change. Only the base URL does.

The alternatives, briefly: SGLang is competitive and often faster on structured output and heavy prefix reuse. TensorRT-LLM is the fastest on NVIDIA hardware and the most painful to operate. llama.cpp's server is for small deployments on modest hardware. Start with vLLM; you can justify moving later.

This is the part nobody tells beginners, and it's just arithmetic.

Every in-flight request holds a KV cache — the attention state for every token it has seen. That cache lives in whatever GPU memory the weights didn't take, and when the pool is full, the next request queues. That's it. That's your concurrency limit.

So on an 80 GB H100 running a 70B model in FP8, the weights take roughly 70 GB and you're left with something like 10 GB for the cache. Divide that by what one request holds:

Read that again, because it's the most useful thing in this issue: doubling your context length halves your concurrency. The 32K window you enabled because it seemed harmless cost you 80% of your capacity.

Two levers, once you see the shape of it:

Reading a long prompt is one big compute burst. While it runs, everyone else's tokens stop. One 32K prompt on a 70B model occupies the GPU for a few hundred milliseconds, and every other user feels it.

Chunked prefill breaks that burst into pieces and interleaves it with ongoing generation. In the benchmark above, median latency barely moved — but p95 time-to-first-token for long inputs fell by about two thirds.

That's the signature of a fix worth having: the average looks the same and the tail stops embarrassing you.

Quantization is how a 70B model fits on one card, so you will use it. The question is what you're paying.

Averaged across a set of 70B-class models:

FP8 looks close to free, and on modern hardware it's the sensible default. But the averages are hiding something, and it's the part that matters for this audience.

The damage isn't evenly spread. Chat and summarisation barely notice. Maths, code generation and agentic tool use notice a lot — the same reporting that puts FP8's average cost at 0.4 points finds it can take a point off maths benchmarks, and that four-bit hurts code generation considerably more than an aggregate score suggests. One team found FP8 cost them several points on a domain classification task while leaving their chat quality untouched.

If you are building agents, you are building on exactly the capability that quantization degrades first.

So: run a hundred prompts from your own workload before you trust a format. Not a public benchmark. Yours. This is the fourth time in six months I've written some version of that sentence, which should tell you something.

Now the question the whole thing is for. GPU, or API?

An H100 rents for around $2.90 an hour on the mid-market GPU clouds as I write this. Hold onto that number while we run it both ways.

Saturated, at 2,380 tokens a second, that card produces about 8.6 million output tokens an hour:

At one request at a time, 122 tokens a second, it produces 0.44 million an hour:

Nineteen times the unit cost, from the same hardware, for the same model. Self-hosting economics are almost entirely a utilisation question, and nothing else in this section matters as much as that.

Now set it against hosted pricing. That H100 runs you about $2,088 a month if you keep it up, and at full tilt it could produce around 6.2 billion output tokens in that month. So what would you have to be buying for the GPU to win?

And that second line is the honest comparison, because the first one isn't apples to apples: a 70B open model is not a frontier model, and if the small hosted model does your job then the small hosted model is your baseline.

So the real rule is: self-hosting wins when you can keep a GPU genuinely busy. Steady, high-volume, predictable traffic. Overnight batch work. A queue you control.

It loses on bursty traffic, because you pay for the peak and run at the average. Business-hours traffic with a nightly trough is the classic case where the spreadsheet says yes and the invoice says no.

Which leaves the reasons from last issue standing unchanged. Data that can't leave your network, reproducibility, no rate limits — those still justify self-hosting on their own. Cost only joins them when the utilisation is there.

A GPU's cost per token is a utilisation number, not a hardware number.

The same card, the same model and the same weights will cost you thirty-four cents or six dollars sixty per million tokens depending on nothing but how many requests you can keep in flight. Everything in this issue — continuous batching, the KV cache ceiling, chunked prefill, how long a context you allow — is a lever on that one quantity.

Which is why the first question isn't which GPU. It's how busy can I keep it, and if you can't answer that, you're not ready to buy one.

Hit reply: are you self-hosting, and did the economics work out the way you expected? I read every reply.

AI is easy to demo. Hard to ship.

Notes on the facts: every throughput number here comes from one published benchmark — Llama 3.3 70B in FP8, a single H100 SXM5, vLLM — so treat the shape of the curve as the lesson and the absolute figures as indicative. Your model, your hardware and your prompt mix will all move them. The GPU price is a mid-market on-demand median from the day of writing; hyperscalers charge appreciably more and spot instances less. The break-even arithmetic uses published list prices on both sides and ignores the engineer-hours of running the thing, which in my experience is the line item that actually decides these arguments.

I'm a research engineer with 18+ years building production systems, 35+ patents, and 17+ publications. I work on infrastructure for multi-agent systems. AIEdTalks is my field notes: real problems, what I tried, and what I learned.

Views are my own.

💬 Hit reply. I read every reply.

▶️ Watch it instead. This week's breakdown is on YouTube. Subscribe to the channel for more.

📨 Know an engineer who'd find this useful? Share your referral link and earn rewards.

💡 Topic ideas or sponsorships: Reply to this email.

Until next time,

AIEdTalks team.

P.S. AI is easy to demo. Hard to ship. That's what this newsletter is about.