Welcome to AIEdTalks’ Newsletter!
In today's edition:
What to install, and what to run on the RAM you have
Pointing your existing OpenAI code at localhost
Three traps, including one that silently cuts your prompt in half
When to stop and use a paid model instead
Let’s dive in.
After reading, please rate today’s edition.
Rate today's Newsletter
Sponsored by
State of Product reveals what AI still hasn’t solved
80% of product professionals say AI helps them ship faster, but customers aren’t seeing value any sooner. Atlassian’s State of Product 2027 explores the gains AI is delivering and the challenges that remain, from decision-making that hasn’t kept pace to gut instinct overriding customer evidence.
Today’s Edition
The local model won't save you money. Run it anyway.
How to run an open model on your own machine, and when to stop.
Almost everyone learning AI engineering starts the same way. They read about API costs, get nervous, and go looking for a model they can run for free on their laptop.
I want to take that reason away from you, because it's wrong, and because the real reasons are better.
The bill you're afraid of
Let's do the arithmetic you've been avoiding.
A small paid model — Claude Haiku 5.5 — currently costs $0.10 per million input tokens and $0.50 per million output tokens.
Say you're iterating hard on a prompt or a small RAG app: 500 calls in a day, 3,000 tokens in and 600 tokens out each time.
input: 500 × 3,000 = 1.5M tokens → $0.15
output: 500 × 600 = 0.3M tokens → $0.15
total = $0.30Thirty cents. Six dollars a month if you code every weekday.
A full eval run over a thousand test cases, 4,000 tokens in and 800 out each, lands at 80 cents — 40 cents if you send it through the Batch API, which is half price.
Your laptop does not run for free either. It runs on electricity, and it runs slowly, and while it's running you can't use it for anything else.
So if you were going local to protect your wallet: the wallet is fine. Go use the API.
Four reasons that do hold up
Now the honest case.
1. The data can't leave. This is the big one, and it's not a beginner concern — it's most of enterprise. Customer records, patient data, source code under NDA, anything inside an air-gapped environment. If the data can't cross your network boundary, the price per token is irrelevant. I work on deployments like this; there is no API call to make.
2. No rate limits. Fan out 10,000 requests to grade an eval set and a hosted API will start returning 429s. Your own GPU returns answers until you stop asking. For bulk, boring, parallel work, that's a real difference.
3. The weights never change. A hosted model gets deprecated, or quietly improved, and your eval numbers move without your code moving. A file on your disk is a file on your disk. If you need results you can reproduce in six months, pin the weights.
4. You'll actually learn how this works. Context windows, tokenizers, sampling parameters, quantization, what “it ran out of memory” feels like. Behind an HTTPS call all of that is invisible. Running it yourself once is worth a month of reading.
Notice what's not on that list. Cost is not on that list.
Ten minutes to a local model
Step 1: Install a runner
You have four options and you only need to know what each one is for.
Ollama — easiest. A CLI and a local server. Start here.
LM Studio — a GUI over the same idea. Good if you'd rather click.
llama.cpp — the engine underneath both, and the thing that reads GGUF files. Use directly when you want control.
vLLM — not for your laptop. This is what you deploy on a real GPU when you need throughput.
On a Mac, MLX is the Apple-silicon-native path and it is noticeably faster than the generic route.
Step 2: Pull a model that fits your RAM
The only question that matters is how much memory you have. Quantized to 4 bits (Q4_K_M, the default), the rough tiers look like this:
Your machine | What to run |
|---|---|
8–16 GB | A 3–4B model (Phi-4-mini class). Fine for chat and simple extraction. |
24 GB VRAM | A 7B coder model, or a quantized ~26B mixture-of-experts model. |
32–64 GB unified | The 26–30B tier comfortably. This is where local gets genuinely useful. |
Less than that | Expect 2–5 tokens/sec as it spills to CPU. Painful, but it runs. |
ollama pull gemma4:26b # if you have the memory
ollama run phi4-mini # if you don'tOne trap in that table: a mixture-of-experts model advertises a small active parameter count, but you still need the full parameter count in memory. A “26B with 4B active” model needs 26B worth of RAM, not 4B worth.
Step 3: Point your code at it
This is the step that matters more than the model you picked. Ollama exposes an OpenAI-compatible endpoint, so your application code doesn't need to know what's behind it.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1/",
api_key="ollama", # required by the client, ignored by the server
)
resp = client.chat.completions.create(
model="phi4-mini",
messages=[{"role": "user", "content": "Say this is a test"}],
)Write your app against this interface and switching between local and hosted is a base URL and a model name. That single decision is what makes everything in the last section of this issue possible.
Three traps, and the first one is nasty
Trap 1: Your prompt is being silently cut in half
This is the one that will cost you a weekend.
Ollama's context window is set at server startup based on how much VRAM it detects. As of February 2026 the tiers are roughly 256K tokens at 47 GiB or more, 32K at 23 GiB or more, and 4K below that — including CPU-only machines. The old “4096 for everyone” figure you'll find in tutorials is out of date, which is almost worse, because now the default depends on the laptop.
When your prompt exceeds the window, nothing errors. The server keeps the first handful of tokens, throws away a block from the middle, keeps the tail, and returns a perfectly ordinary HTTP 200. Your 12,000-token RAG prompt arrives as a few tokens plus the end of itself. The response is fluent, confident, and built on a document it never saw.
Two colleagues on different laptops get different behaviour from identical code. That's the symptom.
And the OpenAI-compatible endpoint I just told you to use makes it worse: it accepts num_ctx in the request and throws it away, still returning 200. So you cannot fix this per-request through that route.
Fix it at the server instead:
OLLAMA_CONTEXT_LENGTH=32768 ollama serveOr bake it into the model so it travels with your project:
FROM gemma4:26b
PARAMETER num_ctx 32768ollama create gemma4-32k -f ModelfileThen check it actually took effect, rather than trusting it.
Trap 2: The compatibility layer is a subset
Ollama's OpenAI layer is explicitly experimental, and the gaps bite exactly where agent work lives:
tool_choice is not supported — you cannot force a tool call. If your agent depends on that, it won't behave the same locally.
logprobs is not supported — no token probabilities, so confidence-based routing and some eval techniques are off the table.
n is not supported — no multiple samples per request, which matters for self-consistency checks.
Tools, streaming, JSON mode and seed do work. Know which side of the line your project sits on before you build on it.
Trap 3: Local eval numbers don't transfer
A prompt tuned against a 4B model is tuned to that model's weaknesses. You'll add scaffolding — repeated instructions, defensive examples, output format reminders — that a frontier model doesn't need and may actually be hurt by.
So: develop locally, but always run your eval set against the model you intend to ship. Those are two different numbers and only one of them is real.
When to reach for the paid model
Stop being clever and pay when:
A user sees the output. Quality is the product. This isn't close.
The task needs real reasoning. Multi-step planning, subtle code changes, anything where being 80% right is the same as being wrong.
You need long context. A 26B model with a 32K window is not going to read your codebase.
You're measuring what you ship. Final evals run against the production model. Always.
Your time costs more than the tokens. Two hours fighting quantization to save six dollars a month is a bad trade, and you already know it.
Stay local when the data can't leave, when you're doing bulk parallel work, when you need reproducibility, or when you're learning.
The pattern that actually works
Not local or paid. Both, deliberately:
One interface. Write against the OpenAI-compatible shape so the provider is configuration, not code.
Develop against local. Fast loop, no bill, no rate limits, offline on a plane.
Validate against paid. Every eval run, before every merge.
Ship paid. Unless reason one on that earlier list applies to you, in which case ship local and budget for the GPU.
Keep both wired up. The day a provider has an outage, a config change is a much better position than a rewrite.
The one idea to remember
Run a model locally to understand the stack, not to economise.
The money was never the point — a thousand-case eval run costs less than a coffee. What you get from local is a boundary you control: your data stays put, nobody rate-limits you, and the weights don't move under your feet.
Those are engineering reasons. “It's free” is not one, and chasing it will cost you a weekend on a context window that was quietly throwing away your prompt.
Your turn
Hit reply: what's running on your machine right now, and what finally made you reach for the API instead? I read every reply.
AI is easy to demo. Hard to ship.
Sources
Claude API pricing — the per-million-token figures and the 50% Batch API discount, October 2026
Ollama OpenAI compatibility — the localhost:11434/v1/ endpoint, and the list of unsupported parameters
Ollama num_ctx: why long prompts get silently truncated — the VRAM-tiered defaults and the two truncation layers
Best open-source LLMs in 2026: coding, local, agentic — model tiers and licences
Best local LLMs 2026 — memory requirements per tier
Notes on the facts: the Claude pricing is from Anthropic's own pricing page and will change — check it rather than trusting this issue in six months. The context-window tiers and the truncation behaviour are documented by multiple independent sources and corroborated by issue threads in the Ollama repository, but they are runner behaviour, not a spec, so verify on your own install. I've deliberately avoided naming “the best” open model: that list turns over every few weeks, while the memory tiers and the selection rule don't. Most of these models are open-weight, not open-source — read the licence before you ship anything commercial on one.
Who's writing this?
I'm a research engineer with 18+ years building production systems, 35+ patents, and 17+ publications. I work on infrastructure for multi-agent systems. AIEdTalks is my field notes: real problems, what I tried, and what I learned.
Views are my own.
👋 Before you go
💬 Hit reply. I read every reply.
▶️ Watch it instead. This week's breakdown is on YouTube. Subscribe to the channel for more.
📨 Know an engineer who'd find this useful? Share your referral link and earn rewards.
💡 Topic ideas or sponsorships: Reply to this email.
Until next time,
AIEdTalks team.
P.S. AI is easy to demo. Hard to ship. That's what this newsletter is about.
