Welcome to AIEdTalks’ Newsletter!
In today's edition:
What brute-force search actually costs, measured
Semantic search in about twenty lines, no database
Three silent failures, including one optimisation that backfires
The three rungs: numpy, Postgres, and a real vector store
Let’s dive in.
After reading, please rate today’s edition.
Rate today's Newsletter
In partnership with
When someone asks AI what to buy, be the answer.
Search didn't die. It moved. Your buyers ask ChatGPT, Gemini, Claude and Perplexity what to buy, and they act on the answer they get.
That answer is assembled from sources most companies have never looked at. If you're not in there, a competitor is.
ChatFeatured is the AEO platform that does the work to get you in AI answers: See exactly what AI says about you, and how you stack up against competitors on every major engine.
Ask the built-in analyst agent why AI skips you, and get the specific fixes. Publish content engineered to get cited, then watch every AI bot that crawls your site.
Vessel grew its AI visibility 2,200%+. Most customers see their first citations inside 7 days.
Today’s Edition
Do you actually need a vector database?
Measured numbers, twenty lines of code, and the point where you should give in and install something.
The standard first move in RAG is to pick a vector database. There are a lot of them, they all have benchmarks, and choosing feels like the serious thing to do.
I want to suggest you skip it, and I'd rather show you numbers than argue.
What brute force actually costs
Semantic search over N documents is one matrix multiply: your query vector against every document vector, then take the top few. No index, no approximation, no server.
I ran it. These are measured on a two-core container — deliberately modest hardware, so treat them as a floor rather than a best case. 1024-dimensional float32 embeddings, median of several runs, top-10 retrieval:
Documents | Search time | Memory |
|---|---|---|
10,000 | 1 ms | 41 MB |
100,000 | 17 ms | 410 MB |
500,000 | 81 ms | 2.0 GB |
1,000,000 | 179 ms | 4.1 GB |
A hundred thousand documents, searched exhaustively, in seventeen milliseconds. On two cores.
Put that next to what happens after retrieval: the model call that uses the results takes one to three seconds. Your retrieval step is under one percent of the latency your user experiences. Shaving it is not where your attention belongs.
Notice also which column becomes a problem first. Speed is still tolerable at a million documents. Memory is the wall — 4.1 GB resident, and you need headroom on top to build it. I know because I killed my own container finding that out.
Twenty lines
import numpy as np
# embed once, at startup
vecs = np.array([embed(d) for d in docs], dtype=np.float32)
vecs /= np.linalg.norm(vecs, axis=1, keepdims=True)
def search(query, k=5):
q = np.array(embed(query), dtype=np.float32)
q /= np.linalg.norm(q)
scores = vecs @ q # cosine, because everything is unit-norm
idx = np.argpartition(-scores, k)[:k]
return [(docs[i], float(scores[i])) for i in idx[np.argsort(-scores[idx])]]That's the whole thing. Normalise once, and cosine similarity becomes a dot product. argpartition finds the top k without sorting the rest.
Save the array with np.save, load it at boot, rebuild when your documents change. For a corpus that updates daily, a nightly rebuild is a cron job, not an architecture.
Embedding the corpus is a one-off cost and a small one: a hundred thousand documents at roughly 500 tokens each is 50 million tokens, which runs about five dollars at current hosted embedding prices. Many models let you truncate the vector — 1024 dimensions down to 512 or 256 — which cuts your memory in proportion and usually costs very little recall. Test it on your own data and take the smallest size that still hits your target.
Three ways this fails silently
Retrieval failures don't throw. They return plausible, confidently wrong neighbours. Here are the three that catch people.
1. You embedded the two sides differently
This is the big one, and the cause is almost always innocent.
Many embedding models are asymmetric — they expect a prefix telling them which side they're embedding. Nomic's model wants search_document: in front of stored text and search_query: in front of a query. E5 and BGE families have their own equivalents.
Leave the prefixes off and nothing complains. Ollama, for instance, passes your input through unchanged and adds nothing. You get vectors back, you get rankings, and the rankings are quietly worse than they should be, because you've pushed the model outside the distribution it was trained on. There are open issues in several projects right now where queries are being embedded as documents for exactly this reason.
The same applies to any mismatch across the two sides: index with one model and query with another, or index at 1024 dimensions and query at 512, and you get silent nonsense. Not an error. Nonsense that typechecks.
Pin the model name, the dimension count and the prefixes in one config object that both paths read. Write it down next to the saved vectors, so a future you who upgrades the model is forced to notice the index is stale.
2. Your documents were truncated before they were embedded
Regular readers will find this familiar. Two issues ago we covered Ollama silently truncating prompts; the embedding path has its own version. truncate defaults to true, so text longer than the model's input limit is clipped without a warning.
And the limits are shorter than people assume. Nomic's v1.5 takes 8,192 tokens, but its v2 MoE variant caps at 512. Feed a 2,000-token document to that and you've embedded the first quarter of it. The vector represents the introduction, and the document never matches queries about its own conclusion.
Check your model's real input limit, and chunk to fit it deliberately rather than letting the runner decide for you.
3. The optimisation that made it seventeen times slower
This one surprised me, so I'm including the measurement.
Halving memory by storing embeddings as float16 is an obvious-looking win. At a hundred thousand documents and 1024 dimensions, it cut memory from 410 MB to 205 MB exactly as expected — and took search from 17 ms to 293 ms.
Seventeen times slower, for half the memory.
The reason is that numpy has no fast path for float16 matrix multiplication, so it converts elementwise and loses the optimised routine that made float32 quick. Float32 isn't the obvious choice because it's smaller. It's the obvious choice because it's the one the fast code path takes.
The general lesson is the one this newsletter keeps arriving at: measure the optimisation. The intuition about memory was correct and the conclusion was still wrong.
A related note in the other direction: if you're running an eval sweep, batch your queries. Sixty-four queries at once came out at 4 ms each, against 16 ms each run one at a time. Same arithmetic, four times the throughput, for the same reason the GPU section of the last issue gave.
When you do need the database
Brute force is not a permanent answer. Here's the ladder, and what moves you up a rung.
Rung 1 — numpy, up to a few hundred thousand documents. Exhaustive, exact, no dependencies, no server. Memory is the constraint and it's predictable: rows × dimensions × 4 bytes.
Rung 2 — pgvector, up to roughly ten million. The move here is usually not scale, it's operations: you want durable storage, concurrent writes, backups and metadata filtering alongside your existing data, and you already run Postgres. That's a real reason. “My corpus got to 200,000” is not.
Rung 3 — a dedicated vector store, beyond that. Tens of millions of vectors upward, sustained high write churn, hybrid keyword-and-vector search at scale, multi-region replication. These are genuine distributed-systems problems and they deserve a system built for them.
One thing worth knowing before you climb, because it's a cost nobody mentions in the tutorials. Approximate indexes interact badly with filtering. The index finds nearest neighbours first, and then your WHERE clause throws out the ones that don't match — so a request for ten results can come back with two. pgvector added iterative scans to mitigate it, and there are knobs for recall and scan limits, and now you own a tuning problem.
With brute force, a filter is a boolean mask over the score array. It's exact, it's three lines, and there is no recall to tune.
That's the honest trade. You climb the ladder for durability, write throughput and scale — all good reasons. You don't climb it for retrieval quality, because the thing you're leaving behind is the only option that was exact.
The one idea to remember
A vector database is an operations decision, not a retrieval decision.
For a hundred thousand documents, exhaustive search takes seventeen milliseconds on two cores and your model call takes two seconds. The database doesn't make your answers better — your chunking, your embedding model and your prompt do that. What it buys you is durability, concurrency and scale, which are things you should buy when you need them, and not before.
Start with an array. Move when something hurts. The thing that hurts first will be memory, and you'll know.
Your turn
Hit reply: what's in your retrieval stack, and did you pick it before or after you measured? I read every reply.
AI is easy to demo. Hard to ship.
Sources
The latency and memory figures are my own, measured on a two-core container with numpy 2.5: 1024-dimensional float32 unit-norm vectors, top-10 retrieval, median of seven runs. The benchmark script is short enough to reproduce from the code above.
Best embedding models for RAG, ranked by MTEB, cost and self-hosting — dimensions, hosted pricing and which models support truncatable embeddings
Nomic Embed Text, explained — the required task prefixes, the silent-truncation default, and context limits per variant
pgvector in production: when Postgres replaces a dedicated vector database — the ten-million-vector threshold, memory sizing and the overfiltering problem
Notes on the facts: my measurements come from one machine with two cores, so the absolute numbers will differ on yours — more cores make brute force faster roughly in proportion, which only strengthens the argument. The float16 result is a numpy-specific effect and does not apply to GPU inference, where half precision is genuinely faster; it's a lesson about measuring, not a claim about the format. The pgvector thresholds are from one practitioner write-up and depend heavily on dimension count and write volume. Embedding prices move; check before you budget.
Who's writing this?
I'm a research engineer with 18+ years building production systems, 35+ patents, and 17+ publications. I work on infrastructure for multi-agent systems. AIEdTalks is my field notes: real problems, what I tried, and what I learned.
Views are my own.
👋 Before you go
💬 Hit reply. I read every reply.
▶️ Watch it instead. This week's breakdown is on YouTube. Subscribe to the channel for more.
📨 Know an engineer who'd find this useful? Share your referral link and earn rewards.
💡 Topic ideas or sponsorships: Reply to this email.
Until next time,
AIEdTalks team.
P.S. AI is easy to demo. Hard to ship. That's what this newsletter is about.
