Welcome to AIEdTalks’ Newsletter!
In today's edition:
More Tools, Dumber Agent
Let’s dive in.
Today’s Edition
AI TOPIC
More Tools, Dumber Agent
GitHub's Copilot team had a problem that will sound familiar. Their coding agent shipped with around 40 tools enabled by default. More tools, more capability — that was the assumption.
Then they cut the default toolset from 40 to 13.
Success rates went up 2–5 percentage points. Latency dropped by roughly 400ms per request. They removed two-thirds of the agent's capabilities and it got better at its job.
Speakeasy ran the cleaner version of this experiment: same task, toolsets of different sizes. At 10 tools, performance was perfect. At 20 tools, large models scored 19 out of 20. At 107 tools, both large and small models failed completely.
The failure wasn't gradual. It was a cliff.
What's actually happening
Every tool you register isn't just a capability. It's a payload.
Each tool ships a JSON schema — name, description, parameters, types — that can run from a few hundred to ~3K tokens. And those schemas are re-sent on every single turn. Connect four MCP servers with 10–15 tools each and you've spent 30–50K tokens of context before the user's request arrives. Backend translation: a service mesh where every request must inline the full OpenAPI spec of every service in the registry, whether the request touches that service or not.
That creates two failures at once:
Context tax. Tool schemas crowd out conversation history, retrieved documents, and the actual task.
Selection confusion. Fifty near-duplicate descriptions (
search_issues,find_issues,query_tickets) compete for attention. The model picks the wrong one, hallucinates parameters, or freezes deciding.
Call it the tool overload cliff. And it hides well: your averages stay decent because simple one-tool tasks keep working while the multi-step workflows quietly break — and it arrives one harmless-looking PR at a time. Nobody adds 100 tools in one diff. Someone connects GitHub, then Slack, then Jira. No single change gets blamed.
Check if you're paying the tax right now:
import json, tiktoken
enc = tiktoken.get_encoding("o200k_base")
schemas = json.dumps([t.to_schema() for t in agent.tools])
tokens = len(enc.encode(schemas))
print(f"{len(agent.tools)} tools = {tokens:,} tokens/turn")
print(f"= {tokens / 200_000:.1%} of a 200K context, every turn")If tool definitions eat more than 10–15% of your context before the user says a word, keep reading.
The fix is retrieval, not restraint
Hard-trimming your toolset trades capability for accuracy. The better fix is the one you already know from RAG: don't show the model everything — retrieve what's relevant. That's the tool router pattern. One embedding index over your tool catalog, one similarity lookup before each model call. Framework-neutral, in Python.
Step 1 — Embed your tool catalog once
Your tools already have names and descriptions. Treat them as documents.
python
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("all-MiniLM-L6-v2") # local, free, fast
def tool_text(tool):
# Embed what the tool is FOR, not just what it's called
params = ", ".join(tool["parameters"].keys())
return f"{tool['name']}: {tool['description']} Parameters: {params}"
catalog = [tool_text(t) for t in ALL_TOOLS]
tool_vectors = model.encode(catalog, normalize_embeddings=True)Runs once at startup. Fifty tools embed in under a second on CPU. No vector database — a numpy array is your index.
Step 2 — Route on every turn
Before each model call, score every tool against the current request and keep the top-k:
python
def route_tools(user_message, k=8):
query_vec = model.encode([user_message], normalize_embeddings=True)
scores = (tool_vectors @ query_vec.T).flatten() # cosine similarity
top_idx = np.argsort(scores)[::-1][:k]
return [ALL_TOOLS[i] for i in top_idx]
tools_this_turn = route_tools(msg)
response = llm_call(messages=history, tools=tools_this_turn)The model sees 8 sharp options instead of 80 blurry ones. Route per turn, not per session — a multi-step task might need search_issues on turn one and send_slack_message on turn four.
Step 3 — Pin the tools that must always exist
Some tools should never be filtered out — your escalation tool, your "ask the user" tool, whatever ends the run safely.
python
PINNED = {"ask_user", "finish_task", "escalate"}
def route_tools(user_message, k=8):
pinned = [t for t in ALL_TOOLS if t["name"] in PINNED]
candidates = [t for t in ALL_TOOLS if t["name"] not in PINNED]
# ... same similarity ranking over candidates ...
return pinned + top_k_candidatesSkip this and you'll learn it the hard way: the one turn where the agent needed to bail out is the one turn the bail-out tool didn't make the cut.
Step 4 — Prove it worked
A router you haven't measured is a superstition. Take 20 tasks your agent handled last week and run them twice:
python
def eval_run(tasks, toolset_fn):
correct = 0
for task in tasks:
tools = toolset_fn(task["message"])
resp = llm_call(messages=[task["message"]], tools=tools)
correct += (resp.tool_called == task["expected_tool"])
return correct / len(tasks)
print("full :", eval_run(TASKS, lambda m: ALL_TOOLS))
print("routed :", eval_run(TASKS, lambda m: route_tools(m)))Log the token delta too. GitHub's router pre-selected the right tools for 72% of calls (up from 19%) and cut token usage roughly 18% per session. Your numbers will differ. Get your numbers.
Two ways the router bites back
Bad descriptions route badly. If your tool description is "Runs a query", the embedding has nothing to grab. Rewrite descriptions to say when to use the tool — "Search open Jira tickets by keyword, assignee, or status" routes; "Query tool" doesn't. Most of your routing accuracy lives here.
k is a dial, not a constant. Too small and the right tool misses the cut on ambiguous requests; too large and you've rebuilt the cliff. Start at 8, then check your eval: if the expected tool ranked 9th–12th on your failures, raise k before you blame the embeddings.
Route your tools, or your tools will bury your prompt. There is no third option.
Ran this on your own toolset? Reply with your before/after numbers — the best ones go in a future issue.
👋 That’s All Folks!
Before you go, just a few public service announcements:
See you soon,
AIEdTalks’ Newsletter Team
