Measurement & correction for MCP tool descriptions
A name, a description, a parameter schema — three strings are the entire interface an agent sees. When the words are off, it picks the wrong tool or pads the job with calls it never needed, and your server takes the blame. Toolmetry measures how often agents get it right, rewrites the descriptions, and re-measures until the improvement is proven.
npx mcp-toolmetry measure ./scenarios
Exhibit A describe_table official sqlite MCP server — the full diff, unedited
Get the schema information for a specific table Get the column schema for one table. Use only when the user asks about a table’s structure — do not call this merely to check that a table exists before running a query.
Procedure
Eval runners tell you the score and stop. Toolmetry is the step after — the rewrite is only kept if the numbers say so.
Every scenario runs N=5 against your live server. Tool choice, arguments, and extra calls are recorded — never a conclusion from a single run.
Failures are grouped by cause: wrong-tool confusion, wrong arguments, or ritual extra calls the task never needed.
An LLM redrafts only the failing descriptions, targeting the diagnosed failure — not a blanket rewrite of your server.
Same scenarios, same N, new words. The before/after delta is the product.
Regressions are thrown away automatically. Only measured improvement ever ships.
Results
Strict success = right tool + right args + no extra calls, over N=5 runs per scenario.
sqliteofficial server
+66.0
Agents ritually inspected the schema before two-thirds of queries. One added sentence fixed it.
memoryofficial server
+34.5
Wrong-tool confusion between overlapping tools; hit rate went 80% → 100%.
gitofficial server
+21.7
Perfect tool choice — but constant ritual git_status / git_log verification calls.
filesystemofficial server
+10.0
A deprecated read_file alias kept trapping agents into the old tool.
Every number above comes from committed run data, and the memory and git rewrites were sent upstream with the evidence attached — servers#4519 and servers#4520.
A stronger agent, Claude Haiku 4.5, baselined at 84.4% on the same suite. Bad descriptions tax cheap agents hardest — and cheap agents are what everyone deploys at scale.
Run it on your server
MeasureScore your server as-is: strict success, hit rate, and extra-call rate per scenario.
npx mcp-toolmetry measure ./scenarios
OptimizeRun the full loop. Produces a before/after report and the winning descriptions.
npx mcp-toolmetry optimize scenarios/your-server.yaml --rounds 2
ShipA proxy rewrites tools/list on the fly — better descriptions without forking the server.
npx mcp-toolmetry proxy --overrides best.json -- <server cmd>
Uncertainty budget
A measurement is only as honest as its error bars.
Per-scenario rates move in 20-point steps. Small deltas need more runs before you believe them.
Two attempts on the same baseline scored +10.0 and −2.2 points. That’s why the loop measures every rewrite and discards regressions automatically.
An improvement measured with one model doesn’t transfer one-to-one to another. Measure with the agent you actually deploy.
If two tools genuinely overlap, no description will save the agent. Renaming or merging them is still your job.