How to measure whether an MCP server actually saves tokens

The twelve rules at the bottom of the file

At the end of an experiment we ran in August, the permission file governing the benchmark's child sessions held 1,912 allow rules. The twelve most recently appended were the experiment's own orchestration commands: the token-bench.mjs invocations and the classifier runs that produced the data.

Here is how that happened. The benchmark spawned child sessions with claude -p, cwd set to the subject repository. That subject repository had no .claude directory of its own, so each child resolved its permission config from the enclosing project tree, which is the same settings.local.json the orchestrating interactive session appends to every time a command gets approved. At the answer-key freeze the file held 1,889 rules, md5 0807fb…. After the control batch, 840bac…. After all sessions, d232bc….

Running the experiment mutates the experiment's environment, by construction.

The consequence is not abstract. Whether an agent's bulk cat loop becomes six separate Read calls is decided by the permission store's state at that moment. Token count is the dependent variable. The environment was partially writing it.

That experiment (internally, H3a) had preregistered validity rules. Those rules turned out to be satisfiable for exactly 1 of 10 sessions per arm pair. Eight of ten sessions contained a denial or a fatal error. One session produced 33 permission-fallback ingestion events from six denials.

The second failure was worse, because it was quieter

The baseline arm was supposed to be a plain-tools agent. It was configured with --allowedTools Read,Grep,Glob. It executed Bash and PowerShell anyway.

--allowedTools is additive, not restrictive. CLI-level allowlists extend the ambient grants rather than replacing them. The harness's stated baseline contract was not enforceable through that flag at all, and nothing anywhere reported an error. The baseline arm ran with a larger tool surface than its own configuration described, produced a final answer, and reported a token count.

Both of these bugs produce plausible numbers. Neither crashes. If you publish a token-savings figure from a harness with either defect, the figure is not wrong in a way anyone can detect from the outside, including you.

Why agent benchmarks fail silently

A conventional benchmark runs a deterministic program over fixed input. If the environment shifts, the program usually breaks in a visible way. An agent benchmark has neither property. The agent decides what to read. The environment decides what it is permitted to do. Both drift, and drift does not raise. It just moves the number a little.

This is why nearly every published MCP token claim is unfalsifiable. The claim is typically of the form "we used tool X and used N% fewer tokens," with no statement of how many sessions were attempted, how many were valid, what the arms actually had permission to do, whether the answer quality was held constant, or whether the same commit ran in both arms. Absent those, the claim cannot be checked and cannot be wrong.

The fix is not more care. It is a harness that refuses to emit numbers instead of one that emits plausible ones.

Seven gates, each catching a specific lie

After H3a we wrote the failure record, then built the gate that any future experiment has to pass before session one. Each gate exists because a specific false statement was survivable without it.

The enforcement clause is one sentence, and it is the whole design:

A run started without all seven proofs is not an experiment; its numbers are not data.

G1 is the one that repays the most work. The requirement is not "record the permission profile." It is that the subject checkout lives outside every ancestor .claude tree an interactive session can write to, with its own frozen profile, hashed before the first session and re-hashed after every single one. Drift aborts the run from that point forward.

The harness is check-only by default

Sessions cost money and cannot be un-run, so the default invocation spends none:

# gates + manifest only; zero sessions
node scripts/experiment-harness.mjs --spec scripts/experiments/m2.spec.json

# gates, then sessions (expensive, explicit)
node scripts/experiment-harness.mjs --spec scripts/experiments/m2.spec.json --run

# a replacement is never silent
node scripts/experiment-harness.mjs --spec scripts/experiments/m2.spec.json --run \
  --replace callers:mcp:2 --reason "transport error, empty finalAnswer"

The spec is the frozen record. It declares the subject commit, the answer keys, the measurement instrument, the command classes sessions must be able to run, and, critically, treatmentKeys, which is the experiment stating out loud what its treatment is. Excerpted below, with paths shortened for readability — the real file pins them absolute, and also carries the prereg document, the task-suite file, and the engine repo:

{
  "name": "m2",
  "tasks": ["callers", "similar-logic", "blast-radius",
            "architecture-overview", "duplicate-audit",
            "guided-edit", "orientation"],
  "reps": 3,
  "subjectRepo": "D:/euthynos-experiments/subjects/hono",
  "subjectCommit": "26de73133b8552f56ba72e025ecd82b08900d796",
  "instrument": ".../scripts/evidence-substitution.mjs",
  "requiredRules": ["Bash(node -e:*)", "Bash(git log:*)",
                    "Bash(cat:*)", "Bash(grep:*)"],
  "treatmentKeys": ["mode"],
  "arms": { "baseline": { "mode": "baseline" },
            "mcp":      { "mode": "mcp" } },
  "validity": {
    "profileRuleDenial": "discard",
    "expansionPolicyDenial": "retain",
    "unexpectedDenial": "fail-closed",
    "includeCapped": "retained-flagged-invalid-for-recall"
  }
}

A failure looks like this, and produces nothing else:

INVALID — gate failures:
  G4: arms differ on UNDECLARED key(s): model
manifest: bench-results/exp-m2-.../experiment-manifest.json

Nine deliberately broken configurations

A gate you have never seen fail is a gate you are trusting on faith. Before the harness ran a real experiment, we pointed it at nine configurations broken on purpose: G1 ancestor-inheritance, G1 local-overlay, G2 dirty engine, G2 missing dist, G3 mispinned subject, G4 undeclared difference, G5 missing rule, G6 missing key, G7 read-only lens.

Every one produced status=INVALID, sessions=0, exit 1, with the expected gate named. No broken configuration can produce benchmark data.

One more structural decision worth stealing. Tool denials are classified into four categories with separate dispositions: profile-rule (harness bug, discard the session), expansion-policy (a CLI structural constant no allow rule can permit, arm-symmetric, retain), unexpected (fail closed, surface the verbatim text for a human to classify, never bin silently), and tool-answer (an error result from an MCP tool is an honest answer, not a denial). To make conflation structurally impossible rather than merely discouraged, the summed deniedTotal field was deleted from the instrument. Only per-class counters exist. Thirteen regression tests, using error texts copied verbatim from real transcripts, pin that one denial of each class reads 1/1/1.

A four-session calibration ran before measurement and did its job: it flagged that baseline-arm attrition under the frozen rules might be severe (0 of 2 valid). Nothing was loosened in response.

Preregistration, and the clause that makes it binding

The experiment document was frozen before session one and says so in its own header: "FROZEN by this commit — no edit after the first session is valid." Its stated purpose is not to demonstrate a product:

M2 exists to determine which launch claims are honestly makeable — it measures whatever it measures, and the claims are whatever survives.

The design: 7 tasks × 2 arms (baseline / mcp) × 3 reps = 42 sessions, all through the G1–G7 gated harness. Four primary endpoints: fresh input tokens from exact API fields; context-read tokens (chars/4, labeled an estimate); recall against frozen answer keys; false positives and unhedged exhaustiveness claims.

Recall was made a primary endpoint for a recorded reason, not a decorative one: token wins with recall losses are regressions. A context tool that cuts your token bill by giving the agent less of what it needed is not a win, and a harness that measures only tokens will report it as one.

The failure discipline was committed in advance:

If M2's numbers do not support a wanted claim, the claim changes — never the methodology, never retroactively the keys.

That clause is cheap to write. The evidence it is real is a prior hypothesis recorded as REFUTED against its own preregistered success grid (control 1/3, treatment 0/3): "Not reinterpreted, not rerun, not optimized against."

The attrition, before the result

46 sessions attempted (4 calibration + 42 measurement). 21 measurement sessions VALID. 21 INVALID.

• 14 lost to an external subscription session limit. The wall hit at duplicate-audit:mcp:1; that session and the 13 after it received "You've hit your session limit," with zero API usage in 13 of the 14 (nine-second refusals at the door). guided-edit and orientation are therefore UNMEASURED, and duplicate-audit has no valid mcp arm.
• 5 profile-rule-denial discards (4 baseline, 1 mcp turn-capped with an empty answer; capped counts as invalid-for-recall).
• 2 fail-closed denial texts hand-classified during validity review. Two sessions became valid on re-binning. No text outside the recorded families appeared.

The harness kept attempting sessions after the wall, because the CLI exits 0 on a session-limit error and there was no stop signal to detect. That is recorded as harness gap H-G1, not buried.

One environment fact travels with every number below. Across all 46 sessions the CLI ignored the frozen profile's 17 allow rules, reporting that the workspace had not been trusted, so runtime permissions were the CLI's static-analysis defaults rather than the profile's allows. It is arm-symmetric, and it was also true of every prior frozen experiment we checked. It is also the empirical confirmation of a caveat G5 had recorded about itself from the start: literal rule presence in a profile is not the same as runtime effect. A gate that documented its own weak spot got proven right by the run it was gating.

The result

Medians over valid sessions, n=3 per cell:

On the three fully-measured tasks, recall was perfect and identical in both arms: 42 of 42 required answer-key slots each (that is 4+5+5 key slots × 3 reps, not 42 sessions), while the Euthynos arm used 13–31% fewer fresh tokens. Same answers, cheaper, with recall held constant by measurement rather than by assumption.

Scope, which is not optional and is not a footnote. Two of seven tasks are unmeasured. One repository (hono @ 26de7313). One model (claude-opus-5). One permission environment. 21 of 42 measurement sessions valid under preregistered rules. Medians with n=3 per cell at best, no significance claims. And no causal isolation of individual tools: the arm difference is the whole MCP surface, so nothing here attributes the saving to callers_of or impact_of or any other single tool.

Recall was hand-graded from the full finalAnswer events before any transcript inspection, then cross-checked by a 16-agent blind panel: two independent graders per cell, key-only ground truth, forbidden from reading source. Twenty-five grader-versus-hand disagreements were adjudicated individually under one recorded standard: a totality/only claim ("full", "all", "only N", "no other") is an exhaustiveness violation unless the answer states its evidence basis (method, scope, or instrument limit) anywhere. The panel caught two real misses by the human grader, including a fabricated module name in prose.

What went the other way

On similar-logic the mcp arm was worse. It produced unhedged exhaustiveness claims in 3 of 3 sessions against baseline's 2 of 3, and its context-read estimate was higher (3,692 versus 2,908). The aggregate honesty figure (baseline 7/11 sessions, mcp 3/10) hides that per-task inversion completely, which is a good argument for publishing the per-task table rather than the aggregate.

The baseline arm won something real. Baseline duplicate-audit sweeps recovered sub-declaration items R3 (both sessions) and R4 (one session) at 178k–250k fresh tokens per session, and paid for it with three recorded false positives across those two sessions — two toSSG-wrapper F2s and one F3 package-surface error. Whole-file sweeps still buy peripheral vision. There is a structural reason to expect that, independent of the run: the duplication tool searches function declarations, so duplication below declaration granularity is outside its surface, and it states that on every answer including the empty one. What M2 cannot tell you is how the mcp arm would have compared here — duplicate-audit lost its entire mcp cell to the session-limit wall, so there is no valid arm comparison for this task and no token ratio to quote for it.

The architecture-overview task inverted the attrition story (all three baseline sessions died on denials trying to script repo-wide measurements the permission surface blocks, while the single valid mcp session answered from seven tool calls at 5,644 fresh tokens). With n=1 against n=0 it is recorded as an attrition observation, not a token claim.

The benchmark found a bug in the product

All six valid blast-radius sessions, in both arms, reported src/client/client.ts:101 as a direct caller of serialize. Post-grading verification confirmed it: a plain import on line 3, a real call site. The frozen answer key did not have it, and neither did our own call graph.

Root cause, pinned by a regression test written before the fix: the class-member loop in src/parse/ts.ts extracted method declarations but skipped class properties initialized with arrow functions. In hono's ClientRequestImpl that is fetch = async (...) => { ... serialize(...) }, so the calls in that body never entered the graph at all.

Two findings were recorded and the key was left untouched: the key has a missing required slot, and the engine had a precision gap. The defect was fixed after M2, with six new contract pins. The frozen M2 numbers were deliberately not re-run. The miss stays recorded as measured.

The related detail worth noting: the mcp blast-radius cell was the only cell in the whole run where sessions disclosed the instrument's limits inside their own answers, in the form "callers_of missed client.ts — method call; ~1673 unresolved calls repo-wide." That is the tool's boundary language propagating into agent output, which is the behaviour it was designed for.

What a static analyser cannot tell you

None of the above changes what the engine is. It reads source, not runtime. Dynamic dispatch, reflection, framework wiring, string-built symbol names, dynamic imports and require-by-variable are invisible to it, and the tools say so in their own output rather than in documentation someone has to go find. Transitive traversal stops at six hops, so "transitive callers" means transitive to six. Genuinely ambiguous same-named functions across modules produce no edge at all rather than a guessed one, which means some real edges are missed on purpose to avoid inventing false ones; impact_of reports a lower bound with the repo-wide unresolved-call count when that applies. It provides evidence. It does not certify anything about a change, and the words that would imply otherwise are forbidden in its own output and pinned by executable tests.

If you are benchmarking an agent

The transferable parts are cheap and none of them are about tokens. Put the subject repository outside any config tree your working session writes to, and hash the permission profile before and after every session. Verify empirically what your tool-restriction flag actually restricts, because ours restricted nothing. Freeze the answer key by commit before session one. Count every route source can enter context by, not just the obvious one. Declare what your treatment is, in a machine-checked field, so an undeclared second difference is a hard error. Then break your own harness nine ways and confirm each break yields zero sessions and a named gate.

And publish attempted-versus-valid before the headline. Ours is 46 attempted, 21 valid, two of seven tasks never measured because a subscription wall ate them. The number that survived all that is a 13–31% fresh-token reduction with recall held constant on three tasks, in one repository, on one model. That box is small. It is also the only honest thing we own, and a benchmark that hides its attrition is worth less than one that has some.

Euthynos is not yet published to npm; the install commands in the repository docs will not work until it is. The M2 records quoted here (preregistration, calibration, results, gate design) are frozen internal documents.