Repository navigation
feat(bench): measure code execution's response-side saving - #1160
Merged
Merged
Conversation
Deploying mcpproxy-docs with
|
| Latest commit: |
cda9a4f
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://5ca8f2c8.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://spec103-codeexec-response-sa.mcpproxy-docs.pages.dev |
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 33520331963 --repo smart-mcp-proxy/mcpproxy-go
|
…he three gaps that blocked it
Every savings figure in bench so far is about the MENU — how many tokens of
tool definitions an agent carries. Code execution's menu is already the
smallest of the three surfaces, but the menu is not where it pays: a script
issues several upstream calls inside the sandbox and only the value it returns
enters the model's context. The intermediate responses, which an agent driving
those tools itself would have paid for in full, never appear. Nothing measured
that.
It could not be measured, for three separate reasons, each fixed here:
1. Sandbox sub-calls emitted hardcoded 0,0 byte lengths. In the activity log 0
means UNKNOWN, not free, so every sub-call was unaccountable — the gap bench
already names ReasonSubCallZeroBytes.
2. Internal built-in calls never populated the byte fields at all, so the
code_execution PARENT was unaccountable too. Fixed centrally in the emitter,
which also closes ReasonInternalNoByteCounts for retrieve_tools and
describe_tool. The value emitted is the full pre-truncation size, which is
what response_bytes is defined to mean; the truncation flag travels beside
it so a consumer can still exclude the row.
3. A PARAMETERLESS tool call records no arguments, so requestCost fell through
to the byte estimate while its siblings were measured. Colliding bases meant
the whole script was withheld — and most fleets have several parameterless
tools. Found on real traffic, not in unit tests.
Measured over 7 scripts and 63 sub-calls on a live 4-server fleet, the saving
ranges from -14.9% (a script that returns its one response verbatim) to +97.0%
(thirty file reads summarised to a count). The negative case is real and is
reported as one: code execution's saving is a function of how much the sandbox
discards, and a script that discards nothing loses tokens.
Accounting rules, each pinned by a test that fails when the guard is removed:
- never-sum: a measured cost is TOKENS and an estimated cost is BYTES; a call
mixing them is withheld rather than added into a number in no unit.
- no silent zeros: a withheld saving carries no figure at all, because a 0
reads as "saved nothing" when the truth is "nothing was measurable".
- no double-counting: sub-calls live under their parent, never in Calls.
- truncation withholds, and does not merely annotate. A truncated component's
byte length is the full pre-truncation size, but baseline means what an
agent would have paid, and that agent receives a cut response — so counting
it overstates the baseline and INFLATES the saving. Caught by cross-model
review; the test that previously asserted the annotate behaviour is
inverted rather than deleted.
Also adds breakevencurve, which computes fleet-size break-even by bootstrapping
random orderings of the 527-tool corpus: definition sizes span 145..6485 bytes,
so a single ordering would be one arbitrary curve rather than an answer.
Corrects specs/103 findings/webui-savings-vs-bench.md: BOTH figures in the
+68.4% / -32.5% contradiction were wrong. The Web-UI number is non-deterministic
(map-order sampling, 52.35%-78.80% over an unchanging fleet), and bench's own
proxy menu was overstated because ProxyModeToolDefs pins code execution on. The
sign of the finding survives; its magnitudes did not.
… state
Spec 103's break-even framing inherited a 2025 claim: that above roughly 40-128
tools a user physically cannot connect everything, so mcpproxy's value is
CAPABILITY rather than token saving. Checked it. It mostly does not hold.
- Cursor's "40-tool cap" is 2025 forum lore, absent from current docs, and
recharacterized by staff as an upstream per-model limit.
- Gemini went 128 -> 512. Anthropic, Bedrock, Claude Code, Kiro, Cline and
Zed document no cap, and the MCP spec caps nothing.
- The 2026 answer to large tool surfaces was not a raised ceiling but
DEFERRED LOADING, which Claude Code, Cursor, VS Code and Kiro now ship
natively — the same mechanism this proxy sells.
The binding constraint moved, and it is far LOWER than any cap: Anthropic
publishes tool-selection degradation past 30-50 available tools, where nothing
errors and quality just falls. Every crossing point on our break-even curve is
above that. Fleet size may be the wrong x-axis — median MCP server is 10 tools
and only 1.2% expose more than 100, so WHICH servers dominates HOW MANY.
Two consequences for what we may claim:
1. Do not say "you couldn't connect these otherwise" without naming the
client. On Claude Code in 2026 you could.
2. Against a deferring client the token baseline is already reduced, so the
defensible claim is retrieval QUALITY above 30-50, not raw token saving.
The all-loaded baseline is honest only on non-deferring surfaces.
The document keeps its own completeness critic, which judges the research NOT
thorough and names three holes — tools/list pagination non-compliance, the
gateway/aggregator tier (AWS AgentCore and Docker MCP Gateway have shipped this
design already), and prompt caching. The caching one matters most and is
unresolved: tool definitions are the canonical cacheable prefix, so a proxy that
varies the exposed tool list per turn invalidates that prefix every turn and can
cost MORE than the static baseline it is measured against. Nothing in bench
models this.
…t does not move break-even
Step 1 of the caching question: can mcpproxy invalidate the provider's prompt
cache, and does that flip the sign of the savings claim? Static audit of every
tool-list mutation path plus the refresh log of one ordinary 4-server startup.
No API spend.
Pinned to primary source (platform.claude.com, verified today): prefix order is
tools -> system -> messages; writes 1.25x (5m) / 2x (1h); reads 0.1x; and
"modifying tool definitions ... invalidates the entire cache".
Findings:
- The DEFAULT retrieve_tools surface is cache-stable. It is rebuilt only on
config.reloaded, never on server churn.
- The code_execution surface fires notifications/tools/list_changed on every
servers.changed for a tool list that structurally cannot change —
buildCodeExecModeTools never iterates upstreams. Measured 9 of 9 broadcasts
gratuitous on one startup.
- The direct surface DOES re-write real tool definitions on ordinary churn
(1 -> 3 -> 18 -> 27 tools across one startup), and that is a genuine full
invalidation. RefreshDirectModeToolsOnSerializationChange is drift-guarded;
the servers.changed path calls the UNGUARDED RefreshDirectModeTools.
- 17 sites emit servers.changed, including server_connected/_disconnected/
_state_changed. Coalescing is a 50ms window, so a reconnect loop is not
collapsed.
Two corrections to my own earlier reasoning, both in the write-up:
1. A gratuitous list_changed is NOT a cache hazard. mcp-go sorts tools/list
(sort.Strings), so an unchanged set returns byte-identical content and the
cache still hits. It is wasted client round-trips, not lost cache.
2. Caching does NOT push break-even further right. Both arms cache the same
way, so the (B - M) factor divides out and the crossover fleet size is
roughly invariant. What caching changes is the ABSOLUTE saving: in a cached
steady state a token bills at 0.1x, so every raw token figure this spec
publishes overstates money value by about 10x.
The mechanisms that do move the line, both against the proxy: discovery
responses are per-turn cache WRITES at 1.25x while the baseline pays nothing
extra, and direct-mode invalidation re-writes the whole prefix including the
conversation. The latter is the only mechanism found that can plausibly flip the
sign, and it is confined to direct mode with a churning fleet.
Recommends guarding both refresh paths before any paid step-2 measurement, so
the experiment does not price avoidable churn as if it were inherent.
mcp-go's SetTools notifies EVERY initialized client unconditionally — it swaps the registry and sends notifications/tools/list_changed without comparing old and new content, despite its own comment saying "when the list of available tools changes". Both routing-mode surfaces rebuilt speculatively on servers.changed, which has 17 emitters including server_connected, server_disconnected and server_state_changed. So every reconnect attempt told every connected client its tool set had moved, and the 50ms coalescer collapses nothing on that timescale. Measured on one ordinary 4-server startup, before and after: code_execution surface : 9 publishes -> 1 direct surface : 10 publishes -> 4 The direct tool_count sequence is the clearest evidence: before: 1 1 1 3 3 18 27 27 27 27 after: 1 3 18 27 Every duplicate gone, every genuine transition preserved. The code-execution surface could never change at all — buildCodeExecModeTools carries built-ins only and never iterates upstreams — so all 9 of its publishes were spurious. Guarded on a content FINGERPRINT rather than by dropping the servers.changed call, so both stay correct if either surface ever gains fleet-dependent content. The direct guard compares TWO fingerprints, and the second is the load-bearing one. Registry and catalog publish as a pair (a handler closes over its catalog entry), and routing can move while the listing stays byte-identical: a display-name collision resolving to a different upstream, or a changed RequiredPermission that gates scoped tokens without appearing in any tool definition. Guarding on the listing alone breaks three existing tests, including spec-102's direct_tool_response_mode hot reload — with no upstream tools a serialization flip changes only the catalog's mode, leaving the listing identical. A listing-only guard would have silently disabled a shipped feature. RenderedDescription is deliberately excluded from the routing fingerprint: it is captured at render time and the signature cache mutates independently of rebuilds, so including it would report a cache warm or evict as a catalog change — the distinction D13 rule 5 exists to preserve. directSurfaceUnchanged is a named predicate rather than an inline && because the reload harness has no upstreams and so cannot stage a routing-only change; the requirement is pinned by a test instead of by a comment. What this does NOT do: it removes wasted client round-trips, not lost provider cache. mcp-go sorts tools/list, so a re-fetch of an unchanged set returns byte-identical content and a cached prefix still hits (see findings/cache-prefix-stability-audit.md). The four remaining direct-mode publishes are genuine tool-definition changes and are genuine invalidations.
…ed can be money lost
The saving I shipped two hours ago sums a tool call's ARGUMENTS with its
RESPONSE. They are not the same kind of token. Arguments are text the MODEL
GENERATED (output, billed ~5x input); the response is text it reads (input,
billed 1x, or 0.1x against a cached prefix). Summing them prices a 5x asymmetry
at 1x — a milder form of the never-sum error this file already polices.
It is not academic. Code execution trades the cheap direction for the expensive
one: it replaces N argument-writings with ONE script, whose cost is fixed
however few calls it replaces. Measured on real traffic, filesystem reads of
~700-byte files:
sub-calls | output base->ce | input base->ce | tokens | USD (uncached / cached in)
1 | 5 -> 44 | 62 -> 26 | -3 | -0.000477 / -0.000574
2 | 24 -> 78 | 210 -> 11 | +145 | -0.000213 / -0.000750
3 | 36 -> 78 | 315 -> 11 | +262 | +0.000282 / -0.000539
5 | 60 -> 78 | 525 -> 12 | +495 | +0.001269 / -0.000116
7 | 84 -> 78 | 735 -> 12 | +729 | +0.002259 / +0.000307
30 | 360 -> 78 | 3150 -> 12 | +3420 | +0.013644 / +0.005171
Break-even is ~2 sub-calls in raw tokens but ~6 in money once input is cached.
Between 3 and 5 the token figure reads "+262 to +495 saved" while the wallet
loses. Publishing the token number alone would be exactly the kind of flattering
arithmetic spec 103 exists to stop.
CodeExecSaving now carries BaselineOutput/BaselineInput and ProxyOutput/
ProxyInput beside the existing totals, and PricedSavingUSD converts them.
Pricing REFUSES two cases rather than returning a plausible number: a withheld
saving (0.0 would read as "free" rather than "not measurable") and an estimated
one (those are BYTE lengths, and pricing bytes per token is wrong by the
bytes-per-token ratio while looking entirely correct).
Adds bench/cmd/iosplit, which reports the split straight from an activity export
— that is how the table above was measured.
Response-side totals are unchanged; this only stops them being read as cost.
…p truncated built-ins inflating usage
Three defects from cross-model review of the branch.
HIGH (bench): a stored-script code_execution was charged for a script the model
never wrote. mcpproxy records the RESOLVED source under "code" alongside the
name so the audit trail shows what ran, but the model sent only
{"script": "<name>", "input": {...}}. Billing the recorded arguments as
model-generated output charges ~5,200 output tokens per run for a ~17-token
request, at roughly 5x the input rate — and it does so in exactly the
configuration stored scripts exist to make cheap, so it understated code
execution by the whole size of the script on every invocation.
billableArguments now drops "code" when a non-empty "script" name is present,
and only then; an inline call genuinely did send its source and keeps it.
MEDIUM (bench): PricedSavingUSD accepted NaN, infinity and negative prices and
returned ok=true, so nonsense propagated silently into a confident dollar
figure. Rejected now, the same way a withheld or byte-estimated saving is.
MEDIUM (runtime): countInTimeBucket added ResponseBytes without regard to
ResponseTruncated. That was harmless while internal calls carried no byte counts
at all; this branch gave them some, so a truncated retrieve_tools began
contributing its FULL pre-truncation size to delivered traffic. The direction
differs by record type and that is the point — an upstream tool_call is cut on
the way into the LOG while the agent got the whole response, so its
pre-truncation length is honest; a built-in is the reverse, the log keeps the
full text and the agent consumed the cut one. Counting the second inflates the
usage timeline in the flattering direction.
Each fix is pinned by a test that fails when the guard is removed, verified by
mutation. Also adds a runbook for measuring the "Twenty-Eight Round Trips"
workload on the machine it was recorded on.
redactedRegistrySummary (#1164) dereferences its argument, and all three registry handlers — add, edit and remove — called it on the SUCCESS path with no nil check. A controller reporting success without an entry therefore panicked inside the handler, and chi's recoverer turned that into a bare 500 with an EMPTY body: the caller saw an unexplained server error on a request that had actually succeeded, and the cause survived only as a stack in the log. Not hypothetical. MockServerController already returns (nil, nil, nil) from these methods, so every test driving those routes through it hit the panic. It surfaced as an intermittent windows-amd64 CI failure where the recovered panic escaped the test process — intermittent because the recoverer usually swallows it, which is exactly why it went unnoticed: the package stayed green while three routes were answering 500 to successful requests. The guard lives in the renderer rather than at the three call sites so a fourth caller cannot reintroduce it. A nil entry now renders the zero summary, which reports no identity and — deliberately — Trusted=false: nothing unknown may be described as trusted. Tests assert both that the routes do not panic and that the body is NON-EMPTY, since an empty 500 is the signature of a recovered panic and a test that only checked the status code would have passed throughout.
Dumbris
force-pushed
the
spec103-codeexec-response-saving
branch
from
September 1, 2026 14:34
040b9f2 to
cda9a4f
Compare
Dumbris
added a commit
that referenced
this pull request
Sep 2, 2026
… token-bench truth-sync (#1200) * chore(roadmap): raise mcp-2026-upgrade to P1 (mcp-go v1.0.0 stable), add activity-storage-bounds, sync token-bench - mcp-2026-upgrade: P3 -> P1. mark3labs/mcp-go v1.0.0 stable shipped 2026-09-02; go.mod still pins v0.57.0. Next step is speckit.plan (#532). - action-log-transparency: new task activity-storage-bounds tracking #1173/#1174 (response truncation never wired), #1176 (unbounded per-server tool_calls buckets) and #1175 (omitempty zero erasure). - token-bench-harness: in_progress -> done; #1141/#1147/#1151/#1153/#1160 are all merged. Clears both --check-github errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(roadmap): keep merged token-bench PRs on the done task, not the in_progress epic Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every savings figure in
bench/so far measures the menu — how many tokens of tool definitions an agent carries. Code execution's menu is already the smallest of the three surfaces, but the menu is not where it pays.It pays on the response side: a script issues several upstream calls inside the sandbox and only the value it returns enters the model's context. The intermediate responses — which an agent driving those tools itself would have paid for in full — never appear. Nothing measured that.
Why it could not be measured
Three separate gaps, each fixed here:
0,0byte lengths. In the activity log0means UNKNOWN, not free — the gap bench already namesReasonSubCallZeroBytes.code_executionparent was unaccountable too. Fixed centrally in the emitter, which also closesReasonInternalNoByteCountsforretrieve_toolsanddescribe_tool.requestCostfell through to the byte estimate while siblings were measured. Colliding bases withheld the whole script — and most fleets have several parameterless tools. Found on real traffic, not in unit tests.What it measures
Over 7 scripts / 63 sub-calls on a live 4-server fleet:
The saving is a function of how much the sandbox discards. A script that discards nothing loses tokens, and that negative is reported rather than floored.
Accounting rules
Each pinned by a test that fails when the guard is removed:
0would read as "saved nothing" when the truth is "nothing was measurable".Calls.baselinemeans what an agent would have paid, and that agent receives a cut response. Counting it overstates the baseline and inflates the saving.Also
bench/cmd/breakevencurvecomputes fleet-size break-even by bootstrapping random orderings of the 527-tool corpus — definition sizes span 145–6,485 bytes, so a single ordering is one arbitrary curve, not an answer. Median crossing: 67 tools / 10 servers (retrieve_tools), 61 tools (code_execution).Corrects
specs/103-token-bench/findings/webui-savings-vs-bench.md: both figures in the +68.4% / −32.5% contradiction were wrong. The Web-UI number is non-deterministic (map-order sampling — 52.35%–78.80% over an unchanging fleet), and bench's own proxy menu was overstated becauseProxyModeToolDefspins code execution on. The sign of the finding survives; the magnitudes did not.Adds
specs/103-token-bench/findings/honest-savings-design.md— a design for replacing the single dashboard percentage with a measured session ledger.