Repository navigation
feat(bench): payload decomposition — where the tokens actually go (US4) - #1147
Merged
Merged
Conversation
…50-T052) Spec 103 US4. Attributes a corpus's rendered payload to name, description, schema and structural overhead, and recomputes the achievable ceiling FOR THAT CORPUS rather than carrying one forward — the error spec 102 made when it projected ~70% from an assumption about one corpus shape. TOKENS ARE NOT ADDITIVE ACROSS CONCATENATION, so the shares cannot be measured by counting components separately: BPE merges across boundaries, and Count(name) + Count(description) + Count(schema) does not equal Count of the concatenation. Independent counts would produce shares that look plausible, never sum, and misattribute every boundary. The payload is therefore tokenized ONCE and each token attributed by byte offset, reusing the span partition AttributeTokens already implements for response cost. T050 asserts the shares partition the payload exactly. RUNNING IT AGAINST THE REAL CORPORA EXPOSED TWO DEFECTS IN MY OWN VERDICT LOGIC, both fixed here, and both of the same kind — a confident statement about a corpus that does not support it: 1. The 527-tool snapshot carries ZERO schemas. The first version reported "deferring schemas cannot exceed 0.0% however it is implemented", which reads as a decisive confirmation of spec 102 and is really a corpus that does not contain the thing under test. Now INAPPLICABLE, with ToolsWithSchema on the struct so the distinction cannot collapse again. 2. The verdict claimed to CONFIRM or CORRECT spec 102 from corpus evidence. Spec 102 measured the production WIRE payload — annotations, mcp.Tool marshalling, the "[server] " prefix; this decomposition covers the corpus rendering, which has none of those. Adjudicating across that gap is answering with evidence about something else, and the code already said so in its Limitation field while the verdict ignored it. Verdicts are now scoped to the corpus and explicitly decline to settle spec 102's figure. THE MEASUREMENT ITSELF IS INTERESTING. On the 45-tool corpus_v2 the payload is 59.4% schema, 36.5% description, 2.4% name, 1.7% structural — schemas are the DOMINANT term in the corpus rendering, which is the opposite of what spec 102 concluded about the wire payload. That does not overturn spec 102; it localises the disagreement to the wire framing and annotations, which is a far more useful place to look than "schemas are cheap". T053 (emitting the block into the report) is deliberately NOT ticked — the decomposition computes, but nothing wires it into ReportV2 yet. go test ./bench/... green; golangci-lint v2 0 issues.
…s null, not zero Completes T053. Spec 103 US4 is done: 37/64. The block type already existed with ShareAnnotationsPct as a plain float64, and the JSON schema marked it required. Both were wrong for the only source that can populate this block today: the frozen corpora carry no annotations field, so the share is NOT MEASURABLE, and emitting 0 would claim a measurement that was never taken — the silent-zero failure the rest of this report works to avoid. So the field is now a pointer that marshals to null, and the schema types it ["number","null"] with a comment explaining that null and zero differ here. A future source that CAN measure annotations — the production wire payload, which is what spec 102 measured — populates it without a schema change. PayloadDecompositionBlockFor refuses fewer than two fleet shapes. FR-024 asks for at least two so the attribution can be seen to MOVE with fleet size, and one shape is exactly the evidence base spec 102 projected from; making that unemittable is the point of the requirement. Two existing reportv2_test fixtures were updated to pointer values rather than having the field dropped: they model a future annotation-capable source, and keeping real values there preserves what they were testing. Gates: go test ./bench/... green, -race green, golangci-lint v2 0 issues, all three schema-validation tests PASS (confirmed not skipped).
Deploying mcpproxy-docs with
|
| Latest commit: |
6608d2e
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://24e858fd.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://103-fleet-and-decomp.mcpproxy-docs.pages.dev |
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 33410171096 --repo smart-mcp-proxy/mcpproxy-go
|
Dumbris
added a commit
that referenced
this pull request
Sep 2, 2026
… token-bench truth-sync (#1200) * chore(roadmap): raise mcp-2026-upgrade to P1 (mcp-go v1.0.0 stable), add activity-storage-bounds, sync token-bench - mcp-2026-upgrade: P3 -> P1. mark3labs/mcp-go v1.0.0 stable shipped 2026-09-02; go.mod still pins v0.57.0. Next step is speckit.plan (#532). - action-log-transparency: new task activity-storage-bounds tracking #1173/#1174 (response truncation never wired), #1176 (unbounded per-server tool_calls buckets) and #1175 (omitempty zero erasure). - token-bench-harness: in_progress -> done; #1141/#1147/#1151/#1153/#1160 are all merged. Clears both --check-github errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(roadmap): keep merged token-bench PRs on the done task, not the in_progress epic Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spec 103 US4 complete (T050–T053). 37/64 tasks.
Attributes a corpus's rendered payload to name / description / schema / structural overhead, and recomputes the achievable ceiling for that corpus rather than carrying one forward — the error spec 102 made when it projected ~70% from an assumption about one corpus shape.
The measurement result
On the 45-tool
corpus_v2:Schemas are the dominant term in the corpus rendering — the opposite of what spec 102 concluded about the wire payload. That does not overturn spec 102. It localises the disagreement to the wire framing and annotations, which is a far more useful place to look than "schemas are cheap."
Why the shares are attributed, not counted
Tokens are not additive across concatenation. BPE merges across component boundaries, so
Count(name) + Count(description) + Count(schema)≠Countof the concatenation. Counting separately yields shares that look plausible, never sum, and misattribute every boundary.So the payload is tokenized once and each token attributed by byte offset, reusing the span partition
AttributeTokensalready implements for response cost. T050 asserts the shares partition the payload exactly.Two defects running it against real corpora exposed — in my own verdict logic
Both were the same kind: a confident statement about a corpus that doesn't support it.
The 527-tool snapshot carries zero schemas. The first version reported "deferring schemas cannot exceed 0.0% however it is implemented" — which reads as a decisive confirmation of spec 102, and is really a corpus that does not contain the thing under test. Now
INAPPLICABLE, withToolsWithSchemaon the struct so the distinction can't collapse again.The verdict claimed to CONFIRM or CORRECT spec 102 from corpus evidence. Spec 102 measured the production wire payload — annotations,
mcp.Toolmarshalling, the[server]prefix. This decomposition covers the corpus rendering, which has none of those. The code already said so in itsLimitationfield while the verdict ignored it. Verdicts are now scoped to the corpus and explicitly decline to settle spec 102's figure.Null is not zero
ShareAnnotationsPctwas a plainfloat64, schema-required. Both were wrong for the only source that can populate this block today: the corpora carry no annotations field, so the share is not measurable, and emitting0would claim a measurement never taken.It's now a pointer marshalling to
null, with the schema typed["number","null"]. A future annotation-capable source — the wire payload — populates it with no schema change.PayloadDecompositionBlockForalso refuses fewer than two fleet shapes: FR-024 wants the attribution to be seen moving with fleet size, and one shape is exactly the evidence base spec 102 projected from.Verification