From 86867be34b78b9c95852e82c8e9a279d03cb87e4 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Wed, 2 Sep 2026 19:34:01 +0300 Subject: [PATCH 1/2] chore(roadmap): raise mcp-2026-upgrade to P1 (mcp-go v1.0.0 stable), add activity-storage-bounds, sync token-bench - mcp-2026-upgrade: P3 -> P1. mark3labs/mcp-go v1.0.0 stable shipped 2026-09-02; go.mod still pins v0.57.0. Next step is speckit.plan (#532). - action-log-transparency: new task activity-storage-bounds tracking #1173/#1174 (response truncation never wired), #1176 (unbounded per-server tool_calls buckets) and #1175 (omitempty zero erasure). - token-bench-harness: in_progress -> done; #1141/#1147/#1151/#1153/#1160 are all merged. Clears both --check-github errors. Co-Authored-By: Claude Fable 5.1 --- ROADMAP.md | 36 ++++++++++++++++++++---------------- roadmap.yaml | 18 +++++++++++++----- 2 files changed, 33 insertions(+), 21 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index f65258bd7..65ef400a5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -74,13 +74,13 @@ graph LR - πŸ”΅ **Release qualification gate (auto-QA matrix blocks the tag)** β€” In progress Β· P0 - πŸ”΅ **Planning/docs truth automation** β€” In progress Β· P2 - πŸ”΅ **Discovery-quality eval harness (Spec 065 second half)** β€” In progress Β· P3 +- βšͺ **MCP protocol upgrade to 2026-07-28 revision** β€” Todo Β· P1 - βšͺ **Windows native tray app** β€” Todo Β· P2 - ⚫ **Server marketplace** β€” Todo Β· P3 Β· parked - ⚫ **Audit SIEM integration** β€” Todo Β· P3 Β· parked - ⚫ **Paid-tier MVP (billing / seats / license)** β€” Todo Β· P3 Β· parked - ⚫ **SDK v1 migration** β€” Todo Β· P3 Β· parked - ⚫ **SSO (server edition)** β€” Todo Β· P3 Β· parked -- βšͺ **MCP protocol upgrade to 2026-07-28 revision** β€” Todo Β· P3 - βšͺ **Security gateway Tracks C/D (per-arg least-privilege + signature provenance)** β€” Todo Β· P3 - 🟒 **Upgrade awareness & guided update** β€” Done Β· P0 - 🟒 **Connect step trust: preview, visible backup, one-click undo** β€” Done Β· P0 @@ -178,13 +178,16 @@ graph LR action_log_tray_menu["Activity in the tray menu (recent tool calls…"] tray_menu_open_telemetry["tray_menu_opened counter: Swift menuWillOpen…"] action_log_retention_tie_in["Tie activity retention/size into the glance v…"] + activity_storage_bounds["Bound every activity-adjacent store: response…"] action_log_glance_view --> action_log_tray_menu action_log_glance_view --> action_log_retention_tie_in classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; + classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff; classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; class sessions_web_ui done; + class activity_storage_bounds in_progress; class action_log_glance_view,action_log_tray_menu,tray_menu_open_telemetry,action_log_retention_tie_in todo; ``` @@ -195,6 +198,7 @@ graph LR | Activity in the tray menu (recent tool calls + security events, jump to full log) | βšͺ Todo | β€” | | tray_menu_opened counter: Swift menuWillOpen (MCPProxyApp.swift:192) -> lightweight POST /api/v1/telemetry/tray-menu-opened -> registry counter -> heartbeat tray_menu_opened_24h | βšͺ Todo | β€” | | Tie activity retention/size into the glance view | βšͺ Todo | β€” | +| Bound every activity-adjacent store: response truncation on the write path (#1173/#1174), per-server tool_calls buckets (#1176), omitempty zero-erasure (#1175) | πŸ”΅ In progress | #1174 | @@ -203,7 +207,7 @@ graph LR > Measure the real token cost of every routing/savings mode combination β€” baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) β€” on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product. -Spec: [103-token-bench](./specs/103-token-bench/) Β· PR: #1141 +Spec: [103-token-bench](./specs/103-token-bench/) Β· PR: #1141 #1147 #1151 #1153 #1160 ```mermaid graph LR @@ -215,15 +219,15 @@ graph LR token_bench_harness --> token_bench_public token_bench_public --> token_bench_blog - classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff; + classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; - class token_bench_harness in_progress; + class token_bench_harness done; class token_bench_public,token_bench_blog,token_bench_telemetry todo; ``` | Task | Status | Refs | | --- | --- | --- | -| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | πŸ”΅ In progress | #1141 | +| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | 🟒 Done | #1141 #1147 #1151 #1153 #1160 | | Run public suites locally (Ο„-bench / BFCL / MCP-specific β€” final list verified by a research pass) and record reproducible results | βšͺ Todo | β€” | | Publish results + methodology on mcpproxy.app/blog | βšͺ Todo | β€” | | Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation | βšͺ Todo | β€” | @@ -381,6 +385,15 @@ graph LR +
+βšͺ MCP protocol upgrade to 2026-07-28 revision β€” Todo Β· P1 + +> STABLE GATE CLEARED 2026-09-02: mark3labs/mcp-go v1.0.0 (stable) released; go.mod still pins v0.57.0. Raised P3->P1 on the 2026-09-02 issue-prioritization pass β€” spec 058 has spec.md only, next step is speckit.plan then tasks (tracker #532). Earlier: UNBLOCKED 2026-08-12: the mcp-go gate cleared β€” v1.0.0-beta.1 (mark3labs/mcp-go#951) ships full 2026-07-28 support with per-request era detection (the pin was v0.55.x, topping out at 2025-11-25). Spec 058 revision MERGED as PR #1033 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that β€” implementation has not started, hence todo): final error-code renumbering (-32020/-32021/-32022), FR-001..006 / FR-014..016 recast as adopt-and-verify, FR-028 legacy-only transport pin as the safe merge state, plus Risks & Watch Items. CROSS-SPEC CONFLICT still open and now the mandatory first task: FR-012 forbids per-connection */list variation; SHIPPED Spec 057 selects toolset by URL path /mcp/p/ β€” Option A/B decided at plan time (acceptance = the 2 tests failing under beta.1). 028 agent-token scoping is already compatible (header-carried). Tracker: #532. + +Spec: [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/) + +
+
βšͺ Windows native tray app β€” Todo Β· P2 Β· MCP-43 @@ -464,15 +477,6 @@ graph LR
-
-βšͺ MCP protocol upgrade to 2026-07-28 revision β€” Todo Β· P3 - -> UNBLOCKED 2026-08-12: the mcp-go gate cleared β€” v1.0.0-beta.1 (mark3labs/mcp-go#951) ships full 2026-07-28 support with per-request era detection (the pin was v0.55.x, topping out at 2025-11-25). Spec 058 revision MERGED as PR #1033 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that β€” implementation has not started, hence todo): final error-code renumbering (-32020/-32021/-32022), FR-001..006 / FR-014..016 recast as adopt-and-verify, FR-028 legacy-only transport pin as the safe merge state, plus Risks & Watch Items. CROSS-SPEC CONFLICT still open and now the mandatory first task: FR-012 forbids per-connection */list variation; SHIPPED Spec 057 selects toolset by URL path /mcp/p/ β€” Option A/B decided at plan time (acceptance = the 2 tests failing under beta.1). 028 agent-token scoping is already compatible (header-carried). Tracker: #532. - -Spec: [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/) - -
-
βšͺ Security gateway Tracks C/D (per-arg least-privilege + signature provenance) β€” Todo Β· P3 @@ -772,16 +776,16 @@ graph LR | Web UI + macOS app UX audit | In progress | P0 | β€” | | | | Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | β€” | [081-release-qa-gate](./specs/081-release-qa-gate/) | | | Action log / transparency β€” info at a glance | In progress | P1 | β€” | | | -| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | #1141 | +| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | #1141 #1147 #1151 #1153 #1160 | | Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | β€” | | | | Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | β€” | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | | | Planning/docs truth automation | In progress | P2 | β€” | | | | Discovery-quality eval harness (Spec 065 second half) | In progress | P3 | β€” | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | | | tpa-db: versioned TPA signature database for the offline scanner | Todo | P1 | β€” | [101-tpa-db](./specs/101-tpa-db/) | | +| MCP protocol upgrade to 2026-07-28 revision | Todo | P1 | β€” | [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/) | | | Windows native tray app `MCP-43` | Todo | P2 | β€” | | | | Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | β€” | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | | | Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | β€” | | | -| MCP protocol upgrade to 2026-07-28 revision | Todo | P3 | β€” | [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/) | | | Security gateway Tracks C/D (per-arg least-privilege + signature provenance) | Todo | P3 | β€” | [054-mcp-security-gateway](./specs/054-mcp-security-gateway/) | | | Server marketplace `MCP-37` | Todo (parked) | P3 | β€” | | | | Audit SIEM integration `MCP-39` | Todo (parked) | P3 | β€” | | | diff --git a/roadmap.yaml b/roadmap.yaml index cd755268a..85dfec4c3 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -253,6 +253,13 @@ epics: status: todo note: "No spec: link on purpose β€” 073-activity-size-retention is the SHIPPED retention backend (13/14); this task is the glance-view tie-in on top of it." depends_on: [action-log-glance-view] + - id: activity-storage-bounds + title: "Bound every activity-adjacent store: response truncation on the write path (#1173/#1174), per-server tool_calls buckets (#1176), omitempty zero-erasure (#1175)" + status: in_progress + priority: P1 + pr: "#1174" + note: "User report 2026-09-02 (#1173): config.db reached 940MB; activity_max_response_size existed but was never read, and server__tool_calls buckets have no size, count or age bound at all (the other ~432MB). #1174 (external contributor, CI green) wires the 64KB cap; #1176 and #1175 are the follow-ons. Retention backend (073) prunes activity records only, not the per-server buckets." + depends_on: [] - id: analytics-dashboard title: Analytics dashboard as default page @@ -392,14 +399,15 @@ epics: status: in_progress priority: P1 spec: specs/103-token-bench - pr: "#1141" + pr: ["#1141", "#1147", "#1151", "#1153", "#1160"] depends_on: [schema-deferred] note: "Measure the real token cost of every routing/savings mode combination β€” baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) β€” on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product." tasks: - id: token-bench-harness title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries" - status: in_progress - pr: "#1141" + status: done + pr: ["#1141", "#1147", "#1151", "#1153", "#1160"] + note: "Shipped 2026-08-31..09-01: #1141 replay MVP, #1147 payload decomposition (US4), #1151 tools/list primitive + live fleet source, #1153 live agent loop + reproducibility + publication, #1160 code-execution response-side saving. Epic stays in_progress: public suites, blog post and heartbeat-v10 counters remain todo." depends_on: [] note: "BUILDS ON the existing bench/ harness - do NOT rebuild token counting, encoding arms, break-even or recall. Already shipped there (PRs #747/#748 MCP-42, #851 spec 083): six encoding arms including direct_deferred (spec 102), cl100k_base counting, live response cost, break-even, RecallAtK/NDCGAtK, the v2 report + dashboard, `make bench-discovery` and CI wiring in bench.yml/eval.yml. 2026-08-31 audit correction: the old note claimed this task absorbs discovery-eval-harness because it would measure retrieval recall - recall is ALREADY measured (eval.yml retrieval-d1 + bench/armindex.go), so that is not this task's contribution. GENUINELY OPEN and unique here: (a) activity-log session REPLAY (nothing in bench/ reads the activity log; internal/contracts/activity.go already carries WorkSessionID, ToolName and Response - watch ResponseTruncated, which would undercount tokens); (b) MEASURED first-call success and retry rate β€” today they are assumed, not observed: bench/session.go defines per-arm literature-derived defaults (armRetryRates, research D8) and bench/report.go only renders and labels them as estimates; (c) tokens-per-SUCCESSFUL-task, which needs a task-completion signal the static harness has no concept of." - id: token-bench-public @@ -715,10 +723,10 @@ epics: - id: mcp-2026-upgrade title: MCP protocol upgrade to 2026-07-28 revision status: todo - priority: P3 + priority: P1 spec: specs/058-mcp-2026-upgrade depends_on: [] - note: "UNBLOCKED 2026-08-12: the mcp-go gate cleared β€” v1.0.0-beta.1 (mark3labs/mcp-go#951) ships full 2026-07-28 support with per-request era detection (the pin was v0.55.x, topping out at 2025-11-25). Spec 058 revision MERGED as PR #1033 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that β€” implementation has not started, hence todo): final error-code renumbering (-32020/-32021/-32022), FR-001..006 / FR-014..016 recast as adopt-and-verify, FR-028 legacy-only transport pin as the safe merge state, plus Risks & Watch Items. CROSS-SPEC CONFLICT still open and now the mandatory first task: FR-012 forbids per-connection */list variation; SHIPPED Spec 057 selects toolset by URL path /mcp/p/ β€” Option A/B decided at plan time (acceptance = the 2 tests failing under beta.1). 028 agent-token scoping is already compatible (header-carried). Tracker: #532." + note: "STABLE GATE CLEARED 2026-09-02: mark3labs/mcp-go v1.0.0 (stable) released; go.mod still pins v0.57.0. Raised P3->P1 on the 2026-09-02 issue-prioritization pass β€” spec 058 has spec.md only, next step is speckit.plan then tasks (tracker #532). Earlier: UNBLOCKED 2026-08-12: the mcp-go gate cleared β€” v1.0.0-beta.1 (mark3labs/mcp-go#951) ships full 2026-07-28 support with per-request era detection (the pin was v0.55.x, topping out at 2025-11-25). Spec 058 revision MERGED as PR #1033 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that β€” implementation has not started, hence todo): final error-code renumbering (-32020/-32021/-32022), FR-001..006 / FR-014..016 recast as adopt-and-verify, FR-028 legacy-only transport pin as the safe merge state, plus Risks & Watch Items. CROSS-SPEC CONFLICT still open and now the mandatory first task: FR-012 forbids per-connection */list variation; SHIPPED Spec 057 selects toolset by URL path /mcp/p/ β€” Option A/B decided at plan time (acceptance = the 2 tests failing under beta.1). 028 agent-token scoping is already compatible (header-carried). Tracker: #532." - id: security-gateway-cd title: Security gateway Tracks C/D (per-arg least-privilege + signature provenance) status: todo From 76f0cd90d2e1d55d5cff3763166fc836f3cd10d1 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Wed, 2 Sep 2026 19:35:54 +0300 Subject: [PATCH 2/2] chore(roadmap): keep merged token-bench PRs on the done task, not the in_progress epic Co-Authored-By: Claude Fable 5.1 --- ROADMAP.md | 4 ++-- roadmap.yaml | 1 - 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 65ef400a5..762f41573 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -207,7 +207,7 @@ graph LR > Measure the real token cost of every routing/savings mode combination β€” baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) β€” on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product. -Spec: [103-token-bench](./specs/103-token-bench/) Β· PR: #1141 #1147 #1151 #1153 #1160 +Spec: [103-token-bench](./specs/103-token-bench/) ```mermaid graph LR @@ -776,7 +776,7 @@ graph LR | Web UI + macOS app UX audit | In progress | P0 | β€” | | | | Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | β€” | [081-release-qa-gate](./specs/081-release-qa-gate/) | | | Action log / transparency β€” info at a glance | In progress | P1 | β€” | | | -| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | #1141 #1147 #1151 #1153 #1160 | +| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | | | Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | β€” | | | | Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | β€” | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | | | Planning/docs truth automation | In progress | P2 | β€” | | | diff --git a/roadmap.yaml b/roadmap.yaml index 85dfec4c3..9f53b98c3 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -399,7 +399,6 @@ epics: status: in_progress priority: P1 spec: specs/103-token-bench - pr: ["#1141", "#1147", "#1151", "#1153", "#1160"] depends_on: [schema-deferred] note: "Measure the real token cost of every routing/savings mode combination β€” baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) β€” on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product." tasks: