Repository navigation
chore(roadmap): truth-sync statuses against code, clear all --check-github warnings - #1136
Merged
Merged
Conversation
…ithub warnings An evidence-based re-derivation of every non-done roadmap.yaml leaf. Each flip below was verified against the tree, git/gh state and the rendering sites, then independently re-checked by an adversarial pass; claims that did not survive that pass were dropped rather than applied. Statuses that were wrong in the "already shipped" direction: - upgrade-nudge-delta todo -> done (#1085, closes #1081). The note asserted TODO(spec-079/FR-002) markers exist and that the checker calls only /releases/latest; both are false. internal/updatecheck/delta.go now computes the delta and all six surfaces render it. - upgrade-nudge epic in_progress -> done (all five tasks now done). - analytics-dashboard in_progress -> done. Both children shipped; the default-landing half is live (router '/' -> Dashboard, guarded by dashboard-default-landing.spec.ts). Spec 069's one open task is a local Playwright sweep that leaves no artifact and can never tick from code. - discovery-eval-harness todo -> done. Both halves ship: cmd/scan-eval for security recall/FP, and the eval.yml retrieval-d1 gate plus bench/'s own RecallAtK/NDCGAtK + recall@5 parity gate for discovery quality. - hygiene-spec-gardener in_progress -> done (#824, #870, #999, #842). Its recorded blocker is cleared: check-spec-evidence.py is on main. Statuses that were wrong in the "claimed progress it does not have" direction: - windows-tray and windows-tray-window in_review -> todo. No Windows-tray PR is open or merged and native/windows/ is a README placeholder, so "in review" pointed at nothing. The section header is renamed to match. - planning-hygiene todo -> in_progress and action-log-transparency todo -> in_progress; both already have done children. Notes corrected where they would misdirect the next implementer: - token-bench-harness claimed it absorbs discovery-eval-harness by measuring retrieval recall. Recall is already measured, and bench/ already covers token counting, encoding arms (incl. direct_deferred), break-even and reporting. The note now records what is genuinely open: activity-log session replay, measured first-call success/retry rate, and tokens-per-successful-task. - token-bench-public: the MCP-specific third already ships via spec 083 (ToolRet, LiveMCPBench snapshot, lap-score); only agent-loop suites remain. specs/098-tools-preflight/tasks.md: tick T001, T024, T031, each confirmed against the tree (doc.go states both invariants T001 requires; toolslist_snapshot_test.go is labelled T024 and green; README carries all three FR-017 examples). T025/T027/T032/T033 are left unchecked — they are one-time process acts or did not survive verification. Deliberately NOT changed: no epics were added for shipped specs (084-100). roadmap.yaml is the working set by design and ROADMAP.md already renders a generated per-spec progress table, so absence of an epic is not drift. scripts/gen-roadmap.py --check-github: 3 warnings -> 0 errors, 0 warnings.
Deploying mcpproxy-docs with
|
| Latest commit: |
492d2d2
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://7479e16b.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://claude-next-roadmap-priority.mcpproxy-docs.pages.dev |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 33361704123 --repo smart-mcp-proxy/mcpproxy-go
|
All six findings from the opencode gpt-5.6-sol review were independently re-verified against the tree before being applied; none were false positives. (The reviewer's initial claim that spec-gardener-auto-approve.yml is missing WAS a false positive from its glob tool and it self-corrected — the file exists at .github/workflows/spec-gardener-auto-approve.yml.) 1. Un-tick specs/098 T024. T024 asks for a golden snapshot test proving tools/list is BYTE-IDENTICAL across all three modes. Spec 099 (FR-014/015) later converted that test from a no-delta gate into an ENUMERATED-delta gate (toolsListAllowedDelta permits describe_tool and quarantine_security to move), so at HEAD the test no longer demonstrates the property T024 states. It may well have held on 098's own branch, but that is not checkable from the tree, and an unverifiable tick is exactly what this PR exists to remove. Badge 29/33 -> 28/33. 2. discovery-eval-harness note: drop the "SUPERSEDED — folded into token-bench-harness" framing. It contradicted itself once the epic was marked done — token-bench-harness is still unbuilt, so nothing could have been folded into it. Both halves shipped independently. 3. Same note: the eval.yml retrieval-d1 job is REPORT-ONLY on pull requests (continue-on-error, guarding against npx/uvx fetch flake) and blocks only on the nightly schedule. Promoting it to PR-blocking is still open (MCP-742). 4. Same note: the SC-003 recall@5 = 0.68 +/- 0.05 parity gate is asserted in bench/armindex_test.go, not armindex.go — the latter supplies the production index wiring, not the assertion. 5. token-bench-harness note: the literature-derived retry defaults live in bench/session.go (armRetryRates, research D8). bench/report.go only renders and labels them as estimates. The substantive point stands: retry rates are assumed, not observed. 6. token-bench-public note: "CI-wired" overstated the LiveMCPBench snapshot. bench.yml's regular job passes only -corpus-v2; the snapshot is runnable via -livemcptool but not run in CI, and its loader has no retrieval golden set, so it measures token/scale rather than recall. ToolRet does have its own bench.yml job. gen-roadmap.py --check and --check-github both still clean.
Both findings verified against the tree before applying. 1. The retrieval-D1 blocking scope was stated too narrowly. eval.yml sets continue-on-error only for github.event_name == 'pull_request', and the workflow triggers on pull_request, workflow_dispatch and schedule — so the job is report-only on PRs and BLOCKING on both nightly and manual dispatch runs, not "only on the nightly schedule". The note now says so, and records that the workflow's own inline comment carries the same omission. 2. discovery-eval-harness is now done but still declared depends_on: [token-bench-harness], which is unbuilt — a done node cannot depend on an open one, and the edge contradicted the corrected note saying both halves shipped independently. Edge removed; the id remains a valid depends_on target for anything that needs it. gen-roadmap.py --check and --check-github both clean.
…und 3 All four verified against GitHub and the tree before applying. 1. windows-tray: "no Windows-tray PR is open or merged" was false. #74 ("feature: windows tray support") merged 2025-10-23 and the cross-platform Go/systray build still ships on Windows. The claim now scopes to what the epic actually tracks — the native WebView2 replacement — for which no PR exists and native/windows/ is a README placeholder. 2. analytics-dashboard: "can never tick from code" conflated two things. A human can run spec 069's T023 Playwright sweep and tick the box; what can never exist is code evidence for it. Reworded so the distinction is explicit. 3. hygiene-spec-gardener: "three gardener runs ... auto-approved by #842" was wrong. Only #999 carries a github-actions APPROVED review. #824 has none and merged ~24 minutes BEFORE the auto-approve workflow landed; #870 records no approving review either. 4. token-bench-public: calling the MCP-specific third "integrated" overstated completion. ToolRet and LiveMCPTool are static retrieval/token corpora and lap-score is a linter — none is a public agent-loop task suite, which is what the task ultimately owes. Also noted for the record: the linux-arm64 Build Binaries failure on the previous commit is a flaky test, not a regression from this docs-only change. TestServiceStartScanDeepOnRunsSourceResolutionAndPass2 (internal/security/scanner) passes locally with -count=3, and two runs of this same branch four minutes apart — 33360429077 and 33360648947, both docs-only — disagreed on it.
Verified: bench/corpusio/toolret.go documents ToolRet as a general tool-retrieval corpus whose documentation fields are "heterogeneous JSON/text, NOT MCP input schemas" (Schema stays nil). Grouping it under "the MCP-specific third" was wrong. The token-bench-public note now says public corpus/linter groundwork is integrated while NO public agent-loop task suite is, and names the ToolRet caveat explicitly so the next implementer does not re-inherit the error. Round 4 found no other issues; the round-3 windows-tray, analytics T023 and gardener corrections were all confirmed accurate against GitHub and the tree.
… round 5) The final review round caught an error in this PR's own flip, and it is the same failure mode the PR was written to fix: an epic marked done while a governing FR is unmet. Spec 065 FR-009 and SC-005 require CI to FAIL on a discovery regression beyond tolerance. The retrieval-D1 job sets continue-on-error for pull_request (D1 depends on npx/uvx fetches, a known flake source), so on the PR path CI does not fail — and eval.yml's own comment records the promotion to PR-blocking as still planned (MCP-742). The HARNESS shipped; the GATE did not. Reverted to in_progress and added the remaining work as a child task (discovery-eval-pr-blocking, MCP-742) so the epic's status has something in the DAG to point at. The reviewer also read CN-002 (frozen corpus, never live) as violated because D1 boots a live mcpproxy serving seven reference servers. Recorded as a weaker tension to adjudicate rather than a violation: #931 pinned all seven upstreams to freeze-era versions and the job gates on the exact corpus ID set, so the corpus is reproducible in practice. Whether that satisfies CN-002 or a committed snapshot is required is a judgement for the spec owner. gen-roadmap.py --check and --check-github both clean (0 errors, 0 warnings).
Dumbris
added a commit
that referenced
this pull request
Aug 31, 2026
…to 10 (#1140) Two follow-ups from the 2026-08-31 roadmap truth-sync and spec-103 work. 1. roadmap.yaml: the token-bench epic now carries spec: specs/103-token-bench, so ROADMAP.md renders its real progress badge (0/64) instead of a blank. The note records what the spec settled, because three of its 13 cross-model review rounds changed the DESIGN rather than the wording and a future reader of the epic should not have to rediscover them: - a recording carries no prompt, conversation or completion oracle, so replay CANNOT show agent behaviour — US1 (deterministic cost) and US2 (live agent loop) are now separate user stories; - replay needs a FLEET INPUT, because the activity export has no fleet snapshot and a menu is a property of the tool definitions; - bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. Also records that the matrix is 5 distinct behaviours, not a 3x2x2 product. 2. CLAUDE.md escalation trigger 4 was doubly stale: it capped cross-model review at 5 rounds and named Codex as the reviewer. The cap is 10 (maintainer decision 2026-08-31) and the reviewer is the opencode CLI with github-copilot/gpt-5.6-sol (maintainer directive 2026-08-14, Codex not to be used). The rationale for the raise is recorded inline: round 5 on #1136 caught a real defect the first four rounds missed — an epic marked done while its spec's gating FR was unmet, which is the exact failure that PR existed to fix. gen-roadmap.py --check and --check-github both clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An evidence-based re-derivation of every non-done
roadmap.yamlleaf. The roadmap had drifted in both directions: items markedtodothat already shipped, and items claimingin_reviewwith no PR behind them.Every flip was verified against the tree, git/
ghstate and the actual rendering sites, then put through five rounds of cross-model review (opencodegpt-5.6-sol). That review produced 14 findings, every one independently re-verified before being applied; one reviewer claim was a false positive (a glob miss onspec-gardener-auto-approve.yml) and was rejected with evidence.Status flips
upgrade-nudge-deltainternal/updatecheck/delta.go+ six rendering surfacesupgrade-nudge(epic)analytics-dashboard/→ Dashboard, guarded bydashboard-default-landing.spec.tshygiene-spec-gardenerwindows-tray,windows-tray-windownative/windows/is a README placeholderplanning-hygiene,action-log-transparencydiscovery-eval-harnessThe
upgrade-nudge-deltanote carried two provably false claims — thatTODO(spec-079/FR-002)markers still exist, and that the checker only calls/releases/latest.grep -rn "spec-079" --include='*.go' internal cmdnow returns nothing.What the review changed about this PR's own conclusions
Round 5 caught this PR committing the exact error it was written to fix.
discovery-eval-harnesswas initially flipped todonebecause both halves of the harness shipped — but spec 065 FR-009/SC-005 require CI to fail on a discovery regression, and the retrieval-D1 job iscontinue-on-erroron pull requests. The harness shipped; the gate did not. Reverted toin_progress, with the remaining work added as a child task (discovery-eval-pr-blocking, MCP-742).Round 1 similarly caught a wrong checkbox tick: T024 asks for a golden snapshot test proving
tools/listis byte-identical across all three modes, but spec 099 later converted that test into an enumerated-delta gate (toolsListAllowedDelta), so the property is no longer demonstrable at HEAD. Un-ticked.Other corrections:
windows-tray's "no Windows-tray PR" was false (#74 merged Windows tray support in 2025 — the epic tracks the native WebView2 replacement); "three gardener runs auto-approved by #842" was false (only #999 has a bot approval, and #824 merged before the workflow existed); ToolRet is not an MCP-specific corpus; the retrieval-D1 gate blocks onworkflow_dispatchtoo, not just nightly; retry defaults live inbench/session.go, notreport.go.Notes corrected where they would misdirect the next implementer
token-bench-harnessclaimed it absorbsdiscovery-eval-harness"because the same harness measures retrieval recall". Recall is already measured, andbench/already covers token counting, encoding arms (includingdirect_deferredfrom spec 102), break-even and reporting. The note now records what is genuinely open: activity-log session replay, measured first-call success/retry rate, and tokens-per-successful-task.token-bench-public: public corpus/linter groundwork ships, but no public agent-loop task suite does — which is what the task actually owes.specs/098-tools-preflight/tasks.md
Ticked T001 and T031, each confirmed against the tree (
doc.gostates both invariants T001 requires; README carries all three FR-017 examples). Badge 26/33 → 28/33. T024/T025/T027/T032/T033 left unchecked — one-time process acts, or claims that did not survive verification.Deliberately not changed
No epics were added for shipped specs 084–100.
roadmap.yamlis the working set by design, andROADMAP.mdalready renders a generated per-spec progress table — so absence of an epic is not drift.Verification
One CI flake was observed and is not caused by this docs-only change:
TestServiceStartScanDeepOnRunsSourceResolutionAndPass2(internal/security/scanner) failed once on linux-arm64 but passes locally with-count=3, and two runs of this same branch four minutes apart — both docs-only — disagreed on it. Cleared on re-run.