Skip to content

[Phase 2] Pilot MCP-enriched review on one downstream repo and measure quality + latency #681

Closed as not planned

Description

@github-actions

Story

As a platform owner,
I want enable a starter MCP server on a single downstream repo, run real reviews, and record a short before/after note on review quality and latency,
so that an evidence-based go/no-go before any broader rollout.

Acceptance Criteria

  1. One downstream repo is selected and MCP review enabled there using the story-1 knob plus a starter server (GitHub MCP secret scanning per the MCP-Powered CI Review Enhancement — Standard #650 weekly-update recommendation, or Context7 as the no-auth fallback).
  2. A small sample of PRs is reviewed with MCP on, and results are captured (framework-aware findings, deprecated-API / secret / dependency catches that surfaced).
  3. The latency delta versus MCP-off is recorded (the idea claims marginal/ms-range; verify against real runs).
  4. A short pilot note records findings and a rollout recommendation; broader rollout remains a separate, explicit human decision.

Tasks / Subtasks

Dev Notes

  • Depends on plumbing (story 1), graceful degradation (story 2), the enablement path (story 3), and the per-stack docs/server guidance (story 4).
  • PREREQUISITE per MCP-Powered CI Review Enhancement — Standard #650: in-flight security-hardening (action allowlisting, input sanitization) must land FIRST. That work is not tracked as a repo issue in the planning context (referenced_issues is empty), so it is an external gate — see open_questions — not a blocked_by edge. Do not start the pilot before it lands.
  • Starter-server choice: the 2026-06-13 weekly update revises the recommended first step to GitHub MCP secret/dependency scanning (GA/preview, no extra auth, security-aligned) over Context7; Context7 remains the zero-auth fallback.
  • Review-quality/latency signal already exists in this repo via scripts/pr_review_health.sh and the daily health workflow — reuse it rather than building a new measurement harness.
  • Operational/human-gated: choosing the repo, enabling MCP on an external repo, and the rollout decision are human calls — hence hands_off.

Project Structure Notes

Primarily operational; any artifact is the pilot note (in the story-4 doc or an issue comment). Reuses existing review-health tooling for measurement.

References

Likely target surface

  • docs/initiatives/
  • scripts/pr_review_health.sh

Story prepared by the BMAD Scrum Master (Bob) for epic #676. Status: ready-for-dev.

Activity

  1. added
    initiativeEpic / initiative tracking issue
    initiative:holdInitiative driver must not release this issue (human hold)
    dev-lead:hands-offExclude from dev-lead agent automation
    on Jun 14, 2026
  2. don-petry commented on Jun 20, 2026

    @don-petry
    Collaborator

    Pilot note — closing (outcome: tool reachable, starter server blocked by plan)

    This records the Phase-2 pilot result and the go/no-go, per AC #4.

    What was validated ✅

    What blocked the starter server ⛔

    • The recommended starter — GitHub MCP secret scanning (run_secret_scanning, per the 2026-06-13 weekly-update revised sequencing) — is a content-scanning API that gates on the repo's advanced_security flag (full GitHub Advanced Security).
    • That flag is distinct from secret-scanning / push-protection (the GitHub Secret Protection product). Secret scanning + push protection are enabled and working on this repo (confirmed by a test PR), but the advanced_security flag is not available on the current plan for this private repo.
    • The tool returns Repository does not have GitHub Advanced Security enabled — identically from two independent tokens (the session token and the workflow's donpetry-bot token), confirming this is a repo/plan feature-flag gate, not an auth or call-shape problem. The call type and params are correct; there is no alternate GitHub API for this capability.

    Latency (AC #3)

    • No meaningful delta to report for the secret-scan tool, since the gated API never executes the scan. MCP connection/handshake overhead in the runs was negligible (sub-second), consistent with the "marginal/ms-range" expectation, but a real before/after for useful work needs a starter server that actually runs.

    Interim secret coverage

    • The repo's Secret scan (gitleaks) CI check covers secrets today, plan-independent — so closing this pilot does not leave a coverage gap.

    Recommendation / go-no-go

    • No-go on the GitHub-MCP-secret-scanning starter on this plan; do not pursue advanced_security purchase solely for the pilot.
    • Pivot to the zero-auth fallback already named in AC test issue from agent #1: Context7, implemented as an MCP server inside the action agent runtime (not just an interactive session, so the agent actually gains the capability). Tracked as a new work item: see the follow-up issue linked below. Broader rollout remains a separate, explicit human decision.

    Closing as not planned for the original starter; the capability work continues under the follow-up.

    Refs: epic #676, #809 (env-mapping fix), discussion #650 (revised sequencing), PR #774 (pilot subject).


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-lead:hands-offExclude from dev-lead agent automationinitiativeEpic / initiative tracking issueinitiative:holdInitiative driver must not release this issue (human hold)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions