You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Initiative: Enhance the PR-review pipeline into a world-class bug hunter — semantic symbol context, issue-type specialist routing, agentic validation, and merged-PR few-shot, eval-gated and dry-run-verified #1088
Our PR-review cascade (scripts/engine.sh triage → deep → audit → action, with the cross-engine rubber-duck pass) already implements several patterns top commercial reviewers use, but it under-delivers on bug-finding depth. Per the sharpened idea (Discussion #906) and a benchmark against CodeRabbit, Greptile, baz.co, and Qt, the biggest gaps are:
Semantic symbol context — the deep tier reviews diff hunks without callers/callees/type-defs for touched symbols (Greptile's core differentiator).
Issue-type routing — one monolithic prompts/deep-review.md handles everything instead of dispatching a specialist prompt per type (security / logic / performance / style).
Agentic validation — suspected bugs are reported without running lint/tests to confirm them, inflating false positives.
Merged-PR few-shot — no learning from this repo's past review→merge histories.
Approach
A prioritized audit first (Phase 1) grounds the roadmap and freezes an immutable baseline; then the four gap-closing enhancements land as independent, PR-sized units (Phase 2/3); then a single Phase-4 convergence gate proves the whole thing on the held-out eval + a merged-PR dry-run before any LIVE_MODE flip. Every prompt/quality change is scored only against the held-out eval split and few-shot is drawn only from the dev split, so we never teach to the test.
This reuses, and must not duplicate, existing in-flight work: the LSP pilot (#839), MCP review enrichment (#676, docs/initiatives/mcp-powered-review.md), the context-adaptive review agent / rubric registry (#610), and the eval-gated self-improving-skills infra (#581). Coordination is called out per story.
Success metric (initiative-level)
On the frozen baseline captured in Phase 1: the deep-review held-out eval (scripts/evals/run-eval.sh deep-review, llm-judge, pass_threshold 0.7) aggregate score strictly improves over the baseline, and the false-positive rate — measured from kind:"finding_verification" records (scripts/lib/token-metrics.sh) aggregated by scripts/token_report.sh — shows no regression, both verified on a DRY_RUN pass over ≥5 real merged PRs before the feature is enabled in LIVE_MODE.
Cost cap
Added context + specialist prompts + validation must not blow up per-review cost: median Effective Tokens (ET) per escalated review must stay ≤ 1.5× the Phase-1 frozen baseline, tracked via scripts/token_report.sh (ET is the fine-grained comparator per AGENTS.md#cost-reporting). The validation/eval loop is bounded to ≤ 5 merged PRs × ≤ 3 iterations per enhancement; any story exceeding the ET ceiling must land behind an opt-in knob (inert by default) rather than change default review cost.
Untracked prerequisites
Reference architectures to benchmark against (learning sources in Discussion Enhance PR Review Agent pipeline #906): CodeRabbit agentic-validation blog, Greptile 'what is AI code review', baz.co 'engineering intuition at scale', Qt code-review skills — these are external articles, not repo issues; the Phase-1 audit must cite ≥2 of them.
All 6 stories are genuine new bug-finding work; nothing overtaken. The epic is well-guarded: Phase 1 (#1089) freezes an immutable baseline first, every quality change is scored only on the held-out split, and there's a hard cost cap (median ET ≤ 1.5× baseline) plus a held Phase-4 convergence gate.
[DEFERRED-Q / coordination] — #1090 (semantic symbol context: callers/callees/type-defs) overlaps the LSP pilot, which was decided NO-GO for Shell and retired (#839, PR #1110). #1090's approach must NOT depend on the retired LSP harness — the Phase-1 audit (#1089) should pick a Shell-viable source (ctags/tree-sitter/gh grep) and re-confirm scope against #610 (context-adaptive review, now promoting via #616), #676 (MCP, done), and #581. Flagging so the audit doesn't rebuild a dead path.
Activated for dev-lead:#1089 (audit/baseline — gates the rest), then #1090/#1091/#1093 (Phase 2), #1092 (Phase 3). #1094 (Phase 4 eval gate + dry-run + human go/no-go before LIVE_MODE) stays initiative:hold + dev-lead:hands-off.
[DEFERRED-Q UPDATE] LSP is permitted for #1090 where appropriate
Maintainer clarification (2026-07-13): LSP is not excluded — if a language server is the appropriate way to source semantic symbol context (callers/callees/type-defs), #1090 may use it. The only hard constraint is the specific Shell NO-GO from #839 (bash-language-server didn't earn its nav-token cost); the door remains open for LSP on TS/JS, Python, Go, etc.
So the Phase-1 audit (#1089) should choose the best per-language source on the merits — LSP where it pays off (typed languages), lighter tools (ctags/tree-sitter/gh grep) where LSP isn't warranted (Shell) — rather than treating LSP as off-limits. — Claude Code
Attached #1596 as a sub-issue: pr-review has no false-negative metric and no production feedback loop.
It closes a gap this initiative names but does not measure. This initiative's success metric is the held-out eval score plus no false-positive regression — both real, but neither detects the inverse: a defect pr-review approved past that a third-party bot then found and that was accepted. That signal is free, perfectly labelled, and currently discarded.
It also gives gap 4 ("merged-PR few-shot — no learning from this repo's past review→merge histories") an actual source of truth. Today the evals/deep-review cases are hand-authored synthetic scenarios; harvested production misses would be real regressions the agent demonstrably failed.
Evidence gathered on 2026-08-30/31 — six accepted defects across five PRs, none raised by pr-review:
.github#995 — the ADR's rolling-vs-fixed-window self-contradiction, found independently by graphite-app and CodeRabbit nine days apart, and it would have shipped as the acceptance criterion for #641/#994.
.github#1017 — contract test compared jq's rendered text, so "50" passed as an integer and "true" as a boolean, in the very test meant to enforce types. Separately, the config's own _exempt_note claims same-set parity with pr-limits.json and nothing enforced it — raised at 22:40, eight minutes after pr-review approved at 22:32.
#1556/#1558/#1559 — three architectural findings now tracked as #1591/#1590/#1589.
Two design constraints called out in #1596 that matter to this initiative directly:
Harvest into dev/ only, never holdout/.holdout-guard.yml and CODEOWNERS exist so the proposer cannot see what it is scored against. Auto-harvesting into holdout/ would silently destroy the control that makes this initiative's success metric meaningful.
Accepted-vs-refuted classification is the crux, not a detail. Roughly 3 of 4 bot findings dispositioned that same day were false positives correctly refuted. A naive "bot opened a thread → pr-review missed it" counter would be dominated by noise and would push the agent toward more false positives — regressing this initiative's own false-positive metric.
Not released to dev-lead independently, to avoid colliding with this initiative's phasing — sequence it as you see fit.
Activity
don-petry commentedon Jul 14, 2026
Reconciled + activated (2026-07-13)
All 6 stories are genuine new bug-finding work; nothing overtaken. The epic is well-guarded: Phase 1 (#1089) freezes an immutable baseline first, every quality change is scored only on the held-out split, and there's a hard cost cap (median ET ≤ 1.5× baseline) plus a held Phase-4 convergence gate.
[DEFERRED-Q / coordination] — #1090 (semantic symbol context: callers/callees/type-defs) overlaps the LSP pilot, which was decided NO-GO for Shell and retired (#839, PR #1110). #1090's approach must NOT depend on the retired LSP harness — the Phase-1 audit (#1089) should pick a Shell-viable source (ctags/tree-sitter/
ghgrep) and re-confirm scope against #610 (context-adaptive review, now promoting via #616), #676 (MCP, done), and #581. Flagging so the audit doesn't rebuild a dead path.Activated for dev-lead: #1089 (audit/baseline — gates the rest), then #1090/#1091/#1093 (Phase 2), #1092 (Phase 3). #1094 (Phase 4 eval gate + dry-run + human go/no-go before LIVE_MODE) stays
initiative:hold+dev-lead:hands-off.Adding
initiative:auto. — Claude Codedon-petry commentedon Jul 14, 2026
[DEFERRED-Q UPDATE] LSP is permitted for #1090 where appropriate
Maintainer clarification (2026-07-13): LSP is not excluded — if a language server is the appropriate way to source semantic symbol context (callers/callees/type-defs), #1090 may use it. The only hard constraint is the specific Shell NO-GO from #839 (bash-language-server didn't earn its nav-token cost); the door remains open for LSP on TS/JS, Python, Go, etc.
So the Phase-1 audit (#1089) should choose the best per-language source on the merits — LSP where it pays off (typed languages), lighter tools (ctags/tree-sitter/
ghgrep) where LSP isn't warranted (Shell) — rather than treating LSP as off-limits. — Claude Codedon-petry commentedon Aug 31, 2026
Attached #1596 as a sub-issue: pr-review has no false-negative metric and no production feedback loop.
It closes a gap this initiative names but does not measure. This initiative's success metric is the held-out eval score plus no false-positive regression — both real, but neither detects the inverse: a defect pr-review approved past that a third-party bot then found and that was accepted. That signal is free, perfectly labelled, and currently discarded.
It also gives gap 4 ("merged-PR few-shot — no learning from this repo's past review→merge histories") an actual source of truth. Today the
evals/deep-reviewcases are hand-authored synthetic scenarios; harvested production misses would be real regressions the agent demonstrably failed.Evidence gathered on 2026-08-30/31 — six accepted defects across five PRs, none raised by pr-review:
.github#995— the ADR's rolling-vs-fixed-window self-contradiction, found independently by graphite-app and CodeRabbit nine days apart, and it would have shipped as the acceptance criterion for#641/#994..github#1017— contract test compared jq's rendered text, so"50"passed as an integer and"true"as a boolean, in the very test meant to enforce types. Separately, the config's own_exempt_noteclaims same-set parity withpr-limits.jsonand nothing enforced it — raised at 22:40, eight minutes after pr-review approved at 22:32.#1556/#1558/#1559— three architectural findings now tracked as#1591/#1590/#1589.Two design constraints called out in #1596 that matter to this initiative directly:
dev/only, neverholdout/.holdout-guard.ymland CODEOWNERS exist so the proposer cannot see what it is scored against. Auto-harvesting intoholdout/would silently destroy the control that makes this initiative's success metric meaningful.Not released to dev-lead independently, to avoid colliding with this initiative's phasing — sequence it as you see fit.