Repository navigation
Open
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Description
Story
As a pipeline maintainer,
I want update prompts/deep-review.md to read the pre-fed metadata/diff files when present instead of running gh pr view / gh pr diff, while keeping dynamic exploration tools available,
so that the deep tier stops spending tool round-trips and reasoning tokens on deterministic re-fetching.
Split from #1104 (dev-lead timed out at the 1800s action budget implementing both prompt files in one pass). This story is the
deep-review.mdhalf; therubber-duck.mdhalf is its sibling. It also carries the deep-review held-out eval regression gate (AC#5), which is specific to this tier.
Acceptance Criteria
- When the prefetch flag is on and
PR_CONTEXT_*files are present,prompts/deep-review.mdinstructs the model to read the pre-fed metadata + diff files instead of runninggh pr view/gh pr difffor the primary PR context. - When the flag is off (or files absent), the prompt falls back to the existing in-loop
gh pr view/gh pr diffsteps unchanged — behavior byte-identical to today. - The prompt explicitly retains permission to use Bash/Read/Grep/Glob and
gh apifor DYNAMIC needs (cross-file reads, org-standard lookups, secret-scan MCP) — only the deterministic metadata+diff fetch is replaced; keep the FORBIDDEN-enumeration guardrail and the secret-scan MCP step intact. - The pre-fed diff the deep tier consumes is the FULL diff (per Story 2), so the reviewable surface is not reduced.
scripts/evals/run-eval.sh deep-review(5 cases, llm-judge scorer, pass_threshold 0.7) is run with the change applied + flag on, and its aggregate score is NOT lower than the flag-off baseline (no regression). Do NOT modify held-out cases (holdout-guard.shenforces immutability).
Dev Notes
- Add
$PR_CONTEXT_METADATA_FILE/$PR_CONTEXT_DIFF_FILEto Inputs the same way$SAFETY_CHECKS_FILE/$ADVISORY_BOT_FEEDBACK_FILE/$DOWNSTREAM_IMPACT_FILEare referenced, with an explicit "if set, read these INSTEAD of gh pr view/diff; if unset, run the gh commands as before" instruction — keeps the flag-off path byte-identical (rollback + holdout-eval stability). - Depends on Story 2's plumbing; independent of the rubber-duck sibling and of Story 4.
References
- prompts/deep-review.md#inputs
- scripts/evals/run-eval.sh
- evals/deep-review/holdout/cases.jsonl
- scripts/lib/holdout-guard.sh
Likely target surface
prompts/deep-review.md
Metadata
Metadata
Assignees
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Activity