Skip to content

[Phase 2] Rewire deep-review.md to consume pre-fed context (split 1/2 of #1104) #1199

Open

Description

@don-petry

Story

As a pipeline maintainer,
I want update prompts/deep-review.md to read the pre-fed metadata/diff files when present instead of running gh pr view / gh pr diff, while keeping dynamic exploration tools available,
so that the deep tier stops spending tool round-trips and reasoning tokens on deterministic re-fetching.

Split from #1104 (dev-lead timed out at the 1800s action budget implementing both prompt files in one pass). This story is the deep-review.md half; the rubber-duck.md half is its sibling. It also carries the deep-review held-out eval regression gate (AC#5), which is specific to this tier.

Acceptance Criteria

  1. When the prefetch flag is on and PR_CONTEXT_* files are present, prompts/deep-review.md instructs the model to read the pre-fed metadata + diff files instead of running gh pr view / gh pr diff for the primary PR context.
  2. When the flag is off (or files absent), the prompt falls back to the existing in-loop gh pr view / gh pr diff steps unchanged — behavior byte-identical to today.
  3. The prompt explicitly retains permission to use Bash/Read/Grep/Glob and gh api for DYNAMIC needs (cross-file reads, org-standard lookups, secret-scan MCP) — only the deterministic metadata+diff fetch is replaced; keep the FORBIDDEN-enumeration guardrail and the secret-scan MCP step intact.
  4. The pre-fed diff the deep tier consumes is the FULL diff (per Story 2), so the reviewable surface is not reduced.
  5. scripts/evals/run-eval.sh deep-review (5 cases, llm-judge scorer, pass_threshold 0.7) is run with the change applied + flag on, and its aggregate score is NOT lower than the flag-off baseline (no regression). Do NOT modify held-out cases (holdout-guard.sh enforces immutability).

Dev Notes

  • Add $PR_CONTEXT_METADATA_FILE / $PR_CONTEXT_DIFF_FILE to Inputs the same way $SAFETY_CHECKS_FILE / $ADVISORY_BOT_FEEDBACK_FILE / $DOWNSTREAM_IMPACT_FILE are referenced, with an explicit "if set, read these INSTEAD of gh pr view/diff; if unset, run the gh commands as before" instruction — keeps the flag-off path byte-identical (rollback + holdout-eval stability).
  • Depends on Story 2's plumbing; independent of the rubber-duck sibling and of Story 4.

References

  • prompts/deep-review.md#inputs
  • scripts/evals/run-eval.sh
  • evals/deep-review/holdout/cases.jsonl
  • scripts/lib/holdout-guard.sh

Likely target surface

  • prompts/deep-review.md

Split from #1104 for epic #1101. Status: ready-for-dev.

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions