Repository navigation
Description
Background
PR #131 added 7 safety checks to the PR-review agent. It was closed unfixable
(28 merge-conflict sentinels) because its supporting script changes were
restructured out from under it by the dev-lead refactor on main. The check
semantics are still worth landing; reference PR #131 branch
claude/issue-43-20260511-2113, commit ed7d462.
The 7 checks: (1) CI-weakening detection, (2) prompt-injection scanning in
workflows, (3) large-PR gating, (4) PR-description quality scoring,
(5) critical-path tracing, (6) duplication search, (7) dependency-risk
assessment.
Revised implementation plan (determinism-first)
Correction to the original premise. The original issue said the checks are
"purely prompt-level additions toagents/pr-reviewer.md, no script changes
needed." That is inaccurate for the operational pipeline.agents/pr-reviewer.md
is only the manually-invoked Copilot profile. The automated cascade
that reviews every PR runsprompts/triage.md(Tier 1, no tools) →
prompts/deep-review.md(Tier 2, hassearch) →prompts/security-audit.md
(Tier 3), driven byscripts/review-one-pr.sh. To make these checks actually
fire on real PRs they must live in the cascade, not only the Copilot profile.
Design principle (required): deterministic wherever possible, LLM only where
unavoidable. Mechanical, pattern-matchable checks must NOT be left to the LLM
to "eyeball" — they must be computed in shell so they are reproducible and
unit-testable. The LLM is reserved for genuinely semantic judgment.
Determinism split
| # | Check | Mechanism | Where |
|---|---|---|---|
| 1 | CI weakening | Deterministic — grep diff for skipped/deleted tests (skip, only, xit, it.skip, test.skip, .todo, @Ignore), if: false, continue-on-error: true, commented-out CI steps, and lowered numeric thresholds (compare -/+ lines in coverage/CI config). Hard-stop. |
new scripts/lib/safety-checks.sh |
| 2 | Prompt injection | Deterministic — scan changed .github/workflows/*.yml for ${{ github.event.* }} used inside a run: step, LLM output interpolated into shell, over-broad token perms, pull_request_target + PR-head checkout without trust gate. Hard-stop. |
scripts/lib/safety-checks.sh |
| 3 | Large-PR gate | Deterministic — numeric from PR_METADATA (changedFiles, additions, deletions) + regex for an implementation-plan/breakdown section in the body. |
scripts/lib/safety-checks.sh |
| 4 | Description quality | Deterministic — regex presence of the 5 required sections (problem statement, risk category, test plan, rollback, monitoring). Count missing; ≥3 missing → escalate. | scripts/lib/safety-checks.sh |
| 7 | Dependency risk | Hybrid — deterministic parse of lockfile/manifest diff hunks (added deps, unpinned/^/~/latest ranges) in the lib; LLM narrative / CVE interpretation in deep-review. |
lib + prompts/deep-review.md |
| 5 | Critical-path trace | LLM (semantic) — data-flow tracing, permission checks on all auth branches, boundary-condition coverage. | prompts/deep-review.md |
| 6 | Duplication search | Hybrid — Tier 2 has the search tool: deterministic candidate gathering, LLM adjudicates true duplication. |
prompts/deep-review.md |
Architecture (mirror the existing downstream-impact feature)
There is already a proven pattern for injecting a deterministic, pre-computed
block into the LLM cascade. Follow it exactly:
scripts/lib/downstream-impact.sh→assemble_downstream_impact()writes a
structured block;review-one-pr.shinlines it into the triage prompt via
downstream_impact_triage_section; gated behindDOWNSTREAM_IMPACT_ENABLED
so that when the flag is off the triage prompt is byte-identical to
pre-feature behavior. Tests:tests/test_downstream_impact_*.bats,
tests/test_downstream_impact_triage_inline.bats.
Do the same for safety checks:
- New
scripts/lib/safety-checks.sh— pure functions that takePR_METADATA
(JSON fromgh pr view) andPR_DIFFand emit a structuredSAFETY_CHECKS
block: per-check findings withseverity,file:line, and two hard-stop
flagsCI_WEAKENING_DETECTED/PROMPT_INJECTION_DETECTED. No network, no
ghcalls — deterministic and side-effect-free so it is unit-testable. - Wire into
scripts/review-one-pr.sh— compute the block before the Tier 1
prompt is assembled and inline it intoTRIAGE_PROMPT_FILE(next to the
DOWNSTREAM_IMPACT/ADVISORY_BOT_FEEDBACKsections). Also write it to a
file (mirrorADVISORY_BOT_FEEDBACK_FILE) so Tier 2/3 can read it. Gate
behind aSAFETY_CHECKS_ENABLEDflag; default it ON (these are
safety-critical), but keep the flag for rollback and holdout-eval stability
(scripts/lib/holdout-guard.sh). When off, the prompt must be byte-identical. prompts/triage.md— consume theSAFETY_CHECKSblock: treat
CI_WEAKENING_DETECTEDandPROMPT_INJECTION_DETECTEDas forced
escalate:trueand never-approve; large-PR gate and description-missing≥3
→ escalate. The prompt must NOT re-derive these mechanically (no tools in
Tier 1) — it consumes the pre-computed verdicts.prompts/deep-review.md— add the LLM-only checks: critical-path trace
(5), duplication adjudication using thesearchtool (6), dependency-risk
narrative (7). Reference the deterministic hard-stops as blocking.agents/pr-reviewer.md— document all 7 checks in the human-facing
Copilot profile so the manually-invoked agent stays consistent with the
cascade (this satisfies the literal original ask; the operational teeth are
in the lib + prompts).
Testing (required — the whole point of determinism)
tests/test_safety_checks.bats— unit-test each deterministic check with
fixture diffs/metadata: CI-weakening true/false (incl. threshold lowering),
prompt-injection true/false, large-PR boundary (just under/over), description
scoring (0/3/5 sections present), dependency parsing (pinned vs unpinned).- Triage-inlining test mirroring
tests/test_downstream_impact_triage_inline.bats:
block present when enabled, and prompt byte-identical when the flag is off. - Register new test files in the
batsjob list /scripts/run-bats.sh
invocation so lint CI runs them.shellcheckmust pass on the new lib.
Guardrails
- Do NOT modify thin caller-stub workflows (
dev-lead.yml,pr-review.yml
header contract, etc.) beyond what AGENTS.md allows. Changes are confined to
scripts/lib/safety-checks.sh(new),scripts/review-one-pr.sh,
prompts/triage.md,prompts/deep-review.md,agents/pr-reviewer.md, and
tests/. - Flag-off path must be byte-identical to current triage output (protects
holdout evals). - Keep the lib free of
gh/network calls so it stays deterministic and testable.
Acceptance criteria
-
scripts/lib/safety-checks.shcomputes all deterministic checks with no network calls;shellcheckclean. -
SAFETY_CHECKSblock inlined into Tier 1 prompt whenSAFETY_CHECKS_ENABLED; byte-identical when off. -
prompts/triage.mdforces escalate/never-approve on the two hard-stops and consumes (does not re-derive) the deterministic verdicts. -
prompts/deep-review.mdcovers critical-path tracing, duplication adjudication, dependency-risk narrative. -
agents/pr-reviewer.mddocuments all 7 checks for the manual Copilot profile. -
tests/test_safety_checks.bats+ triage-inlining test added and wired into thebatsCI job; all green.
🤖 Generated with Claude Code
Activity
don-petry commentedon Jul 2, 2026
Dev-Lead Implementation Plan
Issue: #305 — feat(pr-reviewer): re-implement 7 safety checks against current architecture
Scope
Add the 7 PR-review safety checks determinism-first: a new pure
scripts/lib/safety-checks.shcomputes the mechanical checks (CI-weakening, prompt-injection, large-PR, description-quality, dependency parsing) as aSAFETY_CHECKSblock inlined into the Tier-1 triage prompt (mirroringdownstream-impact), with the semantic checks (critical-path, duplication, dependency-risk narrative) added to the deep-review prompt, all documented in the human-facing Copilot profile.Implementation Checklist
scripts/lib/safety-checks.sh(new) — pure functions overPR_METADATA(JSON) +PR_DIFF; emits aSAFETY_CHECKSblock with per-check findings (severity,file:line) and two hard-stop flagsCI_WEAKENING_DETECTED/PROMPT_INJECTION_DETECTED; nogh/network. Checks 1–4 + dependency parse (7). Plus asafety_checks_triage_sectiongated behindSAFETY_CHECKS_ENABLED(default ON) that inlines the block, byte-identical when off.scripts/review-one-pr.sh— compute the block before the triage prompt is assembled, inline viasafety_checks_triage_sectionnext to the DOWNSTREAM_IMPACT / ADVISORY_BOT_FEEDBACK sections, and write toSAFETY_CHECKS_FILEfor Tier 2/3.prompts/triage.md— consume the pre-computed block:CI_WEAKENING_DETECTED/PROMPT_INJECTION_DETECTEDforceescalate:true+ never-approve; large-PR gate and description-missing≥3 → escalate. Do NOT re-derive (no tools in Tier 1).prompts/deep-review.md— add LLM-only checks: critical-path trace (5), duplication adjudication viasearch(6), dependency-risk narrative (7); reference the deterministic hard-stops as blocking.agents/pr-reviewer.md— document all 7 checks for the manual Copilot profile..github/workflows/lint.yml— register new test files in thebatsjob list.Tests to Write
tests/test_safety_checks.bats— unit-test each deterministic check with fixture diffs/metadata: CI-weakening true/false (incl. threshold lowering), prompt-injection true/false, large-PR boundary (just under/over), description scoring (0/3/5 sections), dependency parsing (pinned vs unpinned).tests/test_safety_checks_triage_inline.bats— block present when enabled; prompt byte-identical when the flag is off (mirrorstest_downstream_impact_triage_inline.bats).Test Command
bats tests/test_safety_checks.bats tests/test_safety_checks_triage_inline.bats(+ fullscripts/run-bats.shlist);bash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jul 2, 2026
Dev-Lead: Implementation Complete
Re-implemented the 7 PR-review safety checks against the current cascade
architecture, determinism-first: the mechanical checks are computed in shell
(reproducible, unit-testable) and the LLM is reserved for genuinely semantic
judgment.
Plan Execution
scripts/lib/safety-checks.sh(new) — pure, no-gh/no-network functions overPR_METADATA+PR_DIFF. Deterministic checks 1–4 + dependency parse (7): CI-weakening (skip/disable markers,if: false,continue-on-error: true, lowered numeric thresholds), prompt-injection (untrustedgithub.event.*in workflow steps,pull_request_target,write-all), large-PR gate, description-quality scoring, dependency-range parsing. Emits aSAFETY_CHECKSblock with the two hard-stop flagsCI_WEAKENING_DETECTED/PROMPT_INJECTION_DETECTED. Plusassemble_safety_checks(writes block + exportsSAFETY_CHECKS_FILE) andsafety_checks_triage_section(default-ON gate).scripts/review-one-pr.sh— sources the lib, computes the block from the already-fetched metadata/diff, writes it toSAFETY_CHECKS_FILEfor Tier 2/3, and inlines it into the triage prompt next to theDOWNSTREAM_IMPACT/ADVISORY_BOT_FEEDBACKsections. Gated behindSAFETY_CHECKS_ENABLED(default ON; explicitfalse⇒ byte-identical triage prompt).prompts/triage.md— consumes (does not re-derive) the verdicts: the two hard-stops forceescalate:true+ never-approve + HIGH; large-PR and description-missing≥3 force escalate.prompts/deep-review.md— added the LLM-only checks: critical-path tracing (5), duplication adjudication via thesearchtool (6), dependency-risk narrative (7); references the deterministic hard-stops as blocking.agents/pr-reviewer.md— documents all 7 checks for the manually-invoked Copilot profile..github/workflows/lint.yml— registered both new test files in thebatsjob list.Test Results (TDD: tests written first, confirmed red, then green)
Files Changed
scripts/lib/safety-checks.sh(new): deterministic safety-check library.scripts/review-one-pr.sh: wire the block into the cascade (compute → inline into triage → export for deep/audit).prompts/triage.md: consume the hard-stops / gates.prompts/deep-review.md: add semantic checks 5/6/7.agents/pr-reviewer.md: document all 7 checks..github/workflows/lint.yml: register the two new bats files.tests/test_safety_checks.bats,tests/test_safety_checks_triage_inline.bats(new).Notes
SAFETY_CHECKS_ENABLED=false, nothing is computed andsafety_checks_triage_sectionemits nothing — the triage prompt is byte-identical to pre-feature behavior (asserted in the triage-inline test).gh/network I/O; the LLM tiers only adjudicate the semantic checks and consume the pre-computed mechanical verdicts.tests/test_batch_fallback.bats("Gemini trust error (55) falls back to Copilot") fails identically on a clean tree and is not in the CIbatslist — out of scope for this issue.feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
test(safety-checks): add regression guards for xit/xdescribe (mawk) +…
test(safety-checks): add regression guards for xit/xdescribe (mawk) +…
test(safety-checks): regression guards for xit/xdescribe (mawk) + cro…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…
feat: implement issue #305 — feat(pr-reviewer): re-implement 7 safety…