Skip to content

pr-review: stop escalating trusted first-party caller-stub / standards-sync PRs to 'needs human' #1358

Closed

Description

@don-petry

Problem

The PR-review agent escalates "needs human review" on PRs that are low/no-risk
and fully aligned to our standards — specifically the org's own plumbing PRs that
ship through the canary rollout strategy. Reviewing the org-wide
needs-human-review corpus (gh search prs --owner petry-projects --label needs-human-review), the escalations are dominated by two classes:

  • Standards-sync PRs — chore: sync N org-standard workflow stub(s) from petry-projects/.github (bot-authored).
  • secrets: inherit migration PRs on caller stubs (e.g. the S7635 /
    GH_PAT_* fallback work).

These are exactly the changes the canary rollout strategy exists to de-risk, yet
they get flagged for a human, adding latency and eroding trust in the automated
gate.

Note: .github-private itself tends to approve these (its review lenses can
read AGENTS.md's caller-stub / canary policy), while consumer repos escalate
because they lack that context. The fix must therefore live in the review logic
itself, not in per-repo docs.

Root cause

Three compounding drivers in the review decision surfaces
(prompts/triage.md, prompts/shared.md, prompts/single-review.md,
prompts/deep-review.md, scripts/lib/safety-checks.sh):

  1. Secret forwarding is misread as secret handling. All three risk
    taxonomies bucket any diff touching "secrets/credentials/tokens" or "GitHub
    Actions security" as HIGH (never auto-approve). But a thin caller stub that
    forwards secrets: inherit (or maps a secret into a secrets:/with:/env:
    block) to a pinned first-party petry-projects/* reusable is the org-standard,
    SonarCloud-suppressed (S7635) pattern — not secret handling.
  2. Bot chore PRs trip the process gates. The deterministic
    DESCRIPTION_MISSING >= 3 check forces escalate, and the "linked issue
    addressed" / "well-structured" decision gates fail on terse, templated,
    issue-less automated bodies.
  3. Canary rollout is not modeled anywhere as a risk reducer, despite the
    org's formal canary strategy (docs/initiatives/agentic-release-strategy.md,
    the pr-review-canary / initiative-driver-canary workflows).

Proposed solution

Introduce a Trusted first-party stub / standards-sync carve-out, encoded in
two layers so it survives even in consumer repos that lack our policy docs:

  1. Deterministic guard (scripts/lib/safety-checks.sh) — a new
    TRUSTED_STUB_SYNC classification computed from structural signals:
    WORKFLOW_ONLY_CHANGE, SECRET_IN_RUN_STEP, THIRD_PARTY_REUSABLE_ADDED,
    STANDARDS_SYNC_PR (bot-authored or standards-sync title). When true, the
    description-quality and large-PR findings are demoted from [escalate] to
    [info]. The two hard-stops (CI_WEAKENING_DETECTED,
    PROMPT_INJECTION_DETECTED) still override, and a secret in a run: step or a
    non-petry-projects reusable disqualifies the carve-out.
  2. Prompt carve-out — the same, self-contained exception on the HIGH bucket
    and decision gates of triage.md, shared.md, single-review.md, and
    deep-review.md, distinguishing secret forwarding from handling, clarifying
    that a standards-sync PR is the opposite of a standards violation, and
    naming canary rollout as a risk reducer.

Scope / boundary (deliberate)

The carve-out applies only to: (a) secrets: inherit / secret-forwarding to a
pinned petry-projects/* reusable, and (b) bot-authored standards-sync /
caller-stub-sync PRs. Genuine secret handling, third-party reusables, and
human-authored PRs continue to escalate exactly as before.

Acceptance criteria

  • A bot-authored, workflow-only standards-sync PR that only forwards
    secrets: inherit to a first-party reusable is classified
    TRUSTED_STUB_SYNC: true and is not escalated on secrets / GitHub
    Actions / missing-description / no-linked-issue grounds.
  • A secret interpolated into a run: step → SECRET_IN_RUN_STEP: true,
    carve-out does not apply (still HIGH/escalate).
  • A non-petry-projects reusable-workflow call added →
    THIRD_PARTY_REUSABLE_ADDED: true, carve-out does not apply.
  • A human-authored PR, or any PR touching non-workflow files, is not
    classified TRUSTED_STUB_SYNC.
  • The two deterministic hard-stops still fire and still force escalate even
    under TRUSTED_STUB_SYNC: true.
  • The four LLM taxonomies carry the same carve-out so a triage-approved stub
    sync is not re-escalated by the single-review / council / deep tiers.
  • Unit tests cover each signal and disqualifier; existing safety-checks,
    triage-prompt, and triage-inline suites stay green.

Risk

Low. The change only narrows escalation for a precisely-bounded, structurally
verified class; it never widens auto-approval for genuine secret handling,
third-party reusables, hard-stops, failing CI, or human PRs. The deterministic
guard is pure/side-effect-free and unit-tested.

Test plan

  • bats tests/test_safety_checks.bats (new Check-8 cases + existing).
  • bats tests/test_triage_prompt.bats, tests/test_safety_checks_triage_inline.bats,
    tests/test_downstream_impact_triage_inline.bats.
  • shellcheck scripts/lib/safety-checks.sh.
  • Behavioral spot-check against a real markets#369-style diff and its
    disqualifier variants.

Rollback

Pure prompt + shell-lib change with no schema/state migration. Revert the PR to
restore prior behavior byte-for-byte; the SAFETY_CHECKS block degrades to its
previous shape when the new lines are absent.

Monitoring

Track the needs-human-review escalation rate on standards-sync / secrets: inherit PRs (should drop toward zero) via the existing reviewer telemetry
(scripts/reviewer_report.sh / pr_review_health.sh), and watch for any
regression in genuine-risk escalations.

Out of scope

  • The escalate path posting no reasoning body onto the PR (label + codeowners
    only) — worth a follow-up but not addressed here.
  • Broadening the carve-out to any workflow-stub change regardless of authorship
    or reusable owner (explicitly rejected — see Scope).

Activity

  1. github-actions commented on Jul 23, 2026

    @github-actions
    Contributor

    Thanks for the thorough write-up — the root cause breakdown across the three compounding drivers and the clear scope boundary make this really actionable. One clarifying question before we start implementation: for the WORKFLOW_ONLY_CHANGE signal that gates TRUSTED_STUB_SYNC, should a PR that touches only .github/workflows/ files always qualify, or should we also require that the changed workflow files themselves are recognized caller-stub patterns (e.g. thin wrappers with no run: steps) — so that a human-authored workflow-only PR that happens to add secrets: inherit to a first-party call doesn't accidentally get the carve-out?

  2. added a commit that references this issue on Jul 23, 2026
  3. added 13 commits that reference this issue on Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions