Skip to content

PR Review Agent Improvements #43

Closed as not planned
Closed as not planned

Description

@don-petry

Background

The current PR Review Agent (agents/pr-reviewer.md) provides a solid cascading-tier framework but lacks specificity in several high-value areas. The articles below identify common failure modes in AI-generated PRs that the agent should actively surface:


Goals

1. Detect CI Weakening (Hard Stop)

Any change that reduces CI coverage or disables safety checks must be an automatic blocker, regardless of risk tier.

2. Flag Code Duplication

Agent-generated PRs frequently reimplement existing utilities. The reviewer should search for duplicate logic before approving.

3. Enforce Correctness Tracing

The agent should trace at least one critical path end-to-end and verify permission checks on all branches, not just summarize the diff.

4. Reject Structureless Large PRs

PRs touching 5+ unrelated files with no implementation plan should be escalated immediately rather than reviewed deeply.

5. Check for Prompt Injection in Workflows

CI workflow files that execute model output or pass untrusted user input to shell commands must be flagged as HIGH risk.

6. Score PR Description Quality

The review output should assess whether the PR description includes: problem statement, risk categorization, test plan, rollback procedure, and monitoring plan — and note what's missing.

7. Structured Review Timeline

Reviews should follow a time-bounded protocol aligned with risk tier to reduce token spend and improve consistency.


Acceptance Criteria

AC-1: CI Weakening Detection

  • The agent detects and flags any of the following as a HIGH severity blocker that prevents approval:
    • Reduced test coverage thresholds in CI config
    • Skipped or deleted test cases
    • Disabled workflow steps or conditional gates added to bypass checks
  • Findings include the specific file path and line number of the weakening change

AC-2: Code Duplication Check

  • For any new utility, helper, middleware, or validation function introduced in the PR, the agent performs a codebase search for similar existing implementations
  • If a near-duplicate is found, the finding is reported as MEDIUM severity with the path to the existing implementation
  • Approval is withheld until duplication is resolved or the author justifies the new implementation

AC-3: Critical Path Tracing

  • For MEDIUM and HIGH risk PRs, the agent traces at least one critical path (input → transform → output) rather than only reviewing changed lines
  • Permission checks on all branches of any auth-related code are verified
  • Boundary conditions (empty, zero, max values) are noted in the findings if not covered by tests

AC-4: Large PR Gating

  • PRs meeting any of the following criteria are immediately escalated (Tier 3) without deep review:
    • Changes span 5+ unrelated files with no stated implementation plan in the PR body
    • PR purpose cannot be summarized in one sentence
    • Only test file changes while CI is failing
  • Escalation comment requests a structured breakdown before further review investment

AC-5: Prompt Injection in Workflows

  • The agent scans any modified .github/workflows/*.yml files for:
    • User-controlled input passed to run: steps without sanitization
    • Model output executed directly as shell commands
    • Tokens with write permissions used where read-only would suffice
  • Any match is reported as HIGH severity with a reference to the specific step

AC-6: PR Description Scoring

  • The review output includes a PR Description Quality section that checks for the presence of:
    • Problem statement (what and why)
    • Risk category (high/medium/low with justification)
    • Test plan (what was tested and how)
    • Rollback procedure
    • Monitoring/observability plan
  • Missing required elements are listed explicitly; PRs missing 3+ elements receive a MEDIUM finding recommending description improvement before merge

AC-7: Tiered Review Protocol

  • Tier 1 (triage) completes without tool calls and classifies: scope, CI weakening indicators, large-PR gate
  • Tier 2 (deep review) follows a structured order: CI/workflow changes → duplication search → critical path trace → security boundaries
  • Tier 3 (escalation) fires for: HIGH risk, CI failing, large structureless PR, prompt injection finding, or 3+ missing description elements
  • Idempotency marker (<!-- pr-review-agent v1 sha=<SHA> -->) is checked before any tool calls are made

Out of Scope

  • Changes to the risk tier thresholds (LOW/MEDIUM/HIGH decision table)
  • Human reviewer workflow or assignment logic
  • Integration with external SAST tools (separate initiative)

Activity

  1. don-petry commented on May 11, 2026

    @don-petry
    CollaboratorAuthor

    Claude finished @don-petry's task in 4m 59s —— View job • claude/issue-43-20260511-2113 • [Create PR ➔](https://github.com/petry-projects/.github-private/compare/main...claude/issue-43-20260511-2113?quick_pull=1&title=Issue%20%2343%3A%20Changes%20from%20Claude&body=This%20PR%20addresses%20issue%20%2343%0A%0AGenerated%20with%20%5BClaude%20Code%5D(https%3A%2F%2Fclaude.ai%2Fcode)


    Implementing PR Review Agent Improvements

    • Gather context (read AGENTS.md and current pr-reviewer.md)
    • Understand the 7 improvement goals and acceptance criteria
    • Implement all 7 ACs in agents/pr-reviewer.md:
      • AC-1: CI Weakening Detection (hard stop blocker with file/line reporting)
      • AC-2: Code Duplication Check (MEDIUM severity, withholds approval)
      • AC-3: Critical Path Tracing (MEDIUM/HIGH risk PRs, auth branches, boundary conditions)
      • AC-4: Large PR Gating (5+ unrelated files, unsummarizable, failing CI + test-only → Tier 3)
      • AC-5: Prompt Injection in Workflows (user-controlled input, model output in shell, write-scoped tokens)
      • AC-6: PR Description Scoring (5-element table, 3+ missing → MEDIUM finding)
      • AC-7: Structured Tiered Protocol (Tier 1 no-tools, Tier 2 fixed order, Tier 3 triggers, idempotency pre-tools)
    • Self-review: fixed idempotency check step ordering (moved to Step 2, after fetch, before further tool calls)
    • Commit and push changes
    • Create pull request
    • Tag code owners (@petry-projects/org-leads)

    Summary

    All 7 acceptance criteria from the issue are implemented in a single file change to agents/pr-reviewer.md. The key additions:

    AC What was added
    AC-1 CI weakening scan in Tier 1 (no tool calls) — pattern list for threshold reductions, skip markers, if: false, continue-on-error: true, deleted tests — hard stop blocker with file:line reporting
    AC-2 Duplication search step in Tier 2 — semantic search for any new utility/helper/middleware, MEDIUM severity, approval withheld
    AC-3 Critical path trace in Tier 2 — input→transform→output, permission checks on all auth branches, boundary condition gaps noted
    AC-4 Large PR gate in Tier 1 — 5+ unrelated files with no plan, unsummarizable purpose, or test-only with failing CI → immediate Tier 3
    AC-5 Prompt injection scan in Tier 2 — ${{ github.event.* }} in run: steps, model output as shell, write-scoped tokens, pull_request_target without trust checks — HIGH severity hard stop
    AC-6 PR Description Quality section in every review output — 5-element table, 3+ missing → MEDIUM finding
    AC-7 Explicit step numbering and ordering: fetch → idempotency check → Tier 1 (no tools) → Tier 2 (fixed order) → Tier 3

    View PR #131 ·

  2. added
    dev-leadFor dev-lead agent pickup
    and removed on Jun 6, 2026
  3. don-petry commented on Oct 21, 2026

    @don-petry
    CollaboratorAuthor

    Closing after the 2026-10-21 backlog triage.

    Why: May wishlist; CI-weakening/duplication/injection checks now in prompts; superseded by #1088.
    Evidence: main:prompts/deep-review*.md, agents/pr-reviewer.md; PR #131 closed

    Reopen if this still reproduces on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions