Skip to content

Implement advisory bot review gate in pr-review agent #457

Closed

Description

@don-petry

Problem Statement

The pr-review agent approves PRs without waiting for advisory bot reviews (Gemini, Copilot, SonarCloud, Codex) to complete. This causes:

  1. PR fix(dev-lead): hold auto-merge while working a PR + push defensively (#452) #453 incident: pr-review approved at 02:51:35, but Copilot review with valid feedback arrived at 02:52:18 (43 seconds too late)
  2. Missed feedback: Valid code reviews are never incorporated because approval already posted
  3. Inconsistent coverage: Since CodeRabbit was fixed to NOT approve, pr-review doesn't wait for any bot reviews

Root Cause Analysis

Current Behavior

  • pr-review workflow triggers on check_suite (completed) event
  • Approves immediately after CI checks pass
  • Missing gate: No check for advisory bot review submissions

CodeRabbit Status

  • Does NOT approve (recent fix)
  • No longer the bottleneck
  • Rate-limits 30% of submissions (hourly quota exhaustion) — treat as optional

Advisory Bot Characteristics (from analysis of 50 recent PRs)

Bot Median P95 Participation Rate-Limits? Reliability
Gemini 50s (0.84m) 348s (5.8m) 80% No ✓ Excellent
Copilot 186s (3.1m) 444s (7.4m) 80% No ✓ Excellent
SonarCloud 794s (13.2m) 93k s (26h)* 80% No ✓ Good
Codex 1,059s (17.7m) 47k s (13.2h) 44% No ✓ Good

SonarCloud/Codex P95 outliers from May compliance blitz; typical PRs much faster

Data Collection

  • Sample: 50 recent merged PRs (May 23 - June 7, 2026)
  • 133+ bot submissions analyzed
  • Clean data methodology: excluded rate-limits, excluded PRs blocked by CodeRabbit

Recommended Solution

Wait Gate Implementation

Tier 1: 900 seconds (15 minutes)

  • Catches Gemini (0.84m), Copilot (3.1m), SonarCloud (13.2m) reliably
  • Catches ~70% of Codex submissions (median 17.7m)

Tier 2: 1,200 seconds (20 minutes)

  • Catches all 4 bots when they participate

Tier 3: 3,600 seconds (60 minutes)

  • Hard timeout before escalation

Smart Detection

  • Don't block on bots that aren't triggered
  • Codex only on 44% of PRs
  • Log which bots participated

Implementation Plan

Files to Modify

  1. scripts/lib/advisory-review-gate.sh (NEW)

    • wait_for_advisory_reviews(pr_url) function
    • Polls for review submissions (Tier 1 → Tier 2 → Tier 3)
  2. scripts/review-one-pr.sh

    • Add gate call after CI gate
    • Handle return codes
  3. tests/test_advisory_review_gate.sh (NEW BATS suite)

    • Test all tier behaviors
    • Mock bot responses
  4. Documentation

    • .claude/pr-review-agent/ADVISORY_REVIEW_GATE.md
    • Update scripts/review-one-pr.sh comments

Success Metrics

  • ✓ All advisory bots reviewed before approval (when triggered)
  • ✓ Typical approval latency: 20 min
  • ✓ <5% of PRs hit hard timeout
  • ✓ Zero regressions

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions