Skip to content

10 of 11 open PRs have never received an approving review: the advisory quorum is unmeetable, the disposition gate revokes what does issue, and nothing reports why #1894

Closed
1 of 1 issue completed

Description

@don-petry

Summary

Ten of the eleven open PRs in this repo have never received an approving review. Not dismissed — never issued. mergeStateStatus is BLOCKED / reviewDecision: REVIEW_REQUIRED on all of them, and the ruleset needs exactly one approval.

The approval path is a chain of individually-defensible gates. Each is locally correct. In series they make approval effectively unreachable, and no component reports the end-to-end outcome, so the fleet looks healthy while nothing can merge.

Measured 2026-09-23

Approving reviews ever issued, across the open queue:

#1891 approved=1     ← the only one, opened hours earlier
#1888 approved=0     #1887 approved=0     #1873 approved=0     #1799 approved=0
#1796 approved=0     #1794 approved=0     #1793 approved=0     #1770 approved=0
#1742 approved=0     #1730 approved=0

Several of these have been open since 2026-09-13.

Gate 1 — the advisory quorum is permanently unmeetable (but is NOT the terminal blocker)

Every PR carrying a partial-evidence marker shows the same ratio:

#1799 submitted=4 required=6 reason=head-age-timeout
#1796 submitted=4 required=6 reason=head-age-timeout
#1794 submitted=4 required=6 reason=quiescence-timeout
#1793 submitted=4 required=6 reason=quiescence-timeout
#1770 submitted=4 required=6 reason=head-age-timeout
#1742 submitted=4 required=6 reason=head-age-timeout

Two problems:

  1. required=6 does not match the registry. reviewer-sources.tsv lists seven logins with advisory_gate=yes (copilot-pull-request-reviewer, gemini-code-assist, chatgpt-codex-connector, sonarqubecloud, qodo-code-review, codeant-ai, graphite-app), and _advisory_gate_load_fallback_bots hardcodes the same seven. The runtime denominator is 6. One member is silently dropped; which one, and why, is not observable.
  2. Most registered advisory bots never participate. On feat: implement issue #1754 — pr-review counts a human-escalation as a posted review — run reports "1 reviews posted" with nothing on the PR #1793 only three bots posted reviews at all — codeant-ai, gemini-code-assist, and coderabbitai — and coderabbitai is advisory_gate=no, so it does not even count. Four of the seven (copilot-pull-request-reviewer, chatgpt-codex-connector, qodo-code-review, graphite-app) contribute nothing. The quorum can never be satisfied on its own terms, so every PR reaches approval only through a timeout fallback. A fallback that fires 100% of the time is the real policy; the quorum is decoration.

To be clear about where this gate ends: check_advisory_reviews records the partial evidence and returns 0 — "Ready to approve." So the advisory gate is not what blocks these PRs. It is working as designed, just always via its escape hatch.

Gate 2 — the disposition gate revokes approvals that do issue

Where an approval did issue, it was taken away:

The maintainer-comment gate (#1290 / #1813) is sound in intent — a maintainer finding posted as a plain PR issue comment creates no review thread, so it neither blocks merge nor reaches dev-lead's fix-reviews prompt, and approving over it ships the defect. The redesign correctly replaced the push-timestamp proxy with a verified disposition.

But it has no path to satisfaction that does not depend on dev-lead running. A disposition is recorded by dev-lead. If dev-lead is suppressed (needs-human-review), rate-limited, cancelled by concurrency, or simply never dispatched for that PR, the comment stays undispositioned forever and the approval can never stand. The gate is fail-closed with no operator override — a maintainer cannot disposition their own comment.

The systemic defect

No single component is wrong. The failure is that nothing owns or reports the end-to-end question "why is this PR not approved?" The advisory gate logs that it is ready to approve. The disposition gate logs a dismissal. Neither knows about the other, and the PR's own surface shows only REVIEW_REQUIRED with no reason. Diagnosing one PR took reading three scripts and correlating GraphQL timeline events by hand.

The consequence is the queue this repo has been carrying for ten days, and it is also what forces every merge through an admin bypass — which discards required_review_thread_resolution along with the approval requirement. A gate chain so strict that the only viable path is bypassing all of it is weaker than a looser chain that actually runs.

Acceptance Criteria

  1. A single deterministic diagnostic answers, for a given PR, why it is not approved — naming the gate, the specific condition, and what would satisfy it. Surfaced where an operator looks (step summary and/or a PR marker), not only in a run log.
  2. The advisory denominator is reconciled with the registry: required equals the count of advisory_gate=yes logins, or the discrepancy is explained and asserted by a test. 6 != 7 must not be silently possible.
  3. Advisory bots that do not participate are handled explicitly — removed from the gate set, or marked non-participating so they are excluded from the denominator rather than waited on and then timed out. A quorum whose fallback fires on every PR is recorded as the policy it actually is.
  4. The disposition gate gains a satisfaction path that does not require dev-lead: an attributable maintainer disposition (a reply, a label, or a marker a human can post) clears their own comment. Fail-closed is right; unsatisfiable is not.
  5. A fleet-level metric: open PRs with zero approving reviews, and their age. Ten PRs unapproved for ten days must set off an alarm. Nothing currently measures this — Runaway PR candidate(s) detected 2026-09-20 #1850/Stalled PR candidate(s) detected 2026-09-20 #1851 count runaway and stalled PRs, neither counts unapprovable ones.
  6. Regression coverage for the end-to-end path: a PR with partial advisory evidence and a dispositioned maintainer comment reaches APPROVED without an admin bypass.

Notes for the implementer

References


✅ MAINTAINER DECISION 2026-09-23 (revised) — advisory quorum is lowered to 4

Revised 2026-09-23: graphite-app stays in the gate. The maintainer bought Graphite's paid plan ($40/mo, unlimited reviews) on 2026-09-22. It went silent on 2026-08-21, exactly 30 days after it was installed, which points to a lapsed trial rather than a broken integration. While it was active (Jul 22–Aug 21) it reviewed 267 org PRs and posted 109 findings on 69 of them; about 72% were correct and another 18% partly correct. The earlier version of this decision removed four logins and set the quorum to 3. That is superseded: remove three, and the quorum is 4.

A maintainer has decided AC #3: the advisory quorum becomes 4, by removing the three logins that never participate from the gate set. This supersedes the "removed or marked non-participating" ambiguity in AC #3 — take the removal option.

The evidence this rests on

Reviews submitted across eight recent PRs (#1888, #1887, #1873, #1799, #1796, #1794, #1793, #1770):

Login PRs reviewed advisory_gate today
gemini-code-assist 8 yes
codeant-ai 8 yes
coderabbitai 5 no (not part of the gate)
copilot-pull-request-reviewer 0 yes
chatgpt-codex-connector 0 yes
qodo-code-review 0 yes
graphite-app 0 yes

sonarqubecloud does not appear because it posts a top-level comment, not a review (creates_threads=no in the registry — "Top-level quality-gate comment only"). It does participate; it simply is not visible in the reviews API. It stays in the gate set.

So the advisory sources that actually produce signal are gemini-code-assist, codeant-ai, sonarqubecloud. graphite-app is kept in the gate as a fourth source: it produced 0 reviews on these PRs only because its plan had lapsed, and it is now paid. The quorum is 4.

The change

Set advisory_gate to no for these three rows in scripts/lib/reviewer-sources.tsv:

  • copilot-pull-request-reviewer
  • chatgpt-codex-connector
  • qodo-code-review

Do NOT change the graphite-app row. It stays advisory_gate=yes. Update its rationale to record why: it was kept in the gate because it moved to a paid plan on 2026-09-22 after its trial lapsed on 2026-08-21.

Because PR #1902 already made the denominator a projection of the registry (AC #2), this data change alone moves required from 7 to 4. Do not hardcode 4 (or any count) anywhere. Tests should assert that required equals the registry's advisory_gate=yes count, not a literal. #1903 separately adds cubic-dev-ai with advisory_gate=yes, which will take the count to 5 when it merges. A literal would break at that point.

Three places must move together or they drift

This is the exact failure mode the registry exists to prevent — the denominator is now read from the registry in three independent spots:

  1. scripts/lib/reviewer-sources.tsv — the source of truth (the three rows above; graphite-app unchanged).
  2. _advisory_gate_load_fallback_bots in scripts/lib/advisory-review-gate.sh — the hardcoded seven-entry fallback map used when the registry cannot be sourced. It keeps graphite-app.
  3. The hardcoded fallback array in scripts/lib/approval-diagnostic.sh, which keeps graphite-app. Currently '["copilot-pull-request-reviewer","gemini-code-assist","chatgpt-codex-connector","sonarqubecloud","qodo-code-review","codeant-ai","graphite-app"]'.

A test must assert all three agree, so a future edit to one cannot silently diverge.

Preserve these

What this does and does not fix

It makes the gate pass honestly rather than through a 10–20 minute timeout fallback on every single PR, which is a real improvement: faster approvals, and the recorded policy finally matches the behaviour. That holds only if Graphite resumes reviewing now that its plan is active. If Graphite is still silent after the plan change, report it as a Graphite-side problem; do not remove it from the gate again without asking the maintainer.

It does not by itself unblock the queue. check_advisory_reviews already returned 0 ("ready to approve") via those fallbacks — the advisory gate was never the terminal blocker. The approvals that do issue are revoked by the disposition gate (Gate 2 above, #1813), which remains unaddressed. AC #4 is still the one that determines whether approvals start landing.


Implementation note — the quorum change was applied by hand, 2026-09-23 (cf5a8c8b on PR #1902)

The registry change this decision calls for was not picked up by dev-lead across two fix-reviews passes, and the reason is worth recording because it will recur:

The instruction was posted as a PR review body, which creates no review thread. fix-reviews walks reviewThreads, so a review with only a body is invisible to it — the same asymmetry #1290 documents for plain issue comments. dev-lead's 03:00 commit reported "address review comments" and changed approval-diagnostic.sh, review-one-pr.sh and a test (+183/−48 etc.) while never touching reviewer-sources.tsv. A reviewer reading only the commit subject would have believed it applied.

Applied directly instead, in one commit:

  • advisory_gate=no for copilot-pull-request-reviewer, chatgpt-codex-connector, qodo-code-review, each with the dropping evidence recorded in its rationale column.
  • graphite-app kept, with its rationale recording why (check-run reporting, lapsed trial, paid plan 2026-09-22).
  • Both fallback copies brought into line: _advisory_gate_load_fallback_bots in advisory-review-gate.sh and the array in approval-diagnostic.sh.
  • A new test in tests/test_reviewer_sources.bats asserting the third copy is a registry projection (the first two were already covered by the existing consistency tests).

Verified locally: reviewer_sources_advisory_gate_logins returns exactly codeant-ai, gemini-code-assist, graphite-app, sonarqubecloud — required=4, nothing hardcoded; the new test's logic passes standalone; shellcheck -S warning clean on both edited scripts. The associative-array tests cannot run on macOS bash 3.2 (declare: -A: invalid option) and were left to CI.

Process lesson for this repo: to steer dev-lead on an open PR, the instruction must land somewhere it actually reads — an inline review comment (which creates a thread), or the issue body for an issue-intent run. A review body and a plain PR comment both look authoritative to a human and are invisible to the agent. This is the same class as #1566 (issue comments never reaching the prompt) and is worth folding into whatever AC #4 produces.

Activity

  1. github-actions commented on Sep 23, 2026

    @github-actions
    Contributor

    Thank you for this exceptionally thorough write-up — the per-PR measurements, gate-by-gate trace, and incident cross-references make the failure chain immediately actionable. Given the breadth of the six acceptance criteria, one clarifying question: should AC #1 (the end-to-end diagnostic surface) ship as a standalone deliverable first so each subsequent gate fix can be verified against it, or are ACs #2–#4 required in the same batch to make the diagnostic output meaningful?

  2. don-petry commented on Sep 23, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1894 — 10 of 11 open PRs have never received an approving review

    Scope (budget-guarded — this PR delivers the keystone only)

    Per the issue's own guidance ("Start with AC #1 … everything else is easier to size once a single PR can explain itself"), this PR delivers a single deterministic approval diagnostic (AC #1) plus the advisory-denominator reconciliation it depends on (AC #2). ACs #3–#6 (non-participating-bot policy, a dev-lead-independent disposition satisfaction path, the fleet zero-approval metric, and the end-to-end regression) are each sizeable and are explicitly deferred to follow-up issues once the diagnostic exists — I have permission from the prompt's budget guard to stop after the highest-value unit and file the rest.

    Implementation Checklist

    • scripts/lib/approval-diagnostic.sh — new PURE library. diagnose_approval <pr_snapshot_json> [advisory_json] [approver] evaluates the approval gate chain in the same order review-one-pr.sh applies it and emits ONE structured JSON verdict naming the blocking gate, the specific unmet condition, and what would satisfy it. render_approval_diagnostic <verdict_json> renders a Markdown block for the step summary / a PR marker.
    • Advisory denominator reconciled with the registry (AC Go-live improvements for PR review agent #2): the diagnostic's advisory.required == count of advisory_gate=yes logins in reviewer-sources.tsv, and it enumerates the specific missing bots by name so 6 != 7 can never be silent.
    • scripts/review-one-pr.sh — surface the diagnostic to $GITHUB_STEP_SUMMARY (guarded, never breaks the run); name which advisory bots were dropped as unavailable in the gate log.
    • .github/workflows/lint.yml — register the new bats test file in the run-bats.sh list.

    Tests to Write

    • tests/test_approval_diagnostic.bats — undispositioned maintainer comment → blocking_gate=maintainer-comment-gate with satisfaction path; advisory partial evidence → required==registry count with missing bots named; already-approved → not blocked; changes-requested-at-head; terminal "approval-not-yet-issued".
    • Assertion that advisory.required equals the registry advisory_gate=yes count (AC Go-live improvements for PR review agent #2, "asserted by a test").

    Test Command

    scripts/run-bats.sh tests/test_approval_diagnostic.bats (full suite via lint.yml run-bats.sh), then bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Sep 23, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Scope delivered this workspace (AC #1 + AC #2 — the keystone)

    Per the issue's own guidance ("Start with AC #1 … everything else is easier to size once a single PR can explain itself") and the prompt's budget guard, this workspace delivers the diagnostic and the advisory-denominator reconciliation it depends on. ACs #3–#6 are deferred to follow-ups (see below).

    Files touched (this workspace, not yet pushed)

    • scripts/lib/approval-diagnostic.sh (new): PURE diagnose_approval <snapshot> [advisory_json] [approver] — evaluates the approval gate chain in the same order review-one-pr.sh applies it and emits ONE verdict naming blocking_gate, condition, and satisfied_by, plus an advisory block with required / submitted / missing / via_timeout. render_approval_diagnostic renders the step-summary Markdown. Fails closed on an unparseable snapshot.
    • scripts/review-one-pr.sh: emit_approval_diagnostic appends the diagnostic to $GITHUB_STEP_SUMMARY (guarded subshell, never affects the review decision), called once CI is green and before the gate chain.
    • scripts/lib/advisory-review-gate.sh: the reduced-denominator log now NAMES which bots were dropped and why (dropped: <bot>=RATE_LIMITED, …), so "6 != 7" is observable at runtime (AC Go-live improvements for PR review agent #2).
    • tests/test_approval_diagnostic.bats (new, 12 cases): blocking-gate identification incl. the pr-review can't approve ANY open PR: maintainer-comment gate needs pushedDate (always null) and treats codeant/qodo/auto-rebase comments as maintainer findings — 18/18 PRs stranded #1813 revocation case (an approval standing at head does not mask an undispositioned comment); advisory.required == registry advisory_gate=yes count; missing bots named; fail-closed; renderer.
    • .github/workflows/lint.yml, tests/dev-lead/unit/test_advisory_review_gate.bats: register the new test; bump the gate line-budget 515→520 with a recorded rationale for the four AC Go-live improvements for PR review agent #2 observability lines.

    How AC #1 / AC #2 are met

    • AC test issue from agent #1 — a single deterministic diagnostic answers "why is this PR not approved?" naming the gate, the condition, and the satisfaction path, surfaced in the step summary (and as a machine-readable JSON line in the run log).
    • AC Go-live improvements for PR review agent #2 — advisory.required is derived from the registry (7), the diagnostic enumerates the missing bots by name, and the runtime gate now names dropped bots; test_approval_diagnostic.bats asserts required equals the registry advisory_gate=yes count so 6 != 7 cannot be silent.

    Deferred to follow-ups (not attempted here — budget guard)

    Local check output (provisional — not the pushed run)

    tests/test_approval_diagnostic.bats: 12/12 ok
    tests/dev-lead/unit/test_advisory_review_gate.bats: ok (line-budget updated)
    .dev-lead/scripts/dev-lead-lint.sh: all checks passed (shellcheck + agent frontmatter)
    

    Note: some test_review_one_pr_* cases fail identically with and without this change in the local bash-3.2/mocking environment (a known AGENTS.md caveat); CI is authoritative.

    Notes

    The diagnostic is read-only and additive — it introduces no new gate and loosens nothing (per the issue's "do not fix by loosening gates" note). It only makes the existing chain explain itself.

  4. don-petry commented on Sep 23, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #1902

    The implementation for issue #1894 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  5. added 5 commits that reference this issue on Sep 23, 2026
  6. added 9 commits that reference this issue on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions