Repository navigation
Description
Summary
Ten of the eleven open PRs in this repo have never received an approving review. Not dismissed — never issued. mergeStateStatus is BLOCKED / reviewDecision: REVIEW_REQUIRED on all of them, and the ruleset needs exactly one approval.
The approval path is a chain of individually-defensible gates. Each is locally correct. In series they make approval effectively unreachable, and no component reports the end-to-end outcome, so the fleet looks healthy while nothing can merge.
Measured 2026-09-23
Approving reviews ever issued, across the open queue:
#1891 approved=1 ← the only one, opened hours earlier
#1888 approved=0 #1887 approved=0 #1873 approved=0 #1799 approved=0
#1796 approved=0 #1794 approved=0 #1793 approved=0 #1770 approved=0
#1742 approved=0 #1730 approved=0
Several of these have been open since 2026-09-13.
Gate 1 — the advisory quorum is permanently unmeetable (but is NOT the terminal blocker)
Every PR carrying a partial-evidence marker shows the same ratio:
#1799 submitted=4 required=6 reason=head-age-timeout
#1796 submitted=4 required=6 reason=head-age-timeout
#1794 submitted=4 required=6 reason=quiescence-timeout
#1793 submitted=4 required=6 reason=quiescence-timeout
#1770 submitted=4 required=6 reason=head-age-timeout
#1742 submitted=4 required=6 reason=head-age-timeout
Two problems:
required=6does not match the registry.reviewer-sources.tsvlists seven logins withadvisory_gate=yes(copilot-pull-request-reviewer, gemini-code-assist, chatgpt-codex-connector, sonarqubecloud, qodo-code-review, codeant-ai, graphite-app), and_advisory_gate_load_fallback_botshardcodes the same seven. The runtime denominator is 6. One member is silently dropped; which one, and why, is not observable.- Most registered advisory bots never participate. On feat: implement issue #1754 — pr-review counts a human-escalation as a posted review — run reports "1 reviews posted" with nothing on the PR #1793 only three bots posted reviews at all —
codeant-ai,gemini-code-assist, andcoderabbitai— andcoderabbitaiisadvisory_gate=no, so it does not even count. Four of the seven (copilot-pull-request-reviewer, chatgpt-codex-connector, qodo-code-review, graphite-app) contribute nothing. The quorum can never be satisfied on its own terms, so every PR reaches approval only through a timeout fallback. A fallback that fires 100% of the time is the real policy; the quorum is decoration.
To be clear about where this gate ends: check_advisory_reviews records the partial evidence and returns 0 — "Ready to approve." So the advisory gate is not what blocks these PRs. It is working as designed, just always via its escape hatch.
Gate 2 — the disposition gate revokes approvals that do issue
Where an approval did issue, it was taken away:
- feat: implement issue #1568 — status=rate-limited is emitted for non-quota holds (reason=blocked) — retires the token half #461 deliberately deferred #1798:
Dismissing approval due to a PR issue comment lacking a verified disposition (#1813) - feat: link dispatched feature-ideation runs in Idea discussions #1796: 1 dismissal citing the same reason
- feat: implement issue #1754 — pr-review counts a human-escalation as a posted review — run reports "1 reviews posted" with nothing on the PR #1793: 1 dismissal citing the same reason
The maintainer-comment gate (#1290 / #1813) is sound in intent — a maintainer finding posted as a plain PR issue comment creates no review thread, so it neither blocks merge nor reaches dev-lead's fix-reviews prompt, and approving over it ships the defect. The redesign correctly replaced the push-timestamp proxy with a verified disposition.
But it has no path to satisfaction that does not depend on dev-lead running. A disposition is recorded by dev-lead. If dev-lead is suppressed (needs-human-review), rate-limited, cancelled by concurrency, or simply never dispatched for that PR, the comment stays undispositioned forever and the approval can never stand. The gate is fail-closed with no operator override — a maintainer cannot disposition their own comment.
The systemic defect
No single component is wrong. The failure is that nothing owns or reports the end-to-end question "why is this PR not approved?" The advisory gate logs that it is ready to approve. The disposition gate logs a dismissal. Neither knows about the other, and the PR's own surface shows only REVIEW_REQUIRED with no reason. Diagnosing one PR took reading three scripts and correlating GraphQL timeline events by hand.
The consequence is the queue this repo has been carrying for ten days, and it is also what forces every merge through an admin bypass — which discards required_review_thread_resolution along with the approval requirement. A gate chain so strict that the only viable path is bypassing all of it is weaker than a looser chain that actually runs.
Acceptance Criteria
- A single deterministic diagnostic answers, for a given PR, why it is not approved — naming the gate, the specific condition, and what would satisfy it. Surfaced where an operator looks (step summary and/or a PR marker), not only in a run log.
- The advisory denominator is reconciled with the registry:
requiredequals the count ofadvisory_gate=yeslogins, or the discrepancy is explained and asserted by a test.6 != 7must not be silently possible. - Advisory bots that do not participate are handled explicitly — removed from the gate set, or marked non-participating so they are excluded from the denominator rather than waited on and then timed out. A quorum whose fallback fires on every PR is recorded as the policy it actually is.
- The disposition gate gains a satisfaction path that does not require dev-lead: an attributable maintainer disposition (a reply, a label, or a marker a human can post) clears their own comment. Fail-closed is right; unsatisfiable is not.
- A fleet-level metric: open PRs with zero approving reviews, and their age. Ten PRs unapproved for ten days must set off an alarm. Nothing currently measures this — Runaway PR candidate(s) detected 2026-09-20 #1850/Stalled PR candidate(s) detected 2026-09-20 #1851 count runaway and stalled PRs, neither counts unapprovable ones.
- Regression coverage for the end-to-end path: a PR with partial advisory evidence and a dispositioned maintainer comment reaches
APPROVEDwithout an admin bypass.
Notes for the implementer
- Do not "fix" this by loosening gates until approval flows. Each gate encodes a real incident (PR comments from maintainers are mechanically inert — they neither block merge nor reach dev-lead #1290/pr-review can't approve ANY open PR: maintainer-comment gate needs pushedDate (always null) and treats codeant/qodo/auto-rebase comments as maintainer findings — 18/18 PRs stranded #1813 the silent-comment defect, [bug] pr-review: approve only on clean PRs, and minimize superseded approvals as Outdated #1766 approving over unresolved threads, Agent-authored infra PRs are hard to merge — approval-identity + bot-review friction #617 superseded bot reviews). The ask is observability plus a reachable satisfaction path for each, not removal.
- Base-branch merge commits do not dismiss approvals here — verified on feat: implement issue #1766 — [bug] pr-review approves PRs with unresolved threads (violates its own decision gate 4) and double-reviews the same SHA #1770, which took four merge-from-main commits with zero
review_dismissedevents. Do not build on the assumption that rebasing costs approvals; it does not. The dismissals in this repo are all our own (Agent-authored infra PRs are hard to merge — approval-identity + bot-review friction #617 and pr-review can't approve ANY open PR: maintainer-comment gate needs pushedDate (always null) and treats codeant/qodo/auto-rebase comments as maintainer findings — 18/18 PRs stranded #1813). - Start with AC test issue from agent #1. Everything else is easier to size once a single PR can explain itself.
References
- PR comments from maintainers are mechanically inert — they neither block merge nor reach dev-lead #1290 / pr-review can't approve ANY open PR: maintainer-comment gate needs pushedDate (always null) and treats codeant/qodo/auto-rebase comments as maintainer findings — 18/18 PRs stranded #1813 — the maintainer-comment gate and its verified-disposition redesign
- [bug] pr-review: approve only on clean PRs, and minimize superseded approvals as Outdated #1766 / PR feat: implement issue #1766 — [bug] pr-review approves PRs with unresolved threads (violates its own decision gate 4) and double-reviews the same SHA #1770 — approving with unresolved threads (adjacent, still open)
- pr-review counts a human-escalation as a posted review — run reports "1 reviews posted" with nothing on the PR #1754 / PR feat: implement issue #1754 — pr-review counts a human-escalation as a posted review — run reports "1 reviews posted" with nothing on the PR #1793 — human-escalation counted as a posted review
- Agent-authored infra PRs are hard to merge — approval-identity + bot-review friction #617 — superseded-bot-review auto-dismissal
- Merge protections are unsatisfiable at N>1 open PRs: one merge to main forces a full re-review of every other PR — ~154 forced re-reviews in 48h, zero merges #1864 — the merge queue; this issue is why its approval half matters more than its rebase half
scripts/lib/advisory-review-gate.sh,scripts/lib/maintainer-comment-gate.sh,scripts/lib/reviewer-sources.tsv,scripts/lib/partial-evidence-marker.sh
✅ MAINTAINER DECISION 2026-09-23 (revised) — advisory quorum is lowered to 4
Revised 2026-09-23:
graphite-appstays in the gate. The maintainer bought Graphite's paid plan ($40/mo, unlimited reviews) on 2026-09-22. It went silent on 2026-08-21, exactly 30 days after it was installed, which points to a lapsed trial rather than a broken integration. While it was active (Jul 22–Aug 21) it reviewed 267 org PRs and posted 109 findings on 69 of them; about 72% were correct and another 18% partly correct. The earlier version of this decision removed four logins and set the quorum to 3. That is superseded: remove three, and the quorum is 4.
A maintainer has decided AC #3: the advisory quorum becomes 4, by removing the three logins that never participate from the gate set. This supersedes the "removed or marked non-participating" ambiguity in AC #3 — take the removal option.
The evidence this rests on
Reviews submitted across eight recent PRs (#1888, #1887, #1873, #1799, #1796, #1794, #1793, #1770):
| Login | PRs reviewed | advisory_gate today |
|---|---|---|
gemini-code-assist |
8 | yes |
codeant-ai |
8 | yes |
coderabbitai |
5 | no (not part of the gate) |
copilot-pull-request-reviewer |
0 | yes |
chatgpt-codex-connector |
0 | yes |
qodo-code-review |
0 | yes |
graphite-app |
0 | yes |
sonarqubecloud does not appear because it posts a top-level comment, not a review (creates_threads=no in the registry — "Top-level quality-gate comment only"). It does participate; it simply is not visible in the reviews API. It stays in the gate set.
So the advisory sources that actually produce signal are gemini-code-assist, codeant-ai, sonarqubecloud. graphite-app is kept in the gate as a fourth source: it produced 0 reviews on these PRs only because its plan had lapsed, and it is now paid. The quorum is 4.
The change
Set advisory_gate to no for these three rows in scripts/lib/reviewer-sources.tsv:
copilot-pull-request-reviewerchatgpt-codex-connectorqodo-code-review
Do NOT change the graphite-app row. It stays advisory_gate=yes. Update its rationale to record why: it was kept in the gate because it moved to a paid plan on 2026-09-22 after its trial lapsed on 2026-08-21.
Because PR #1902 already made the denominator a projection of the registry (AC #2), this data change alone moves required from 7 to 4. Do not hardcode 4 (or any count) anywhere. Tests should assert that required equals the registry's advisory_gate=yes count, not a literal. #1903 separately adds cubic-dev-ai with advisory_gate=yes, which will take the count to 5 when it merges. A literal would break at that point.
Three places must move together or they drift
This is the exact failure mode the registry exists to prevent — the denominator is now read from the registry in three independent spots:
scripts/lib/reviewer-sources.tsv— the source of truth (the three rows above;graphite-appunchanged)._advisory_gate_load_fallback_botsinscripts/lib/advisory-review-gate.sh— the hardcoded seven-entry fallback map used when the registry cannot be sourced. It keepsgraphite-app.- The hardcoded fallback array in
scripts/lib/approval-diagnostic.sh, which keepsgraphite-app. Currently'["copilot-pull-request-reviewer","gemini-code-assist","chatgpt-codex-connector","sonarqubecloud","qodo-code-review","codeant-ai","graphite-app"]'.
A test must assert all three agree, so a future edit to one cannot silently diverge.
Preserve these
- Do not change
creates_threadsordev_lead_trustedon any row. The [Phase 4] A review bot that can create blocking threads but is not in dev-lead's TRUSTED_BOTS creates an unclearable merge block (graphite-app; root cause of the 2026-08-02 deadlock) #1425 invariant — any row withcreates_threads=yesMUST havedev_lead_trusted=yes— is unrelated toadvisory_gateand must survive untouched. The three removed bots remain trusted for dev-lead and remain on the reviewer scorecard; they are only removed from the approval-wait set. - Keep
sonarqubecloudandgraphite-appatadvisory_gate=yes. - Record in the registry's row rationale why each login was dropped (zero reviews observed across the eight PRs above, 2026-09-23), so a future maintainer re-adding one knows what evidence to re-check.
What this does and does not fix
It makes the gate pass honestly rather than through a 10–20 minute timeout fallback on every single PR, which is a real improvement: faster approvals, and the recorded policy finally matches the behaviour. That holds only if Graphite resumes reviewing now that its plan is active. If Graphite is still silent after the plan change, report it as a Graphite-side problem; do not remove it from the gate again without asking the maintainer.
It does not by itself unblock the queue. check_advisory_reviews already returned 0 ("ready to approve") via those fallbacks — the advisory gate was never the terminal blocker. The approvals that do issue are revoked by the disposition gate (Gate 2 above, #1813), which remains unaddressed. AC #4 is still the one that determines whether approvals start landing.
Implementation note — the quorum change was applied by hand, 2026-09-23 (cf5a8c8b on PR #1902)
The registry change this decision calls for was not picked up by dev-lead across two fix-reviews passes, and the reason is worth recording because it will recur:
The instruction was posted as a PR review body, which creates no review thread. fix-reviews walks reviewThreads, so a review with only a body is invisible to it — the same asymmetry #1290 documents for plain issue comments. dev-lead's 03:00 commit reported "address review comments" and changed approval-diagnostic.sh, review-one-pr.sh and a test (+183/−48 etc.) while never touching reviewer-sources.tsv. A reviewer reading only the commit subject would have believed it applied.
Applied directly instead, in one commit:
advisory_gate=noforcopilot-pull-request-reviewer,chatgpt-codex-connector,qodo-code-review, each with the dropping evidence recorded in its rationale column.graphite-appkept, with its rationale recording why (check-run reporting, lapsed trial, paid plan 2026-09-22).- Both fallback copies brought into line:
_advisory_gate_load_fallback_botsinadvisory-review-gate.shand the array inapproval-diagnostic.sh. - A new test in
tests/test_reviewer_sources.batsasserting the third copy is a registry projection (the first two were already covered by the existing consistency tests).
Verified locally: reviewer_sources_advisory_gate_logins returns exactly codeant-ai, gemini-code-assist, graphite-app, sonarqubecloud — required=4, nothing hardcoded; the new test's logic passes standalone; shellcheck -S warning clean on both edited scripts. The associative-array tests cannot run on macOS bash 3.2 (declare: -A: invalid option) and were left to CI.
Process lesson for this repo: to steer dev-lead on an open PR, the instruction must land somewhere it actually reads — an inline review comment (which creates a thread), or the issue body for an issue-intent run. A review body and a plain PR comment both look authoritative to a human and are invisible to the agent. This is the same class as #1566 (issue comments never reaching the prompt) and is worth folding into whatever AC #4 produces.
Activity
github-actions commentedon Sep 23, 2026
Thank you for this exceptionally thorough write-up — the per-PR measurements, gate-by-gate trace, and incident cross-references make the failure chain immediately actionable. Given the breadth of the six acceptance criteria, one clarifying question: should AC #1 (the end-to-end diagnostic surface) ship as a standalone deliverable first so each subsequent gate fix can be verified against it, or are ACs #2–#4 required in the same batch to make the diagnostic output meaningful?
don-petry commentedon Sep 23, 2026
Dev-Lead Implementation Plan
Issue: #1894 — 10 of 11 open PRs have never received an approving review
Scope (budget-guarded — this PR delivers the keystone only)
Per the issue's own guidance ("Start with AC #1 … everything else is easier to size once a single PR can explain itself"), this PR delivers a single deterministic approval diagnostic (AC #1) plus the advisory-denominator reconciliation it depends on (AC #2). ACs #3–#6 (non-participating-bot policy, a dev-lead-independent disposition satisfaction path, the fleet zero-approval metric, and the end-to-end regression) are each sizeable and are explicitly deferred to follow-up issues once the diagnostic exists — I have permission from the prompt's budget guard to stop after the highest-value unit and file the rest.
Implementation Checklist
scripts/lib/approval-diagnostic.sh— new PURE library.diagnose_approval <pr_snapshot_json> [advisory_json] [approver]evaluates the approval gate chain in the same order review-one-pr.sh applies it and emits ONE structured JSON verdict naming the blocking gate, the specific unmet condition, and what would satisfy it.render_approval_diagnostic <verdict_json>renders a Markdown block for the step summary / a PR marker.advisory.required== count ofadvisory_gate=yeslogins inreviewer-sources.tsv, and it enumerates the specific missing bots by name so6 != 7can never be silent.scripts/review-one-pr.sh— surface the diagnostic to$GITHUB_STEP_SUMMARY(guarded, never breaks the run); name which advisory bots were dropped as unavailable in the gate log..github/workflows/lint.yml— register the new bats test file in therun-bats.shlist.Tests to Write
tests/test_approval_diagnostic.bats— undispositioned maintainer comment → blocking_gate=maintainer-comment-gate with satisfaction path; advisory partial evidence → required==registry count with missing bots named; already-approved → not blocked; changes-requested-at-head; terminal "approval-not-yet-issued".advisory.requiredequals the registryadvisory_gate=yescount (AC Go-live improvements for PR review agent #2, "asserted by a test").Test Command
scripts/run-bats.sh tests/test_approval_diagnostic.bats(full suite via lint.ymlrun-bats.sh), thenbash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Sep 23, 2026
Dev-Lead: Progress (in progress — not yet durable)
Scope delivered this workspace (AC #1 + AC #2 — the keystone)
Per the issue's own guidance ("Start with AC #1 … everything else is easier to size once a single PR can explain itself") and the prompt's budget guard, this workspace delivers the diagnostic and the advisory-denominator reconciliation it depends on. ACs #3–#6 are deferred to follow-ups (see below).
Files touched (this workspace, not yet pushed)
scripts/lib/approval-diagnostic.sh(new): PUREdiagnose_approval <snapshot> [advisory_json] [approver]— evaluates the approval gate chain in the same orderreview-one-pr.shapplies it and emits ONE verdict namingblocking_gate,condition, andsatisfied_by, plus anadvisoryblock withrequired/submitted/missing/via_timeout.render_approval_diagnosticrenders the step-summary Markdown. Fails closed on an unparseable snapshot.scripts/review-one-pr.sh:emit_approval_diagnosticappends the diagnostic to$GITHUB_STEP_SUMMARY(guarded subshell, never affects the review decision), called once CI is green and before the gate chain.scripts/lib/advisory-review-gate.sh: the reduced-denominator log now NAMES which bots were dropped and why (dropped: <bot>=RATE_LIMITED, …), so "6 != 7" is observable at runtime (AC Go-live improvements for PR review agent #2).tests/test_approval_diagnostic.bats(new, 12 cases): blocking-gate identification incl. the pr-review can't approve ANY open PR: maintainer-comment gate needs pushedDate (always null) and treats codeant/qodo/auto-rebase comments as maintainer findings — 18/18 PRs stranded #1813 revocation case (an approval standing at head does not mask an undispositioned comment);advisory.required== registryadvisory_gate=yescount; missing bots named; fail-closed; renderer..github/workflows/lint.yml,tests/dev-lead/unit/test_advisory_review_gate.bats: register the new test; bump the gate line-budget 515→520 with a recorded rationale for the four AC Go-live improvements for PR review agent #2 observability lines.How AC #1 / AC #2 are met
advisory.requiredis derived from the registry (7), the diagnostic enumerates the missing bots by name, and the runtime gate now names dropped bots;test_approval_diagnostic.batsassertsrequiredequals the registryadvisory_gate=yescount so6 != 7cannot be silent.Deferred to follow-ups (not attempted here — budget guard)
Local check output (provisional — not the pushed run)
Note: some
test_review_one_pr_*cases fail identically with and without this change in the local bash-3.2/mocking environment (a known AGENTS.md caveat); CI is authoritative.Notes
The diagnostic is read-only and additive — it introduces no new gate and loosens nothing (per the issue's "do not fix by loosening gates" note). It only makes the existing chain explain itself.
don-petry commentedon Sep 23, 2026
Dev-Lead: Implementation Complete — PR #1902
The implementation for issue #1894 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
9e64baa12aba4b48d9629052e15028c85a01201bmain: 6 file(s), +504/-5 linesReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
feat(#1894): lower the advisory quorum to 4 — drop three non-particip…
fix(#1894): re-apply the quorum-of-4 decision, reverted by b6e4e65, a…
feat: implement issue #1910 — #1894 AC #4: give maintainers a dev-lea…
feat: implement issue #1918 — Approval gate can never clear bot statu…
feat: implement issue #1894 — 10 of 11 open PRs have never received a…
feat: implement issue #1997 — Lower the advisory quorum to 5: drop th…
feat: implement issue #1894 — 10 of 11 open PRs have never received a…
feat(#1894): answer "why isn't this PR approved?" from the real gate …
feat(#1894): ship why-not-approved as an org-wide agent profile, not …
test(#1894): make the diagnostic's read-only assertion able to fail
fix(#1894): diagnose mode refuses every GitHub write; no diagnosis fr…
fix(#1894): lead with the holding gate on an approved PR; keep adviso…
fix(#1894): diagnose-mode gh guard fails closed on unknown subcommands
fix(#1894): an approved PR held by the human escalation leads with th…