Skip to content

dev-lead never dispositions a neutral bot overview (CodeAnt "PR Risk: Low Risk"), but pr-review's comment gate still counts it — the PR can never be approved #2178

Closed

Description

@don-petry

Problem

A finding-bot's clean summary with no finding in it — for example CodeAnt's ## CodeAnt PR Risk: Low Risk … The PR appears safe to merge — holds pr-review's undispositioned-pr-comment gate forever:

  • dev-lead (fix-bot-comment) classifies it as a neutral overview, correctly concludes there is nothing to fix, and posts no disposition reply. prompts/dev-lead/fix-bot-comment.md line 36 says "A neutral overview is not an actionable finding … there is nothing to fix" and does not say to disposition it. The "never leave it undispositioned" rule (dev-lead's own unmarked acknowledgement comments block the approval they exist to unblock — 5 self-inflicted gate blockers on one PR #1919, line 151 onward) covers operational notices (rate-limit, trial-ended), not overviews.
  • pr-review (scripts/lib/maintainer-comment-gate.sh) clears a bot comment only when it is minimized RESOLVED or matches the bot's info_status_pattern in scripts/lib/reviewer-sources.tsv. codeant-ai has no pattern (-), so the overview blocks.
  • scripts/maintainer-resolve-comment.sh refuses it too: a bot comment qualifies only when it matches that bot's info_status_pattern.

So no actor can clear it. This is independent of #2152 (dev-lead: every informational comment disposition fails verification in fleet repos): with that fix live, the gate still does not clear.

Evidence

petry-projects/.github PR #1269 (feat: implement issue #1267), comment IC_kwDORyesfc8AAAABauHIXQ, dev-lead v139-ring0 at 71920a24 (the #2154 fix is present):

Run 38020265393, 2026-10-10 03:2xZ — model output:

Bot: codeant-ai[bot]
Issues addressed: 0
Reason: The comment is a neutral overview summarizing the PR as safe to merge, with no specific, actionable defects …
Status: No-changes — ready for merge

Harness, same run:

##[notice]skipping comment IC_kwDORyesfc8AAAABauHIXQ — no authorized dev-lead disposition reply from don-petry found; leaving open (#1813)
##[notice]minimized comment IC_kwDORyesfc8AAAABaxemig RESOLVED (disposition=informational is_human=false)
##[warning]fix-bot-comment: comment IC_kwDORyesfc8AAAABauHIXQ has no verified disposition after this pass (state: no) — withholding the terminal marker so the bot-comment retry can re-dispatch within its attempt limits (#2017)

The second line shows the #2154 fix working on a different comment in the same pass. The earlier run 38012118914 (before that fix) ended the same way for this comment. Each retry spends a full pass and changes nothing.

The same note blocked petry-projects/.github-private PR #2096 (fix(pr-review): honour DRY_RUN in the advisory-gate marker posts) on 2026-10-09; that PR was merged by admin bypass.

Options

  1. Prompt: in fix-bot-comment.md, a neutral overview with no finding still gets exactly one informational disposition on the original comment, like an operational notice. The harness then minimizes it.
  2. Registry: give codeant-ai an info_status_pattern for its clean risk summary (for example anchored on ^## CodeAnt PR Risk: Low Risk), so the gate auto-clears it and dev-lead never spends a pass on it. Check that the pattern cannot match a summary that also lists findings.

Option 2 removes the wasted passes; option 1 covers overviews from bots with no registered pattern. They are compatible.

Acceptance criteria

  • A codeant-ai comment whose whole body is a "PR Risk: Low Risk … safe to merge" summary does not hold undispositioned-pr-comment after at most one dev-lead pass (or none, with option 2).
  • A CodeAnt comment that carries a finding or a suggestion is still counted and still needs a real disposition.
  • Covered by a test in the gate's and/or the prompt-contract test suites.

Activity

  1. github-actions commented on Oct 10, 2026

    @github-actions
    Contributor

    Thanks for the detailed report and the run evidence, which makes the gap between dev-lead and the pr-review gate easy to follow. Both proposed options look compatible, so before choosing, one question: does CodeAnt's clean "PR Risk: Low Risk" summary always arrive as its own comment, or can the same comment also contain findings or suggestions (for example under the same heading)? That decides whether an anchored info_status_pattern can safely auto-clear it without hiding a real finding.

  2. don-petry commented on Oct 10, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2178 — dev-lead never dispositions a neutral bot overview (CodeAnt "PR Risk: Low Risk"), but pr-review's comment gate still counts it

    Scope

    Option 1 (prompt): fix-bot-comment.md tells dev-lead that a neutral overview posted as a PR issue comment still gets exactly one informational disposition on the original comment. The harness then verifies the disposition and minimizes the comment RESOLVED, which clears undispositioned-pr-comment after one pass. A neutral overview with any finding or suggestion in it does not qualify.

    Option 2 (registry pattern for codeant-ai) is not taken. The registry contract (#1995) and tests/test_reviewer_sources.bats say finding-producing reviewers (codeant-ai) carry -. The info-status rule also needs the bot's complete, fixed notice body anchored at both ends, and CodeAnt's risk summary is model-written bullet prose (see the live body of IC_kwDORyesfc8AAAABauHIXQ). No pattern can guarantee that such a body has no finding in it, so this would fail AC2's "cannot match a summary that also lists findings" check.

    Implementation Checklist

    Tests to Write

    Test Command

    bats tests/dev-lead/unit/test_fix_reviews.bats tests/dev-lead/unit/test_maintainer_comment_gate.bats tests/dev-lead/unit/test_agent_marker_convention.bats tests/test_reviewer_sources.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Oct 10, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2192

    The implementation for issue #2178 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  4. added a commit that references this issue on Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions