Skip to content

Harness cannot resolve a bot review thread whose correct response is 'no change needed' — false positives block merge permanently #1743

Closed

Description

@don-petry

Summary

A bot review thread whose correct response is "no change needed" can never be resolved by the harness, so it blocks merge forever under required_review_thread_resolution.

Confirmed live on PR #1710 (ADR-0007), which sat blocked for over four hours in exactly this state.

The deadlock

Two rules are individually correct and jointly form a trap:

  1. Agents must not resolve threads themselves. prompts/dev-lead/fix-reviews.md, fix-bot-comment.md and review-changes.md all forbid calling resolveReviewThread "under any circumstance" — resolution is the harness's job (dev-lead-fix-reviews.sh). Correct: it stops an agent dismissing feedback it did not address.
  2. The harness only resolves on a commit-producing pass. Per the Follow-up to #1609 (slice 2): gate dev-lead thread resolution on the pass having advanced the PR head #1617 gate, "a pass that advances the PR head triggers resolution; a no-commit pass resolves nothing."

When a bot finding is a false positive, the correct engineering response produces no commit. Rule 2 therefore withholds resolution, and rule 1 forbids anyone in the loop from resolving it. The thread is structurally unreachable, and merge is blocked on a finding that is wrong.

The gap is not "agents should be allowed to resolve threads" — rule 1 is right. It is that "no change needed" is not an expressible disposition anywhere in the pipeline.

Evidence — PR #1710

  • Blocking thread: gemini-code-assist on docs/architecture/adr/0007-one-agent-ingress-stub-per-repo.md:13, low, claiming the inline code span for template_stub_drift.sh is missing a closing backtick.
  • The finding is false: line 13 carries 4 backticks (two balanced pairs); lines 9–17 carry 12 (balanced); markdownlint-cli2 reports 0 errors; the rendered ADR shows both filenames as inline code with following prose rendering normally.
  • dev-lead was triggered and behaved correctly — it did not alter correct text to appease the finding:
    <!-- dev-lead-fix-reviews pr=1710 sha=3a534433 intent=fix-bot-comment status=no-changes -->
    Issues addressed: 0
    Files changed: none
    
  • PR state throughout: MERGEABLE, donpetry-bot APPROVED, zero failing checks, zero pending checks. Sole blocker: one unresolved thread.

Secondary defect found in the same run — intent mis-routing

The @dev-lead mention explicitly asked for a fix-reviews pass on the outstanding gemini review thread. The pass ran as intent=fix-bot-comment and addressed the newest bot issue comment (SonarCloud's quality-gate status, logged as "Skipped (informational): 1"). It never examined the blocking thread.

So the mention path cannot currently be aimed at a specific unresolved review thread — the intent classifier routes on comment recency/type, not on what is actually blocking the merge. Worth splitting out if it is a separate concern.

Note on #1692 / #1621 S2

The maintainer-disposition work that landed in 8e951e7 is the opposite control: acv_latest_maintainer_disposition matches REQUIRED|MUST BE FIXED|BLOCKING|REQUESTING CHANGES and withholds resolution until a verified fix postdates it. It gives a maintainer a way to say "this MUST be fixed". There is no counterpart for "this is a false positive, no change is coming" — which is the missing half.

Acceptance criteria

  1. A false-positive bot thread on an otherwise-green PR can reach resolution without fabricating a commit, and without an agent calling resolveReviewThread to dismiss feedback it did not address.
  2. The disposition is explicit and attributable — whoever asserts "no change needed" is recorded, by the same standard acv_latest_maintainer_disposition applies to the blocking direction. A silent auto-resolve is not an acceptable fix.
  3. The no-changes pass outcome carries that disposition through to the harness, rather than terminating with the thread untouched as it does today.
  4. Rules 1 and 2 above are preserved for the normal case: an agent still cannot resolve a thread it merely disagrees with, and a genuine fix still requires a commit.
  5. Regression coverage: a bats case for a no-changes pass on a false-positive bot thread, asserting the thread reaches resolution; and a case asserting a maintainer thread and a REQUIRED disposition still block, per [Phase 4] Close the maintainer review-gate gap — dev-lead PRs cannot be blocked by the owner (shared identity + agent-resolved threads) #1415 and [#1621 S2] Verify the addressed-marker against the pushed diff + honour maintainer dispositions #1692 AC4.

Activity

  1. github-actions commented on Sep 8, 2026

    @github-actions
    Contributor

    Thank you for this exceptionally thorough write-up — the deadlock analysis, live evidence from PR #1710, and the acceptance criteria all make the core defect very clear. One question before we scope the fix: you flagged the intent mis-routing (the @dev-lead mention running as fix-bot-comment instead of targeting the blocking review thread) as a secondary defect and noted it may be worth splitting out — would you like that tracked as a separate issue, or should it be part of the acceptance criteria here so both gaps are closed in the same pass?

  2. don-petry commented on Sep 13, 2026

    @don-petry
    CollaboratorAuthor

    Second failure mode: maintainer steering on dev-lead's own PRs is dropped as self-review

    Found 2026-09-13 while trying to get dev-lead to address real (not false-positive) review findings on PR #1772 — [Phase 2] Make caller_stub_freeze and validate-caller-inputs ingress-aware (#1725). This compounds the deadlock above: it's not just that "no change needed" can't resolve a thread — a maintainer also cannot direct dev-lead to look at a thread at all on a PR dev-lead authored.

    What happened

    1. PR feat: implement issue #1725 — [Phase 2] Make caller_stub_freeze and validate-caller-inputs ingress-aware (job-scoped freeze + if:-as-event-filter check) #1772 had 5 unresolved bot review threads, including two verified-real Major defects (an if:-filter bypass via indexed access in validate-ingress-if.sh, and caller_stub_freeze.sh yielding a non-empty extraction for an absent job).
    2. dev-lead had run three passes, all intent=fix-bot-comment status=no-changes. Its reasoning shows it evaluated SonarCloud's quality-gate comment and described coderabbitai's review as "DISMISSED (review was dismissed, comments not active)" — while that review's inline threads were still unresolved and blocking the merge. Dismissing a review hid its live threads from dev-lead.
    3. I posted maintainer REQUIRED replies on the two defect threads, the [#1621 S2] Verify the addressed-marker against the pushed diff + honour maintainer dispositions #1692 disposition mechanism.
    4. dev-lead classified the resulting events as a skip — run 34735788836:
      [intent] type=skip reason=self-review context={"pr_number":1772}
      
      dev-lead commits and opens PRs as don-petry, and the maintainer replying is also don-petry, so the classifier cannot tell a maintainer's steering from dev-lead talking to itself.
    5. The only reason a fix-reviews pass started at all: coderabbitai auto-replied to my comment ("Use this command on a human-authored review finding…"), and that bot event routed to fix-reviews. The steering reached dev-lead only by accident, through a third-party bot's boilerplate.
    6. A sanctioned dev-lead-reviews-retry dispatch (intent_type=fix-reviews) sent at the same time was cancelled in the dev-lead-pr-1772 concurrency group (cancel-in-progress: false, so only the newest pending run survives) when coderabbitai's second auto-reply queued behind it. Explicit retry is easily displaced by incidental bot chatter.

    Why it belongs with this issue

    Taken together, the two failure modes mean that on a dev-lead-authored PR:

    Situation What a maintainer can do today
    Bot finding is a false positive Nothing automated can resolve it (#1617 no-commit gate)
    Bot finding is real but dev-lead mis-routes past it Replies are dropped as self-review; review-dismissal hides the threads

    A human's only levers are resolving threads by hand or hoping a bot's auto-reply wakes the right intent.

    Additional acceptance criteria

    1. A marker-less maintainer comment or reply on a dev-lead-authored PR is not classified self-review. The self-review guard must key on dev-lead's own markers or the harness's identity, not on the shared account login, since don-petry is both the committer and a maintainer.
    2. A dismissed review's still-unresolved inline threads are visible to fix-reviews. Dismissing the review body must not remove live blocking threads from the agent's input.
    3. Pass selection prefers unresolved blocking review threads over newer informational bot issue comments (SonarCloud status, rate-limit notices). A fix-bot-comment pass on a PR with unresolved blocking threads should at least surface that those threads exist.
    4. A dev-lead-reviews-retry dispatch is not silently displaced by an incidental bot event in the same concurrency group — or the displacement is logged with a reason, so an operator can tell an explicit retry was dropped.
  3. don-petry commented on Sep 13, 2026

    @don-petry
    CollaboratorAuthor

    Confirmed live today — I hit this twice in one session. Releasing.

    Re-verified 2026-09-13. Two independent instances this morning, both of which I had to break by hand as a maintainer:

    1. .github-private#1730 — one thread on scripts/lib/shadow-suppress.sh:47. codeant-ai raised it, dev-lead correctly refuted it as out of scope for an inert split, and codeant-ai itself acknowledged ("Customized review instruction saved!"). Correct outcome: no change needed. The thread stayed unresolved from 2026-09-08 until I resolved it manually today — five days blocking merge on a settled disagreement.
    2. .github#1084 — the attribution finding on scripts/agents-md-cycle-log.sh:109, which I refuted as working-as-designed. No code change will ever come, so nothing can ever resolve it. I resolved it by hand.

    This is exactly the deadlock in the summary: agents are forbidden from calling resolveReviewThread (fix-reviews.md, fix-bot-comment.md, review-changes.md all say "under any circumstance"), and the harness resolves only on evidence of a pushed diff. A thread whose correct answer is "no change needed" produces no diff, so it has no path to resolution under required_review_thread_resolution: true.

    Both rules are right. #1691/#1699 removed the model's self-resolution path for good reason — this workstream has a documented history of dev-lead falsely resolving threads, once resolving all 18 on a PR including a Critical syntax break with no commit at all. The fix is not to give the model the mutation back.

    The missing case is a maintainer-attested refutation. #1692/#1704 taught the harness to verify an addressed-marker against the pushed diff. What it needs alongside that is a second disposition — refuted, requiring a human or an explicit maintainer disposition comment, never the authoring agent's own say-so — which resolves without a diff.

    Acceptance criteria

    Also clearing needs-human-review: applied by issue-triage at 2026-09-08T17:29, at filing time (#1778).

  4. don-petry commented on Sep 13, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1743 — Harness cannot resolve a bot review thread whose correct response is 'no change needed'

    Scope

    Add the missing counterpart to acv_latest_maintainer_disposition: an explicit, attributable "no change needed" maintainer disposition that lets a false-positive bot thread reach resolution on a no-commit pass — without an agent ever calling resolveReviewThread and without fabricating a commit.

    Design

    Symmetric to the existing blocking control. The blocking gate (acv_latest_maintainer_disposition, REQUIRED|MUST BE FIXED|…) withholds resolution until a fix postdates a maintainer's assertion. The new control recognizes a marker-less human maintainer comment asserting "no change needed / false positive / working as intended" and permits resolution of a bot thread even when the pass produced no commit. A later REQUIRED disposition still supersedes it (preserves #1415/#1692). The agent's own opinion never resolves anything — only the attributable human disposition does (preserves rules 1 & 2).

    Implementation Checklist

    • scripts/lib/addressed-claim-verify.sh — add _ACV_NOCHANGE_RE_UPPER regex + pure acv_latest_nochange_disposition() (mirror of acv_latest_maintainer_disposition: returns rc0+latest ISO date / rc1 none / rc2 unparseable).
    • scripts/dev-lead-fix-reviews.sh — add resolve_nochange_disposition_threads() net: enumerate unresolved bot-originated threads, resolve those carrying a no-change disposition not superseded by a later REQUIRED one. Runs regardless of the Follow-up to #1609 (slice 2): gate dev-lead thread resolution on the pass having advanced the PR head #1617 resolution gate (its authorization is the human disposition, not head movement).
    • scripts/dev-lead-fix-reviews.sh — wire the net into the fix-reviews, fix-bot-comment, and review-changes intent branches so the no-commit path carries the disposition through to resolution instead of dead-ending.

    Tests to Write

    Test Command

    bats tests/dev-lead/unit/ (dev-lead unit suite); plus bash .dev-lead/scripts/dev-lead-lint.sh

    Out of scope

    The secondary intent-mis-routing defect (mention path routing fix-bot-comment vs targeting a specific blocking review thread) is explicitly flagged in the issue as "worth splitting out if it is a separate concern"; it is not part of AC 1–5 and is left for a separate issue.

  5. don-petry commented on Sep 13, 2026

    @don-petry
    CollaboratorAuthor

    AC9 reproduced three times on 2026-09-13 — bot chatter cancels real work under runner saturation

    Concurrency group dev-lead-pr-<n> uses cancel-in-progress: false, so a PR gets one running dev-lead run and only the newest pending run survives. With runners saturated (at 04:43Z dev-lead.yml had 18 queued + 19 pending runs fleet-wide), a no-op run can hold the running slot for 10+ minutes. During that window, any later event on the PR replaces whatever real work is pending.

    # PR Real work that was cancelled Displaced by
    1 #1789 auto-rebase's <!-- auto-rebase-conflict: --> sentinel comment (04:28:56) → rebase intent run 34737987378, cancelled the review-changes pass and its thread-reply events
    2 #1772 my dev-lead-reviews-retry dispatch, intent_type=fix-reviews (03:35:04) → run 34735847428, cancelled coderabbitai's boilerplate reply to a maintainer comment (03:37:01)
    3 #1789 my dev-lead-reviews-retry dispatch, intent_type=rebase (04:38:05) → run 34738346802, cancelled at 04:42:18 pr-review-agent's <!-- pr-review-agent ci-pending-ack --> comment (04:42:14) → issue_comment run 34738509837

    In case 3, the run holding the slot the whole time was 34738175535. It was triggered by dev-lead's own dev-lead-fix-reviews … status=applied marker comment (04:33:39), which the classifier skips as a self-comment, and it sat queued waiting for a runner from 04:33:39 onward.

    So a PR with a merge conflict cannot get its rebase: the designed sentinel path and the sanctioned retry were both cancelled by automated acknowledgement comments, while the slot was held by a run that does nothing.

    Additional acceptance criteria

    1. Events that can only ever classify as skip (self-comments, pr-review-agent ack/rate-limit markers, known bot boilerplate) are filtered before a run is queued, for example with a job-level if: on the caller. They must not enter the dev-lead-pr-<n> concurrency group, where they displace real work.
    2. rebase (auto-rebase conflict sentinel or retry dispatch) and fix-reviews retries are not displaceable by lower-value events: separate concurrency lanes, priority, or a pending-displacement guard.
  6. don-petry commented on Sep 13, 2026

    @don-petry
    CollaboratorAuthor

    In progress — no-change disposition net (provisional)

    Implementation is drafted locally on the working branch (not yet committed/pushed). Summary of the approach, matching the plan posted earlier:

    • Pure verifier — added acv_latest_nochange_disposition() to scripts/lib/addressed-claim-verify.sh, the mirror of the existing acv_latest_maintainer_disposition. It recognises an explicit, attributable, marker-less human "no change needed / false positive / working as intended / by design" disposition and returns the latest such timestamp (fail-closed on an unparseable date). Governed by the same standard as the REQUIRED counterpart, so the agent cannot manufacture its own authorization.
    • Harness net — added resolve_nochange_disposition_threads() to scripts/dev-lead-fix-reviews.sh, modelled on resolve_addressed_bot_threads (paginated bot-thread enumeration + per-candidate re-fetch of full comments, fail-closed isResolved). It resolves a bot-originated thread carrying a no-change disposition unless a strictly-newer REQUIRED disposition supersedes it. Wired into the fix-reviews, fix-bot-comment, and review-changes branches to run regardless of the Follow-up to #1609 (slice 2): gate dev-lead thread resolution on the pass having advanced the PR head #1617 resolution gate — its authorization is the human disposition, not head movement — so a genuine false positive (a no-commit pass) is carried through to resolution instead of dead-ending.
    • Scope guards preserved: only bot-originated threads are touched (maintainer threads never auto-resolved, [Phase 4] Close the maintainer review-gate gap — dev-lead PRs cannot be blocked by the owner (shared identity + agent-resolved threads) #1415); the agent still never calls resolveReviewThread itself.

    Regression coverage added: unit tests for the new verifier and harness tests for (a) a false-positive bot thread with a maintainer no-change disposition resolving on a no-commit pass, and (b) a bot thread with only a REQUIRED disposition, a maintainer-originated thread, and a REQUIRED-supersedes-no-change case all still blocking.

    Verification pending final push; will update once the durable record is available.

  7. don-petry commented on Sep 13, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #1799

    The implementation for issue #1743 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  8. added a commit that references this issue on Sep 21, 2026
  9. don-petry commented on Oct 21, 2026

    @don-petry
    CollaboratorAuthor

    Closing after the 2026-10-21 backlog triage.

    Why: Fixed: harness resolves bot thread on maintainer no-change disposition.
    Evidence: PRs #2084 (#2079), #2239 (#2161) merged 2026-10-10/11

    Reopen if this still reproduces on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reports

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions