Skip to content

dev-lead fix-bot reverts its own fix on neutral Copilot 'overview' comments → net no-op PRs that falsely close compliance issues #1340

Closed

Description

@don-petry

Summary

The dev-lead fix-bot step misinterprets a neutral Copilot "pull request overview" comment (state COMMENTED, merely describing the diff) as a change-request, and reverts the exact lines its own feat commit added. The base…head compare then returns zero changed files. Because the PR body says Closes #N, merging the no-op would auto-close the compliance issue while the drift/finding remains unfixed — the idempotent audit then re-opens it, and the cycle repeats.

Confirmed evidence (found during a 50-cap open-PR drain, 2026-07-20)

In both cases pr-review's cascade caught the no-op (MAJOR finding), but the PRs then sat at fix-requested indefinitely because dev-lead's fix pass reads only reviewThreads (see #1290) and found nothing actionable.

Root cause

fix-bot treats any reviewer comment referencing a removed/added block as an instruction to restore/revert it, without checking the reviewer's review state (a COMMENTED overview is not CHANGES_REQUESTED) or whether reverting nets the diff to zero.

Proposed fixes

  1. Never act on COMMENTED/overview reviews as change-requests — only CHANGES_REQUESTED (or explicit actionable thread comments).
  2. No-op guard: after fix-bot runs, if base…head diff is empty, abort and flag rather than pushing — a PR that nets to zero must never carry Closes #N.
  3. Coordinate with chore(deps): bump the actions group across 1 directory with 5 updates #806 (thread resolution) and PR comments from maintainers are mechanically inert — they neither block merge nor reach dev-lead #1290 (comment inertness) — same fix-loop subsystem.

Impact

Systemic: this is a likely contributor to the broad fix-requested PR backlog that fills the 50-PR automation cap. Closing the no-op PRs is only a temporary drain — regeneration will reproduce them until fix-bot stops self-cancelling.

Activity

  1. github-actions commented on Jul 21, 2026

    @github-actions
    Contributor

    Thanks for the thorough write-up and the concrete PR evidence — this is a well-documented root cause report. To help us reproduce and patch this reliably: can you share the exact condition in fix-bot's comment-parsing logic that causes it to treat a COMMENTED-state overview as a change-request (e.g., a specific field check or regex match in the script)?

  2. added
    dev-leadFor dev-lead agent pickup
    and removed
    needs-human-reviewFlagged by automated PR review agent
    on Jul 21, 2026
  3. don-petry commented on Jul 21, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1340 — dev-lead fix-bot reverts its own fix on neutral Copilot 'overview' comments → net no-op PRs that falsely close compliance issues

    Scope

    Stop the dev-lead fix pass from self-cancelling a PR: (1) never treat a neutral COMMENTED/overview review as a change-request, and (2) add a script-level no-op guard that refuses to push a fix whose net base…head diff is empty, flagging it for a human instead.

    Implementation Checklist

    • scripts/dev-lead-fix-reviews.sh — add pr_nets_to_zero helper (empty net origin/<base>...HEAD diff = net-zero)
    • scripts/dev-lead-fix-reviews.sh — add flag_noop_pr helper (deduped human-attention comment + needs-human-review label + disable auto-merge + suppress the EXIT-trap auto-merge restore)
    • scripts/dev-lead-fix-reviews.sh — wire the guard into the active commit_and_push for fix-reviews/fix-bot-comment intents: after commit, before push, abort+flag on net-zero (return 3)
    • scripts/dev-lead-fix-reviews.sh — teach the fix-reviews and fix-bot-comment call sites to handle rc=3 (no applied/no-changes/retry marker, no auto-merge re-enable)
    • prompts/dev-lead/fix-reviews.md — Phase-0 + constraint: a COMMENTED/overview review is neutral, never a change-request; never revert the PR's own changes to satisfy it
    • prompts/dev-lead/fix-bot-comment.md — same guidance: a neutral overview/summary is not an actionable finding

    Tests to Write

    • no-op guard: fix-reviews net-zero diff flags PR (needs-human comment + disable-auto) and does NOT push
    • no-op guard: fix-bot-comment net-zero diff flags and does NOT push
    • no-op guard: non-empty net diff pushes normally and posts applied (guard does not fire)
    • no-op guard: net-zero path does not re-enable auto-merge
    • prompt: fix-reviews.md instructs not to treat COMMENTED/overview reviews as change-requests
    • prompt: fix-bot-comment.md states a neutral overview is not an actionable finding

    Test Command

    bats tests/dev-lead/unit/test_fix_reviews.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  4. don-petry commented on Jul 21, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: issue #1340 implemented — no-op guard + neutral-overview prompt guidance

    Problem. The fix-bot/fix-reviews pass could misread a neutral Copilot "pull request overview" (review state = COMMENTED, which only describes the diff) as a change-request and revert the exact lines its own feat commit added. That nets the origin/<base>...HEAD diff to zero. Because the PR body says Closes #N, merging that no-op auto-closes the compliance issue while the finding stays unfixed (the idempotent audit then re-opens it) — the fix silently cancels itself.

    Fix — two layers (defense in depth):

    1. Prevention (prompts). Both prompts/dev-lead/fix-reviews.md and prompts/dev-lead/fix-bot-comment.md now state that a COMMENTED/"pull request overview" review is neutral and must never be treated as a change-request — the agent must not revert/undo its own committed changes to "address" it. Added both as an inline blockquote (right after the Tier-1/CHANGES_REQUESTED guidance) and a Constraints bullet, each citing #1340.

    2. Backstop (code no-op guard). scripts/dev-lead-fix-reviews.sh:

    • pr_nets_to_zero [base] — returns true when git diff --name-only origin/<base>...HEAD is empty. Fails safe: if the base ref can't be resolved (no ref, fetch fails, no common ancestor) it warns and returns false, so an unverifiable state never blocks a legitimate push.
    • flag_noop_pr <intent> — posts one deduped needs-human-review comment (marker-keyed per PR+intent), adds the label, and runs gh pr merge --disable-auto. Sets _AM_NEEDS_RESTORE=0 so the EXIT-trap auto-merge restore can't silently re-enable a self-cancelling PR. Honors DEV_LEAD_DRY_RUN.
    • Wired into commit_and_push after the commit and before push_no_clobber, scoped to fix-reviews|fix-bot-comment only. On net-zero it flags and returns new code 3 (0=pushed, 1=nothing to push, 3=no-op aborted). Both call sites branch on cp_rc==3: emit a warning, post no applied/no-changes marker, and skip thread-resolution + try_enable_auto_merge.

    Tests (TDD, written first). Added 6 tests to tests/dev-lead/unit/test_fix_reviews.bats:

    Verification.

    • bats tests/dev-lead/unit/test_fix_reviews.bats → 93/93 pass
    • Full bats tests/dev-lead/unit/ → only pre-existing failure #260 (REVIEW_MCP_DEBUG, unrelated — confirmed failing on a clean tree via git stash); zero regressions from this change
    • shellcheck scripts/dev-lead-fix-reviews.sh → clean (only pre-existing SC1091 info)
    • dev-lead-lint.sh → passed

    Files changed: scripts/dev-lead-fix-reviews.sh, prompts/dev-lead/fix-reviews.md, prompts/dev-lead/fix-bot-comment.md, tests/dev-lead/unit/test_fix_reviews.bats.

    No commit/push performed — CI handles git operations.

  5. added a commit that references this issue on Jul 21, 2026
  6. added 13 commits that reference this issue on Aug 2, 2026
  7. added 2 commits that reference this issue on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions