Skip to content

dev-lead: unprocessed bot review threads are never retried — review-thread counterpart of #2017 #2046

Closed

Description

@don-petry

Summary

Bot review threads that dev-lead never processes are never retried. They sit with no reply and no resolution, and under required_review_thread_resolution the PR cannot merge. This is the review-thread counterpart of #2017, whose fix (#2022) covers only undispositioned PR issue comments.

Evidence — PR #1953 (as of 2026-10-03 13:45Z)

Eleven unresolved bot threads have no reply at all, though both bots are trusted (dev_lead_trusted=yes in scripts/lib/reviewer-sources.tsv):

Reviewer Posted Threads (file:line)
cubic-dev-ai 2026-10-01 11:35Z canary_report.sh 121, 659, 220; tests/canary_report.bats 199
chatgpt-codex-connector 2026-10-02 22:49–23:19Z canary_report.sh 195, 192, 314, 619
chatgpt-codex-connector 2026-10-03 13:37Z canary_report.sh 475, 226; token-report.yml 50

dev-lead's pass history on the PR shows why they were missed:

  • Every pass after the cubic review (2026-10-01 11:35Z through 23:14Z) ran with intent=fix-bot-comment, the PR issue-comment path. None was a review-thread pass.
  • The only intent=fix-reviews pass (2026-10-02 23:10Z, sha 571cc02) revisited just the five older, already-deferred Codex threads and reported status=not-applied. It never picked up the cubic threads or the newer Codex threads.

The job logs were not available from the investigating session, so the exact reason is unconfirmed. Possible causes:

Proposed fix

Extend the #2017 retry scan (scripts/dev-lead-retry.sh / scripts/lib/bot-comment-retry.sh) to review threads:

  1. For each open PR, find unresolved, non-outdated, bot-originated review threads from a trusted reviewer source whose originating comment has no our-account reply, after the same grace period the issue-comment scan uses.
  2. Dispatch one deduplicated fix-reviews pass per PR covering those thread ids. Reuse the retry-marker claim, attempt caps (per rate-limit window and in total) and rate-limit awareness from feat: implement issue #2017 — A lost fix-bot-comment run is never retried — a bot comment stays undispositioned and the PR stalls at the maintainer-comment gate forever (#2009) #2022.
  3. When attempts are exhausted, apply needs-human-review or post a notice so the stall is visible. This was also raised as a non-blocking note on feat: implement issue #2017 — A lost fix-bot-comment run is never retried — a bot comment stays undispositioned and the PR stalls at the maintainer-comment gate forever (#2009) #2022.
  4. Also check whether the fix-reviews thread query enumerates all unresolved bot threads, not only those from the triggering reviewer or review.

Acceptance criteria

  • An unreplied, unresolved bot thread older than the grace period triggers exactly one retry fix-reviews pass. The pass is deduplicated across the cron scan and pr-review's gate hook.
  • Threads with an our-account reply, and outdated or maintainer threads, are not retried.
  • The attempt caps are honoured, and exhaustion is surfaced (label or notice), not silent.
  • A fix-reviews pass sees every unresolved trusted-bot thread on the PR. A regression test covers threads from multiple reviewers and from reviews older than the triggering one.
  • bats coverage for the decision rules.

Generated by Claude Code

Activity

  1. don-petry commented on Oct 3, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2046 — dev-lead: unprocessed bot review threads are never retried — review-thread counterpart of #2017

    Scope

    Extend the #2017 retry sweep to unreplied trusted-bot review threads (one deduplicated fix-reviews retry per PR, capped, exhaustion surfaced by a notice), and make every fix-reviews / review-changes pass enumerate all unresolved threads.

    Likely root cause for AC4

    dev-lead-fix-reviews.sh builds OPEN_THREADS_JSON from reviewThreads(first:50) without pagination. GitHub returns threads oldest first, so on a PR with more than 50 threads (PR #1953) a pass only sees the oldest page. On #1953 that page held the five already-deferred Codex threads, and the cubic and newer Codex threads never reached the engine. This matches the status=not-applied pass on sha 571cc02.

    Implementation Checklist

    • scripts/lib/open-review-threads.sh (new): fetch_open_review_threads <repo> <pr>. Paginated (first:100, $endCursor) and not filtered by reviewer. Returns every unresolved thread.
    • scripts/dev-lead-fix-reviews.sh: the fix-reviews and review-changes branches both use the helper instead of the single 50-thread page.
    • scripts/lib/bot-thread-retry.sh (new):
      • pure btr_retry_decisions with per-thread reasons: replied, grace-period, retry-pending, rate-limited, retry-attempts-exhausted, dispatch;
      • paginated thread fetch;
      • retry and exhaustion marker builders.
      • Candidates are unresolved, non-outdated threads whose originating comment is a Bot from a trusted reviewer source. Maintainer, outdated and resolved threads are never candidates. The grace period, pending window, attempt caps and rate-limit awareness mirror bot-comment-retry.sh, and only our own automation's markers count.
    • scripts/dev-lead-retry.sh: scan_pr_for_unreplied_bot_threads.
      • Gates: open PR, dev-lead authored, same-repo head, pr_resume_suppressed.
      • Posts the claim marker before dispatch, then settles, re-lists and keeps the earliest marker.
      • Sends one dev-lead-reviews-retry intent_type=fix-reviews dispatch per PR.
      • Posts a single deduplicated exhaustion notice once a thread hits the cap.
      • Wired into scan_repo after the bot-comment scan, keeping one dispatch per PR per scan.
    • scripts/review-one-pr.sh: the gate hook runs the thread scan when the bot-comment scan dispatched nothing. Both callers share the same marker claim, so the dispatch is deduplicated.
    • tests/dev-lead/integration/test_prompt_coverage.sh: point the __typename selection check at the new helper, where the query now lives.

    Tests to Write

    • tests/dev-lead/unit/test_bot_thread_retry.bats, decision matrix:
      • an unreplied old bot thread dispatches;
      • an our-account reply, an outdated thread, a resolved thread, a maintainer thread and an untrusted bot are each excluded;
      • grace period, pending, rate-limited and both attempt caps are honoured;
      • exhaustion is reported;
      • forged markers are ignored.
    • Sweep wiring:
      • exactly one fix-reviews dispatch;
      • the marker is posted before the dispatch;
      • a concurrent claim backs off;
      • exhaustion posts one notice, and no duplicate notice follows.
    • fetch_open_review_threads regression: a two-page response with threads from multiple reviewers and older reviews returns every unresolved thread, and the query paginates.

    Test Command

    bats tests/dev-lead/unit/ + bash tests/dev-lead/integration/test_prompt_coverage.sh + bash .dev-lead/scripts/dev-lead-lint.sh

  2. don-petry commented on Oct 3, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2048

    The implementation for issue #2046 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  3. don-petry commented on Oct 4, 2026

    @don-petry
    CollaboratorAuthor

    Root cause found for one cause of this pattern: dev-lead's fix-reviews pass reads reviewThreads(first:50) with no pagination, so on PR #1953 (66 threads) the single open thread was invisible — filed as #2056.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions