Skip to content

Follow-up to #1567: threads resolved with NO commit at all — the shipped evidence bar gates status=applied, not thread resolution (near-miss: fleet-breaking merge on .github#1024) #1609

Closed

Description

@don-petry

RE-SCOPED 2026-08-31 by the maintainer after the timeout escalation (run 33433995544, 2701s vs a 2700s budget).
Per dev-lead's own escalation guidance, this issue is now slice 1 of 2 and covers only the pure helper library. The wiring into dev-lead-fix-reviews.sh moved to the follow-up issue linked at the bottom. Do not modify scripts/dev-lead-fix-reviews.sh in this issue.
This is the same re-scoping that unblocked #1589 after an identical timeout.

SCOPE OF THIS ISSUE (slice 1) — the pure library only

Add scripts/lib/resolution-integrity.sh, a pure, side-effect-free helper. No network calls, no git invocations, no gh invocations, no sourcing of other dev-lead libs.

Required function:

ri_may_resolve <sha_before> <sha_after>
  • Returns 0 (may resolve) iff both arguments are non-empty AND they differ.
  • Returns non-zero (must not resolve) in every other case — including either argument empty, both empty, or the two equal. Fail closed.
  • No output on stdout other than what the tests below assert; no exit (it is sourced, so return).

Acceptance criteria

  • scripts/lib/resolution-integrity.sh exists, is shellcheck-clean, and is safe to source under set -euo pipefail.
  • ri_may_resolve implements exactly the truth table above.
  • Unit tests in tests/dev-lead/unit/test_resolution_integrity.bats covering, at minimum: both empty → deny; before empty → deny; after empty → deny; equal SHAs → deny; differing SHAs → allow.
  • The file declares each function exactly once (grep -c '^ri_may_resolve()' → 1). This issue exists because of the #1485 duplicate-declaration class; do not reintroduce it.
  • No other file is changed. In particular scripts/dev-lead-fix-reviews.sh is untouched.

Keeping the helper pure is deliberate: it is the part that can be exhaustively unit-tested without a PR fixture, and it is what makes slice 2 small.


Why this is needed — the incident record (context, not scope)

Third occurrence — and this one defeated the gate on a fleet-breaking change

The fix for this class merged at 13:34Z (#1604 → 9b3bc2b0) and a more severe instance occurred at 14:03Z, on petry-projects/.github#1024.

Between 14:00Z and 14:03Z, all 18 open threads on #1024 went to resolved — clean thread surface, REVIEW_REQUIRED, one approval from merge. No commit was made: head unchanged at 30375c5c, verified at that SHA via the contents API:

bash -n scripts/apply-repo-settings.sh   → line 498: syntax error near unexpected token `}'
grep -c '^_watched_paths()'         canary-rollout.sh   → 2
grep -c '^ensure_required_labels()' compliance-audit.sh → 3

The resolved set included a 🔴 Critical thread on the unmatched brace. All 28 threads were reopened; the merge is re-blocked and dev-lead:hands-off is applied.

Severity is materially higher than the earlier cases

The two occurrences on #1021 cost review cycles. This one would have merged:

  • scripts/apply-repo-settings.sh unparseable — it drives the fleet-wide apply-repo-settings workflow delivered hours earlier by #984/#985. Every scheduled run org-wide would fail at parse.
  • The #1485 trunk-corruption class into petry-projects/.github — the one org-infra repo with no duplicate-decl-gate (petry-projects/.github#1025).

The gap in the shipped fix (root cause, confirmed)

#1604/#1567 moved the bar to "the commit is non-trivial AND touches the regions the review named" — but that is an evidence check on the status=applied claim only (scripts/lib/review-change-evidence.sh, finalize_review_application).

It does not gate thread resolution. In scripts/dev-lead-fix-reviews.sh, the three resolution nets — resolve_bot_outdated_threads, resolve_actor_outdated_threads, and resolve_addressed_bot_threads — run in the no-changes path as well as the applied path (fix-reviews, fix-bot-comment, review-changes). resolve_addressed_bot_threads (#1547) resolves a non-outdated bot thread purely on a reply marker with no commit required — the exact vector that resolved the Critical brace thread.

This incident produced no commit and no status=applied claim at all; the threads were simply marked resolved.


Follow-up slices (do NOT implement here)

  • Slice 2 — wiring (Follow-up to #1609 (slice 2): gate dev-lead thread resolution on the pass having advanced the PR head #1617): source the lib in scripts/dev-lead-fix-reviews.sh, add resolution_gate_open, and wrap the three resolve_* call sites in the fix-reviews / fix-bot-comment / review-changes intents; emit a ::notice:: naming this issue when resolution is skipped; update the existing tests that assert resolution in a no-commit pass.
  • Slice 3 (not yet filed) — pr-review-side invalidation: an approval is void if threads were resolved with no intervening head change. This is the control that would have caught all three occurrences independently.

Related: #1596, #1607, #1610, petry-projects/.github#1024, petry-projects/.github#1025.

Activity

  1. github-actions commented on Aug 31, 2026

    @github-actions
    Contributor

    Thank you for the thorough write-up and for catching this before the merge went through — the near-miss on #1024 makes the severity very clear. To confirm the scope of the fix needed: can you clarify whether the pr-review fix pass currently has any check that a thread may only transition to resolved when the head SHA has changed since the finding was recorded, or does the shipped guard in #1604 only fire when a status=applied claim is present?

  2. don-petry commented on Aug 31, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1609 — Follow-up to #1567: threads resolved with NO commit at all — the shipped evidence bar gates status=applied, not thread resolution

    Scope

    Confirm the #1604 gap and close it: gate all auto-resolution of review threads in dev-lead-fix-reviews.sh on the fix pass having advanced the PR head this pass (a new commit was created + pushed). A pass that produces no commit resolves zero threads — the near-miss on petry-projects/.github#1024, where 18 open threads (incl. a 🔴 Critical unmatched-brace finding) were marked resolved with the head unchanged and no commit.

    Root cause (confirmed)

    #1604's evidence check (review-change-evidence.sh) only gates the status=applied claim. The three thread-resolution nets — resolve_bot_outdated_threads, resolve_actor_outdated_threads, resolve_addressed_bot_threads — run in the no-changes path too (fix-reviews L1485-91, fix-bot-comment L1530-36, review-changes L1602-07). resolve_addressed_bot_threads (#1547) resolves a non-outdated bot thread purely on a reply marker, with no commit required — the exact vector that resolved the Critical brace thread.

    Implementation Checklist

    Tests to Write

    Test Command

    bats tests/dev-lead/unit/ and the repo bats suite via scripts/run-bats.sh; lint via bash .dev-lead/scripts/dev-lead-lint.sh

    Note on AC #4

    The pr-review-side approval invalidation ("an approval is void if threads were resolved with no intervening head change") is framed as "Consider" in the issue; the dev-lead-side gate above closes AC #1–#3 at the source. AC #4 is recorded as a follow-up rather than built in this pass to keep scope to what the issue requires.

  3. don-petry commented on Aug 31, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: issue #1609 implementation failed — will retry

    All engines were rate-limited. This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

    Last 40 lines of engine output (redacted)
        at throwErrorIfNotOK (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:267240:24)
        at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
        at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:267003:7
        at async Models.generateContentStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:268099:16)
        at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:310468:19
        at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:287277:23
        at async retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:307305:23)
        at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:331095:28)
        at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:330848:29)
        at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:331770:24) {
      status: 429
    }
    Attempt 10 failed: Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. . Max attempts reached
    Error when talking to Gemini API Full report available at: /tmp/gemini-client-error-Turn.run-sendMessageStream-2026-08-31T14-40-46-128Z.json RetryableQuotaError: Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. 
        at classifyGoogleError (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:306592:14)
        at retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:307328:29)
        at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
        at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:331095:28)
        at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:330848:29)
        at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:331770:24)
        at async GeminiClient.processTurn (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:345315:22)
        at async GeminiClient.sendMessageStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-7HKQGPWB.js:345412:14)
        at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-OYYGXMHL.js:11527:26
        at async main (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-OYYGXMHL.js:16954:5) {
      cause: {
        code: 429,
        message: 'Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. ',
        details: []
      },
      retryDelayMs: undefined,
      status: 429
    }
    {
      "session_id": "c88bc728-2797-49e2-a00a-d44cf50d616b",
      "error": {
        "type": "Error",
        "message": "Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. ",
        "code": 429
      }
    }
    
    
  4. don-petry commented on Aug 31, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1609 — Follow-up to #1567: threads resolved with NO commit at all — the shipped evidence bar gates status=applied, not thread resolution

    Root cause

    The #1604/#1567 evidence bar (scripts/lib/review-change-evidence.sh, finalize_review_application) gates the status=applied claim on a substantive commit that touches the reviewed regions. But it does not gate thread resolution. In dev-lead-fix-reviews.sh, resolve_bot_outdated_threads, resolve_actor_outdated_threads, and resolve_addressed_bot_threads run in both the applied and the no-changes paths (fix-reviews, fix-bot-comment, review-changes). A pass that makes no commit can still mark threads resolved — exactly the .github#1024 near-miss (head unchanged, 18 threads incl. a Critical resolved).

    Scope

    Add a pass-level resolution-integrity gate: a dev-lead fix pass may auto-resolve review threads only when it advanced the PR head (a commit landed since the finding). A no-commit pass resolves zero threads.

    Implementation Checklist

    • scripts/lib/resolution-integrity.sh — new pure helper ri_may_resolve <sha_before> <sha_after> (allow iff both non-empty AND differ; fail closed otherwise).
    • scripts/dev-lead-fix-reviews.sh — source the lib; add resolution_gate_open <cp_rc> (dry-run passes through so functions still announce; else compares HEAD_SHA pre-head to git rev-parse HEAD post-head, falling back to cp_rc==0 only when a SHA is unavailable).
    • Wrap the three resolve_* call sites in the fix-reviews, fix-bot-comment, and review-changes intents with the gate; emit a ::notice:: when resolving zero threads. Capture cp_rc in review-changes.

    Tests to Write

    • tests/dev-lead/unit/test_resolution_integrity.bats — unit tests for ri_may_resolve (differ→allow; equal→deny; empty operand→deny).
    • tests/dev-lead/unit/test_fix_reviews.bats — AC Add @claude delegation, auto-merge, and rebase handling #3 integration: a fix pass that makes no change (HEAD_SHA == real head) asserts no resolveReviewThread mutation; positive control where a commit advances the head still resolves.

    Test Command

    bats tests/dev-lead/unit/test_resolution_integrity.bats tests/dev-lead/unit/test_fix_reviews.bats tests/dev-lead/unit/test_review_change_evidence.bats then bash .dev-lead/scripts/dev-lead-lint.sh

  5. don-petry commented on Aug 31, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: cannot implement issue #1609 — needs human attention

    A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.

    This hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise vars.ACTION_TIMEOUT_SEC, then re-apply the dev-lead label.

    Note: the engine had produced changes before the timeout. They were on branch dev-lead/issue-1609-20260831-2004 on the runner and were not pushed, so they are not recoverable from this run (the runner is ephemeral). The redacted session snippet below shows what it attempted — treat it as context, not restorable output. Re-apply dev-lead after splitting the issue or raising the budget to regenerate the work.

  6. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-lead:needs-humandev-lead could not complete this issue; needs human attention
    dev-leadFor dev-lead agent pickup
    on Aug 31, 2026
  7. don-petry commented on Aug 31, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1609 — Follow-up to #1567: threads resolved with NO commit at all — add the pure resolution-integrity.sh helper (slice 1 of 2)

    Scope

    Add scripts/lib/resolution-integrity.sh, a pure, side-effect-free helper exposing ri_may_resolve <sha_before> <sha_after> that fails closed (returns 0 only when both SHAs are non-empty AND differ). No wiring into dev-lead-fix-reviews.sh — that is slice 2 (#1617).

    Implementation Checklist

    • Create scripts/lib/resolution-integrity.sh — pure, set -euo pipefail-safe to source, no network/git/gh/other-lib deps, ri_may_resolve declared exactly once
    • Create tests/dev-lead/unit/test_resolution_integrity.bats
    • No other file changed (in particular scripts/dev-lead-fix-reviews.sh untouched); new test auto-runs via test-dev-lead.yml's bats tests/dev-lead/unit/ glob

    Tests to Write

    • both empty -> deny (non-zero)
    • before empty -> deny
    • after empty -> deny
    • equal SHAs -> deny
    • differing non-empty SHAs -> allow (0)
    • ri_may_resolve declared exactly once (grep -c '^ri_may_resolve()' -> 1)
    • lib is safe to source under set -euo pipefail

    Test Command

    bats tests/dev-lead/unit/test_resolution_integrity.bats and bash .dev-lead/scripts/dev-lead-lint.sh

  8. don-petry commented on Aug 31, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    • Added the pure helper scripts/lib/resolution-integrity.sh exposing ri_may_resolve <sha_before> <sha_after> (fail-closed: allow only when both SHAs are non-empty AND differ; no network/git/gh/other-lib deps; return-only, source-safe under set -euo pipefail).
    • Added tests/dev-lead/unit/test_resolution_integrity.bats covering the full truth table (both empty / before empty / after empty / equal → deny; differing → allow) plus source-safety and the single-declaration guard.

    Local check output (provisional — not the pushed run)

    shellcheck scripts/lib/resolution-integrity.sh   → clean
    bash .dev-lead/scripts/dev-lead-lint.sh          → all checks passed
    grep -c '^ri_may_resolve()' …                    → 1
    

    Files touched (this workspace, not yet pushed)

    • scripts/lib/resolution-integrity.sh: new pure helper (slice 1 only).
    • tests/dev-lead/unit/test_resolution_integrity.bats: new unit tests (auto-run by test-dev-lead.yml's bats tests/dev-lead/unit/ glob — no CI list edit needed).

    Notes

  9. don-petry commented on Aug 31, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #1618

    The implementation for issue #1609 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  10. added a commit that references this issue on Aug 31, 2026
  11. added a commit that references this issue on Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions