Skip to content

feat: implement issue #2139 — Retry cron should hold retries until a Claude window's real reset, read from the usage telemetry (split 1 of 2 from the weekly-cap issue) - #2144

Closed
don-petry wants to merge 1 commit into
mainfrom
dev-lead/issue-2139-20261009-1927
Closed

don-petry wants to merge 1 commit into
mainfrom
dev-lead/issue-2139-20261009-1927

Conversation

@don-petry

@don-petry don-petry commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Retry cron should hold retries until a Claude window's real reset, read from the usage telemetry (split 1 of 2 from the weekly-cap issue)

From the issue: Split 1 of 2 from #1863 (which timed out at dev-lead's maximum action budget on its first attempt and asked to be split). As the fleet maintainer, I want the retry cron to ask the usage endpoint whether a Claude window is exhausted and when it resets, so that a weekly cap suppresses re-dispatch until its real reset instead of retrying every two hours for up to seven days.

Risk

Medium — changes GitHub Actions workflow behavior, which is exercised only post-merge; verify via the affected workflow runs.

Test plan

Tests added/updated: tests/dev-lead/unit/test_dev_lead_retry_usage_hold.bats. Verification: bash scripts/dev-lead-lint.sh (shellcheck --severity=warning) ran pre-commit; the bats suite runs in CI.

Rollback

Revert this PR. No non-revertible side effects (no tags, migrations, or external state).

Monitoring

Watch the affected workflow run(s) in the Actions tab and this PR's Lint check for regressions.

Closes #2139

View guided diff Turn on auto-fix

…Claude window's real reset, read from the usage telemetry (split 1 of 2 from the weekly-cap issue)
@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because your trial has ended. Ask your workspace admin to add credits to resume reviews. Manage billing

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 36 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 1bfa13db-472d-4f39-9026-b63d9a609dbf

📥 Commits

Reviewing files that changed from the base of the PR and between 0ae6227 and 1956950.


📒 Files selected for processing (3)
  • .github/workflows/lint.yml
  • scripts/dev-lead-retry.sh
  • tests/dev-lead/unit/test_dev_lead_retry_usage_hold.bats


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T19:38:04.736240Z 1956950 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #2144
No changes were committed, but the PR still can't be marked done: required checks SonarCloud, agent-shield / AgentShield, duplicate-decl-gate are still pending. The retry cron will re-attempt automatically. Next attempt after: 2026-10-09T20:04:21Z

@don-petry

Copy link
Copy Markdown
Collaborator Author

Note

@don-petry I reviewed this PR and no code changes were needed, but I can't mark it done yet: required checks SonarCloud, agent-shield / AgentShield, duplicate-decl-gate are still pending. I'll re-check automatically.
Next attempt after: 2026-10-09T20:04:21Z

@don-petry
don-petry enabled auto-merge (squash) October 9, 2026 19:34
@sonarqubecloud

sonarqubecloud Bot commented Oct 9, 2026

Copy link
Copy Markdown

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-09T20:35:09Z.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/dev-lead-retry.sh Resolved
Comment thread scripts/dev-lead-retry.sh Resolved
Comment thread scripts/dev-lead-retry.sh Resolved
Comment thread scripts/dev-lead-retry.sh Resolved
@don-petry
don-petry disabled auto-merge October 9, 2026 19:37

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 195695000b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/dev-lead-retry.sh
USAGE_HOLD_WINDOWS="weekly_all session"
USAGE_HOLD_LIMIT_PCT=100
# The public library (petry-projects/.github scripts/lib/agent-rate-limit.sh).
AGENT_RATE_LIMIT_LIB="${AGENT_RATE_LIMIT_LIB:-$SCRIPT_DIR/lib/agent-rate-limit.sh}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Wire the telemetry reader into the retry workflow

The production checkout does not contain the default scripts/lib/agent-rate-limit.sh—a repo-wide search finds it only under tests/fixtures—and .github/workflows/dev-lead-retry.yml also never exposes CLAUDE_CODE_OAUTH_TOKEN to the scan step. Consequently every scheduled run returns from usage_hold_init without fetching telemetry, so the new hold is permanently inactive and weekly exhaustion continues triggering two-hour retries. The workflow needs to obtain the public reader and pass the OAuth secret rather than treating these absent production inputs as a normal no-hold result.

AGENTS.md reference: AGENTS.md:L168-L171

Useful? React with 👍 / 👎.

Comment thread scripts/dev-lead-retry.sh
fi

# Read the Claude usage windows once for the whole run (#2139).
usage_hold_init

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Initialize the hold on the event-first resume path

This initialization runs only from main, but scripts/dev-lead-resume.sh sources this file and calls scan_pr_for_rate_limits directly. On a successful check or submitted review, USAGE_HOLD_EPOCH therefore remains empty and usage_hold_active returns false, allowing a rate-limited retry during an exhausted weekly window even after the cron wiring is fixed. Initialize the usage hold before the resume path scans the PR so all retry entry points honor the real reset.

Useful? React with 👍 / 👎.

Comment thread scripts/dev-lead-retry.sh
local reviews_status
reviews_status=$(echo "$comments_json" | jq -r \
--arg pat "$reviews_pattern" \
'[.[] | select(. | test($pat))] | .[0] | capture("status=(?<s>[a-z-]+)") | .s // ""' \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Classify the latest review marker before applying the hold

This selects the first matching marker even though comments are chronological and a PR can accumulate several retry outcomes for the same SHA and intent. If an older status=blocked marker is followed by a newer status=rate-limited marker, the exhausted-window hold is bypassed and the cron re-dispatches during the cap; in the reverse order, a current non-quota blocked retry is incorrectly held. Select the latest applicable marker, as the issue-retry path already does, before deciding whether this is a quota retry.

Useful? React with 👍 / 👎.

Comment thread scripts/dev-lead-retry.sh
--arg pat "$reviews_pattern" \
'[.[] | select(. | test($pat))] | .[0] | capture("status=(?<s>[a-z-]+)") | .s // ""' \
2>/dev/null || true)
if [ "$reviews_status" = "rate-limited" ] && usage_hold_active; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply the usage hold to bot recovery retries

The hold is checked only inside the fix-ci/fix-reviews/issue paths, while scan_repo subsequently invokes scan_pr_for_undispositioned_bot_comments and scan_pr_for_unreplied_bot_threads whenever this scan returns zero. A weekly-cap fix-bot-comment marker has no reset and is excluded from RETRYABLE_REVIEW_INTENTS, so its dedicated scanner dispatches again; an exhausted fix-reviews marker can likewise be followed by the separate thread scanner once its local pending window expires. These cron retry paths need to consult the same active usage hold before dispatching.

Useful? React with 👍 / 👎.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Closing: reviewed against production, this design cannot work as written. The retry workflow passes no CLAUDE_CODE_OAUTH_TOKEN and the public agent-rate-limit.sh library exists only under tests/fixtures, so the hold would log 'reader unavailable' on every run and never engage (the tests passed because they mocked both). The maintainer decided not to give a second workflow the Claude token. Issue 2139 is rewritten so the cron reads the dry-run budget poller's published result instead, and is blocked on that poller (issue 2029, PR 2031) merging. The log line, once-per-run read, fail-open behaviour and no-weekday-literal rule from this attempt are kept in the new spec.

@don-petry don-petry closed this Oct 9, 2026
Comment thread scripts/dev-lead-retry.sh
USAGE_HOLD_WINDOWS="weekly_all session"
USAGE_HOLD_LIMIT_PCT=100
# The public library (petry-projects/.github scripts/lib/agent-rate-limit.sh).
AGENT_RATE_LIMIT_LIB="${AGENT_RATE_LIMIT_LIB:-$SCRIPT_DIR/lib/agent-rate-limit.sh}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: No scripts/lib/agent-rate-limit.sh exists in this checkout, so this default path is unreadable, the reader check exits early, and no usage request or hold is made.

Assessment: 🟠 Major · 🔁 Occurrence: Often · 🏷️ Incomplete implementation

Use CodeAnt Skill Fix in Cursor Fix in VSCode Claude

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** scripts/dev-lead-retry.sh
**Line:** 187:187
**Comment:**
	*Incomplete Implementation: No `scripts/lib/agent-rate-limit.sh` exists in this checkout, so this default path is unreadable, the reader check exits early, and no usage request or hold is made.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

Comment thread scripts/dev-lead-retry.sh
Comment on lines +612 to +615
reviews_status=$(echo "$comments_json" | jq -r \
--arg pat "$reviews_pattern" \
'[.[] | select(. | test($pat))] | .[0] | capture("status=(?<s>[a-z-]+)") | .s // ""' \
2>/dev/null || true)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: This selects the oldest matching marker, not the latest status. An old rate-limited marker before a newer blocked marker makes an exhausted window wrongly suppress the blocked retry.

Assessment: 🟠 Major · 🔁 Occurrence: Rarely · 🏷️ Logic error

Use CodeAnt Skill Fix in Cursor Fix in VSCode Claude

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** scripts/dev-lead-retry.sh
**Line:** 612:615
**Comment:**
	*Logic Error: This selects the oldest matching marker, not the latest status. An old rate-limited marker before a newer blocked marker makes an exhausted window wrongly suppress the blocked retry.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

Comment thread scripts/dev-lead-retry.sh
--arg pat "$reviews_pattern" \
'[.[] | select(. | test($pat))] | .[0] | capture("status=(?<s>[a-z-]+)") | .s // ""' \
2>/dev/null || true)
if [ "$reviews_status" = "rate-limited" ] && usage_hold_active; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: The resume caller invokes scan_pr_for_rate_limits without running usage_hold_init; this guard sees an empty epoch and dispatches quota-limited PR retries before reset.

Assessment: 🟠 Major · 🔁 Occurrence: Sometimes · 🏷️ Incomplete implementation

Use CodeAnt Skill Fix in Cursor Fix in VSCode Claude

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** scripts/dev-lead-retry.sh
**Line:** 616:616
**Comment:**
	*Incomplete Implementation: The resume caller invokes `scan_pr_for_rate_limits` without running `usage_hold_init`; this guard sees an empty epoch and dispatches quota-limited PR retries before reset.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Claude usage-window hold mechanism to the retry cron script to prevent redundant retries when account-wide Claude limits are exhausted, along with comprehensive unit tests. The implementation queries the usage telemetry endpoint once per run and holds rate-limited retries until the active window resets, failing open if telemetry is unavailable. Feedback on the changes suggests assigning command substitutions to variables first and capturing their exit status safely to prevent premature script termination under set -e, ensuring robust fail-open behavior.

Comment thread scripts/dev-lead-retry.sh
Comment on lines +237 to +241
[ "$(arl_token_window_active "$body" "$w")" = "false" ] && continue
pct=$(arl_token_extract_percent "$body" "$w")
[[ "$pct" =~ ^[0-9]+$ ]] || continue
[ "$pct" -ge "$USAGE_HOLD_LIMIT_PCT" ] || continue
reset=$(arl_token_extract_resets_at "$body" "$w")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In Shell scripts running under set -e, executing command substitutions directly inside conditional lists (such as [ ... ]) can cause failures to be silently swallowed. Additionally, unguarded assignments like pct=$(...) and reset=$(...) can cause the script to terminate prematurely if the telemetry reader functions fail, which violates the "fail open" design.

Assign the command substitutions to variables first, capturing their exit status safely to ensure robust fail-open behavior.

Suggested change
[ "$(arl_token_window_active "$body" "$w")" = "false" ] && continue
pct=$(arl_token_extract_percent "$body" "$w")
[[ "$pct" =~ ^[0-9]+$ ]] || continue
[ "$pct" -ge "$USAGE_HOLD_LIMIT_PCT" ] || continue
reset=$(arl_token_extract_resets_at "$body" "$w")
active=$(arl_token_window_active "$body" "$w") || active="false"
[ "$active" = "false" ] && continue
pct=$(arl_token_extract_percent "$body" "$w") || pct=""
[[ "$pct" =~ ^[0-9]+$ ]] || continue
[ "$pct" -ge "$USAGE_HOLD_LIMIT_PCT" ] || continue
reset=$(arl_token_extract_resets_at "$body" "$w") || reset=""
References
  1. In Shell scripts running under set -e, avoid executing command substitutions directly inside conditional lists (such as if statements). Instead, assign the command substitution to a variable and capture its exit status first (e.g., output="$(cmd)" || status=$?), then perform the conditional check on the captured status to prevent failures from being silently swallowed.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (partial)

A commit was pushed, but not every requested change was applied. Per requested item:

  • scripts/dev-lead-retry.sh:616 — applied
  • scripts/dev-lead-retry.sh:1426 — not applied
  • scripts/dev-lead-retry.sh:187 — not applied
  • scripts/dev-lead-retry.sh:614 — applied

The unaddressed items above still need work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants