Repository navigation
feat: implement issue #2139 — Retry cron should hold retries until a Claude window's real reset, read from the usage telemetry (split 1 of 2 from the weekly-cap issue) - #2144
Conversation
…Claude window's real reset, read from the usage telemetry (split 1 of 2 from the weekly-cap issue)
|
ⓘ Qodo reviews are paused because your trial has ended. Ask your workspace admin to add credits to resume reviews. Manage billing |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 36 minutes. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configuration
📒 Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Dev-Lead — waiting on PR blockers (intent: review-changes)PR: #2144 |
|
Note @don-petry I reviewed this PR and no code changes were needed, but I can't mark it done yet: required checks |
|
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-09T20:35:09Z. |
There was a problem hiding this comment.
All reported issues were addressed across 3 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 195695000b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| USAGE_HOLD_WINDOWS="weekly_all session" | ||
| USAGE_HOLD_LIMIT_PCT=100 | ||
| # The public library (petry-projects/.github scripts/lib/agent-rate-limit.sh). | ||
| AGENT_RATE_LIMIT_LIB="${AGENT_RATE_LIMIT_LIB:-$SCRIPT_DIR/lib/agent-rate-limit.sh}" |
There was a problem hiding this comment.
Wire the telemetry reader into the retry workflow
The production checkout does not contain the default scripts/lib/agent-rate-limit.sh—a repo-wide search finds it only under tests/fixtures—and .github/workflows/dev-lead-retry.yml also never exposes CLAUDE_CODE_OAUTH_TOKEN to the scan step. Consequently every scheduled run returns from usage_hold_init without fetching telemetry, so the new hold is permanently inactive and weekly exhaustion continues triggering two-hour retries. The workflow needs to obtain the public reader and pass the OAuth secret rather than treating these absent production inputs as a normal no-hold result.
AGENTS.md reference: AGENTS.md:L168-L171
Useful? React with 👍 / 👎.
| fi | ||
|
|
||
| # Read the Claude usage windows once for the whole run (#2139). | ||
| usage_hold_init |
There was a problem hiding this comment.
Initialize the hold on the event-first resume path
This initialization runs only from main, but scripts/dev-lead-resume.sh sources this file and calls scan_pr_for_rate_limits directly. On a successful check or submitted review, USAGE_HOLD_EPOCH therefore remains empty and usage_hold_active returns false, allowing a rate-limited retry during an exhausted weekly window even after the cron wiring is fixed. Initialize the usage hold before the resume path scans the PR so all retry entry points honor the real reset.
Useful? React with 👍 / 👎.
| local reviews_status | ||
| reviews_status=$(echo "$comments_json" | jq -r \ | ||
| --arg pat "$reviews_pattern" \ | ||
| '[.[] | select(. | test($pat))] | .[0] | capture("status=(?<s>[a-z-]+)") | .s // ""' \ |
There was a problem hiding this comment.
Classify the latest review marker before applying the hold
This selects the first matching marker even though comments are chronological and a PR can accumulate several retry outcomes for the same SHA and intent. If an older status=blocked marker is followed by a newer status=rate-limited marker, the exhausted-window hold is bypassed and the cron re-dispatches during the cap; in the reverse order, a current non-quota blocked retry is incorrectly held. Select the latest applicable marker, as the issue-retry path already does, before deciding whether this is a quota retry.
Useful? React with 👍 / 👎.
| --arg pat "$reviews_pattern" \ | ||
| '[.[] | select(. | test($pat))] | .[0] | capture("status=(?<s>[a-z-]+)") | .s // ""' \ | ||
| 2>/dev/null || true) | ||
| if [ "$reviews_status" = "rate-limited" ] && usage_hold_active; then |
There was a problem hiding this comment.
Apply the usage hold to bot recovery retries
The hold is checked only inside the fix-ci/fix-reviews/issue paths, while scan_repo subsequently invokes scan_pr_for_undispositioned_bot_comments and scan_pr_for_unreplied_bot_threads whenever this scan returns zero. A weekly-cap fix-bot-comment marker has no reset and is excluded from RETRYABLE_REVIEW_INTENTS, so its dedicated scanner dispatches again; an exhausted fix-reviews marker can likewise be followed by the separate thread scanner once its local pending window expires. These cron retry paths need to consult the same active usage hold before dispatching.
Useful? React with 👍 / 👎.
|
Closing: reviewed against production, this design cannot work as written. The retry workflow passes no CLAUDE_CODE_OAUTH_TOKEN and the public agent-rate-limit.sh library exists only under tests/fixtures, so the hold would log 'reader unavailable' on every run and never engage (the tests passed because they mocked both). The maintainer decided not to give a second workflow the Claude token. Issue 2139 is rewritten so the cron reads the dry-run budget poller's published result instead, and is blocked on that poller (issue 2029, PR 2031) merging. The log line, once-per-run read, fail-open behaviour and no-weekday-literal rule from this attempt are kept in the new spec. |
| USAGE_HOLD_WINDOWS="weekly_all session" | ||
| USAGE_HOLD_LIMIT_PCT=100 | ||
| # The public library (petry-projects/.github scripts/lib/agent-rate-limit.sh). | ||
| AGENT_RATE_LIMIT_LIB="${AGENT_RATE_LIMIT_LIB:-$SCRIPT_DIR/lib/agent-rate-limit.sh}" |
There was a problem hiding this comment.
Suggestion: No scripts/lib/agent-rate-limit.sh exists in this checkout, so this default path is unreadable, the reader check exits early, and no usage request or hold is made.
Assessment: 🟠 Major · 🔁 Occurrence: Often · 🏷️ Incomplete implementation
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** scripts/dev-lead-retry.sh
**Line:** 187:187
**Comment:**
*Incomplete Implementation: No `scripts/lib/agent-rate-limit.sh` exists in this checkout, so this default path is unreadable, the reader check exits early, and no usage request or hold is made.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix| reviews_status=$(echo "$comments_json" | jq -r \ | ||
| --arg pat "$reviews_pattern" \ | ||
| '[.[] | select(. | test($pat))] | .[0] | capture("status=(?<s>[a-z-]+)") | .s // ""' \ | ||
| 2>/dev/null || true) |
There was a problem hiding this comment.
Suggestion: This selects the oldest matching marker, not the latest status. An old rate-limited marker before a newer blocked marker makes an exhausted window wrongly suppress the blocked retry.
Assessment: 🟠 Major · 🔁 Occurrence: Rarely · 🏷️ Logic error
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** scripts/dev-lead-retry.sh
**Line:** 612:615
**Comment:**
*Logic Error: This selects the oldest matching marker, not the latest status. An old rate-limited marker before a newer blocked marker makes an exhausted window wrongly suppress the blocked retry.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix| --arg pat "$reviews_pattern" \ | ||
| '[.[] | select(. | test($pat))] | .[0] | capture("status=(?<s>[a-z-]+)") | .s // ""' \ | ||
| 2>/dev/null || true) | ||
| if [ "$reviews_status" = "rate-limited" ] && usage_hold_active; then |
There was a problem hiding this comment.
Suggestion: The resume caller invokes scan_pr_for_rate_limits without running usage_hold_init; this guard sees an empty epoch and dispatches quota-limited PR retries before reset.
Assessment: 🟠 Major · 🔁 Occurrence: Sometimes · 🏷️ Incomplete implementation
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** scripts/dev-lead-retry.sh
**Line:** 616:616
**Comment:**
*Incomplete Implementation: The resume caller invokes `scan_pr_for_rate_limits` without running `usage_hold_init`; this guard sees an empty epoch and dispatches quota-limited PR retries before reset.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
Code Review
This pull request introduces a Claude usage-window hold mechanism to the retry cron script to prevent redundant retries when account-wide Claude limits are exhausted, along with comprehensive unit tests. The implementation queries the usage telemetry endpoint once per run and holds rate-limited retries until the active window resets, failing open if telemetry is unavailable. Feedback on the changes suggests assigning command substitutions to variables first and capturing their exit status safely to prevent premature script termination under set -e, ensuring robust fail-open behavior.
| [ "$(arl_token_window_active "$body" "$w")" = "false" ] && continue | ||
| pct=$(arl_token_extract_percent "$body" "$w") | ||
| [[ "$pct" =~ ^[0-9]+$ ]] || continue | ||
| [ "$pct" -ge "$USAGE_HOLD_LIMIT_PCT" ] || continue | ||
| reset=$(arl_token_extract_resets_at "$body" "$w") |
There was a problem hiding this comment.
In Shell scripts running under set -e, executing command substitutions directly inside conditional lists (such as [ ... ]) can cause failures to be silently swallowed. Additionally, unguarded assignments like pct=$(...) and reset=$(...) can cause the script to terminate prematurely if the telemetry reader functions fail, which violates the "fail open" design.
Assign the command substitutions to variables first, capturing their exit status safely to ensure robust fail-open behavior.
| [ "$(arl_token_window_active "$body" "$w")" = "false" ] && continue | |
| pct=$(arl_token_extract_percent "$body" "$w") | |
| [[ "$pct" =~ ^[0-9]+$ ]] || continue | |
| [ "$pct" -ge "$USAGE_HOLD_LIMIT_PCT" ] || continue | |
| reset=$(arl_token_extract_resets_at "$body" "$w") | |
| active=$(arl_token_window_active "$body" "$w") || active="false" | |
| [ "$active" = "false" ] && continue | |
| pct=$(arl_token_extract_percent "$body" "$w") || pct="" | |
| [[ "$pct" =~ ^[0-9]+$ ]] || continue | |
| [ "$pct" -ge "$USAGE_HOLD_LIMIT_PCT" ] || continue | |
| reset=$(arl_token_extract_resets_at "$body" "$w") || reset="" |
References
- In Shell scripts running under
set -e, avoid executing command substitutions directly inside conditional lists (such asifstatements). Instead, assign the command substitution to a variable and capture its exit status first (e.g.,output="$(cmd)" || status=$?), then perform the conditional check on the captured status to prevent failures from being silently swallowed.
Dev-Lead — fix-reviews (partial)A commit was pushed, but not every requested change was applied. Per requested item:
The unaddressed items above still need work. |



Problem
Retry cron should hold retries until a Claude window's real reset, read from the usage telemetry (split 1 of 2 from the weekly-cap issue)
From the issue: Split 1 of 2 from #1863 (which timed out at dev-lead's maximum action budget on its first attempt and asked to be split). As the fleet maintainer, I want the retry cron to ask the usage endpoint whether a Claude window is exhausted and when it resets, so that a weekly cap suppresses re-dispatch until its real reset instead of retrying every two hours for up to seven days.
Risk
Medium — changes GitHub Actions workflow behavior, which is exercised only post-merge; verify via the affected workflow runs.
Test plan
Tests added/updated:
tests/dev-lead/unit/test_dev_lead_retry_usage_hold.bats. Verification:bash scripts/dev-lead-lint.sh(shellcheck --severity=warning) ran pre-commit; the bats suite runs in CI.Rollback
Revert this PR. No non-revertible side effects (no tags, migrations, or external state).
Monitoring
Watch the affected workflow run(s) in the Actions tab and this PR's Lint check for regressions.
Closes #2139