Skip to content

A weekly Claude cap parses as 'unknown reset', so the retry cron re-dispatches every 2 hours against a block that can last 7 days #1863

Open

Description

@don-petry

Summary

The rate-limit reset parser only understands the 5-hour window's message shape. A weekly cap block parses to "unknown", and every downstream consumer treats unknown as "retry now" — so the fleet re-dispatches every two hours against a block that lasts up to seven days.

The chain

1. The regex matches only a time-of-day. parse_reset_time / parse_reset_time_files (scripts/engine.sh:1793, :1810):

grep -oiE 'resets [0-9]{1,2}:[0-9]{2}(am|pm)'

A weekly-limit message carries a day-of-week form (the weekly window resets Tuesday ~11:00 America/Chicago), which this pattern cannot match. No match ⇒ the sidecar /tmp/dev-lead-rate-limit-reset is written empty.

2. The clamp cannot express a multi-day reset. _emit_reset_iso (scripts/engine.sh:1766) resolves the parsed H:MM against today, and if that is already past, against tomorrow. Its range is bounded at ~24h by construction. Even if the regex matched a weekly reset, the clamp would collapse it to tomorrow.

3. Unknown is treated as "go". scripts/dev-lead-retry.sh:97:

is_reset_in_future() {
  local reset_iso="$1"
  [ -z "$reset_iso" ] && return 1  # unknown reset = don't skip

So status=rate-limited reset= (empty) does not suppress the retry. The cron (15 */2 * * *) re-dispatches every two hours.

Consequence

During a weekly-cap exhaustion the fleet does not back off. Each retry re-walks the engine fallback chain, burning Gemini and Copilot quota and consuming MAX_ATTEMPTS on work that provably cannot proceed, then escalates to dev-lead:needs-human for a reason that was never the PR's fault. The per-PR automation budget (10 actions since the last human) and MAX_ATTEMPTS=3 are the only things bounding it, and neither is reset-aware.

The blast radius is bounded — a hard cap fails fast and cheaply on the Claude side — but the behaviour is backwards: the one situation where waiting is provably correct is the one situation where the fleet does not wait.

Acceptance Criteria

  1. The reset parser recognises a multi-day / day-of-week reset form in addition to the existing resets H:MMam/pm, and emits a correct ISO-8601 UTC timestamp for it.
  2. _emit_reset_iso's today/tomorrow clamp no longer caps a legitimately-further-out reset. A reset up to 7 days ahead round-trips unchanged.
  3. Prefer the authoritative value over parsing prose: when the Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565 telemetry envelope is available, take resets_at from it for the relevant window and use the text parse only as a fallback. Never hardcode a weekday.
  4. is_reset_in_future keeps its fail-open default for a genuinely unknown reset (do not invert it — an unparseable 5-hour block must still retry), but an empty reset on a marker whose window is known to be weekly suppresses the retry until that window's resets_at.
  5. The retry cron logs which window it is honouring and until when, so a deliberate multi-day wait is legible rather than looking like a stalled fleet.
  6. Bats coverage for: the weekly form parsing correctly; a >24h reset surviving the clamp; and the retry cron skipping while a weekly reset is in the future.

Notes for the implementer

References


Dispatch sequencing — deliberately not yet released to dev-lead (2026-09-21)

This issue is fully specified but intentionally carries no dev-lead label yet. Two reasons:

  1. It is blocked by status=rate-limited is emitted for non-quota holds (reason=blocked) — retires the token half #461 deliberately deferred #1568 (overlapping marker semantics, PR feat: implement issue #1568 — status=rate-limited is emitted for non-quota holds (reason=blocked) — retires the token half #461 deliberately deferred #1798 open).
  2. The weekly Claude budget stood at 79% with two days left in the window, and every new dev-lead run opens another PR into the rebase treadmill described in Merge protections are unsatisfiable at N>1 open PRs: one merge to main forces a full re-review of every other PR — ~154 forced re-reviews in 48h, zero merges #1864 — which multiplies re-review cost across all open PRs, not just the new one. Adding parallel agent work before pr-review re-runs the full tier stack for a conflict-free base merge — carry the prior verdict forward when the diff-vs-merge-base is unchanged #1865 lands makes the budget problem worse, not better.

Release condition: add the dev-lead label once #1865 has merged, or after the weekly window resets (resets_at from the usage endpoint — do not assume a weekday), whichever comes first, and once the blocking issue above is resolved.

Activity

  1. github-actions commented on Sep 21, 2026

    @github-actions
    Contributor

    Thanks for the thorough write-up — the chain from empty sidecar through fail-open retry to multi-day re-dispatch is exactly the kind of thing that's hard to spot without hitting a weekly cap in the wild. To make sure the parser fix targets the right text: could you share the actual message string the weekly-cap response carries? The issue describes the expected shape ('resets Tuesday ~11:00 America/Chicago'), but having the literal API response text (or a sanitised excerpt) will let us write and verify the regex against the real input rather than a reconstruction of it.

  2. don-petry commented on Sep 22, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1863 — A weekly Claude cap parses as 'unknown reset', so the retry cron re-dispatches every 2 hours against a block that can last 7 days

    Scope

    Teach the reset parser to recognise a weekly / day-of-week reset form (preferring the authoritative resets_at from the #1565 telemetry envelope when present), stop the today/tomorrow clamp from collapsing a multi-day reset, tag rate-limit markers with the known window kind, and make the retry cron fail-safe (suppress) — with legible logging — when a marker's window is known to be weekly.

    Implementation Checklist

    • scripts/engine.sh — add _next_weekday_iso (next occurrence of a weekday at H:MM in UTC, up to 7 days out, no clamp)
    • scripts/engine.sh — add _reset_from_envelope reader (authoritative resets_at from the Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565 envelope, ${DEV_LEAD_USAGE_ENVELOPE:-/tmp/dev-lead-usage-envelope.json}), tolerant of absence
    • scripts/engine.sh — extend parse_reset_time / parse_reset_time_files to: prefer envelope, else detect weekly (day-of-week) form, else existing resets H:MMam/pm; write a window-kind sidecar /tmp/dev-lead-rate-limit-window (weekly | 5h | empty)
    • scripts/engine.sh — _emit_reset_iso clamp stays for the ≤24h H:MM (5h) case only; weekly path bypasses the clamp
    • scripts/dev-lead-fix-reviews.sh — post_reviews_rate_limited emits window=<kind> on the marker when the sidecar is set
    • scripts/dev-lead-fix-issue.sh — issue marker emits window=<kind> when the sidecar is set
    • scripts/dev-lead-retry.sh — parse window= from markers; keep is_reset_in_future fail-open default; suppress + log when reset is in future OR (reset empty AND window=weekly); log which window is honoured and until when

    Tests to Write

    • tests/dev-lead/unit/test_engine_chain.bats — weekly day-of-week form parses to a correct future ISO-8601 UTC (>24h ahead, survives clamp); envelope resets_at preferred over prose; window-kind sidecar set
    • tests/dev-lead/unit/test_dev_lead_retry.bats — retry skips while a weekly reset is in the future; empty reset + window=weekly suppresses (fail-safe); empty reset + unknown/5h window still retries (fail-open preserved)

    Test Command

    bats tests/dev-lead/unit/ (CI: .github/workflows/test-dev-lead.yml), plus bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Sep 22, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: cannot implement issue #1863 — needs human attention

    A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.

    This hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise vars.ACTION_TIMEOUT_SEC, then re-apply the dev-lead label.

    Partial work was checkpoint-pushed to dev-lead/issue-1863-20260922-1659. The engine produced changes before the timeout; rather than lose them, they were committed with the checkpoint(dev-lead): INCOMPLETE — marker and pushed to that branch. This work is partial and unreviewed — no pull request was opened and no review threads were resolved, so it must not be merged as-is. Use it as a salvage point: continue the branch by hand, or re-apply dev-lead after splitting the issue or raising the budget.

  4. don-petry commented on Sep 22, 2026

    @don-petry
    CollaboratorAuthor

    dev-lead is withholding action on this item.

    It is labeled dev-lead:needs-human (escalated for a human to take over), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.

    To re-enable automated pickup: remove the dev-lead:needs-human label.

  5. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Split, 2026-10-09. dev-lead timed out at its maximum action budget (2700s) on the first attempt and asked for this to be split, so this issue is now an umbrella and keeps the dev-lead:needs-human label on purpose so dev-lead does not pick it up whole. The release conditions in the dispatch note are met (#1865 closed, #1568 merged via PR #1798, the telemetry adapter merged via PR #1888).

    AC1 and AC2 (a day-of-week parser and the 24h clamp) are deliberately not scheduled: no real weekly-cap message exists in the repo, and building the parser on a guessed format is the failure to avoid. Revisit once #2140 has captured a real sample, and only if the telemetry-based fix leaves a gap. Close this umbrella when #2139 and #2140 are done.

  6. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Update 2026-10-09: the first child attempt (PR 2144) was closed after review: it called the usage endpoint from the retry cron, but the retry workflow has no Claude token and the public rate-limit library is not in the production tree, so it would have shipped inert. The child issue now reads the dry-run budget poller's published result instead, so only one workflow holds the credential; it is blocked on the poller (issue 2029, PR 2031).

  7. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Update 2026-10-09: the first child is done. PR 2159 (issue 2139) is merged: the retry cron reads the dry-run budget poller's published result and holds status=rate-limited retries while a session or weekly window is exhausted with a future reset. It needs no token, fails open on every error (no artifact, expired, stale beyond the poller's own staleness bound, unparseable, missing window), reads the artifact once per run, and never reads weekly_scoped. It is inert until the poller starts publishing: the poller's first scheduled run has not happened yet because GitHub drops about 60 percent of scheduled ticks here (tracked, with a fix, in #2160). Remaining on this umbrella: #2140 (capture redacted samples of rate-limit text the parser cannot read, released to dev-lead now). The day-of-week parser and the clamp (AC 1 and 2) stay unscheduled until a real sample exists. Close this umbrella when #2140 is merged and the poller has produced at least one OK record that the cron has read.

  8. don-petry commented on Oct 11, 2026

    @don-petry
    CollaboratorAuthor

    Status 2026-10-11: the evidence this issue is waiting for cannot arrive yet. The fleet monitor report's "Unparsed rate-limit messages" section has not been produced since 2026-09-20, because "Fleet Monitor aborts every run: legacy_attr_file used before it is created" (#2277) aborts the script before any report is written. The fix is in progress under that issue; the first report after it lands is the one to read for a real weekly-cap message.

  9. don-petry commented on Oct 11, 2026

    @don-petry
    CollaboratorAuthor

    Read of the "Unparsed rate-limit messages" data, 2026-10-11 17:40 UTC. The fleet monitor now completes (run 38158033007), but that section goes to the run's step summary only and the report issue is truncated before it, so I read the source instead: the kind: rate_limit_sample records in this repository's token-usage-* artifacts from about the last 25 hours (212 records, 15 message shapes; partial, I stopped the download to spare the shared API quota).

    No weekly Claude cap message is in them. Nothing in the 212 samples mentions a week, and none is a Claude message at all.

    What is there instead:

    Shape Count What it is
    "hit your limit", "quota exceeded", "rate limit exceeded", "quota exceeded for today", "429 too many requests", "429 too many", and a fake Gemini response with 120/20/40 token stats 11 to 24 each, about 150 in all Test fixture strings. CI test runs record them through the real sampler and upload them in a token-usage artifact, so the section is mostly noise from the test suite.
    "[gemini] every API key is cooling down after a rate limit (or capped) on model … — not called" about 57 across 6 shapes The fleet's own message for a skipped Gemini call, not a provider message. Recording it as an unparsed provider message is a second source of noise.
    Gemini HTTP 402, "Your prepayment credits are depleted" 4, from 2026-10-10 19:00 to 2026-10-11 03:35 UTC The only real provider refusal in the sample. It is a billing state, not a rate limit with a reset.

    So this issue is still waiting for a real weekly-cap message, and the place it will appear needs cleaning before it is useful: test runs should not feed the sampler's ledger, and the fleet's own "not called" line should not be recorded as an unparsed provider message. The first point also matters for "Gemini trickle under a budget pause" (#2147), whose first acceptance criterion is that records feeding a dispatch decision come from trusted runs; I have passed it to that work rather than filing it twice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickupdev-lead:needs-humandev-lead could not complete this issue; needs human attention

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions