Repository navigation
Description
Summary
The rate-limit reset parser only understands the 5-hour window's message shape. A weekly cap block parses to "unknown", and every downstream consumer treats unknown as "retry now" — so the fleet re-dispatches every two hours against a block that lasts up to seven days.
The chain
1. The regex matches only a time-of-day. parse_reset_time / parse_reset_time_files (scripts/engine.sh:1793, :1810):
grep -oiE 'resets [0-9]{1,2}:[0-9]{2}(am|pm)'A weekly-limit message carries a day-of-week form (the weekly window resets Tuesday ~11:00 America/Chicago), which this pattern cannot match. No match ⇒ the sidecar /tmp/dev-lead-rate-limit-reset is written empty.
2. The clamp cannot express a multi-day reset. _emit_reset_iso (scripts/engine.sh:1766) resolves the parsed H:MM against today, and if that is already past, against tomorrow. Its range is bounded at ~24h by construction. Even if the regex matched a weekly reset, the clamp would collapse it to tomorrow.
3. Unknown is treated as "go". scripts/dev-lead-retry.sh:97:
is_reset_in_future() {
local reset_iso="$1"
[ -z "$reset_iso" ] && return 1 # unknown reset = don't skipSo status=rate-limited reset= (empty) does not suppress the retry. The cron (15 */2 * * *) re-dispatches every two hours.
Consequence
During a weekly-cap exhaustion the fleet does not back off. Each retry re-walks the engine fallback chain, burning Gemini and Copilot quota and consuming MAX_ATTEMPTS on work that provably cannot proceed, then escalates to dev-lead:needs-human for a reason that was never the PR's fault. The per-PR automation budget (10 actions since the last human) and MAX_ATTEMPTS=3 are the only things bounding it, and neither is reset-aware.
The blast radius is bounded — a hard cap fails fast and cheaply on the Claude side — but the behaviour is backwards: the one situation where waiting is provably correct is the one situation where the fleet does not wait.
Acceptance Criteria
- The reset parser recognises a multi-day / day-of-week reset form in addition to the existing
resets H:MMam/pm, and emits a correct ISO-8601 UTC timestamp for it. _emit_reset_iso's today/tomorrow clamp no longer caps a legitimately-further-out reset. A reset up to 7 days ahead round-trips unchanged.- Prefer the authoritative value over parsing prose: when the Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565 telemetry envelope is available, take
resets_atfrom it for the relevant window and use the text parse only as a fallback. Never hardcode a weekday. is_reset_in_futurekeeps its fail-open default for a genuinely unknown reset (do not invert it — an unparseable 5-hour block must still retry), but an empty reset on a marker whose window is known to be weekly suppresses the retry until that window'sresets_at.- The retry cron logs which window it is honouring and until when, so a deliberate multi-day wait is legible rather than looking like a stalled fleet.
- Bats coverage for: the weekly form parsing correctly; a >24h reset surviving the clamp; and the retry cron skipping while a weekly reset is in the future.
Notes for the implementer
- Do not hardcode "Tuesday". The window's reset moves, and the authoritative source is
resets_atfrom the usage endpoint (ADR §4.1). A weekday literal is exactly the mistake the public ADR calls out. - status=rate-limited is emitted for non-quota holds (reason=blocked) — retires the token half #461 deliberately deferred #1568 is retiring the overloaded
status=rate-limitedtoken for non-quota holds. Coordinate: this issue assumesstatus=rate-limitedmeans a genuine quota block. Land status=rate-limited is emitted for non-quota holds (reason=blocked) — retires the token half #461 deliberately deferred #1568's semantics first if they conflict, and say so rather than duplicating the token change here.
References
- Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565 — telemetry envelope carrying the authoritative
resets_at - status=rate-limited is emitted for non-quota holds (reason=blocked) — retires the token half #461 deliberately deferred #1568 / PR feat: implement issue #1568 — status=rate-limited is emitted for non-quota holds (reason=blocked) — retires the token half #461 deliberately deferred #1798 —
status=rate-limitedemitted for non-quota holds (overlapping marker semantics) - dev-lead falsely reports "all AI engines are currently rate-limited" when the real cause is stale cancelled check runs #461 — earlier work on the same marker vocabulary
scripts/engine.sh:1766(clamp),:1793/:1810(parsers),scripts/dev-lead-retry.sh:97(the fail-open)
Dispatch sequencing — deliberately not yet released to dev-lead (2026-09-21)
This issue is fully specified but intentionally carries no dev-lead label yet. Two reasons:
- It is blocked by status=rate-limited is emitted for non-quota holds (reason=blocked) — retires the token half #461 deliberately deferred #1568 (overlapping marker semantics, PR feat: implement issue #1568 — status=rate-limited is emitted for non-quota holds (reason=blocked) — retires the token half #461 deliberately deferred #1798 open).
- The weekly Claude budget stood at 79% with two days left in the window, and every new dev-lead run opens another PR into the rebase treadmill described in Merge protections are unsatisfiable at N>1 open PRs: one merge to main forces a full re-review of every other PR — ~154 forced re-reviews in 48h, zero merges #1864 — which multiplies re-review cost across all open PRs, not just the new one. Adding parallel agent work before pr-review re-runs the full tier stack for a conflict-free base merge — carry the prior verdict forward when the diff-vs-merge-base is unchanged #1865 lands makes the budget problem worse, not better.
Release condition: add the dev-lead label once #1865 has merged, or after the weekly window resets (resets_at from the usage endpoint — do not assume a weekday), whichever comes first, and once the blocking issue above is resolved.
Activity
github-actions commentedon Sep 21, 2026
Thanks for the thorough write-up — the chain from empty sidecar through fail-open retry to multi-day re-dispatch is exactly the kind of thing that's hard to spot without hitting a weekly cap in the wild. To make sure the parser fix targets the right text: could you share the actual message string the weekly-cap response carries? The issue describes the expected shape ('resets Tuesday ~11:00 America/Chicago'), but having the literal API response text (or a sanitised excerpt) will let us write and verify the regex against the real input rather than a reconstruction of it.
don-petry commentedon Sep 22, 2026
Dev-Lead Implementation Plan
Issue: #1863 — A weekly Claude cap parses as 'unknown reset', so the retry cron re-dispatches every 2 hours against a block that can last 7 days
Scope
Teach the reset parser to recognise a weekly / day-of-week reset form (preferring the authoritative
resets_atfrom the #1565 telemetry envelope when present), stop the today/tomorrow clamp from collapsing a multi-day reset, tag rate-limit markers with the known window kind, and make the retry cron fail-safe (suppress) — with legible logging — when a marker's window is known to be weekly.Implementation Checklist
scripts/engine.sh— add_next_weekday_iso(next occurrence of a weekday at H:MM in UTC, up to 7 days out, no clamp)scripts/engine.sh— add_reset_from_envelopereader (authoritativeresets_atfrom the Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565 envelope,${DEV_LEAD_USAGE_ENVELOPE:-/tmp/dev-lead-usage-envelope.json}), tolerant of absencescripts/engine.sh— extendparse_reset_time/parse_reset_time_filesto: prefer envelope, else detect weekly (day-of-week) form, else existingresets H:MMam/pm; write a window-kind sidecar/tmp/dev-lead-rate-limit-window(weekly|5h| empty)scripts/engine.sh—_emit_reset_isoclamp stays for the ≤24h H:MM (5h) case only; weekly path bypasses the clampscripts/dev-lead-fix-reviews.sh—post_reviews_rate_limitedemitswindow=<kind>on the marker when the sidecar is setscripts/dev-lead-fix-issue.sh— issue marker emitswindow=<kind>when the sidecar is setscripts/dev-lead-retry.sh— parsewindow=from markers; keepis_reset_in_futurefail-open default; suppress + log when reset is in future OR (reset empty AND window=weekly); log which window is honoured and until whenTests to Write
tests/dev-lead/unit/test_engine_chain.bats— weekly day-of-week form parses to a correct future ISO-8601 UTC (>24h ahead, survives clamp); enveloperesets_atpreferred over prose; window-kind sidecar settests/dev-lead/unit/test_dev_lead_retry.bats— retry skips while a weekly reset is in the future; empty reset +window=weeklysuppresses (fail-safe); empty reset + unknown/5h window still retries (fail-open preserved)Test Command
bats tests/dev-lead/unit/(CI:.github/workflows/test-dev-lead.yml), plusbash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Sep 22, 2026
Dev-Lead: cannot implement issue #1863 — needs human attention
A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.
timeoutThis hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise
vars.ACTION_TIMEOUT_SEC, then re-apply thedev-leadlabel.don-petry commentedon Sep 22, 2026
dev-lead is withholding action on this item.
It is labeled
dev-lead:needs-human(escalated for a human to take over), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.To re-enable automated pickup: remove the
dev-lead:needs-humanlabel.don-petry commentedon Oct 9, 2026
Split, 2026-10-09. dev-lead timed out at its maximum action budget (2700s) on the first attempt and asked for this to be split, so this issue is now an umbrella and keeps the
dev-lead:needs-humanlabel on purpose so dev-lead does not pick it up whole. The release conditions in the dispatch note are met (#1865 closed, #1568 merged via PR #1798, the telemetry adapter merged via PR #1888).resets_atfrom the usage telemetry and holds retries (covers AC3, AC4, AC5 and the cron half of AC6). Released to dev-lead first.AC1 and AC2 (a day-of-week parser and the 24h clamp) are deliberately not scheduled: no real weekly-cap message exists in the repo, and building the parser on a guessed format is the failure to avoid. Revisit once #2140 has captured a real sample, and only if the telemetry-based fix leaves a gap. Close this umbrella when #2139 and #2140 are done.
don-petry commentedon Oct 9, 2026
Update 2026-10-09: the first child attempt (PR 2144) was closed after review: it called the usage endpoint from the retry cron, but the retry workflow has no Claude token and the public rate-limit library is not in the production tree, so it would have shipped inert. The child issue now reads the dry-run budget poller's published result instead, so only one workflow holds the credential; it is blocked on the poller (issue 2029, PR 2031).
don-petry commentedon Oct 9, 2026
Update 2026-10-09: the first child is done. PR 2159 (issue 2139) is merged: the retry cron reads the dry-run budget poller's published result and holds status=rate-limited retries while a session or weekly window is exhausted with a future reset. It needs no token, fails open on every error (no artifact, expired, stale beyond the poller's own staleness bound, unparseable, missing window), reads the artifact once per run, and never reads weekly_scoped. It is inert until the poller starts publishing: the poller's first scheduled run has not happened yet because GitHub drops about 60 percent of scheduled ticks here (tracked, with a fix, in #2160). Remaining on this umbrella: #2140 (capture redacted samples of rate-limit text the parser cannot read, released to dev-lead now). The day-of-week parser and the clamp (AC 1 and 2) stay unscheduled until a real sample exists. Close this umbrella when #2140 is merged and the poller has produced at least one OK record that the cron has read.
don-petry commentedon Oct 11, 2026
Status 2026-10-11: the evidence this issue is waiting for cannot arrive yet. The fleet monitor report's "Unparsed rate-limit messages" section has not been produced since 2026-09-20, because "Fleet Monitor aborts every run: legacy_attr_file used before it is created" (#2277) aborts the script before any report is written. The fix is in progress under that issue; the first report after it lands is the one to read for a real weekly-cap message.
don-petry commentedon Oct 11, 2026
Read of the "Unparsed rate-limit messages" data, 2026-10-11 17:40 UTC. The fleet monitor now completes (run 38158033007), but that section goes to the run's step summary only and the report issue is truncated before it, so I read the source instead: the
kind: rate_limit_samplerecords in this repository'stoken-usage-*artifacts from about the last 25 hours (212 records, 15 message shapes; partial, I stopped the download to spare the shared API quota).No weekly Claude cap message is in them. Nothing in the 212 samples mentions a week, and none is a Claude message at all.
What is there instead:
So this issue is still waiting for a real weekly-cap message, and the place it will appear needs cleaning before it is useful: test runs should not feed the sampler's ledger, and the fleet's own "not called" line should not be recorded as an unparsed provider message. The first point also matters for "Gemini trickle under a budget pause" (#2147), whose first acceptance criterion is that records feeding a dispatch decision come from trusted runs; I have passed it to that work rather than filing it twice.