Skip to content

Retry cron should hold retries until a Claude window's real reset, read from the budget poller's published result (split 1 of 2 from the weekly-cap issue) #2139

Closed

Description

@don-petry

Story

Split 1 of 2 from #1863 (which timed out at dev-lead's maximum action budget and asked to be split). As the fleet maintainer, I want the retry cron to read the dry-run budget poller's published result to learn whether a Claude window is exhausted and when it really resets, so that a weekly cap suppresses re-dispatch until its reset instead of retrying every two hours for up to seven days.

Blocked by #2029 (PR 2031, the dry-run poller). Do not start until it has merged. Also blocked by #2148 (tests for the artifact trust filter in bp_download_latest_log, which this cron will rely on): do not start until that has merged too. The dev-lead label is removed until then and will be re-added by the maintainer.

Revision note (2026-10-09)

The first attempt (PR 2144, closed) had the cron call the usage endpoint itself. Reviewed against production, that cannot work: the retry workflow passes no CLAUDE_CODE_OAUTH_TOKEN, and the public agent-rate-limit.sh library exists only under tests/fixtures/, not in the production tree, so the hold would have logged "reader unavailable" on every run and done nothing. Its tests passed because they mocked both. The maintainer decided not to give a second workflow the Claude token. The cron gets no token and no copy of the library. One workflow (the poller) holds the credential; the cron consumes its output.

Why this does not parse prose

No real weekly-cap message exists anywhere in this repo, so a day-of-week parser would be built on a guessed format. Do not parse the rate-limit text. The poller's records already carry the authoritative values.

What the poller publishes (verify against the merged code, do not trust this summary)

The poller workflow uploads an artifact named budget-poller-log (JSONL, 14-day retention). Each record has a poll field ("ok" for a good read) and, among others, session_pct, weekly_all_pct, session_resets_at, weekly_all_resets_at. scripts/lib/budget-poller.sh already has bp_download_latest_log, bp_last_ok and bp_stale_hours. Reuse them; do not duplicate.

Scope

  1. In scripts/dev-lead-retry.sh, once per cron run, download the latest budget-poller-log artifact with the existing helper, take the latest poll=="ok" record with bp_last_ok, and hold status=rate-limited retries while a window in that record is at or above 100% and its *_resets_at is in the future. Hold until that reset. Honour both weekly_all and session; when both are exhausted, honour the later reset. Never weekly_scoped (per-model, handled by model fallback).
  2. Staleness bound. Ignore a record older than the poller's own staleness window (bp_stale_hours). A stale reading means no hold. The poller being down must never silently extend a hold.
  3. Fail open exactly as today for every failure: no artifact, expired artifact, download error, unparseable record, missing window, no poll=="ok" record, or the poller not yet running. Behave as the current "unknown reset means retry" path. Do not invert is_reset_in_future's default.
  4. Log the outcome once per run in a greppable form (for example retry-cron: weekly window exhausted, holding retries until <ISO> (poll <age> old)), and log the fail-open cases distinctly.
  5. Workflow change, minimal: add actions: read to dev-lead-retry.yml permissions (needed to download the artifact) and nothing else. Do not add any secret (no CLAUDE_CODE_OAUTH_TOKEN) and do not widen any other permission. Say so in the PR.

Acceptance criteria

  1. Tests with a mocked artifact download: weekly window at 100% with a future reset suppresses the retry; the same with a past reset does not; a session window at 100% holds only until its own reset; below-limit windows do not hold.
  2. Tests that every failure mode in scope 3 and a stale record fall through to the existing retry behaviour unchanged.
  3. A test that the artifact is downloaded at most once per cron run however many items are retried.
  4. A test that the workflow file contains no CLAUDE_CODE_OAUTH_TOKEN and that actions: read is the only permission added.
  5. Log lines from scope 4 asserted by tests.
  6. No weekday literal anywhere in the change (a test greps for it). No live network in tests; shellcheck clean; existing tests pass unchanged.
  7. A test that proves the production wiring, not only the mocks: run the hold logic against a fixture copied byte-for-byte from a real record shape produced by the merged poller (generate it with the poller's own bp_build_record), so the reader and the writer cannot drift apart.

Out of scope

Parsing the weekly message text (the capture issue), the _emit_reset_iso clamp, giving this workflow any credential, and any variable writes.

Related: #1863, #2029 (blocker), #1565, #2140.

Activity

  1. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2139 — Retry cron should hold retries until a Claude window's real reset, read from the usage telemetry

    Scope

    Before re-dispatching a status=rate-limited item, the retry cron consults the usage adapter once per run and holds rate-limited retries until an exhausted Claude window's (weekly_all / session) real resets_at; every telemetry failure falls open to today's behaviour.

    Implementation Checklist

    • scripts/dev-lead-retry.sh: source lib/usage-telemetry.sh; add usage_hold_init (called once from main, before any scan subshell, so the result is inherited — one adapter call per run) that reads the envelope via usage_telemetry_fetch and the window fields via the public library's arl_token_extract_percent / arl_token_extract_resets_at / arl_token_window_active (no body re-parsing, no weekday literals)
    • usage_hold_active: true only while now < the honoured window's resets_at
    • Apply the hold to the three rate-limited paths only (fix-ci status=rate-limited, review intents with status=rate-limited, issue status=rate-limited); blocked/history-unavailable/partial/failed are not quota and are untouched. is_reset_in_future unchanged.
    • One grep-able log line per run: retry-cron: <weekly|5-hour> window exhausted, holding retries until <ISO>
    • Fail open (no hold) on: library not resolvable, no token, non-200 (429/500), malformed body, missing windows, unparseable resets_at
    • .github/workflows/dev-lead-retry.yml does not expose CLAUDE_CODE_OAUTH_TOKEN — no secret added; the step is a fail-open no-op there (stated in the PR)
    • .github/workflows/lint.yml: add the new bats file to the list

    Tests to Write (tests/dev-lead/unit/test_dev_lead_retry_usage_hold.bats, mocked curl, pinned public-library fixture)

    • weekly at limit + future reset → retry suppressed; + past reset → dispatched
    • 5-hour at limit → suppressed only until its own reset
    • below-limit windows / inactive window → no suppression
    • no token, 429, 500, malformed JSON, missing windows, missing library → existing behaviour unchanged
    • adapter invoked at most once across several retried items
    • log line asserted
    • no weekday literal in the change (grep)

    Test Command

    bats tests/dev-lead/unit/test_dev_lead_retry_usage_hold.bats tests/dev-lead/unit/test_dev_lead_retry*.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  2. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    • usage_hold_init runs once from main before the scan. Scan subshells inherit its result, so the endpoint is read once per cron run. It reads weekly_all / session through arl_token_window_active / arl_token_extract_percent / arl_token_extract_resets_at. When percent ≥ 100 and resets_at is in the future, it records a hold and logs retry-cron: <weekly|5-hour> window exhausted, holding retries until <ISO> once.
    • usage_hold_active is checked only on the status=rate-limited paths (fix-ci, review intents, issues), after the marker's own reset= check. blocked, history-unavailable, partial and failed are not quota and are not held. is_reset_in_future is unchanged.
    • Fail open: these all record no hold, so behaviour is exactly as before:
      • public reader missing (no request is made)
      • no token
      • non-200 response
      • malformed body
      • missing window
      • unparseable resets_at

    Scope item 4: token availability

    .github/workflows/dev-lead-retry.yml does not expose CLAUDE_CODE_OAUTH_TOKEN to the scan step. The public agent-rate-limit.sh is also not present under scripts/lib/ at runtime; it exists here only as the pinned test fixture. So in production the hold is a fail-open no-op today, and normal retries are unchanged. No secret, permission or workflow env was added.

    Turning it on is a separate, deliberate change with two parts:

    1. Expose the token to the step.
    2. Make the library available, either at scripts/lib/agent-rate-limit.sh or through AGENT_RATE_LIMIT_LIB.

    Files touched (this workspace, not yet pushed)

    • scripts/dev-lead-retry.sh: sources lib/usage-telemetry.sh; adds the usage-window hold and wires it into the three rate-limited paths and main.
    • tests/dev-lead/unit/test_dev_lead_retry_usage_hold.bats: new suite. It uses a mocked curl and the pinned public library, with no network. It covers:
      • weekly future/past reset, 5-hour reset expiry, below-limit, inactive and weekly_scoped windows
      • 429, 500, no token, malformed JSON, missing windows and missing library
      • one adapter call per main run across 4 items
      • the log line
      • a weekday-literal grep
    • .github/workflows/lint.yml: registers the new bats file.

    Notes

    If both windows are exhausted, the later reset is honoured. The hold also stops the #2008 stale-edit re-dispatch on a held PR (held=1), the same as an active marker reset already does.

  3. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2144

    The implementation for issue #2139 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  4. changed the title [-]Retry cron should hold retries until a Claude window's real reset, read from the usage telemetry (split 1 of 2 from the weekly-cap issue)[/-] [+]Retry cron should hold retries until a Claude window's real reset, read from the budget poller's published result (split 1 of 2 from the weekly-cap issue)[/+] on Oct 9, 2026
  5. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Oct 9, 2026
  6. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2139 — Retry cron should hold retries until a Claude window's real reset, read from the budget poller's published result

    Scope

    dev-lead-retry.sh reads the budget-poller-log artifact once per cron run, using the existing bp_download_latest_log, bp_last_ok and bp_staleness helpers. If a fresh poll=="ok" record shows session or weekly_all at 100% or more with a future *_resets_at, status=rate-limited retries are held until the later of those resets. Every failure fails open, which matches today's behaviour. No token is added and no rate-limit text is parsed.

    Implementation Checklist

    • scripts/dev-lead-retry.sh: source lib/budget-poller.sh, then add:
    • .github/workflows/dev-lead-retry.yml: add only actions: read. No secret, and no other permission change.

    Tests to Write (tests/dev-lead/unit/test_dev_lead_retry_budget_hold.bats)

    • Weekly window at 100% with a future reset holds. A past reset does not hold. A session window at 100% holds only until its own reset. Below-limit windows do not hold. When both windows are exhausted, the later reset wins. weekly_scoped is ignored.
    • Fail-open cases each fall through to today's retry with a distinct log line: no artifact or expired, download/API error, unparseable record, no poll=="ok" record, missing window or reset, and a stale record.
    • The artifact is downloaded once per main run across several repos and items.
    • Workflow: no CLAUDE_CODE_OAUTH_TOKEN, and permissions are exactly contents: write plus actions: read.
    • Log lines asserted. No weekday literal in the changed files (test builds the names with date, so it contains no literals).
    • Production wiring: a record built by the poller's own bp_build_record, served through bp_download_latest_log with mocked gh/unzip, drives the hold end to end.

    Test Command

    bats tests/dev-lead/unit/test_dev_lead_retry_budget_hold.bats tests/dev-lead/unit/test_dev_lead_retry*.bats tests/dev-lead/unit/test_budget_poller.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  7. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2159

    The implementation for issue #2139 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  8. added a commit that references this issue on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions