Skip to content

GitHub drops about 60% of scheduled ticks here: give the budget poller a second cron slot and a staleness window that matches #2160

Closed

Description

@don-petry

Problem

The dry-run budget poller (.github/workflows/budget-poller.yml, merged in PR 2031) is scheduled hourly (17 * * * *) and is schedule-only by design (no workflow_dispatch, because a manual dispatch could run branch code that receives the Claude token). Its liveness rule warns when no OK record is newer than bp_stale_hours (default 3 hours), and the retry cron (#2139) ignores a record older than that same bound.

GitHub does not deliver scheduled ticks reliably in this repository. Measured 2026-10-09 on dev-lead-retry.yml (cron 15 */2 * * *, 12 ticks a day), counting runs with event=schedule per day over the earliest 100 runs retained: 4, 5, 5, 5, 6, 4, 4 of 12, about 40% delivery, steady across days. The once-daily schedules (engine-token-liveness.yml, actions-fleet-monitor.yml) fire once a day. The poller itself missed its first two hourly slots after merge (20:17 and 21:17 UTC) with no run listed at all, not even a queued one, which is what dropped ticks look like. GitHub documents scheduled runs as best-effort and delayable under load; this repo also runs 100+ queued workflow runs most of the day.

Consequences if nothing changes: with roughly 40% delivery, a gap over 3 hours is routine, so the "no OK record" warning would be red a large part of the time (a false alarm that trains people to ignore it), and the retry-cron hold would fail open (retry as before) more often than intended.

Scope

  1. Redundant cron slots. Add a second schedule entry at a different minute (for example 47 * * * *) alongside 17 * * * *. The existing concurrency: group: budget-poller already prevents overlap. Two independent chances per hour should lift delivery well above 40%; do not claim a figure, measure it (scope 3).
  2. Staleness window matches reality. Raise the default bp_stale_hours (and the matching documented default) to 6 hours, still overridable by BUDGET_POLLER_STALE_HOURS. State the reasoning in AGENTS.md: the weekly window moves slowly, so a 6-hour-old reading is still meaningful, and a stale record still fails open.
  3. Measure delivery. Have the poller record, in each OK record or its summary, the scheduled time it was meant to run for and the actual start time (the GITHUB_EVENT_NAME, and the cron expression that fired from github.event.schedule), so the real delivery rate and delay can be read from the artifacts instead of guessed. Fleet monitor reports runs received in the last 24 hours against the 48 expected.
  4. Keep every existing guarantee: still schedule-only, still no write path, still read-only permissions, no new secret.

Acceptance criteria

  1. A static test that the workflow has exactly the two cron entries, no workflow_dispatch, no write path and the same permissions as before.
  2. Tests for the new default staleness (6h) and that BUDGET_POLLER_STALE_HOURS still overrides it; the retry-cron tests that depend on the old default keep passing or are updated with the reason stated.
  3. A test that the recorded scheduling fields appear in a record built by bp_build_record and that an older record without them still parses (carry-forward compatibility).
  4. Shellcheck clean; existing tests pass unchanged except where the default changed.

Out of scope

Giving the poller a manual trigger, or changing the retry cron's cadence. Related: #2029, #2139, #1565, PR 2031.

Activity

  1. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2160 — GitHub drops about 60% of scheduled ticks here: give the budget poller a second cron slot and a staleness window that matches

    Scope

    Add a second hourly cron slot (47 * * * *) to the dry-run budget poller, raise the default staleness window from 3h to 6h, and record the trigger and the scheduled-vs-actual start time in each record. The fleet monitor will also report runs received in the last 24h against the 48 expected. All existing guarantees stay: schedule-only, no write path, read-only permissions, no new secret.

    Implementation Checklist

    • .github/workflows/budget-poller.yml: add - cron: '47 * * * *'; pass github.event.schedule to the poll step as BUDGET_POLLER_CRON (GITHUB_EVENT_NAME is already set by the runner). Permissions and triggers are otherwise unchanged.
    • scripts/lib/budget-poller.sh: set BUDGET_POLLER_STALE_HOURS_DEFAULT=6. bp_build_record takes optional args 13 (event) and 14 (cron) and adds trigger_event, trigger_cron, scheduled_for / scheduled_epoch and start_delay_s. The scheduled time is derived from an M * * * * cron and the start time; it is null for anything else. Add BUDGET_POLLER_EXPECTED_RUNS_PER_DAY=48. bp_fleet_section reports runs received in the last 24h against 48, plus the median start delay when known.
    • scripts/budget_poller.sh: forward the event and cron to bp_build_record; add the new fields to the job-summary table.
    • scripts/fleet_monitor.sh: update the default stale-hours comments (3 → 6).
    • AGENTS.md + docs/agentic-interaction-model.md: document the two slots, the 6h default with its reasoning (the weekly window moves slowly, so a 6h reading is still meaningful, and a stale record still fails open), the new record fields, and the delivery line.

    Tests to Write (tests/dev-lead/unit/test_budget_poller.bats)

    • Static: exactly two cron entries (17 * * * *, 47 * * * *), no workflow_dispatch, no write scope, permissions exactly contents: read + actions: read
    • Static: the expected-runs constant equals the cron entries × 24
    • bp_stale_hours defaults to 6, and BUDGET_POLLER_STALE_HOURS still overrides it. A 5h-old OK record is fresh by default and a 7h-old one is stale.
    • bp_build_record records the trigger event, cron, scheduled time and delay. It also handles a non-schedule or unparseable cron (null scheduled fields).
    • The end-to-end poller run records GITHUB_EVENT_NAME / BUDGET_POLLER_CRON
    • Carry-forward compatibility: an older record without the new fields still parses (bp_last_ok, burn rate, fleet section)
    • Fleet section: "runs received in the last 24h: N of 48 expected"
    • Updated with the reason stated: the two existing tests that seeded 4h/5h-old records as stale under the old 3h default now use 7h

    Note: the retry cron (#2139) is not on main in this repo yet, so no retry-cron test depends on the old default. The retry cron should pick up the 6h bound through bp_stale_hours when it lands.

    Test Command

    bats tests/dev-lead/unit/test_budget_poller.bats (plus the full bats suites from lint.yml), bash .dev-lead/scripts/dev-lead-lint.sh

  2. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    • Tests were written first (red), then the implementation; all steps of the plan are done in this workspace.

    Files touched (this workspace, not yet pushed)

    • .github/workflows/budget-poller.yml: adds the second cron slot 47 * * * * and passes github.event.schedule to the poll step as BUDGET_POLLER_CRON (via env:, not interpolated into run:). There are no other trigger, permission, or secret changes.
    • scripts/lib/budget-poller.sh: BUDGET_POLLER_STALE_HOURS_DEFAULT=6 and BUDGET_POLLER_EXPECTED_RUNS_PER_DAY=48. bp_build_record takes optional args 13/14 and adds trigger_event, trigger_cron, scheduled_for, scheduled_epoch and start_delay_s. A new bp_delivery_line reports runs received in the last 24h against 48 expected, with the median start delay, in the fleet section.
    • scripts/budget_poller.sh: forwards GITHUB_EVENT_NAME / BUDGET_POLLER_CRON, and adds the new fields to the job-summary table.
    • scripts/fleet_monitor.sh, AGENTS.md, docs/agentic-interaction-model.md: doc updates covering the two slots, the 6h default with its reasoning, the new record fields and the delivery line.
    • tests/dev-lead/unit/test_budget_poller.bats: 12 new tests. Two existing tests changed their seed age with the reason stated inline: they seeded 4h/5h-old records as stale under the old 3h default, which are now inside the 6h window, so they now use 7h.

    Notes

  3. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2163

    The implementation for issue #2160 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions