Skip to content

Harden canary-rollout _run_json: surface real error + backoff + cut call volume (fixes empty gates blocking promotion) #803

Closed

Description

@don-petry

Problem

The Canary Rollout gate (scripts/canary-rollout.sh, _run_json, ~L378-403)
fetches per-workflow run history with gh run list --repo <r> --workflow <wf> in a
tight per-agent × per-tier loop. Under the fleet-wide burst these calls hit
transient failures ("_run_json: failed to fetch run list … after 3 attempts",
~7/run across both repos), which yield empty gates (gate= for ring '' [] (triage=)) → agents don't promote. This is the actual blocker on dev-lead's
next→ring0 promotion (its gate shows cum_fail=0, triage=- — no health data, NOT
a real regression). It also breaks sync-issues (regression auto-tracking:
issue sync failed (non-fatal)), so real regressions like feature-ideation's go
un-tracked.

Root cause is almost certainly secondary rate-limiting from the per-workflow
call volume — but we can't confirm because _run_json swallows the underlying
error
(the retry wrapper prints only "transient failure", never the HTTP
status/Retry-After). That observability gap must be closed first.

Scope (all in scripts/canary-rollout.sh)

  1. Surface the real error (do first). In _run_json, capture gh's stderr and
    include the HTTP status / message in the ::warning::/::error:: lines
    (distinguish 403 secondary-rate-limit vs 5xx vs auth). No more blind "transient
    failure". Keep fail-closed: a persistent failure must remain a hard error, never
    an empty-but-green gate.
  2. Backoff. Replace the fixed CANARY_GH_RETRY_SLEEP linear retry with
    exponential backoff + jitter; honor Retry-After / x-ratelimit-reset when
    present; raise default attempts. (Keep the CANARY_GH_RETRIES /
    CANARY_GH_RETRY_SLEEP env overrides.)
  3. Cut call volume (the real fix). Replace per-workflow gh run list
    enumeration with far fewer calls — e.g. one paginated gh run list --repo <repo>
    (or a single GraphQL query) per repo, filtered to the registered workflows
    locally. This removes the burst that triggers the secondary limit.
  4. Make sync-issues resilient. A run-history fetch failure must not silently
    drop the REGRESSION→needs-human issue upsert; on partial data, still upsert what
    it can and annotate the gap (so regressions are never silently un-tracked).

Acceptance criteria

  • A forced fetch failure prints the real HTTP status/reason, not just "transient".
  • Backoff is exponential+jittered; Retry-After honored.
  • Per-evaluation gh run list call count is materially reduced (state the before/after in the PR).
  • sync-issues still opens/updates the regression issue when some fetches fail.
  • Fails closed: no empty-data gate is ever reported as promotable.
  • Existing bats for canary-rollout stay green; add a case for _run_json error surfacing.

Activity

  1. don-petry commented on Jul 19, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #803 — Harden canary-rollout _run_json: surface real error + backoff + cut call volume (fixes empty gates blocking promotion)

    Scope

    Harden scripts/canary-rollout.sh run-history fetching so a fleet-wide burst no longer yields empty gates: surface the real gh error, use exponential+jittered backoff honoring Retry-After, collapse per-workflow enumeration to one paginated per-repo fetch (cached), and make the gate + sync-issues fail closed and stay resilient on partial fetch failures.

    Implementation Checklist

    • _run_json (scope 1): capture gh stderr, classify (403 secondary-rate-limit vs 5xx vs auth), and include the real HTTP status/reason in the ::warning::/::error:: lines. Keep non-zero on sustained failure (fail closed).
    • _run_json (scope 2): replace the linear retry with exponential backoff + jitter; honor Retry-After / x-ratelimit-reset parsed from stderr; raise default attempts (3→5). Keep CANARY_GH_RETRIES / CANARY_GH_RETRY_SLEEP overrides + the 30s self-backoff cap.
    • Call-volume cut (scope 3): add a per-repo run-list cache (_repo_runs_json) that issues ONE paginated gh run list --repo <repo> per repo (dropping --workflow), widening the window as needed; _run_json filters workflowName + createdAt locally. Collapses repeated (repo,wf,since) and cross-agent shared-repo reads into a single call.
    • Gate resilience (scope 5): gate-gathering helpers record a data gap (via a gap file) instead of aborting under set -e; _frontier_state forces state=BLOCKED + triage=INCOMPLETE on any gap so an empty-data gate is never promotable, while preserving a real REGRESSION/SUSPECT signal from readable repos.
    • cmd_evaluate: surface triage=INCOMPLETE with an ::error:: naming the fetch gap.
    • sync-issues resilience (scope 4): a fetch gap keeps the agent BLOCKED, so the blocker issue is still opened/updated (+needs-human) rather than closed; annotate the partial-data gap in the blocker body/evidence.

    Tests to Write (tests/canary_rollout.bats)

    • _run_json prints the real HTTP status/reason (403 secondary-rate-limit) on failure — not just "transient".
    • Backoff honors a Retry-After value parsed from gh stderr (sleep called with it); jitter stays within the 30s cap.
    • Per-repo call-volume reduction: two _run_json calls for the same repo (two workflows / two in-window sinces) issue a single gh run list (counter assertion); --workflow no longer passed.
    • Local workflowName filtering returns only the queried workflow's runs from a mixed-workflow payload.
    • Gate fails closed: on a sustained fetch failure _frontier_state yields BLOCKED/INCOMPLETE, never PROMOTE/COMPLETE.
    • sync-issues still upserts the blocker issue (does not close it) when some fetches fail.

    Test Command

    bats tests/canary_rollout.bats (plus bash .dev-lead/scripts/dev-lead-lint.sh for shellcheck)

  2. don-petry commented on Jul 19, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: cannot implement issue #803 — needs human attention

    A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.

    This hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise vars.ACTION_TIMEOUT_SEC, then re-apply the dev-lead label.

    Note: the engine had produced changes before the timeout. They were on branch dev-lead/issue-803-20260719-1935 on the runner and were not pushed, so they are not recoverable from this run (the runner is ephemeral). The redacted session snippet below shows what it attempted — treat it as context, not restorable output. Re-apply dev-lead after splitting the issue or raising the budget to regenerate the work.

  3. removed
    dev-leadFor dev-lead agent pickup
    dev-lead:needs-humandev-lead could not complete this issue; needs human attention
    on Jul 19, 2026
  4. don-petry commented on Jul 19, 2026

    @don-petry
    ContributorAuthor

    Dev-lead timed out at the 2100s action budget implementing this in one pass. Per the agreed approach, split into two independently-implementable stories (this issue is now the parent tracker):

    De-labeled dev-lead/dev-lead:needs-human here; the splits carry the dev-lead label. Closes when both merge.

  5. added 2 commits that reference this issue on Jul 19, 2026
  6. don-petry commented on Oct 21, 2026

    @don-petry
    ContributorAuthor

    Closing after the 2026-10-21 backlog triage.

    Why: Both splits delivered.
    Evidence: #810->PR #816, #819->PR #835 merged; #820 CLOSED

    Reopen if this still reproduces on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions