You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Canary Rollout gate (scripts/canary-rollout.sh, _run_json, ~L378-403)
fetches per-workflow run history with gh run list --repo <r> --workflow <wf> in a
tight per-agent × per-tier loop. Under the fleet-wide burst these calls hit
transient failures ("_run_json: failed to fetch run list … after 3 attempts",
~7/run across both repos), which yield empty gates (gate= for ring '' [] (triage=)) → agents don't promote. This is the actual blocker on dev-lead's next→ring0 promotion (its gate shows cum_fail=0, triage=- — no health data, NOT
a real regression). It also breaks sync-issues (regression auto-tracking: issue sync failed (non-fatal)), so real regressions like feature-ideation's go
un-tracked.
Root cause is almost certainly secondary rate-limiting from the per-workflow
call volume — but we can't confirm because _run_jsonswallows the underlying
error (the retry wrapper prints only "transient failure", never the HTTP
status/Retry-After). That observability gap must be closed first.
Scope (all in scripts/canary-rollout.sh)
Surface the real error (do first). In _run_json, capture gh's stderr and
include the HTTP status / message in the ::warning::/::error:: lines
(distinguish 403 secondary-rate-limit vs 5xx vs auth). No more blind "transient
failure". Keep fail-closed: a persistent failure must remain a hard error, never
an empty-but-green gate.
Backoff. Replace the fixed CANARY_GH_RETRY_SLEEP linear retry with
exponential backoff + jitter; honor Retry-After / x-ratelimit-reset when
present; raise default attempts. (Keep the CANARY_GH_RETRIES / CANARY_GH_RETRY_SLEEP env overrides.)
Cut call volume (the real fix). Replace per-workflow gh run list
enumeration with far fewer calls — e.g. one paginated gh run list --repo <repo>
(or a single GraphQL query) per repo, filtered to the registered workflows
locally. This removes the burst that triggers the secondary limit.
Make sync-issues resilient. A run-history fetch failure must not silently
drop the REGRESSION→needs-human issue upsert; on partial data, still upsert what
it can and annotate the gap (so regressions are never silently un-tracked).
Acceptance criteria
A forced fetch failure prints the real HTTP status/reason, not just "transient".
Backoff is exponential+jittered; Retry-After honored.
Per-evaluation gh run list call count is materially reduced (state the before/after in the PR).
sync-issues still opens/updates the regression issue when some fetches fail.
Fails closed: no empty-data gate is ever reported as promotable.
Existing bats for canary-rollout stay green; add a case for _run_json error surfacing.
Harden scripts/canary-rollout.sh run-history fetching so a fleet-wide burst no longer yields empty gates: surface the real gh error, use exponential+jittered backoff honoring Retry-After, collapse per-workflow enumeration to one paginated per-repo fetch (cached), and make the gate + sync-issues fail closed and stay resilient on partial fetch failures.
Implementation Checklist
_run_json (scope 1): capture gh stderr, classify (403 secondary-rate-limit vs 5xx vs auth), and include the real HTTP status/reason in the ::warning::/::error:: lines. Keep non-zero on sustained failure (fail closed).
_run_json (scope 2): replace the linear retry with exponential backoff + jitter; honor Retry-After / x-ratelimit-reset parsed from stderr; raise default attempts (3→5). Keep CANARY_GH_RETRIES / CANARY_GH_RETRY_SLEEP overrides + the 30s self-backoff cap.
Call-volume cut (scope 3): add a per-repo run-list cache (_repo_runs_json) that issues ONE paginated gh run list --repo <repo> per repo (dropping --workflow), widening the window as needed; _run_json filters workflowName + createdAt locally. Collapses repeated (repo,wf,since) and cross-agent shared-repo reads into a single call.
Gate resilience (scope 5): gate-gathering helpers record a data gap (via a gap file) instead of aborting under set -e; _frontier_state forces state=BLOCKED + triage=INCOMPLETE on any gap so an empty-data gate is never promotable, while preserving a real REGRESSION/SUSPECT signal from readable repos.
cmd_evaluate: surface triage=INCOMPLETE with an ::error:: naming the fetch gap.
sync-issues resilience (scope 4): a fetch gap keeps the agent BLOCKED, so the blocker issue is still opened/updated (+needs-human) rather than closed; annotate the partial-data gap in the blocker body/evidence.
Tests to Write (tests/canary_rollout.bats)
_run_json prints the real HTTP status/reason (403 secondary-rate-limit) on failure — not just "transient".
Backoff honors a Retry-After value parsed from gh stderr (sleep called with it); jitter stays within the 30s cap.
Per-repo call-volume reduction: two _run_json calls for the same repo (two workflows / two in-window sinces) issue a single gh run list (counter assertion); --workflow no longer passed.
Local workflowName filtering returns only the queried workflow's runs from a mixed-workflow payload.
Gate fails closed: on a sustained fetch failure _frontier_state yields BLOCKED/INCOMPLETE, never PROMOTE/COMPLETE.
sync-issues still upserts the blocker issue (does not close it) when some fetches fail.
Test Command
bats tests/canary_rollout.bats (plus bash .dev-lead/scripts/dev-lead-lint.sh for shellcheck)
Dev-Lead: cannot implement issue #803 — needs human attention
A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.
This hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise vars.ACTION_TIMEOUT_SEC, then re-apply the dev-lead label.
Note: the engine had produced changes before the timeout. They were on branch dev-lead/issue-803-20260719-1935 on the runner and were not pushed, so they are not recoverable from this run (the runner is ephemeral). The redacted session snippet below shows what it attempted — treat it as context, not restorable output. Re-apply dev-lead after splitting the issue or raising the budget to regenerate the work.
Dev-lead timed out at the 2100s action budget implementing this in one pass. Per the agreed approach, split into two independently-implementable stories (this issue is now the parent tracker):
Activity
don-petry commentedon Jul 19, 2026
Dev-Lead Implementation Plan
Issue: #803 — Harden canary-rollout _run_json: surface real error + backoff + cut call volume (fixes empty gates blocking promotion)
Scope
Harden
scripts/canary-rollout.shrun-history fetching so a fleet-wide burst no longer yields empty gates: surface the realgherror, use exponential+jittered backoff honoringRetry-After, collapse per-workflow enumeration to one paginated per-repo fetch (cached), and make the gate +sync-issuesfail closed and stay resilient on partial fetch failures.Implementation Checklist
_run_json(scope 1): captureghstderr, classify (403 secondary-rate-limit vs 5xx vs auth), and include the real HTTP status/reason in the::warning::/::error::lines. Keep non-zero on sustained failure (fail closed)._run_json(scope 2): replace the linear retry with exponential backoff + jitter; honorRetry-After/x-ratelimit-resetparsed from stderr; raise default attempts (3→5). KeepCANARY_GH_RETRIES/CANARY_GH_RETRY_SLEEPoverrides + the 30s self-backoff cap._repo_runs_json) that issues ONE paginatedgh run list --repo <repo>per repo (dropping--workflow), widening the window as needed;_run_jsonfiltersworkflowName+createdAtlocally. Collapses repeated (repo,wf,since) and cross-agent shared-repo reads into a single call.set -e;_frontier_stateforcesstate=BLOCKED+triage=INCOMPLETEon any gap so an empty-data gate is never promotable, while preserving a real REGRESSION/SUSPECT signal from readable repos.cmd_evaluate: surfacetriage=INCOMPLETEwith an::error::naming the fetch gap.sync-issuesresilience (scope 4): a fetch gap keeps the agent BLOCKED, so the blocker issue is still opened/updated (+needs-human) rather than closed; annotate the partial-data gap in the blocker body/evidence.Tests to Write (tests/canary_rollout.bats)
_run_jsonprints the real HTTP status/reason (403 secondary-rate-limit) on failure — not just "transient".Retry-Aftervalue parsed fromghstderr (sleep called with it); jitter stays within the 30s cap._run_jsoncalls for the same repo (two workflows / two in-window sinces) issue a singlegh run list(counter assertion);--workflowno longer passed.workflowNamefiltering returns only the queried workflow's runs from a mixed-workflow payload._frontier_stateyieldsBLOCKED/INCOMPLETE, neverPROMOTE/COMPLETE.sync-issuesstill upserts the blocker issue (does not close it) when some fetches fail.Test Command
bats tests/canary_rollout.bats(plusbash .dev-lead/scripts/dev-lead-lint.shfor shellcheck)don-petry commentedon Jul 19, 2026
Dev-Lead: cannot implement issue #803 — needs human attention
A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.
timeoutThis hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise
vars.ACTION_TIMEOUT_SEC, then re-apply thedev-leadlabel.don-petry commentedon Jul 19, 2026
Dev-lead timed out at the 2100s action budget implementing this in one pass. Per the agreed approach, split into two independently-implementable stories (this issue is now the parent tracker):
_run_jsonsurface real error + exponential jittered backoff (small, high-impact).De-labeled
dev-lead/dev-lead:needs-humanhere; the splits carry thedev-leadlabel. Closes when both merge.feat: implement issue #810 — [#803 split 1/2] _run_json: surface real…
fix(canary): cache run-list per (repo,workflow) + not-found→empty (#819…
don-petry commentedon Oct 21, 2026
Closing after the 2026-10-21 backlog triage.
Why: Both splits delivered.
Evidence: #810->PR #816, #819->PR #835 merged; #820 CLOSED
Reopen if this still reproduces on main.