Repository navigation
ci(canary-rollout): raise the gate job timeout 30 -> 60 min so a sweep finishes and leaves logs - #1266
Conversation
…p finishes and leaves logs (#1259) The last 7 consecutive scheduled sweeps (2026-10-07 15:43Z onward, on the #1260 and #1262 code) were killed at the 30 min timeout, sync-issues never ran, and GitHub keeps no logs for a timeout-cancelled job, so the CANARY_TIMING output could not be read. 60 min matches the other long-running jobs in the org standards. Lower it again once the gate steps are back near their pre-#1238 ~10 min. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MjXPsBTaL6WV9AhCH2ajuv
|
@donpetry-bot please review head d403ce0: a one-value change ( Generated by Claude Code |
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
|
ⓘ Qodo reviews are paused because your trial has ended. Ask your workspace admin to add credits to resume reviews. Manage billing |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (2)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 SummarySummary by CodeRabbit
WalkthroughThe canary workflow job timeout increases from 30 to 60 minutes. Updated comments describe recent scheduled-run timeouts. A test checks that the timeout is present and at least 45 minutes. ChangesCanary rollout
Priority: ➖ Normal Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix Merge Risk: ⚪ Minimal · up to The longer timeout gives scheduled canary sweeps more time to finish. No actionable issue blocks merging after normal checks. Pre-merge checks |
|
|
@don-petry I'm on it — starting a fresh review now. Results will appear in a few minutes. |
Dev-Lead — waiting on PR blockers (intent: review-changes)PR: #1266 |
|
Note @don-petry I reviewed this PR and no code changes were needed, but I can't mark it done yet: required checks |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d403ce0dff
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
All reported issues were addressed across 1 file
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
CodeAnt PR Risk: Low Risk
Assessed commit: |
… 30-min value cannot return silently (#1259) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MjXPsBTaL6WV9AhCH2ajuv
|
@donpetry-bot please review head df65af1. Since the first round: Codex P1 asked for a regression test, so a new bats test pins the gate job Generated by Claude Code |
|
@don-petry I'm on it — starting a fresh review now. Results will appear in a few minutes. |
|
|
FYI on the red Generated by Claude Code |
|
Acknowledged: Gemini notice that it cannot review these file types; no finding, no action. |
|
Acknowledged: Qodo billing/trial-ended notice; no finding in this PR, no action. |
|
Acknowledged: Codex review-status table (completed on df65af1); contains no findings. |
|
Acknowledged: CodeRabbit summary reports no actionable comments, pre-merge checks all passed, and no rate-limit, architecture-review, outside-diff or nitpick findings; nothing to change. |
|
Acknowledged: CodeAnt rates this PR Low Risk; no findings. |
|
Acknowledged: SonarCloud quality gate passed with 0 new issues; no action. |
Dev-Lead — fix-reviews (no-changes)Agent reasoning |
|
@donpetry-bot re-requesting a review of head df65af1: two mention-acks (19:22Z, 19:26Z) but no verdict after ~45 min, so it looks stalled. State: every required check and third-party review is green on this head (CodeRabbit approved, no open threads, dev-lead no-changes), the only content is the Generated by Claude Code |
|
@don-petry I'm on it — starting a fresh review now. Results will appear in a few minutes. |



Summary
One-line change (plus an explanatory comment):
timeout-minutes: 30->60on thecanaryjob in.github/workflows/canary-rollout.yml. Requested/approved by the repo owner in the session driving issue 1259.Why. The last 7 consecutive scheduled sweeps (from 2026-10-07 15:43Z through 2026-10-09 15:31Z, on both the #1260 and #1262 code) were all killed at the 30-minute job timeout:
Two consequences: (1) the second gate step (sync-issues: blocker issues + fleet-status summary) never ran in those sweeps, so those outputs went stale; (2) GitHub keeps no logs for a job cancelled by its timeout (
BlobNotFound, also for the steps that finished), so thecanary timingoutput added in #1260 could not be read, and the effect of #1262 (date-bounded ingress lists) cannot be measured. The one good sample (10-07 06:21Z, run 37581060882) showed 99% of each gate step insidegh, ~80% of it ingh run list.60 minutes lets a sweep finish and leave logs. It matches the 60-minute jobs elsewhere in the org standards (
standards/ci-standards.md). It is meant to be revisited: once the gate steps are back near their pre-#1238 ~10 min each, the timeout can come back down.Risk
Low. A genuine hang is bounded at 60 instead of 30 minutes. Because the
fleetconcurrency group hascancel-in-progress: false, a longer sweep can delay the next scheduled one (GitHub keeps one pending run), which is already the case today with the 35-minute cancelled runs. No script or gate logic changes.Test plan
canary-rollout.yml: canary job timeout-minutes ...bats guards (declares one; at least 20 min) pass.Rollback
Revert this PR.
Next
After merge, a manual
workflow_dispatch(also approved by the owner) produces one completed run whosecanary timingblocks can be read, to see whether the ingress run-list cost (#1262) is still dominant. Refs #1259, #1262, #1260.🤖 Generated with Claude Code
https://claude.ai/code/session_01MjXPsBTaL6WV9AhCH2ajuv
Generated by Claude Code