Skip to content

ci(canary-rollout): raise the gate job timeout 30 -> 60 min so a sweep finishes and leaves logs - #1266

Merged
don-petry merged 2 commits into
mainfrom
claude/sleepy-bardeen-28vd7h
Oct 9, 2026
Merged

don-petry merged 2 commits into
mainfrom
claude/sleepy-bardeen-28vd7h

Conversation

@don-petry

@don-petry don-petry commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

One-line change (plus an explanatory comment): timeout-minutes: 30 -> 60 on the canary job in .github/workflows/canary-rollout.yml. Requested/approved by the repo owner in the session driving issue 1259.

Why. The last 7 consecutive scheduled sweeps (from 2026-10-07 15:43Z through 2026-10-09 15:31Z, on both the #1260 and #1262 code) were all killed at the 30-minute job timeout:

scheduled run (UTC) duration conclusion
10-09 15:31 35.0 min cancelled
10-09 06:34 35.0 min cancelled
10-08 21:26 42.5 min cancelled
10-08 15:48 35.0 min cancelled
10-08 06:32 35.0 min cancelled
10-07 21:30 35.1 min cancelled
10-07 15:43 35.1 min cancelled
10-07 06:21 28.4 min success

Two consequences: (1) the second gate step (sync-issues: blocker issues + fleet-status summary) never ran in those sweeps, so those outputs went stale; (2) GitHub keeps no logs for a job cancelled by its timeout (BlobNotFound, also for the steps that finished), so the canary timing output added in #1260 could not be read, and the effect of #1262 (date-bounded ingress lists) cannot be measured. The one good sample (10-07 06:21Z, run 37581060882) showed 99% of each gate step inside gh, ~80% of it in gh run list.

60 minutes lets a sweep finish and leave logs. It matches the 60-minute jobs elsewhere in the org standards (standards/ci-standards.md). It is meant to be revisited: once the gate steps are back near their pre-#1238 ~10 min each, the timeout can come back down.

Risk

Low. A genuine hang is bounded at 60 instead of 30 minutes. Because the fleet concurrency group has cancel-in-progress: false, a longer sweep can delay the next scheduled one (GitHub keeps one pending run), which is already the case today with the 35-minute cancelled runs. No script or gate logic changes.

Test plan

  • Existing canary-rollout.yml: canary job timeout-minutes ... bats guards (declares one; at least 20 min) pass.
  • YAML parses. The workflow's behavior is otherwise unchanged.

Rollback

Revert this PR.

Next

After merge, a manual workflow_dispatch (also approved by the owner) produces one completed run whose canary timing blocks can be read, to see whether the ingress run-list cost (#1262) is still dominant. Refs #1259, #1262, #1260.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MjXPsBTaL6WV9AhCH2ajuv


Generated by Claude Code

View guided diff Turn on auto-fix

…p finishes and leaves logs (#1259)

The last 7 consecutive scheduled sweeps (2026-10-07 15:43Z onward, on the #1260 and #1262 code) were
killed at the 30 min timeout, sync-issues never ran, and GitHub keeps no logs for a timeout-cancelled
job, so the CANARY_TIMING output could not be read. 60 min matches the other long-running jobs in
the org standards. Lower it again once the gate steps are back near their pre-#1238 ~10 min.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MjXPsBTaL6WV9AhCH2ajuv
@don-petry
don-petry requested a review from a team as a code owner October 9, 2026 19:21
@don-petry
don-petry enabled auto-merge (squash) October 9, 2026 19:21

Copy link
Copy Markdown
Contributor Author

@donpetry-bot please review head d403ce0: a one-value change (timeout-minutes 30 -> 60 on the canary job) plus an explanatory comment, requested by the owner. Evidence in the PR body: the last 7 scheduled sweeps were all cancelled at the 30-min timeout, and GitHub keeps no logs for timeout-cancelled jobs, so the #1260/#1262 timing output cannot be read. Existing timeout guards pass; YAML parses.


Generated by Claude Code

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because your trial has ended. Ask your workspace admin to add credits to resume reviews. Manage billing

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T19:27:34.430418Z df65af1 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: acd31d58-2fbc-449b-adde-97e8d068cfe3

📥 Commits

Reviewing files that changed from the base of the PR and between 2e3d523 and df65af1.


📒 Files selected for processing (2)
  • .github/workflows/canary-rollout.yml
  • tests/canary_rollout.bats

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.



📝 Summary

Summary by CodeRabbit

  • Chores
    • Canary rollout runs can now continue for up to 60 minutes, reducing the chance that longer-running checks are stopped before they finish.
    • Added a check to ensure the canary timeout remains at least 45 minutes.

Walkthrough

The canary workflow job timeout increases from 30 to 60 minutes. Updated comments describe recent scheduled-run timeouts. A test checks that the timeout is present and at least 45 minutes.

Changes

Canary rollout

Layer / File(s) Summary
Scheduled job timeout
.github/workflows/canary-rollout.yml, tests/canary_rollout.bats
The job timeout increases to 60 minutes. Comments describe recent scheduled runs that timed out before sync-issues finished. The test requires a timeout of at least 45 minutes.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix


Merge Risk: ⚪ Minimal · up to df65a

The longer timeout gives scheduled canary sweeps more time to finish. No actionable issue blocks merging after normal checks.

Pre-merge checks | Passed 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check Passed The title clearly and concisely identifies the main change: increasing the canary gate job timeout from 30 to 60 minutes to allow sweeps to finish and preserve logs.
Description check Passed The description directly explains the timeout change, its reason, risk, testing, rollback, and follow-up plan. It is clearly related to the changeset.
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@don-petry
don-petry disabled auto-merge October 9, 2026 19:22
@donpetry-bot

Copy link
Copy Markdown
Contributor

@don-petry I'm on it — starting a fresh review now. Results will appear in a few minutes.

@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #1266
No changes were committed, but the PR still can't be marked done: required checks SonarCloud, agent-shield / AgentShield are still pending. The retry cron will re-attempt automatically. Next attempt after: 2026-10-09T19:52:39Z

@don-petry

Copy link
Copy Markdown
Contributor Author

Note

@don-petry I reviewed this PR and no code changes were needed, but I can't mark it done yet: required checks SonarCloud, agent-shield / AgentShield are still pending. I'll re-check automatically.
Next attempt after: 2026-10-09T19:52:39Z

@don-petry
don-petry enabled auto-merge (squash) October 9, 2026 19:22

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d403ce0dff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/canary-rollout.yml Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .github/workflows/canary-rollout.yml Resolved
@don-petry
don-petry disabled auto-merge October 9, 2026 19:23
@codeant-ai

codeant-ai Bot commented Oct 9, 2026

Copy link
Copy Markdown

CodeAnt PR Risk: Low Risk

  • The PR appears safe to merge; it raises the scheduled canary job timeout to address repeated timeouts that left fleet status stale.

Assessed commit: d403ce0dfffd

… 30-min value cannot return silently (#1259)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MjXPsBTaL6WV9AhCH2ajuv

Copy link
Copy Markdown
Contributor Author

@donpetry-bot please review head df65af1. Since the first round: Codex P1 asked for a regression test, so a new bats test pins the gate job timeout-minutes floor at 45 (verified failing at 30, passing at 60; suite 449/449). cubic's per-step-timeout suggestion is answered on its thread (whole-sweep time up to 42.5 min; per-step sizing needs the logs this PR exists to get). Note: dev-lead's "added a >= 60 test, sha 6919773" claim refers to a commit that is not on this branch; the real change is df65af1.


Generated by Claude Code

@donpetry-bot

Copy link
Copy Markdown
Contributor

@don-petry I'm on it — starting a fresh review now. Results will appear in a few minutes.

@sonarqubecloud

sonarqubecloud Bot commented Oct 9, 2026

Copy link
Copy Markdown

@don-petry
don-petry enabled auto-merge (squash) October 9, 2026 19:28

Copy link
Copy Markdown
Contributor Author

FYI on the red dev-lead / dispatch check on the old head d403ce0 (run 37979865538): it is not a defect in this PR. dev-lead's fix pass tried to push its own commit while my df65af1 (the timeout-floor test) landed; it could not rebase onto it, refused to force-push over the unincorporated commit ("stopping for human review (#1607)"), and retracted its own claim. Its suite note also says every failing test already failed on the pre-pass head. The current head is df65af1; nothing to fix in the diff.


Generated by Claude Code

@don-petry
don-petry disabled auto-merge October 9, 2026 19:33
@don-petry
don-petry enabled auto-merge (squash) October 9, 2026 19:34
@don-petry

Copy link
Copy Markdown
Contributor Author

Acknowledged: Gemini notice that it cannot review these file types; no finding, no action.

@don-petry

Copy link
Copy Markdown
Contributor Author

Acknowledged: Qodo billing/trial-ended notice; no finding in this PR, no action.

@don-petry

Copy link
Copy Markdown
Contributor Author

Acknowledged: Codex review-status table (completed on df65af1); contains no findings.

@don-petry

Copy link
Copy Markdown
Contributor Author

Acknowledged: CodeRabbit summary reports no actionable comments, pre-merge checks all passed, and no rate-limit, architecture-review, outside-diff or nitpick findings; nothing to change.

@don-petry

Copy link
Copy Markdown
Contributor Author

Acknowledged: CodeAnt rates this PR Low Risk; no findings.

@don-petry

Copy link
Copy Markdown
Contributor Author

Acknowledged: SonarCloud quality gate passed with 0 new issues; no action.

@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — fix-reviews (no-changes)

Agent reasoning
- **Issue comments:** I posted an `informational` disposition on six bot comments, none of which contained a finding:
  - **Gemini:** a notice that it can't review these file types.
  - **Qodo:** a trial-ended notice.
  - **Codex:** a review-status table.
  - **CodeRabbit:** a summary with no actionable comments and all pre-merge checks passed. It has no rate-limit, architecture-review, outside-diff or nitpick findings.
  - **CodeAnt:** a Low Risk rating.
  - **SonarCloud:** quality gate passed with 0 new issues.
- **Left alone:** I didn't touch the comments from `don-petry` and `donpetry-bot`. They are review requests, an FYI and an acknowledgement, not findings.
- **Tests:** Not run, since there were no changes.
Files changed: none.

Copy link
Copy Markdown
Contributor Author

@donpetry-bot re-requesting a review of head df65af1: two mention-acks (19:22Z, 19:26Z) but no verdict after ~45 min, so it looks stalled. State: every required check and third-party review is green on this head (CodeRabbit approved, no open threads, dev-lead no-changes), the only content is the timeout-minutes 30 -> 60 change plus a bats floor test (>= 45).


Generated by Claude Code

@donpetry-bot

Copy link
Copy Markdown
Contributor

@don-petry I'm on it — starting a fresh review now. Results will appear in a few minutes.

@don-petry
don-petry merged commit 24f7aaa into main Oct 9, 2026
34 of 49 checks passed
@don-petry
don-petry deleted the claude/sleepy-bardeen-28vd7h branch October 9, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants