Repository navigation
ci: back off for two minutes on every GitHub download - #303
Merged
Merged
Conversation
GitHub release assets failed with HTTP 500 for about a minute again on main (run 38581317265): the native tool installer gave up after 7 s of retries in e2e-hp3.3-3 and gitops-flux-cert-manager. Every CI download from GitHub now backs off for about two minutes, like the CI image build and the Gateway API CRD install already do. The MetalLB install, which had no retry, shares kubectlWithBackoff with the CRD install.
Collaborator
Author
|
Gitar review |
Starting a review of this PR now. Results will appear in the dashboard comment. |
GitHub answers HTTP 502 for 100-job pages of a rerun attempt of the 107-job CI run (run 38581317265, attempt 2) and serves 50-job pages, so both verification gates and the failure reporter failed on the rerun.
Collaborator
Author
|
Gitar review |
Starting a review of this PR now. Results will appear in the dashboard comment. |
phihos
added this pull request to the merge queue
Oct 14, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 14, 2026
The merge-queue scale job (run 38607403533) passed TestScale and then failed its history comparison on an HTTP 502 listing an earlier run's jobs, even in pages of 50. GitHub API reads now retry 500, 502, 503, and 504 responses for about a minute; other errors still fail at once.
Collaborator
Author
|
Gitar review |
Starting a review of this PR now. Results will appear in the dashboard comment. |
Code Review ✅ Approved🟡 Medium risk · Extending GitHub download and API retries, plus retrying manifest applies, could prolong CI jobs or repeat partially completed cluster changes. Adds retry and backoff for CI downloads from GitHub, using VerificationRules No rules evaluated Functional validation Not enabledAuto-approval Not enabledWas this helpful? React with 👍 / 👎 | Gitar |
phihos
added this pull request to the merge queue
Oct 14, 2026
phihos
added a commit
that referenced
this pull request
Oct 14, 2026
The `scale` job failed in #303's first run with admission timeouts during the seed storm (#304). The evidence showed that renders got slow, not why: the job captured no profiles. This makes every CI `scale` run profile itself. - **Controller.** `MUTEX_PROFILE_FRACTION` and `BLOCK_PROFILE_RATE` enable the mutex and block profiles on `/debug/pprof`. Both are off by default because sampling lock waits costs every contended path, and invalid values fail startup. - **Scale tier.** e2e scale runs set a mutex fraction of 10 and a block rate of 1 ms. - **`profile-scale.py`.** It samples in back-to-back 30-second windows from controller start until the test finishes, instead of once at 100/400/600 Ingresses, which missed the Gateway phase where #304 struck. Each `.pilot/scale-profile/window-NNN/` holds, per controller, CPU, heap, mutex, block, goroutines, and metrics before and after, plus `crictl stats` for every container on the node at the window's start and end. - **CI.** The `scale` job sets `PILOT_PROFILE_SCALE=1`, and the profiles go into the existing `ci-evidence-scale` artifact. Verified with a local `TestScale` run, node pinned to 2 cores: 19 windows, no capture errors, 43 MB, and the test passed. That run already narrowed #304 (details in the issue): - the controllers used 1.35 of the node's 1.83 busy cores; - 40–50% of their CPU went to template execution; - 99.9% of mutex contention sat on the session lock in `RenderIncrementalTextFragment`. Local checks: `make lint`, `make test-github-verification`, and `make test-unit PKG=./cmd/haptic/ TEST_RUN_PATTERN=TestConfigureContentionProfiling`. This PR also fixes a verifier race the merge queue used to hide. GitHub builds a PR's test merge on the base branch's current tip, which can be newer than the base named in the event. That happened when #305 merged while this PR's CI was starting (run 38631852057: the merge was onto `ff22ec07`, the event named `25ead279`). The verifier rejected the run, and a rerun would have replayed the same mismatch. It now records the merge's actual base and accepts it only if it descends from the declared base.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub release assets failed with HTTP 500 for about a minute several times today. PR #44 handled the CI image build and the Gateway API CRD install. The first
mainrun after the merge (run 38581317265) then failede2e-hp3.3-3andgitops-flux-cert-managerin the native tool installer, which gave up after 7 s of retries.curl --retry 7 --retry-all-errors: exponential backoff for about two minutes, the same as the CI image build. This covers the native tool installer, the Ingress conformance echo sources, the benchmark's Gateway API bundle, and the schema fetcher.kubectlWithBackoffwith the Gateway API CRD install.The scenario installers (cert-manager, Argo CD, Flux) already retry through
fetchManifest. The--retry 30loops poll a local registry.Local checks:
make lint,make test-github-verification,make test-gateway-benchmark-harness, andgo vet -tags e2e ./tests/e2e/.GitHub also answers HTTP 502 for 100-job pages of a rerun attempt (run 38581317265, attempt 2), so both verification gates and the failure reporter failed on the rerun. GitHub serves 50-job pages, and
scripts/ci/github.pyandverify-run.pynow read job lists in pages of 50.The merge-queue run then passed
TestScalebut failed the scale-history comparison on another 502 listing an earlier run's jobs, even in pages of 50. GitHub API reads in both clients now retry 500, 502, 503, and 504 responses for about a minute; other errors still fail at once.