Repository navigation
Enforce a 15-minute CI feedback budget across the portfolio #654
Description
🤖 Generated by the Agentic Engineer
User Story
As a maintainer and contributor,
I want every pull request's complete CI feedback in less than 15 minutes,
So that CI does not dominate task lead time and every regression forces workflow optimization or redesign before merge.
Maintainer direction, 2026-10-19: implement a portfolio-wide baseline budget check after demonstrating sound GitHub Actions optimization practice, then optimize the outliers. This issue is the implementation and rollout tracker, not evidence that the guard is installed.
Evidence and measurement contract
A fresh inventory found 22 active public mapped repositories. The last three successful first-attempt repository-owned CI runs in each produced 66 complete, paginated job observations. Archived repositories were excluded; private-product rollout and evidence stay private. This success-only sample does not establish coverage of failed/cancelled attempts, all PRs or every organization-required workflow.
Measure from native workflow creation to actual final relevant check/job completion, including runner wait, setup, dependency serialization, execution and teardown. Never use workflow updated_at as completion. Counterexample: monorepo run 39277933824 has a 39m42 update span but its complete 96 jobs finish after 10m34. World run 39277625663 has a 55m13 update span but its 32 jobs finish after 29m53. These are regression fixtures for the future collector, not optimization claims.
Confirmed repository-owned CI outliers in the 66-run sample:
| Repository | Over-budget samples | Worst native job-completion span |
|---|---|---|
| agent-skills | 1/3 | 46m21s |
| ksail | 3/3 | 45m52s |
| world-at-ruin | 3/3 | 38m49s |
| platform-tenant-template | 1/3 | 29m21s |
| data-product-controller | 3/3 | 17m18s |
| platform | 3/3 | 17m11s |
The isolated agent-skills and platform-tenant-template outliers have much faster subsequent samples; diagnose the specific admission/dependency/runner cause rather than presuming persistent slow tests.
The workflow name CI is not the full portfolio feedback path. At the exact KSail revision of run39189008611, independent required Go validation took over 22 minutes as well. Conversely, go-template's own CI takes seconds while the independent Go/scaffold validations take minutes. Native rulesets also require code-quality, scanning and other workflows. Inventory those paths explicitly before claiming a repository is under budget.
Proposed direction
- A hard per-revision budget of less than 900 seconds, not an average or median target. Track PR and merge-group execution separately without disguising total delivery lead time; releases/CD remain separately visible and may not absorb mandatory pre-merge checks merely to game this budget.
- Use immutable subject repository/head/event/run-attempt identities and native timestamps. Keep the original admission clock across retries; a re-run or a late update must not manufacture a fresh 15-minute allowance. Superseded heads cannot approve the current head.
- Fail closed on incomplete pagination, duplicate/mismatched job IDs, missing expected workflows/checks, malformed or reversed timestamps, cancelled/failed current attempts and API errors. A fast failed run is not successful CI.
- Preserve all existing quality, security, test-scope, provenance, cleanup and production-serialization outcomes. A budget failure names its real critical path and links the run; the PR must optimize or redesign that path, not relax the ceiling.
- Prefer extending an existing end-of-CI aggregation surface or an event-driven immutable-source observer over a runner sleeping/polling for 15 minutes. Prove the observer cannot pass early, race a late required workflow, depend on itself or leave a stale green result after a new head/retry.
- Shared implementation belongs to this catalogue. Organization enforcement and adoption belong to declarative GitHub configuration, never imperative ruleset writes. Reuse existing identities and least-privilege metadata access; do not expand credentials merely to collect timing.
- New behavior starts default-off with both states tested; publish, pilot against genuine positive/negative PRs, then activate required enforcement across every active mapped repository. Track and remove the temporary rollout switch. Optional flags and passing YAML are not completed adoption.
Optimization order and protected floors
- Profile exact job/step critical paths and separate admission/runner waits from execution. Balance matrices and parallelize genuinely independent work rather than raising concurrency blindly.
- Eliminate duplicate setup/build work with correctly scoped immutable artifacts and complete cache keys. Exercise both cold and warm states; never reuse a stale verdict, cross trust boundaries or remove build inputs from a key without correctness proof.
- Use dependency caching for dependencies and artifacts for produced outputs; keep untrusted restores and trusted cache writes separate.
- Cancel superseded read-only PR work where safe, but never interrupt protected publication/deployment or required cleanup to meet the budget.
- Use affected-test selection only with an independently verified input/dependency closure and full-coverage fallback. Test-directory paths can be production inputs (platform#5077).
- Redesign genuinely serial long-running scenarios without dropping their outcomes. Moving an unchanged slow mandatory suite after merge is not a speedup.
Acceptance Criteria
- Reviewed design records exact admission/end clocks, expected native check/workflow inventory, retry/supersession semantics, PR/merge-group coverage, failure behavior, privacy, rollback and publication/adoption sequence.
- TDD covers 899/900/901-second boundaries, missing/partial/duplicate API data, API errors after plausible output, mutable update timestamps, clock skew, current-head cancellation, late/missing required workflows, retries and stale-head races.
- The actual immutable-source wrapper is tested in omitted/off/on states, with least-privilege positive and refusal cases; no candidate code or untrusted artifact executes with reporting authority.
- Shared guard is merged, released and exercised from genuine consumers; its own overhead is included and measured.
- Required declarative enforcement is merged, published, reconciled and independently read back for every active mapped repository; none is silently skipped because its workflow is named differently or its repository is private.
- Every confirmed outlier is optimized through reviewed merged PRs, retaining its intended outcomes and showing comparable cold/warm before/after evidence.
- All in-scope current-revision CI completes in less than 900 seconds in the declared verification cohort; failures/unknowns remain failures/unknowns, not exclusions or waivers.
- A deliberately over-budget/refused candidate is blocked, an under-budget complete candidate passes, and the temporary rollout controls are retired after verified adoption.
Alternatives Considered
A blanket job-level timeout-minutes: 15 does not constrain a multi-job critical path and can terminate validation or cleanup. A mutable workflow-update timestamp creates false failures. Checking only ci.yaml, only successful fast jobs or only a median misses real feedback delays. A permanently optional timing report is not enforcement.
Sources and related work
Primary guidance reviewed: workflow diagnostics, concurrency, matrix parallelism, cache scope and trust, artifacts, workflow-run security and native run API.
Related existing work, not replacements: #475 (organization contention), #617 (accurate affected-test consumers), #565 (cache-write correctness), platform#4569 (parallel validation), platform#5077 (production-input closure), ksail#7857 (binary cache correctness), ksail#7869 (separate release-performance experiment).
Rough size: large, cross-repository delivery. Ship shared measurement/enforcement, then each critical-path redesign as independently reviewed increments; do not claim the whole target from one fast run.
Implementation resumed after a fresh passing preflight on 2026-10-19. First delivery slice: #656 (exact single-run measurement); temporary rollout-switch retirement: #657. Whole-PR required enforcement and outlier optimization remain open acceptance gates.
Activity
devantler commentedon Oct 20, 2026
Additional exact-head evidence from KSail PR #7873, source
e973aaa9b9b73196cb6cd5c88be6bd58b00e6472, CI run 39426063004, first attempt created2026-10-20T12:55:16Z:12:56:07–13:21:54Z(25m47s). Its coverage-generation step itself took 23 minutes; this is not merely a cache-restore delay.13:22:02Z; mirror warming then ran13:22:10–13:31:59Z(9m49s).Final result: the complete paginated job inventory contains 77 unique completed jobs, with no failing conclusion. The required aggregate ended at
14:20:28Z: 85m12s from run creation to actual final job completion. This supersedes the earlier partial, 76-job lower-bound snapshot. Native run-level status had said queued while actual jobs were running; use actual job completion and complete expected-workflow closure, not that aggregate state or mutable update timestamps, for the budget verdict. All other current-head checks also passed before the protected merge.Setup attribution, not a speculative optimization claim:
For a separate bounded sample, shared automation #670 completed its original PR run in 378 seconds from native creation to final actual job completion, across 245 unique completed jobs. Its post-merge CI is also green. That is not a comparable before/after optimization and does not establish portfolio compliance.
The whole-PR enforcement, consumer rollout and outlier-redesign acceptance criteria here remain open. Prioritize measured coverage and producer-transfer costs over an unmeasured preload-only change; retain all five mirror-archive integrity checks, required test coverage and the Go admission barrier. This sample cannot meet 15 minutes merely by timing individual jobs, increasing their timeout or moving required validation after merge.
Separate publication evidence, not part of the PR-CI budget: v7.204.0 CD passed at merged source
f009f425180fa93dd5a6104b8c1a409bda9c1a22, across 14 unique completed jobs. Creation14:24:51Zto final job completion15:33:05Zis 68m14s. The CLI release job alone took 55m47s, versus 36m44s for the previous release's corresponding job. Both executed GoReleaser 2.18.3; the inspected release workflows, builder configuration and module files are unchanged, but source and uncontrolled execution conditions differ. This is an outlier observation, not a proven regression cause or optimization result. Protected publication gates and every platform target stayed enabled. The immutable release, both generated Homebrew merges, live guide and four real published-CLI fixture cases are verified.devantler commentedon Oct 25, 2026
Updated complete native outlier evidence: KSail #7992, CI run 40209883898, attempt 1 at
861cb9ac1e2fad898e64e8fde2b43f0d05e2f0bbwas created at 16:03:52 UTC. Its last actual job completed at 17:03:25 UTC on 25 October, proving 59m33s of repository CI feedback time. This uses the complete final 68-job inventory, with 64 success and 4 skipped; no failed or incomplete job was dropped. It uses actual job completion, not workflow updated_at or the observation clock.This measures that complete CI run, not the entire PR feedback path or an optimization. Matrix work was still in progress after the earlier observed lower bound of 52 minutes. Existing build/cache preparation and matrix admission contribute to the observed path, but this sample does not prove organization-wide contention or a particular cache miss root cause. Preserve the original admission clock and all validation/cleanup outcomes when designing the repair.
The catalogue direct-runner deadlines delivered in #758 address hung-runner bounds, not this complete feedback target. Keep #654 open for whole-PR trusted enforcement and actual outlier redesign. The release-replay PR makes no 15-minute or CI speedup claim; its current review found a valid manifest-input selection gap, so it remains draft pending repair and fresh validation.
devantler commentedon Oct 26, 2026
Fresh diagnostic evidence from devantler-tech/ksail#8000 at
a0e91351bb5b3ec3f24f22b32795dc88225ad893, native CI run 40361754948. The final identity-checked inventory contained 68 unique jobs: 61 succeeded, 4 were skipped and 3 failed (two test legs plus the aggregate). First-attempt feedback took 86m54s. This is a failed baseline observation, not a controlled optimization comparison or a successful verification cohort.The two completed jobs entered runners within three seconds of creation. Their overrun therefore includes substantial execution/setup time independently of runner queues. Later jobs also waited roughly 21–34 minutes for runners. Both mechanisms need attention; increasing matrix concurrency alone does not establish a solution.
The completed Vanilla job's native log narrows the restore cost further: the mirror artifact contains 2,851,289,961 bytes; download started at 17:06:44.667 UTC and completed with the expected SHA-256 at 17:20:56.347 UTC, 14m12s later. Transfer therefore accounts for most of that job's 15m39s restoration step. Its separate 521,243,244-byte binary artifact took about 2m21s to download. These are measured producer/consumer bytes and timestamps, not a claim about the provider's capacity or billing plan.
A credential-free local comparison used that exact binary artifact after matching its native archive digest. Streaming the same 521,243,120 binary bytes through Zlib level 1 produced 245,924,607 bytes (52.8% smaller); level 6 produced 227,941,998 bytes (56.3% smaller). Every round trip reproduced the same binary SHA-256. Local unzip/compress/verify time was 5.7s at level 1 and 14.4s at level 6; these are local diagnostic timings, not hosted runner results. The binary was never executed. The current producer explicitly selects level 0. This supports a small binary-transport experiment, separate from the larger mirror payload. GitHub's artifact action guidance recommends selecting compression according to actual compressibility and CPU cost.
The K3s and Vanilla Calico-migration legs subsequently failed before their migration assertions: the 2.85 GB mirror artifact download exhausted the action's five retries. Other legs downloaded that same digest successfully. Preserve this failed attempt in the overall feedback measurement; a later retry cannot turn the original observation into an under-budget pass.
A separate constant-memory pass over the exact 2,851,289,961-byte mirror ZIP produced 2,829,500,771 bytes at raw-deflate level 1: only 0.764% smaller. The complete round trip reproduced its native SHA-256. Nothing was extracted, stored or executed. This probes compression of the whole existing ZIP, not the exact per-file GitHub ZIP uploader, and its 958.5s includes download and verification. It supports retaining level 0 for this already-compressed mirror payload.
The measured binary-only experiment is tracked in devantler-tech/ksail#8018 and included in #8000; hosted validation is still pending. The larger mirror problem requires measuring the selected image set and download/decompression/load costs, then comparing a bounded candidate at the same revision and runner class. Preserve the current cluster/provider matrix, offline/pull behavior, cache trust boundaries and cleanup outcomes. Existing devantler-tech/ksail#7348 concerns mirror-cache retention and should be checked for overlap before opening another cache issue.
No checks were skipped, no timeout was raised, and this observation does not complete the less-than-900-second acceptance criterion. The separate release-build experiment in devantler-tech/ksail#7869 remains separate from this pre-merge system-test bottleneck.