Repository navigation
Description
Why
The canary-ring promotion model (epic #495) advances a release next → ring0 → ring1 → stable, gated at every step by a soak on the prior ring. Today that gate is qualitative — the runbook says "confirm its callers' runs are healthy before advancing" — which can't be automated or applied consistently. The first real promotion of the six #482 reusables is in flight (.github-private#870, next now at v2.1.0), so we need a concrete, mechanical soak standard the promotion can be held to (and that canary-rollout.sh / #501 can eventually enforce for cross-repo agents).
What the data says (why a naive rule fails)
14-day run volume on the next tier (.github-private), executed = success+failure (skips excluded); 0 failures across all six → baseline ≈ 0%:
| reusable | exec'd/day | 50%/day | notes |
|---|---|---|---|
| agent-shield | ~71 | ~36 | high volume |
| dependency-audit | ~71 | ~36 | high volume |
| pr-review-mention | ~52 | ~26 | 38% skipped |
| auto-rebase | 9 | ~5 | medium |
| dependabot-automerge | 3 | ~1.5 | 96% skipped |
| dependabot-rebase | 0 | 0 | no runs in 14d |
A literal "N hours + 50% of daily average" breaks three ways:
- The time floor and the count aren't independent — at 71 runs/day, 36 runs takes ~12h, so the count silently dominates for busy reusables and the time floor never binds.
- It collapses at the low end —
dependabot-rebase(0 runs) gets no sample gate and may not fire in any short window;dependabot-automergeis 96% skips. Only time can protect these. - Skips and volume-capping distort the average — must count executed runs only, and the busy three are so high-volume that 50% balloons the soak.
Proposed standard
A ring advance passes for a reusable when all hold (evaluated per reusable, per ring):
- Time floor: ≥ 12 h since the channel tag moved.
- Sample: executed healthy runs ≥
clamp(round(0.5 × daily_avg₁₄d), 5, 25)daily_avg₁₄d= executed runs (success+failure; exclude skipped/cancelled) over the last 14 days, computed across the tier's repos (next→.github-private;ring0→.github;ring1→TalkTerm+bmad;stable→broad fleet).- floor 5 (below that a count is noise), cap 25 (beyond that is diminishing returns and just stalls promotion).
- count only runs started after the cut (so they exercise the new candidate).
- Health:
failure_rate ≤ baseline + εAND zerostartup_failures— the count is the sample size; this is the pass/fail (reusesdecide_gate/failure_rate_permille). - Low-volume fallback: if
0.5 × daily_avg < 5, drop the count gate and require 24 h + ≥ 1 executed healthy run instead (coversdependabot-automerge,dependabot-rebase).
Net effect on the in-flight promotion: busy three gate ~12 h (floor-bound); auto-rebase ~12–13 h (5 runs); the two dependabot reusables on the 24 h fallback — i.e. two waves, not six independent timers.
Open questions (to decide here)
- Per-ring floor. 12 h is sized for
next(first real exposure; doesn't span a daily cycle at 12 h, but the count drags busy ones there anyway). Shouldring0/ring1/stableuse a shorter floor (e.g. 4–6 h) since the candidate already soaked upstream, or stay uniform at 12 h? - Sample bounds. Are
MIN=5/MAX=25right? (Rationale: ~25 clean runs makes a ~30% regression a <0.1% miss; <5 is statistically meaningless.) - ε for the failure-rate comparison — baseline is ~0 today; allowed delta = 0, or a permille threshold (e.g. ≤ baseline + 50‰)?
- Per-reusable vs batch promotion (batching lets a 0-volume reusable gate the whole cohort into the 24 h path).
- Run attribution — "since cut" by created-after-timestamp (proxy) vs detecting the run actually resolved the candidate SHA.
- Where the standard lives —
petry-projects/.github/standards/(org standard) cross-referenced from.github-private/docs/release/{versioning,runbook}.md. - Enforcement — a standalone
soak-check(reads channels + run history → per-reusable PASS/WAIT + reason) for the interim, then consumed bycanary-rollout.shfor cross-repo agents (feat: implement issue #500 — pr-review: dispatcher fails on review_requested (empty gh api URL) + agent comment runaway #501).
Acceptance
- Soak standard written (numbers + rationale + per-ring floor decision) in
standards/and cross-linked from the release runbook/versioning docs. - A mechanical
soak-checkthat, given a reusable + tier, emits PASS/WAIT with the gating reason (time / sample / health / fallback). - Wired into the promotion path (interim: manual
soak-checkbefore eachcut-release.sh --channel <ring> --push; target:canary-rollout.shconsumes it for cross-repo agents).
References
- Epic docs(standards): add Advanced Security (GHAS) org standard #495; promotion in flight
.github-private#870; automated gate.github-private#501; cross-repo cut wiring.github-private#959. - Current qualitative gate:
.github-private/docs/release/runbook.md("Promotion is gated at every ring") +versioning.md.
Until this lands, the in-flight #870 promotion uses a provisional 12 h + healthy-runs judgment before advancing each ring.
Activity
don-petry commentedon Jun 27, 2026
Dev-Lead Implementation Plan
Issue: #548 — Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health
Scope
Add a concrete, mechanical release soak standard to
standards/plus a pure-bash, unit-testedsoak-check(decision core + CLI) that emits per-reusable/per-ring PASS/WAIT with the gating reason (time / sample / health / fallback).Open-question decisions (made here, documented in the standard)
next= 12 h;ring0/ring1/stable= 6 h (candidate already soaked upstream; outer rings stay count-bound at their higher volume).--epsilon-permille). Absorbs one transient failure without masking a real regression.standards/release-soak-standard.md, cross-linked fromstandards/ci-standards.md+ the canary-rings ADR. (Cross-link from.github-private/docs/release/{versioning,runbook}.mdis a follow-up — that repo isn't editable from here.)scripts/soak-check.shfor the interim; consumed bycanary-rollout.sh(feat: implement issue #500 — pr-review: dispatcher fails on review_requested (empty gh api URL) + agent comment runaway #501) later.Implementation Checklist
scripts/lib/soak-gate.sh— pure decision core:soak_ring_floor_hours,soak_sample_threshold(clamp/round),soak_is_low_volume,soak_failure_rate_permille,soak_decide_gatescripts/soak-check.sh— CLI: gathers 14 d executed-run metrics across the tier's repos + cut-time viagh, supports flag overrides, prints PASS/WAITstandards/release-soak-standard.md— numbers + rationale + per-ring decision + soak-check usagestandards/ci-standards.md+docs/initiatives/agent-canary-rings-adr.md— cross-link the new standard.github/workflows/soak-check-tests.yml— shellcheck + bats gate for the new filesTests to Write (TDD — before implementation)
test/scripts/lib/soak-gate.bats— per-ring floors, sample clamp at 5/25, half-up rounding, permille, low-volume fallback (24 h + ≥1), and the full WAIT(time/sample/health-startup/health-rate)/PASS decision matrixtest/scripts/soak-check/soak-check.bats— end-to-end PASS + WAIT scenarios via aghstub, flag overrides, and usage errorsTest Command
bats --print-output-on-failure test/scripts/lib/soak-gate.bats test/scripts/soak-check/and the repo lintbash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jun 27, 2026
Applied live to the in-flight #870
nextsoak (2026-06-27)Ran the proposed gate against all six reusables (
next→v2.1.0, cut2026-06-27T13:57Z). Verdict: all WAIT on the time floor (cut age ~2h; 0 failures; samples already accruing — the busy three had 11/12/148 runs vs a 26 target, so the floor is the binding gate, as intended). Concrete advance windows:cut+12h= 2026-06-28T01:57Zcut+24h= 2026-06-28T13:57ZEdge case the implementation must pin down (open question #2):
auto-rebasehas a 14-day average of 9.1/day →0.5 × avg = 4.55. Underclamp(round(4.55), 5, 25)the count target rounds to 5 (count-mode), but the fallback trigger0.5×avg < 5is true → it lands in the 24h fallback instead. Soround-then-clamp-to-5andfallback-if-<5disagree right at the boundary. Decide one rule, e.g.:0.5×avg < 5(auto-rebase → 24h fallback) — what the live run did; orround(0.5×avg) ≥ 5after clamping (auto-rebase → 12h + 5 runs).Recommend (b) — the clamp floor (
MIN=5) already encodes "5 is the smallest usable sample," so a workflow that can realistically produce 5 runs should be count-gated, and fallback should mean "can't even get 5." That makes the fallback conditionclamp-target unreachable in a reasonable window, not a raw<5on the average.don-petry commentedon Jun 27, 2026
Dev-Lead: issue #548 implementation failed — will retry
The engine failed (
engine-error— e.g. a timeout or transient engine error). This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply thedev-leadlabel to retry now.engine-errordon-petry commentedon Jun 27, 2026
Dev-Lead Implementation Plan
Issue: #548 — Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health
Scope
Ship a concrete, mechanical soak standard (numbers + rationale) in
standards/, plus a unit-testedsoak-checkthat, given a reusable + tier, emits PASS/WAIT with the gating reason, and cross-link it from the canary-rings ADR.Decisions on the open questions
next= 12 h,ring0/ring1= 6 h,stable= 4 h. Downstream rings already soaked upstream, so independent dwell shrinks — but never to 0 (low-volume callers need wall-clock time; some failure modes only surface on a multi-hour cron cadence).MIN=5,MAX=25as proposed.0‰(baseline ≈ 0 today), overridable via--epsilon-permille.standards/release-soak.md, cross-linked fromdocs/initiatives/agent-canary-rings-adr.md(the host-repo runbook/versioning docs live in.github-privateand are out of this repo's scope).soak-check.shfor the interim (manual gate before eachcut-release.sh --channel <ring> --push); consumable later bycanary-rollout.sh.Implementation Checklist
standards/release-soak.md— the standard (time floor table, sample clamp, health gate, low-volume fallback, rationale, decisions).scripts/lib/soak-gate.sh— pure decision core (no gh):soak_sample_target,soak_time_floor_hours,soak_decide.scripts/soak-check.sh— CLI: validates reusable+tier (sourcesring-pins.sh), gathers metrics via gh (overridable by flags), prints PASS/WAIT + gating reason, exits 0=PASS / 20=WAIT.docs/initiatives/agent-canary-rings-adr.md— cross-link the new standard + soak-check..github/workflows/soak-check-tests.yml— shellcheck + bats gate mirroringstandards-deploy-tests.yml.Tests to Write
test/scripts/lib/soak-gate.bats— sample clamp (floor/cap/fallback), per-ring floors, time/sample/health gate ordering, fallback path, PASS path.test/scripts/soak-check/cli.bats— arg validation, override-flag path, PASS/WAIT exit codes, gh-stub gathering path.Test Command
bats test/scripts/lib/soak-gate.bats test/scripts/soak-check/plusbash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jun 27, 2026
Dev-Lead: issue #548 implementation failed — will retry
The engine failed (
engine-error— e.g. a timeout or transient engine error). This was attempt 2 of 3; dev-lead will retry automatically. You can also re-apply thedev-leadlabel to retry now.engine-errordon-petry commentedon Jun 27, 2026
Dev-Lead Implementation Plan
Issue: #548 — Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health
Scope
Add a mechanical release-soak standard to
standards/(resolving the open questions with rationale) plus a standalone, pure-coredsoak-checkthat, given a reusable + tier, emits PASS/WAIT with the gating reason (time / sample / health / fallback) — the interim gate before each ring promotion, later consumable bycanary-rollout.sh.Decisions baked into the standard (open questions)
next=12h,ring0/ring1=6h,stable=4h (inner-soaked candidates re-confirm on local traffic but don't re-prove from scratch). Low-volume fallback floor is a uniform 24h (time is the only protection for near-zero-volume reusables).MIN=5/MAX=25.0permille while baseline ≈ 0 (zero new regressions tolerated during soak); ε is a documented, configurable knob.standards/release-soak.md, cross-linked fromci-standards.mdand the canary-rings ADR.Implementation Checklist
standards/release-soak.md— the standard (numbers + rationale + per-ring floor table)standards/ci-standards.md— cross-link the new standard from the staged-promotion sectiondocs/initiatives/agent-canary-rings-adr.md— cross-link the soak standardscripts/lib/soak-gate.sh— pure decision core (ring floor, required-sample clamp, low-volume detection, failure-rate permille,soak_decide→ PASS/WAIT+reason)scripts/soak-check.sh— CLI wrapper: gather run metrics + cut time viagh, call the core, print per-reusable PASS/WAITTests to Write
test/scripts/lib/soak-gate.bats— clamp bounds (floor 5 / cap 25), low-volume boundary (avg<10), per-ring floor, failure-rate permille rounding, andsoak_decidefor each reason (time / sample / health / fallback) + PASStest/scripts/soak-check/cli.bats— usage/arg validation + a stubbed-ghPASS and WAIT pathTest Command
npx bats test/scripts/lib/soak-gate.bats test/scripts/soak-check(+bash .dev-lead/scripts/dev-lead-lint.shfor shellcheck)don-petry commentedon Jun 27, 2026
Dev-Lead: cannot implement issue #548 — needs human attention
Automatic retries are exhausted (attempt 3 of 3). The most recent failure was
engine-error.Please review the failures above. To retry anyway, remove the
dev-lead:needs-humanlabel and re-apply thedev-leadlabel.don-petry commentedon Jun 30, 2026
Decisions resolved — graduated gate (supersedes the flat 12h+50%)
Owner calls are in; this unblocks the
needs-humanitems. The single fixed gate becomes a graduated, per-transition gate: light/fast early rings, the full metric only atstable, with one signal threading through.Gate per transition
.githubinfra (consumer-less channel; signal =.githubdogfood HEAD)clamp(round(0.25·avg), 3, 15)clamp(round(0.5·avg), 5, 25)The original 12h + 50% + clean is intact — it's now exactly the
ring1→stablegate. Earlier rings are fractional steps (¼, ½ of the sample; 4h, 8h dwell). Cumulative dwell to stable ≈ 24h, distributed with rising rigor.Resolutions to the open questions
avg= mean of 14-day per-day counts after capping each day at 3× the median day (kills outlier spikes like the pr-review-mention loop, which otherwise inflates the target to an unreachable 25). ✅nextcaller, soaks atring0where.githubruns it via local ref). ✅nextruns, so it stays strong even where a tier is low-volume.Human role: override only (not a required gate)
No required-reviewer blocking. Default is autonomous through
stable; humans intervene via a declarativecontrol.yml(pause / hold_at / pin / force / rollback) — see the auto-promoter design issue (linked below). Failure-triage (candidate-regression vs pre-existing/environmental) stays mandatory before any advance.cut-release.sh --promote(the channel-move primitive the gate engine drives) is up in combsmith/.github-private#992.fix(#613): address advisory-bot findings (Gemini/Copilot/CodeRabbit)
feat(#613): relocate canary-rollout engine + bootstrap into .github (…
feat(#613): enable canary-rollout schedule in .github (cutover) (#628)
3 remaining items
don-petry commentedon Jul 13, 2026
Closing — the soak/ring-promotion standard is defined and fully implemented.
The original acceptance boxes were never ticked but are all met in substance (the standard lives as ADRs rather than a `standards/` prose doc — noted for anyone who wants it relocated).
feat(#613): relocate canary-rollout engine + bootstrap into .github (…
feat(#1008): onboard initiative-planner + idea-triage (organic-traffi…
feat(#613): relocate canary-rollout engine + bootstrap into .github (…
feat(#1008): onboard initiative-planner + idea-triage (organic-traffi…
feat(canary): differs-aware benign classes (version_independent) — #668…
feat(#613): relocate canary-rollout engine + bootstrap into .github (…
feat(#1008): onboard initiative-planner + idea-triage (organic-traffi…
feat(#613): relocate canary-rollout engine + bootstrap into .github (…
feat(#1008): onboard initiative-planner + idea-triage (organic-traffi…
feat(canary): differs-aware benign classes (version_independent) — #668…
feat(#613): relocate canary-rollout engine + bootstrap into .github (…
feat(#1008): onboard initiative-planner + idea-triage (organic-traffi…
feat(canary): differs-aware benign classes (version_independent) — #668…