Skip to content

Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health #548

Closed

Description

@don-petry

Why

The canary-ring promotion model (epic #495) advances a release next → ring0 → ring1 → stable, gated at every step by a soak on the prior ring. Today that gate is qualitative — the runbook says "confirm its callers' runs are healthy before advancing" — which can't be automated or applied consistently. The first real promotion of the six #482 reusables is in flight (.github-private#870, next now at v2.1.0), so we need a concrete, mechanical soak standard the promotion can be held to (and that canary-rollout.sh / #501 can eventually enforce for cross-repo agents).

What the data says (why a naive rule fails)

14-day run volume on the next tier (.github-private), executed = success+failure (skips excluded); 0 failures across all six → baseline ≈ 0%:

reusable exec'd/day 50%/day notes
agent-shield ~71 ~36 high volume
dependency-audit ~71 ~36 high volume
pr-review-mention ~52 ~26 38% skipped
auto-rebase 9 ~5 medium
dependabot-automerge 3 ~1.5 96% skipped
dependabot-rebase 0 0 no runs in 14d

A literal "N hours + 50% of daily average" breaks three ways:

  1. The time floor and the count aren't independent — at 71 runs/day, 36 runs takes ~12h, so the count silently dominates for busy reusables and the time floor never binds.
  2. It collapses at the low end — dependabot-rebase (0 runs) gets no sample gate and may not fire in any short window; dependabot-automerge is 96% skips. Only time can protect these.
  3. Skips and volume-capping distort the average — must count executed runs only, and the busy three are so high-volume that 50% balloons the soak.

Proposed standard

A ring advance passes for a reusable when all hold (evaluated per reusable, per ring):

  • Time floor: ≥ 12 h since the channel tag moved.
  • Sample: executed healthy runs ≥ clamp(round(0.5 × daily_avg₁₄d), 5, 25)
    • daily_avg₁₄d = executed runs (success+failure; exclude skipped/cancelled) over the last 14 days, computed across the tier's repos (next→.github-private; ring0→.github; ring1→TalkTerm+bmad; stable→broad fleet).
    • floor 5 (below that a count is noise), cap 25 (beyond that is diminishing returns and just stalls promotion).
    • count only runs started after the cut (so they exercise the new candidate).
  • Health: failure_rate ≤ baseline + ε AND zero startup_failures — the count is the sample size; this is the pass/fail (reuses decide_gate / failure_rate_permille).
  • Low-volume fallback: if 0.5 × daily_avg < 5, drop the count gate and require 24 h + ≥ 1 executed healthy run instead (covers dependabot-automerge, dependabot-rebase).

Net effect on the in-flight promotion: busy three gate ~12 h (floor-bound); auto-rebase ~12–13 h (5 runs); the two dependabot reusables on the 24 h fallback — i.e. two waves, not six independent timers.

Open questions (to decide here)

  1. Per-ring floor. 12 h is sized for next (first real exposure; doesn't span a daily cycle at 12 h, but the count drags busy ones there anyway). Should ring0/ring1/stable use a shorter floor (e.g. 4–6 h) since the candidate already soaked upstream, or stay uniform at 12 h?
  2. Sample bounds. Are MIN=5 / MAX=25 right? (Rationale: ~25 clean runs makes a ~30% regression a <0.1% miss; <5 is statistically meaningless.)
  3. ε for the failure-rate comparison — baseline is ~0 today; allowed delta = 0, or a permille threshold (e.g. ≤ baseline + 50‰)?
  4. Per-reusable vs batch promotion (batching lets a 0-volume reusable gate the whole cohort into the 24 h path).
  5. Run attribution — "since cut" by created-after-timestamp (proxy) vs detecting the run actually resolved the candidate SHA.
  6. Where the standard lives — petry-projects/.github/standards/ (org standard) cross-referenced from .github-private/docs/release/{versioning,runbook}.md.
  7. Enforcement — a standalone soak-check (reads channels + run history → per-reusable PASS/WAIT + reason) for the interim, then consumed by canary-rollout.sh for cross-repo agents (feat: implement issue #500 — pr-review: dispatcher fails on review_requested (empty gh api URL) + agent comment runaway #501).

Acceptance

  • Soak standard written (numbers + rationale + per-ring floor decision) in standards/ and cross-linked from the release runbook/versioning docs.
  • A mechanical soak-check that, given a reusable + tier, emits PASS/WAIT with the gating reason (time / sample / health / fallback).
  • Wired into the promotion path (interim: manual soak-check before each cut-release.sh --channel <ring> --push; target: canary-rollout.sh consumes it for cross-repo agents).

References

Until this lands, the in-flight #870 promotion uses a provisional 12 h + healthy-runs judgment before advancing each ring.

Activity

  1. added
    ciCI/CD pipeline issues
    dev-leadFor dev-lead agent pickup
    on Jun 27, 2026
  2. don-petry commented on Jun 27, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #548 — Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health

    Scope

    Add a concrete, mechanical release soak standard to standards/ plus a pure-bash, unit-tested soak-check (decision core + CLI) that emits per-reusable/per-ring PASS/WAIT with the gating reason (time / sample / health / fallback).

    Open-question decisions (made here, documented in the standard)

    1. Per-ring floor: next = 12 h; ring0/ring1/stable = 6 h (candidate already soaked upstream; outer rings stay count-bound at their higher volume).
    2. Sample bounds: keep MIN=5 / MAX=25.
    3. ε (failure-rate delta): 50‰, configurable (--epsilon-permille). Absorbs one transient failure without masking a real regression.
    4. Per-reusable vs batch: gate evaluated per reusable, per ring; batching is an operational layer (a 0-volume member drags the cohort to the 24 h fallback) — documented, not enforced in the core.
    5. Run attribution: "since cut" via created-after-timestamp proxy (documented limitation; SHA-resolution is the target).
    6. Where it lives: standards/release-soak-standard.md, cross-linked from standards/ci-standards.md + the canary-rings ADR. (Cross-link from .github-private/docs/release/{versioning,runbook}.md is a follow-up — that repo isn't editable from here.)
    7. Enforcement: standalone scripts/soak-check.sh for the interim; consumed by canary-rollout.sh (feat: implement issue #500 — pr-review: dispatcher fails on review_requested (empty gh api URL) + agent comment runaway #501) later.

    Implementation Checklist

    • scripts/lib/soak-gate.sh — pure decision core: soak_ring_floor_hours, soak_sample_threshold (clamp/round), soak_is_low_volume, soak_failure_rate_permille, soak_decide_gate
    • scripts/soak-check.sh — CLI: gathers 14 d executed-run metrics across the tier's repos + cut-time via gh, supports flag overrides, prints PASS/WAIT
    • standards/release-soak-standard.md — numbers + rationale + per-ring decision + soak-check usage
    • standards/ci-standards.md + docs/initiatives/agent-canary-rings-adr.md — cross-link the new standard
    • .github/workflows/soak-check-tests.yml — shellcheck + bats gate for the new files

    Tests to Write (TDD — before implementation)

    • test/scripts/lib/soak-gate.bats — per-ring floors, sample clamp at 5/25, half-up rounding, permille, low-volume fallback (24 h + ≥1), and the full WAIT(time/sample/health-startup/health-rate)/PASS decision matrix
    • test/scripts/soak-check/soak-check.bats — end-to-end PASS + WAIT scenarios via a gh stub, flag overrides, and usage errors

    Test Command

    bats --print-output-on-failure test/scripts/lib/soak-gate.bats test/scripts/soak-check/ and the repo lint bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Jun 27, 2026

    @don-petry
    ContributorAuthor

    Applied live to the in-flight #870 next soak (2026-06-27)

    Ran the proposed gate against all six reusables (next → v2.1.0, cut 2026-06-27T13:57Z). Verdict: all WAIT on the time floor (cut age ~2h; 0 failures; samples already accruing — the busy three had 11/12/148 runs vs a 26 target, so the floor is the binding gate, as intended). Concrete advance windows:

    • count-mode (agent-shield, dependency-audit, pr-review-mention): cut+12h = 2026-06-28T01:57Z
    • fallback (auto-rebase, dependabot-automerge, dependabot-rebase): cut+24h = 2026-06-28T13:57Z

    Edge case the implementation must pin down (open question #2): auto-rebase has a 14-day average of 9.1/day → 0.5 × avg = 4.55. Under clamp(round(4.55), 5, 25) the count target rounds to 5 (count-mode), but the fallback trigger 0.5×avg < 5 is true → it lands in the 24h fallback instead. So round-then-clamp-to-5 and fallback-if-<5 disagree right at the boundary. Decide one rule, e.g.:

    • (a) fallback strictly when 0.5×avg < 5 (auto-rebase → 24h fallback) — what the live run did; or
    • (b) count-mode whenever round(0.5×avg) ≥ 5 after clamping (auto-rebase → 12h + 5 runs).

    Recommend (b) — the clamp floor (MIN=5) already encodes "5 is the smallest usable sample," so a workflow that can realistically produce 5 runs should be count-gated, and fallback should mean "can't even get 5." That makes the fallback condition clamp-target unreachable in a reasonable window, not a raw <5 on the average.

  4. don-petry commented on Jun 27, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: issue #548 implementation failed — will retry

    The engine failed (engine-error — e.g. a timeout or transient engine error). This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

  5. don-petry commented on Jun 27, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #548 — Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health

    Scope

    Ship a concrete, mechanical soak standard (numbers + rationale) in standards/, plus a unit-tested soak-check that, given a reusable + tier, emits PASS/WAIT with the gating reason, and cross-link it from the canary-rings ADR.

    Decisions on the open questions

    1. Per-ring floor: next = 12 h, ring0/ring1 = 6 h, stable = 4 h. Downstream rings already soaked upstream, so independent dwell shrinks — but never to 0 (low-volume callers need wall-clock time; some failure modes only surface on a multi-hour cron cadence).
    2. Sample bounds: MIN=5, MAX=25 as proposed.
    3. ε: default 0‰ (baseline ≈ 0 today), overridable via --epsilon-permille.
    4. Per-reusable evaluation; batching is left to the operator/promotion script (so a 0-volume reusable can't drag the cohort).
    5. Run attribution: "since cut" via created-after-timestamp proxy (documented limitation).
    6. Lives in standards/release-soak.md, cross-linked from docs/initiatives/agent-canary-rings-adr.md (the host-repo runbook/versioning docs live in .github-private and are out of this repo's scope).
    7. Enforcement: standalone soak-check.sh for the interim (manual gate before each cut-release.sh --channel <ring> --push); consumable later by canary-rollout.sh.

    Implementation Checklist

    • standards/release-soak.md — the standard (time floor table, sample clamp, health gate, low-volume fallback, rationale, decisions).
    • scripts/lib/soak-gate.sh — pure decision core (no gh): soak_sample_target, soak_time_floor_hours, soak_decide.
    • scripts/soak-check.sh — CLI: validates reusable+tier (sources ring-pins.sh), gathers metrics via gh (overridable by flags), prints PASS/WAIT + gating reason, exits 0=PASS / 20=WAIT.
    • docs/initiatives/agent-canary-rings-adr.md — cross-link the new standard + soak-check.
    • .github/workflows/soak-check-tests.yml — shellcheck + bats gate mirroring standards-deploy-tests.yml.

    Tests to Write

    • test/scripts/lib/soak-gate.bats — sample clamp (floor/cap/fallback), per-ring floors, time/sample/health gate ordering, fallback path, PASS path.
    • test/scripts/soak-check/cli.bats — arg validation, override-flag path, PASS/WAIT exit codes, gh-stub gathering path.

    Test Command

    bats test/scripts/lib/soak-gate.bats test/scripts/soak-check/ plus bash .dev-lead/scripts/dev-lead-lint.sh

  6. don-petry commented on Jun 27, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: issue #548 implementation failed — will retry

    The engine failed (engine-error — e.g. a timeout or transient engine error). This was attempt 2 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

  7. don-petry commented on Jun 27, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #548 — Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health

    Scope

    Add a mechanical release-soak standard to standards/ (resolving the open questions with rationale) plus a standalone, pure-cored soak-check that, given a reusable + tier, emits PASS/WAIT with the gating reason (time / sample / health / fallback) — the interim gate before each ring promotion, later consumable by canary-rollout.sh.

    Decisions baked into the standard (open questions)

    1. Per-ring floor — next=12h, ring0/ring1=6h, stable=4h (inner-soaked candidates re-confirm on local traffic but don't re-prove from scratch). Low-volume fallback floor is a uniform 24h (time is the only protection for near-zero-volume reusables).
    2. Sample bounds — keep MIN=5 / MAX=25.
    3. ε — 0 permille while baseline ≈ 0 (zero new regressions tolerated during soak); ε is a documented, configurable knob.
    4. Per-reusable evaluation (batch is a wrapper concern; the gate is per reusable/per ring).
    5. Run attribution — "since cut" by created-after channel-tag timestamp (proxy), documented as such.
    6. Location — standards/release-soak.md, cross-linked from ci-standards.md and the canary-rings ADR.

    Implementation Checklist

    • standards/release-soak.md — the standard (numbers + rationale + per-ring floor table)
    • standards/ci-standards.md — cross-link the new standard from the staged-promotion section
    • docs/initiatives/agent-canary-rings-adr.md — cross-link the soak standard
    • scripts/lib/soak-gate.sh — pure decision core (ring floor, required-sample clamp, low-volume detection, failure-rate permille, soak_decide → PASS/WAIT+reason)
    • scripts/soak-check.sh — CLI wrapper: gather run metrics + cut time via gh, call the core, print per-reusable PASS/WAIT

    Tests to Write

    • test/scripts/lib/soak-gate.bats — clamp bounds (floor 5 / cap 25), low-volume boundary (avg<10), per-ring floor, failure-rate permille rounding, and soak_decide for each reason (time / sample / health / fallback) + PASS
    • test/scripts/soak-check/cli.bats — usage/arg validation + a stubbed-gh PASS and WAIT path

    Test Command

    npx bats test/scripts/lib/soak-gate.bats test/scripts/soak-check (+ bash .dev-lead/scripts/dev-lead-lint.sh for shellcheck)

  8. don-petry commented on Jun 27, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: cannot implement issue #548 — needs human attention

    Automatic retries are exhausted (attempt 3 of 3). The most recent failure was engine-error.

    Please review the failures above. To retry anyway, remove the dev-lead:needs-human label and re-apply the dev-lead label.

  9. don-petry commented on Jun 30, 2026

    @don-petry
    ContributorAuthor

    Decisions resolved — graduated gate (supersedes the flat 12h+50%)

    Owner calls are in; this unblocks the needs-human items. The single fixed gate becomes a graduated, per-transition gate: light/fast early rings, the full metric only at stable, with one signal threading through.

    Gate per transition

    transition tier entered (blast radius) dwell fresh sample (source tier, robust avg) cumulative health approval
    next → ring0 .github infra (consumer-less channel; signal = .github dogfood HEAD) 4h ≥ clamp(round(0.25·avg), 3, 15) 0 fail / 0 startup since first cut auto
    ring0 → ring1 TalkTerm + bmad (2 external low-traffic) 8h ≥ clamp(round(0.5·avg), 5, 25) 0 fail / 0 startup (cumulative) auto
    ring1 → stable broad fleet (4 production) 12h (24h major/first) ring1 low-volume → ≥1 run, lean on cumulative 0 fail / 0 startup and failure-rate ≤ baseline+ε over the cumulative window auto

    The original 12h + 50% + clean is intact — it's now exactly the ring1→stable gate. Earlier rings are fractional steps (¼, ½ of the sample; 4h, 8h dwell). Cumulative dwell to stable ≈ 24h, distributed with rising rigor.

    Resolutions to the open questions

    1. Per-ring floor → graduated 4 / 8 / 12h (not uniform). ✅
    2. Sample bounds → per-transition clamps as above. ✅
    3. ε → 0 by default (cumulative window, so robust). ✅
    4. Per-reusable gating (not batch). ✅
    5. Robust baseline → avg = mean of 14-day per-day counts after capping each day at 3× the median day (kills outlier spikes like the pr-review-mention loop, which otherwise inflates the target to an unreachable 25). ✅
    6. No source-tier consumer → sample is N/A → dwell-only at that tier; real soak happens at the next tier that runs it (e.g. dependabot-rebase has no next caller, soaks at ring0 where .github runs it via local ref). ✅
    7. Cumulative health is the through-line: zero failures / startup_failures across every tier since first cut — dominated by the high-volume next runs, so it stays strong even where a tier is low-volume.

    Human role: override only (not a required gate)

    No required-reviewer blocking. Default is autonomous through stable; humans intervene via a declarative control.yml (pause / hold_at / pin / force / rollback) — see the auto-promoter design issue (linked below). Failure-triage (candidate-regression vs pre-existing/environmental) stays mandatory before any advance.

    cut-release.sh --promote (the channel-move primitive the gate engine drives) is up in combsmith/.github-private#992.

  10. removed
    dev-lead:needs-humandev-lead could not complete this issue; needs human attention
    on Jun 30, 2026
  11. added 3 commits that reference this issue on Jul 8, 2026
  12. 3 remaining items

  13. don-petry commented on Jul 13, 2026

    @don-petry
    ContributorAuthor

    Closing — the soak/ring-promotion standard is defined and fully implemented.

    • Standard: documented in `docs/initiatives/agent-canary-rings-adr.md` + `docs/initiatives/canary-rollout-adr.md` (dwell floors + traffic-scaled sample + cumulative run-health; channels next→ring0→ring1→stable; empty-ring dwell waivers).
    • Mechanical soak-check: `canary-rollout.sh evaluate` emits the PASS/WAIT verdict with the gating reason (PROMOTE / SOAKING / BLOCKED[triage] / AWAITING_CONFIRMATION / COMPLETE) — the pure core `decide_graduated` in `scripts/lib/canary-rollout.sh`.
    • Wired into promotion: scheduled `autocut → promote-all` consumes the gate; now also CI-tested (ci: add canary-rollout-tests.yml — enforce the engine's bats suite (CI gap from #613/#624) #685) with 153 bats.
    • Extended: Enhancement: canary gate should assess correctness, not just run-reliability #668 delivered the correctness layer on top (differs-aware benign / SUSPECT / decision telemetry / require_confirmation / auto-downgrade).

    The original acceptance boxes were never ticked but are all met in substance (the standard lives as ADRs rather than a `standards/` prose doc — noted for anyone who wants it relocated).

  14. added 7 commits that reference this issue on Aug 3, 2026
  15. added 3 commits that reference this issue on Aug 31, 2026
  16. added 3 commits that reference this issue on Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciCI/CD pipeline issuesdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions