Repository navigation
Description
#994 implements #993's orchestration well (report-only timer, control.yml overrides, GitHub Deployments, rollback, STATUS.md). Two gaps remain before it can drive the fleet unattended — implement the spec below on the #994 branch.
A. Gate must implement #548's GRADUATED standard (currently a flat health-over-window check)
Per transition, evaluate on the SOURCE tier (the tier currently running the candidate), counting EXECUTED runs (success+failure; exclude skipped/cancelled) since the candidate's cut:
- next→ring0: dwell ≥ 4h + sample ≥
clamp(round(0.25·avg), 3, 15)(dwell-only if the source tier has no caller — seesoak_start_ring) - ring0→ring1: dwell ≥ 8h + CUMULATIVE-HEALTH-ONLY — WAIVE the fresh sample. ring0 is the consumer-less channel;
.githubper-workflow dogfood volume is too low to ever hit the clamp floor (observed: dependabot-automerge/pr-review-mention had 0.githubruns and stalled). The meaningful sample already came from high-volumenext. - ring1→stable: dwell ≥ 12h + ≥ 1 ring1 run + cumulative-clean
- ALWAYS: cumulative health = ZERO failures / ZERO startup_failures across EVERY tier since the candidate's OWN first cut.
Two hard requirements the flat window misses:
- Per-candidate cumulative window — measure health since the candidate's own
<name>/vX.Y.Ztagger.date, NOT a rolling N-day window. (Critical: pr-review-mention v2.1.1 was cut 2026-06-28; the pre-fix v2.1.0 loop failure on 06-27 must be EXCLUDED — a shared/rolling window false-flags it.) - Robust baseline —
avg= mean of the 14-day per-day executed counts AFTER capping each day at 3× the median day. (Neutralizes spikes like the 2500-run pr-review-mention loop, which otherwise inflates the target to an unreachable 25.)
Failure triage (before any advance, on a failure in the window): classify (a) CANDIDATE REGRESSION — confirm by diffing the reusable at the candidate SHA vs the prior channel SHA → HALT + auto-hold + alert + recommend rollback; (b) PRE-EXISTING/ENVIRONMENTAL (identical in prior version, or comment-cap/rate-limit/infra/data) → report, do NOT rollback, do NOT advance. A raw failure count is not enough.
Dwell floors + sample fractions should be registry-configurable (per-transition knobs), defaulting to the above. Standard: .github#548 (the definitive spec).
B. Auth via GitHub App (replace the raw PAT)
The workflow currently reads GH_PAT_WORKFLOWS. Switch to a GitHub App installation token (scoped, rotatable, auditable):
- Mint at runtime with
actions/create-github-app-token@<pinned-sha>fromSOAK_APP_ID+SOAK_APP_PRIVATE_KEYsecrets (the org will provision the App + secrets separately). - Use the minted token as
GH_TOKENfor cut-release.sh (cross-repo tag moves onpetry-projects/.github) and forPOST /deployments. - App permissions required: Contents: write (tag moves, on the host repos), Deployments: write (traceability), Actions: read (gate reads workflow-run history across all tier repos), Metadata: read. Falls back to nothing — hard-error if the App token can't be minted (don't silently degrade to github.token, which can't move
.githubtags).
Acceptance
- Gate matches feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548 (graduated dwell/sample, robust baseline, per-candidate window, ring0 waiver, triage) with bats coverage.
- Workflow authenticates via the App token; cross-repo promotion + deployments work under it.
- Report-only run reproduces the hand-validated verdicts for the current in-flight state (agent-shield+dependency-audit@stable; auto-rebase/dependabot-rebase/dependabot-automerge/pr-review-mention@ring1).
Activity
don-petry commentedon Jul 1, 2026
Dev-Lead Implementation Plan
Issue: #999 — Harden #993 soak-promote to spec: #548 graduated gate + GitHub App auth
Scope
Replace the flat health-over-window gate in the existing
canary-rolloutsystem with#548's GRADUATED standard (per-transition dwell/sample floors, robust spike-capped baseline,
per-candidate cumulative window since the candidate's own cut, ring0→ring1 sample waiver, and
candidate-regression-vs-environmental failure triage), and switch the workflow's mover credential
from the raw
GH_PAT_WORKFLOWSPAT to a runtime-minted GitHub App installation token (scoped,rotatable) used for cross-repo tag moves and GitHub Deployments — hard-erroring if the App token
cannot be minted. (No rename: the shipped system is
canary-rollout; it is hardened in place.)Implementation Checklist
standards/canary-rings.json— add per-agentgateblock: per-transition knobs(
next->ring0,ring0->ring1,ring1->stable) with dwell-hour floors, sample fraction +clamp bounds + waivers, plus baseline window (14d) and spike-cap multiple (3×), defaulting to feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548.
scripts/lib/canary-rollout.sh— replace the flat decision core with the graduated core:clamp,round_div,median_x2,robust_sample_target(spike-capped baseline → clamp),dwell_met,iso_after(per-candidate window predicate),decide_graduated(dwell + sample + cumulative-health, sample-waiver aware),
classify_failure(triage:REGRESSION vs PRE_EXISTING/ENVIRONMENTAL). Remove flat
decide_gate/min_healthy_runs/failure_rate_permille.scripts/canary-rollout.sh— rewire_frontier_state/evaluateto the graduated core:resolve candidate cut date (release-tag tagger.date), gather the per-candidate cumulative
window (executed = success+failure; cumulative failures + startup_failures across ALL tiers
since cut), source-tier sample count, dwell; load the transition knobs; apply the ring0→ring1
sample waiver and the no-caller dwell-only rule; run triage on any in-window failure.
.github/workflows/canary-rollout.yml— mint an App token withactions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1(v3.2.0) fromSOAK_APP_ID+SOAK_APP_PRIVATE_KEY; hard-error if minting fails; use it asGH_TOKENfor cut/promote/rollback and for
POST /deployments; drop theGH_PAT_WORKFLOWSfallback.Tests to Write
robust_sample_target: spike neutralization — a 2500-run day capped at 3× median so thetarget stays in [3,15] instead of inflating to 25.
clamp/round_div/median_x2primitives.decide_graduated: PROMOTE (dwell+sample met, clean), SOAKING (dwell or sample short),BLOCKED (any cumulative failure/startup_failure), ring0→ring1 sample-waiver PROMOTE on
cumulative-clean, dwell-only (no-caller) PROMOTE.
iso_after: per-candidate window excludes a pre-cut failure (the pr-review-mention 06-27 case).classify_failure: reusable-differs → REGRESSION; identical/env category → PRE_EXISTING.canary-rings.json:gateblock shape + per-transition defaults.resolve_members+evaluate/promoteorchestrator stub tests green.Test Command
bats tests/canary_rollout.batsdon-petry commentedon Jul 1, 2026
Auth: finalized names (App provisioned)
GitHub App petry-projects-release-manager, App ID 4193127, is created. Wire section B to these exact names (not the earlier placeholders):
vars.RELEASE_MANAGER_APP_ID— org variable, already set (= 4193127, scoped to.github-private).secrets.RELEASE_MANAGER_APP_KEY— org secret, the App private key (owner is provisioning it).Workflow token minting:
Hard-error if the token can't be minted — do NOT silently fall back to
github.token(it can't move tags onpetry-projects/.github). Replace the currentGH_PAT_WORKFLOWSreferences with this.don-petry commentedon Jul 1, 2026
Dev-Lead: Implementation Complete
Hardened the in-flight
canary-rolloutpromotion system (the #993/#994 "soak-promote"orchestration; kept its shipped name — no rename) to the #548 graduated standard, and
switched its mover credential to a runtime-minted GitHub App token.
Plan Execution
graduated per-transition core in
scripts/lib/canary-rollout.sh:robust_sample_targetcaps each per-day executed count at 3× themedian day before averaging, so a runaway loop day (e.g. 2500 runs) can't inflate the
target — it stays in
clamp(round(0.25·avg), 3, 15)instead of pinning to the ceiling.<agent>/vX.Y.Ztagger date (candidate_cut_date) and measures health withgh --created ">=<cut>", so a pre-cut failure of a prior version is excluded (thepr-review-mention 06-27-before-06-28-cut case; covered by
iso_after).next→ring0= 4h +clamp(round(0.25·avg),3,15)(dwell-onlywhen the source tier has no caller);
ring0→ring1= 8h + sample waived(cumulative-health-only);
ring1→stable= 12h + ≥1 ring1 run.decide_graduatedreturns BLOCKED on any breach (beats dwell+sample).classify_failure→ REGRESSION (reusable differs from the priorchannel AND non-environmental) = HALT + hold + recommend rollback; PRE_EXISTING
(reusable identical, or comment-cap/rate-limit/infra/data) = report only, no rollback,
no advance.
_reusable_differsdiffs the reusable blob at candidate vs prior-channel SHA.standards/canary-rings.json(
.gate), defaulting to the feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548 values..github/workflows/canary-rollout.ymlnowmints a scoped installation token via
actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1(v3.2.0,SHA looked up via the API) from
SOAK_APP_ID/SOAK_APP_PRIVATE_KEY, hard-errorsif the token is empty (no fallback to
github.token), uses it as the checkout tokenand
GH_TOKENfor the gate's cross-repo reads + tag moves, and records a GitHubDeployment (
POST /deployments) on a real promotion for traceability.Test Results
Full repo suite (all bats files in
lint.yml): 913/913 pass, 0 failures..dev-lead/scripts/dev-lead-lint.sh: pass (shellcheck --severity=warning on 83 files,agent-profile + CODEOWNERS validation).
shellcheckon the changed scripts: clean.Files Changed
scripts/lib/canary-rollout.sh— pure graduated core (clamp, round_div, median_x2,robust_sample_target, dwell_met, iso_after, decide_graduated, classify_failure,
transition_key); removed the flat decide_gate/min_healthy_runs/failure_rate_permille.
scripts/canary-rollout.sh— rewired_frontier_state/evaluate/promote to the graduatedcore: candidate cut date, per-candidate window gh reads (executed = success+failure;
cumulative failures + startup_failures across all tiers), source-tier sample + dwell,
ring0 sample waiver, no-caller dwell-only, triage, and
/home/runner/work/_temp/_runner_file_commands/set_output_4c92173f-96e7-4021-92c9-3fba3dfc7bddpromotion signals.standards/canary-rings.json— added the.gateblock (baseline window 14d, spike-cap 3×,per-transition dwell/sample knobs).
.github/workflows/canary-rollout.yml— App-token mint + hard-error guard + deployment step.tests/canary_rollout.bats— rewrote gate tests for the graduated core (incl. the 2500-runspike-neutralization and per-candidate-window cases) and added three orchestrator
integration verdicts: PROMOTE (clean dwell+sample), BLOCKED+REGRESSION, BLOCKED+PRE_EXISTING.
Notes
STATUS.md / Deployments. The shipped code is
canary-rollout(channel tags ARE the state;no control.yml/STATUS.md). I hardened it in place rather than renaming/re-architecting.
for agent-shield/dependency-audit/auto-rebase/dependabot-*/pr-review-mention, but
canary-rings.json(and the workflow'sagentinput) only configure dev-lead — theonly agent feat: implement issue #993 — Autonomous fleet-wide soak-and-promote workflow for all first-party reusables #994 wired. The gate LOGIC is agent-agnostic and proven via bats; reproducing
those specific verdicts requires adding each agent's real ring membership + gate block to
the SoT and a live report-only run, which needs their actual cut dates/run history. I did
not guess their ring memberships. Recommend a follow-up to register those agents.
evaluate) is safe: it makes only read-onlygh run listcalls and nevermoves a tag; the 4h schedule forces
evaluate. Dwell is derived from the earliest sourcerun in-window (falling back to candidate age) since channel-tag move timestamps aren't
reconstructable without a state store.
don-petry commentedon Jul 1, 2026
✅ Auth provisioned and ready to test. Both are registered on the org (scoped to
.github-private): variableRELEASE_MANAGER_APP_ID=4193127 and secretRELEASE_MANAGER_APP_KEY(App: petry-projects-release-manager). Section B is unblocked — mint the token viaactions/create-github-app-tokenfrom these names, then a report-only run should authenticate and reachpetry-projects/.githubfor cross-repo tag reads. No human dependency remains on auth.don-petry commentedon Jul 1, 2026
Monitor validation of #1002 — gate ✅, one hard auth blocker 🔴
Independently validated #1002 (
canary-rollout) against #548: the graduated gate is correctly implemented — per-transition dwell (4/8/12h),sample_fraction_permille250/500 +sample_clamp_min/max, per-candidate cumulative window, robust spike-capped baseline, ring0→ring1 sample waiver, and REGRESSION-vs-PRE_EXISTING triage. Report-only on the 4h timer, dispatch-gated promote/rollback, GitHub Deployments, and hard-fail-if-no-App-token are all present. Consolidating onto the on-maincanary-rollout(vs the unmergedsoak-promote) is the right call. Nice work.🔴 Blocker — wrong auth identifiers (workflow will fail every run until fixed)
canary-rollout.ymlreferences:But the provisioned auth (live on the org, scoped to .github-private) is:
vars.RELEASE_MANAGER_APP_ID— a variable (= 4193127), NOT a secretsecrets.RELEASE_MANAGER_APP_KEY— the secretFix:
Also update the two doc/error strings that name
SOAK_APP_ID + SOAK_APP_PRIVATE_KEY(header comment + the "Refusing to fall back" error). Since the workflow correctly refusesgithub.token, it hard-errors on every run until this is corrected — this is why a report-only run hasn't produced a clean report yet.Reconcile: close #994
soak-promote.*/release/registry.yml/control.ymlfrom #994 are NOT on main and are superseded bycanary-rollout(#1002) +standards/canary-rings.json. Please close #994 to avoid two-system confusion. (Note: canary-rollout's override model is imperative —workflow_dispatch: promote|rollback+--override— rather than #994's declarativecontrol.yml; that's an acceptable design choice, just confirming it's intentional.)Also
#1002 is
BEHIND— rebase before merge. After the auth fix + rebase + CI-green, the 4h report-only run should authenticate and emit a clean per-version gate report; I'll validate that report against hand-derivation and confirm READY TO FLIP.don-petry commentedon Jul 2, 2026
fix(canary-rollout): use provisioned RELEASE_MANAGER App auth names (#…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
2 remaining items
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…
feat: implement issue #999 — Harden #993 soak-promote to spec: #548 g…