Skip to content

Harden #993 soak-promote to spec: #548 graduated gate + GitHub App auth #999

Closed

Description

@don-petry

#994 implements #993's orchestration well (report-only timer, control.yml overrides, GitHub Deployments, rollback, STATUS.md). Two gaps remain before it can drive the fleet unattended — implement the spec below on the #994 branch.

A. Gate must implement #548's GRADUATED standard (currently a flat health-over-window check)

Per transition, evaluate on the SOURCE tier (the tier currently running the candidate), counting EXECUTED runs (success+failure; exclude skipped/cancelled) since the candidate's cut:

  • next→ring0: dwell ≥ 4h + sample ≥ clamp(round(0.25·avg), 3, 15) (dwell-only if the source tier has no caller — see soak_start_ring)
  • ring0→ring1: dwell ≥ 8h + CUMULATIVE-HEALTH-ONLY — WAIVE the fresh sample. ring0 is the consumer-less channel; .github per-workflow dogfood volume is too low to ever hit the clamp floor (observed: dependabot-automerge/pr-review-mention had 0 .github runs and stalled). The meaningful sample already came from high-volume next.
  • ring1→stable: dwell ≥ 12h + ≥ 1 ring1 run + cumulative-clean
  • ALWAYS: cumulative health = ZERO failures / ZERO startup_failures across EVERY tier since the candidate's OWN first cut.

Two hard requirements the flat window misses:

  1. Per-candidate cumulative window — measure health since the candidate's own <name>/vX.Y.Z tagger.date, NOT a rolling N-day window. (Critical: pr-review-mention v2.1.1 was cut 2026-06-28; the pre-fix v2.1.0 loop failure on 06-27 must be EXCLUDED — a shared/rolling window false-flags it.)
  2. Robust baseline — avg = mean of the 14-day per-day executed counts AFTER capping each day at 3× the median day. (Neutralizes spikes like the 2500-run pr-review-mention loop, which otherwise inflates the target to an unreachable 25.)

Failure triage (before any advance, on a failure in the window): classify (a) CANDIDATE REGRESSION — confirm by diffing the reusable at the candidate SHA vs the prior channel SHA → HALT + auto-hold + alert + recommend rollback; (b) PRE-EXISTING/ENVIRONMENTAL (identical in prior version, or comment-cap/rate-limit/infra/data) → report, do NOT rollback, do NOT advance. A raw failure count is not enough.

Dwell floors + sample fractions should be registry-configurable (per-transition knobs), defaulting to the above. Standard: .github#548 (the definitive spec).

B. Auth via GitHub App (replace the raw PAT)

The workflow currently reads GH_PAT_WORKFLOWS. Switch to a GitHub App installation token (scoped, rotatable, auditable):

  • Mint at runtime with actions/create-github-app-token@<pinned-sha> from SOAK_APP_ID + SOAK_APP_PRIVATE_KEY secrets (the org will provision the App + secrets separately).
  • Use the minted token as GH_TOKEN for cut-release.sh (cross-repo tag moves on petry-projects/.github) and for POST /deployments.
  • App permissions required: Contents: write (tag moves, on the host repos), Deployments: write (traceability), Actions: read (gate reads workflow-run history across all tier repos), Metadata: read. Falls back to nothing — hard-error if the App token can't be minted (don't silently degrade to github.token, which can't move .github tags).

Acceptance

  • Gate matches feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548 (graduated dwell/sample, robust baseline, per-candidate window, ring0 waiver, triage) with bats coverage.
  • Workflow authenticates via the App token; cross-repo promotion + deployments work under it.
  • Report-only run reproduces the hand-validated verdicts for the current in-flight state (agent-shield+dependency-audit@stable; auto-rebase/dependabot-rebase/dependabot-automerge/pr-review-mention@ring1).

Refs #993, #994, .github#548, .github#869 (standard doc).

Activity

  1. don-petry commented on Jul 1, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #999 — Harden #993 soak-promote to spec: #548 graduated gate + GitHub App auth

    Scope

    Replace the flat health-over-window gate in the existing canary-rollout system with
    #548's GRADUATED standard (per-transition dwell/sample floors, robust spike-capped baseline,
    per-candidate cumulative window since the candidate's own cut, ring0→ring1 sample waiver, and
    candidate-regression-vs-environmental failure triage), and switch the workflow's mover credential
    from the raw GH_PAT_WORKFLOWS PAT to a runtime-minted GitHub App installation token (scoped,
    rotatable) used for cross-repo tag moves and GitHub Deployments — hard-erroring if the App token
    cannot be minted. (No rename: the shipped system is canary-rollout; it is hardened in place.)

    Implementation Checklist

    • standards/canary-rings.json — add per-agent gate block: per-transition knobs
      (next->ring0, ring0->ring1, ring1->stable) with dwell-hour floors, sample fraction +
      clamp bounds + waivers, plus baseline window (14d) and spike-cap multiple (3×), defaulting to feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548.
    • scripts/lib/canary-rollout.sh — replace the flat decision core with the graduated core:
      clamp, round_div, median_x2, robust_sample_target (spike-capped baseline → clamp),
      dwell_met, iso_after (per-candidate window predicate), decide_graduated
      (dwell + sample + cumulative-health, sample-waiver aware), classify_failure (triage:
      REGRESSION vs PRE_EXISTING/ENVIRONMENTAL). Remove flat decide_gate / min_healthy_runs /
      failure_rate_permille.
    • scripts/canary-rollout.sh — rewire _frontier_state/evaluate to the graduated core:
      resolve candidate cut date (release-tag tagger.date), gather the per-candidate cumulative
      window (executed = success+failure; cumulative failures + startup_failures across ALL tiers
      since cut), source-tier sample count, dwell; load the transition knobs; apply the ring0→ring1
      sample waiver and the no-caller dwell-only rule; run triage on any in-window failure.
    • .github/workflows/canary-rollout.yml — mint an App token with
      actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 (v3.2.0) from
      SOAK_APP_ID + SOAK_APP_PRIVATE_KEY; hard-error if minting fails; use it as GH_TOKEN
      for cut/promote/rollback and for POST /deployments; drop the GH_PAT_WORKFLOWS fallback.

    Tests to Write

    • robust_sample_target: spike neutralization — a 2500-run day capped at 3× median so the
      target stays in [3,15] instead of inflating to 25.
    • clamp / round_div / median_x2 primitives.
    • decide_graduated: PROMOTE (dwell+sample met, clean), SOAKING (dwell or sample short),
      BLOCKED (any cumulative failure/startup_failure), ring0→ring1 sample-waiver PROMOTE on
      cumulative-clean, dwell-only (no-caller) PROMOTE.
    • iso_after: per-candidate window excludes a pre-cut failure (the pr-review-mention 06-27 case).
    • classify_failure: reusable-differs → REGRESSION; identical/env category → PRE_EXISTING.
    • canary-rings.json: gate block shape + per-transition defaults.
    • Keep resolve_members + evaluate/promote orchestrator stub tests green.

    Test Command

    bats tests/canary_rollout.bats

  2. don-petry commented on Jul 1, 2026

    @don-petry
    CollaboratorAuthor

    Auth: finalized names (App provisioned)

    GitHub App petry-projects-release-manager, App ID 4193127, is created. Wire section B to these exact names (not the earlier placeholders):

    • vars.RELEASE_MANAGER_APP_ID — org variable, already set (= 4193127, scoped to .github-private).
    • secrets.RELEASE_MANAGER_APP_KEY — org secret, the App private key (owner is provisioning it).

    Workflow token minting:

    - id: app-token
      uses: actions/create-github-app-token@<pinned-sha>   # v2
      with:
        app-id: ${{ vars.RELEASE_MANAGER_APP_ID }}
        private-key: ${{ secrets.RELEASE_MANAGER_APP_KEY }}
        owner: petry-projects
        repositories: >-
          .github,.github-private,TalkTerm,bmad-bgreat-suite,markets,broodly,ContentTwin,google-app-scripts
    # then: GH_TOKEN: ${{ steps.app-token.outputs.token }} for cut-release.sh + POST /deployments

    Hard-error if the token can't be minted — do NOT silently fall back to github.token (it can't move tags on petry-projects/.github). Replace the current GH_PAT_WORKFLOWS references with this.

  3. don-petry commented on Jul 1, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Hardened the in-flight canary-rollout promotion system (the #993/#994 "soak-promote"
    orchestration; kept its shipped name — no rename) to the #548 graduated standard, and
    switched its mover credential to a runtime-minted GitHub App token.

    Plan Execution

    • A — Graduated gate (feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548). Replaced the flat health-over-window check with a
      graduated per-transition core in scripts/lib/canary-rollout.sh:
      • Robust baseline: robust_sample_target caps each per-day executed count at 3× the
        median day before averaging, so a runaway loop day (e.g. 2500 runs) can't inflate the
        target — it stays in clamp(round(0.25·avg), 3, 15) instead of pinning to the ceiling.
      • Per-candidate cumulative window: the orchestrator resolves the candidate's own
        <agent>/vX.Y.Z tagger date (candidate_cut_date) and measures health with
        gh --created ">=<cut>", so a pre-cut failure of a prior version is excluded (the
        pr-review-mention 06-27-before-06-28-cut case; covered by iso_after).
      • Graduated dwell/sample: next→ring0 = 4h + clamp(round(0.25·avg),3,15) (dwell-only
        when the source tier has no caller); ring0→ring1 = 8h + sample waived
        (cumulative-health-only); ring1→stable = 12h + ≥1 ring1 run.
      • ALWAYS: cumulative ZERO failures / ZERO startup_failures across every tier since cut →
        decide_graduated returns BLOCKED on any breach (beats dwell+sample).
      • Failure triage: classify_failure → REGRESSION (reusable differs from the prior
        channel AND non-environmental) = HALT + hold + recommend rollback; PRE_EXISTING
        (reusable identical, or comment-cap/rate-limit/infra/data) = report only, no rollback,
        no advance. _reusable_differs diffs the reusable blob at candidate vs prior-channel SHA.
      • Knobs are registry-configurable per transition in standards/canary-rings.json
        (.gate), defaulting to the feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548 values.
    • B — GitHub App auth + deployments. .github/workflows/canary-rollout.yml now
      mints a scoped installation token via
      actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 (v3.2.0,
      SHA looked up via the API) from SOAK_APP_ID/SOAK_APP_PRIVATE_KEY, hard-errors
      if the token is empty (no fallback to github.token), uses it as the checkout token
      and GH_TOKEN for the gate's cross-repo reads + tag moves, and records a GitHub
      Deployment (POST /deployments) on a real promotion for traceability.

    Test Results

    ok 28 classify_failure: environmental category → PRE_EXISTING even if reusable changed
    ok 29 next_channel_in_order: walks the ring order
    ok 30 next_channel_in_order: last ring → empty
    ok 31 transition_key: frontier maps to its source→frontier key
    ok 32 canary-rings.json: valid JSON + dev-lead host + ordered rings
    ok 33 canary-rings.json: gate block carries #548 per-transition defaults
    ok 34 orchestrator: resolve_members expands $host / $org_infra / * 
    ok 35 orchestrator: evaluate prints a per-ring gate report and exits 0 (read-only)
    ok 36 orchestrator: promote --override --dry-run shows the move but never pushes
    ok 37 orchestrator: PROMOTE verdict — dwell + sample met on a clean per-candidate window
    ok 38 orchestrator: BLOCKED + REGRESSION — in-window failure with a changed reusable
    ok 39 orchestrator: BLOCKED + PRE_EXISTING — in-window failure but reusable unchanged
    

    Full repo suite (all bats files in lint.yml): 913/913 pass, 0 failures.
    .dev-lead/scripts/dev-lead-lint.sh: pass (shellcheck --severity=warning on 83 files,
    agent-profile + CODEOWNERS validation). shellcheck on the changed scripts: clean.

    Files Changed

    • scripts/lib/canary-rollout.sh — pure graduated core (clamp, round_div, median_x2,
      robust_sample_target, dwell_met, iso_after, decide_graduated, classify_failure,
      transition_key); removed the flat decide_gate/min_healthy_runs/failure_rate_permille.
    • scripts/canary-rollout.sh — rewired _frontier_state/evaluate/promote to the graduated
      core: candidate cut date, per-candidate window gh reads (executed = success+failure;
      cumulative failures + startup_failures across all tiers), source-tier sample + dwell,
      ring0 sample waiver, no-caller dwell-only, triage, and /home/runner/work/_temp/_runner_file_commands/set_output_4c92173f-96e7-4021-92c9-3fba3dfc7bdd promotion signals.
    • standards/canary-rings.json — added the .gate block (baseline window 14d, spike-cap 3×,
      per-transition dwell/sample knobs).
    • .github/workflows/canary-rollout.yml — App-token mint + hard-error guard + deployment step.
    • tests/canary_rollout.bats — rewrote gate tests for the graduated core (incl. the 2500-run
      spike-neutralization and per-candidate-window cases) and added three orchestrator
      integration verdicts: PROMOTE (clean dwell+sample), BLOCKED+REGRESSION, BLOCKED+PRE_EXISTING.

    Notes

    • Scope reconciliation: the issue describes feat: implement issue #993 — Autonomous fleet-wide soak-and-promote workflow for all first-party reusables #994 as "soak-promote" with control.yml /
      STATUS.md / Deployments. The shipped code is canary-rollout (channel tags ARE the state;
      no control.yml/STATUS.md). I hardened it in place rather than renaming/re-architecting.
    • Multi-agent verdict reproduction (follow-up): the acceptance lists in-flight verdicts
      for agent-shield/dependency-audit/auto-rebase/dependabot-*/pr-review-mention, but
      canary-rings.json (and the workflow's agent input) only configure dev-lead — the
      only agent feat: implement issue #993 — Autonomous fleet-wide soak-and-promote workflow for all first-party reusables #994 wired. The gate LOGIC is agent-agnostic and proven via bats; reproducing
      those specific verdicts requires adding each agent's real ring membership + gate block to
      the SoT and a live report-only run, which needs their actual cut dates/run history. I did
      not guess their ring memberships. Recommend a follow-up to register those agents.
    • Report-only (evaluate) is safe: it makes only read-only gh run list calls and never
      moves a tag; the 4h schedule forces evaluate. Dwell is derived from the earliest source
      run in-window (falling back to candidate age) since channel-tag move timestamps aren't
      reconstructable without a state store.
  4. don-petry commented on Jul 1, 2026

    @don-petry
    CollaboratorAuthor

    ✅ Auth provisioned and ready to test. Both are registered on the org (scoped to .github-private): variable RELEASE_MANAGER_APP_ID=4193127 and secret RELEASE_MANAGER_APP_KEY (App: petry-projects-release-manager). Section B is unblocked — mint the token via actions/create-github-app-token from these names, then a report-only run should authenticate and reach petry-projects/.github for cross-repo tag reads. No human dependency remains on auth.

  5. don-petry commented on Jul 1, 2026

    @don-petry
    CollaboratorAuthor

    Monitor validation of #1002 — gate ✅, one hard auth blocker 🔴

    Independently validated #1002 (canary-rollout) against #548: the graduated gate is correctly implemented — per-transition dwell (4/8/12h), sample_fraction_permille 250/500 + sample_clamp_min/max, per-candidate cumulative window, robust spike-capped baseline, ring0→ring1 sample waiver, and REGRESSION-vs-PRE_EXISTING triage. Report-only on the 4h timer, dispatch-gated promote/rollback, GitHub Deployments, and hard-fail-if-no-App-token are all present. Consolidating onto the on-main canary-rollout (vs the unmerged soak-promote) is the right call. Nice work.

    🔴 Blocker — wrong auth identifiers (workflow will fail every run until fixed)

    canary-rollout.yml references:

    app-id:      ${{ secrets.SOAK_APP_ID }}
    private-key: ${{ secrets.SOAK_APP_PRIVATE_KEY }}
    

    But the provisioned auth (live on the org, scoped to .github-private) is:

    • vars.RELEASE_MANAGER_APP_ID — a variable (= 4193127), NOT a secret
    • secrets.RELEASE_MANAGER_APP_KEY — the secret

    Fix:

    app-id:      ${{ vars.RELEASE_MANAGER_APP_ID }}
    private-key: ${{ secrets.RELEASE_MANAGER_APP_KEY }}

    Also update the two doc/error strings that name SOAK_APP_ID + SOAK_APP_PRIVATE_KEY (header comment + the "Refusing to fall back" error). Since the workflow correctly refuses github.token, it hard-errors on every run until this is corrected — this is why a report-only run hasn't produced a clean report yet.

    Reconcile: close #994

    soak-promote.* / release/registry.yml / control.yml from #994 are NOT on main and are superseded by canary-rollout (#1002) + standards/canary-rings.json. Please close #994 to avoid two-system confusion. (Note: canary-rollout's override model is imperative — workflow_dispatch: promote|rollback + --override — rather than #994's declarative control.yml; that's an acceptable design choice, just confirming it's intentional.)

    Also

    #1002 is BEHIND — rebase before merge. After the auth fix + rebase + CI-green, the 4h report-only run should authenticate and emit a clean per-version gate report; I'll validate that report against hand-derivation and confirm READY TO FLIP.

  6. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Jul 2, 2026
  7. don-petry commented on Jul 2, 2026

    @don-petry
    CollaboratorAuthor
    Already working on this: an open PR exists for issue #999.
  8. added 2 commits that reference this issue on Aug 2, 2026
  9. 2 remaining items

  10. added 14 commits that reference this issue on Aug 7, 2026
  11. added a commit that references this issue on Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions