Skip to content

canary #668 increment 3: opt-in human confirmation at ring1→stable (require_confirmation, dev-lead) #677

Closed

Description

@don-petry
No description provided.

Activity

  1. don-petry commented on Jul 12, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #677 — canary #668 increment 3: opt-in human confirmation at ring1→stable (require_confirmation, dev-lead)

    Scope

    Implement Layer 3 of the #668 design: an opt-in, evidence-carrying human go/no-go at the ring1->stable boundary for dev-lead only, via a new AWAITING_CONFIRMATION gate state that the scheduled promote-all never auto-advances and that a deliberate promote <agent> --confirm dispatch clears. Engine-only; every other agent keeps lights-out promotion (default-off, byte-identical).

    Design (per the #668 delivery-plan step 3)

    • Pure core unchanged (scripts/lib/canary-rollout.sh): the overlay lives in the orchestrator.
    • AWAITING_CONFIRMATION = reliability verdict is PROMOTE and the frontier transition sets require_confirmation: true and no confirmation is present. The confirmation is the dispatch (--confirm) — no new state store, consistent with "channel tags ARE the state".
    • --confirm is not --override: it advances only an otherwise-PROMOTE (reliability-clean) AWAITING_CONFIRMATION state; it can never bypass a BLOCKED gate.

    Implementation Checklist

    • standards/canary-rings.json — add require_confirmation: true to dev-lead gate.transitions.ring1->stable + a _confirm_note
    • scripts/canary-rollout.sh _frontier_state — overlay PROMOTE → AWAITING_CONFIRMATION when the transition requires confirmation
    • scripts/canary-rollout.sh cmd_promote — add --confirm flag; advance AWAITING_CONFIRMATION only with --confirm (or --override); informative hold message otherwise
    • scripts/canary-rollout.sh cmd_evaluate — report the new state
    • scripts/canary-rollout.sh — add _confirm_body (evidence-carrying: compare <stable>...<cand> diff link, run stats, the promote --confirm instruction)
    • scripts/canary-rollout.sh cmd_sync_issues — upsert/auto-close a canary-confirm:<agent> issue (label canary-confirm + needs-human); show the state in the fleet dashboard
    • scripts/canary-rollout.sh — usage/help strings for --confirm
    • .github/workflows/canary-rollout.yml — add a confirm boolean workflow_dispatch input, threaded into the promote args

    Tests to Write (TDD, tests/canary_rollout.bats)

    • registry: dev-lead ring1->stable.require_confirmation == true; default-absent for opted-out agents
    • _frontier_state/evaluate: dev-lead at ring1->stable, clean + dwell + sample met → AWAITING_CONFIRMATION (not PROMOTE)
    • promote without --confirm on AWAITING_CONFIRMATION → holds (no tag move), prints the confirm instruction
    • promote --confirm --dry-run → advances (shows the gh-api move)
    • promote --confirm on a BLOCKED+REGRESSION frontier → still held (proves --confirm ≠ --override)
    • sync-issues → opens a canary-confirm issue with the compare link + needs-human; dashboard shows the state
    • sync-issues auto-closes a stale confirm issue once the agent is no longer awaiting
    • _confirm_body renders the compare diff link + the promote --confirm guidance

    Test Command

    bats tests/canary_rollout.bats (plus bash .dev-lead/scripts/dev-lead-lint.sh for shellcheck)

  2. don-petry commented on Jul 12, 2026

    @don-petry
    ContributorAuthor

    ✅ Issue #677 complete — canary #668 increment 3: opt-in human confirmation at ring1->stable

    Layer 3 of the #668 design: an opt-in human go/no-go at a single ring boundary for a correctness-sensitive agent, implemented via a new AWAITING_CONFIRMATION gate state. dev-lead only initially; everything default-off and byte-identical for opted-out agents.

    Plan Execution

    • Phase 1 — Scope & Plan (plan comment posted before touching source)
    • Phase 2 — Test-First TDD (8 new failing tests written & confirmed red first)
    • Phase 3 — Implement (orchestrator overlay + registry knob + workflow input)
    • Phase 4 — Verify (full suite green + dev-lead-lint.sh exit 0)
    • Phase 5 — Rubber Duck (git diff self-review, all four files)
    • Phase 6 — Completion report (this comment)

    Semantics delivered

    • New state AWAITING_CONFIRMATION. When _frontier_state computes reliability PROMOTE and the transition sets require_confirmation: true, the frontier holds in AWAITING_CONFIRMATION instead of advancing. Reliability still clears next/ring0/ring1 unattended — only ring1->stable gates.
    • The confirmation IS the dispatch — no persistent state store. A deliberate promote <agent> --confirm (Actions-audit-logged) advances it; the scheduled promote-all never sets --confirm, so it holds indefinitely without a human.
    • --confirm is NOT --override. It advances only a reliability-clean AWAITING_CONFIRMATION state and can never bypass a BLOCKED gate (verified by test 131). The overlay only fires from PROMOTE, and the BLOCKED+REGRESSION/SUSPECT guard still requires override.
    • Evidence-carrying issue. sync-issues files/updates a marker-keyed canary-confirm:<agent> issue (label canary-confirm, --add-label needs-human) carrying a compare/<prior>...<cand> diff link + run stats, and auto-closes it once the agent is no longer awaiting (promote or candidate change).

    Test Results

    1..134
    134 ok / 0 not ok
    

    All 8 new increment-3 tests (127–134) green; no existing test modified or skipped. dev-lead-lint.sh (shellcheck --severity=warning -x on 21 scripts + frontmatter + CODEOWNERS): all checks passed, exit 0. JSON/YAML valid.

    New tests:

    • 127 — dev-lead ring1->stable opts into require_confirmation; opted-out agents default-off
    • 128 — reliability PROMOTE holds as AWAITING_CONFIRMATION
    • 129 — does NOT advance without --confirm (scheduled sweep holds)
    • 130 — promote --confirm advances a reliability-clean frontier
    • 131 — --confirm cannot advance a BLOCKED gate
    • 132 — _confirm_body renders compare diff link + --confirm instruction
    • 133 — sync-issues opens a canary-confirm issue
    • 134 — sync-issues auto-closes a stale confirm issue

    Files Changed

     .github/workflows/canary-rollout.yml |   9 ++
     scripts/canary-rollout.sh            | 123 +++++++++++++++++++++++--
     standards/canary-rings.json          |   4 +-
     tests/canary_rollout.bats            | 170 +++++++++++++++++++++++++++++++++++
     4 files changed, 299 insertions(+), 7 deletions(-)
    
    • standards/canary-rings.json — dev-lead ring1->stable gains require_confirmation: true + _confirm_note. All other agents null (default-off).
    • scripts/canary-rollout.sh — _frontier_state overlay; cmd_evaluate notice; cmd_promote --confirm handling + hold message; new _confirm_body; cmd_sync_issues confirm-issue upsert/close + canary-confirm label; dashboard legend; CLI usage/header.
    • .github/workflows/canary-rollout.yml — confirm boolean dispatch input → CONFIRM env → --confirm arg in the promote case.

    Notes

    • Pure decision core (scripts/lib/canary-rollout.sh) unchanged per the design mandate — the entire Layer 3 overlay lives in the orchestrator.
    • Default-off invariant held: adding the knob to dev-lead breaks no existing test; the positional _frontier_state output stays single-token so downstream parsing is unaffected.
    • Per task constraints, no commit/push/PR — CI handles all git operations.
  3. added a commit that references this issue on Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions