Skip to content

canary #668 increment 4: decision telemetry (Layer 2) — decision-step emission + decide_decision_shift #683

Closed

Description

@don-petry

Increment 4 of #668 (Layer 2 of the design) — decision telemetry on organic runs, the actual correctness signal. Additive-with-default-off; agents without gate.correctness behave byte-identically. Follows #672/#676/#678 conventions; keep lib/canary-rollout.sh pure.

Insight

For a canary you don't need the right answer — you need to detect the candidate answers differently than the incumbent on comparable traffic. Both #668 pathologies (dispatching when it shouldn't; never dispatching) are distribution pathologies: dispatch-rate ≈100% or ≈0% vs a stable-tier baseline. Detect gross always/never/gross shifts — precisely a canary's job.

Part A — Emission (pr-auto-review-reusable.yml)

Mark the decision as a no-op step per outcome branch so the taken branch's step name encodes the decision:
decision: dispatched, decision: skip-draft, decision: skip-checks-pending, decision: skip-changes-requested, decision: skip-unresolved-threads (mirror the existing readiness branches — one decision: <class> step on each early-exit + the dispatch path). Only the taken branch runs; the rest are skipped. No artifacts, no state store, no consumer-stub change — the gate already fetches gh run view --json jobs (memoized in _RUN_SIG_CACHE); decision steps ride that same surface. The reusable edit itself soaks through the rings like any candidate (self-hosting).

Part B — Consumption (scripts/canary-rollout.sh + scripts/lib/canary-rollout.sh + standards/canary-rings.json)

Registry (opt-in, pr-auto-review only initially):

"gate": {
  "correctness": {
    "decision_step_prefix": "decision: ",
    "min_candidate_sample": 10,
    "min_baseline_sample": 20,
    "max_shift_permille": 400
  }
}

Engine (orchestrator, in _frontier_state): when gate.correctness is present, sample up to K=15 candidate runs on the source tier and K baseline runs on the stable tier (prior version, trailing window); tally decision-class shares from step names (prefix-matched). Pure core gains decide_decision_shift <cand_counts_json> <base_counts_json> <knobs> → OK | SHIFT | INSUFFICIENT:

  • INSUFFICIENT (either sample below its min, or unparseable/absent decision steps) → no effect — degrades to today's behavior; low-traffic agents never stall on a signal they can't produce.
  • SHIFT (any decision class's share moves ≥ max_shift_permille) → hold with triage SUSPECT (correctness variant), and sync-issues renders a decision-mix table (candidate vs baseline shares) in the blocker issue.
  • OK → no effect.

Never rolls back, never auto-acts — SHIFT only holds (worst case: a few hours of human latency). Composition drift (e.g. a draft-PR burst) is the main false-positive; the mix table makes it a ~30-second human dismissal.

Tests (tests/canary_rollout.bats)

  • decide_decision_shift: OK (shares within threshold); SHIFT (one class ≥ 40pp move); INSUFFICIENT (candidate<min, baseline<min, empty/unparseable steps).
  • share tally from mixed step names (prefix match; ignore non-decision: steps).
  • SHIFT surfaces SUSPECT + mix table; INSUFFICIENT is a no-op (agent promotes normally on reliability).
  • agent without gate.correctness → unchanged.

Out of scope

  • Extending telemetry to dependabot-automerge / auto-rebase (later; pr-auto-review first).
  • Statistical rigor beyond gross-shift (n=10–25 detects always/never/gross only — by design).
  • Layer 0 pre-merge decision cores (increment 5); SUSPECT→PRE_EXISTING auto-downgrade (increment 6).

Acceptance

  • lib/ pure; all schema additions default-off; every agent without gate.correctness byte-identical.
  • pr-auto-review runs emit decision: <class> steps; engine tallies + decide_decision_shift gates SHIFT→hold(SUSPECT)+mix table, INSUFFICIENT→no-op.
  • bats green; lint + ShellCheck clean.

Activity

  1. don-petry commented on Jul 12, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #683 — canary #668 increment 4: decision telemetry (Layer 2) — decision-step emission + decide_decision_shift

    Scope

    Additive, default-off decision telemetry: pr-auto-review runs emit decision: <class> no-op steps; the canary engine tallies candidate-vs-baseline decision-class shares and a new pure core decide_decision_shift gates a gross distribution shift → hold(SUSPECT)+mix table. Agents without gate.correctness stay byte-identical.

    Implementation Checklist

    • Part A — .github/workflows/pr-auto-review-reusable.yml: emit a decision output on each outcome branch (draft/checks-pending/changes-requested/unresolved-threads/dispatched) + add 5 no-op decision: <class> steps gated by if: so only the taken branch runs.
    • Registry — standards/canary-rings.json: add opt-in gate.correctness block (decision_step_prefix, min_candidate_sample: 10, min_baseline_sample: 20, max_shift_permille: 400) + _correctness_note to pr-auto-review only.
    • Pure core — scripts/lib/canary-rollout.sh: add decide_decision_shift <cand_counts_json> <base_counts_json> <knobs_json> → OK | SHIFT | INSUFFICIENT (lib stays pure).
    • Engine — scripts/canary-rollout.sh: helpers to tally candidate (source-tier, per-candidate window) + baseline (stable-tier, trailing window) decision counts from gh run view --json jobs step names; in _frontier_state, when gate.correctness present + reliability PROMOTE, overlay SHIFT → BLOCKED with triage SUSPECT (correctness variant); propagate a mix_shift field. INSUFFICIENT/OK → no effect.
    • sync-issues — render a candidate-vs-baseline decision-mix table in the blocker issue for the correctness-SHIFT variant; update cmd_evaluate/cmd_promote/cmd_sync_issues field readers.

    Tests to Write (tests/canary_rollout.bats)

    • decide_decision_shift: OK (within threshold); SHIFT (one class ≥ 40pp move); INSUFFICIENT (candidate<min, baseline<min, empty/unparseable).
    • decision-class share tally from mixed step names (prefix match; ignore non-decision: steps).
    • engine: SHIFT holds as BLOCKED+SUSPECT; sync-issues surfaces the mix table; INSUFFICIENT is a no-op (promotes on reliability).
    • registry: pr-auto-review opts into gate.correctness; every other agent has none (default-off).

    Test Command

    ./node_modules/.bin/bats tests/canary_rollout.bats and bash .dev-lead/scripts/dev-lead-lint.sh

  2. added 3 commits that reference this issue on Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions