Repository navigation
Description
Increment 4 of #668 (Layer 2 of the design) — decision telemetry on organic runs, the actual correctness signal. Additive-with-default-off; agents without gate.correctness behave byte-identically. Follows #672/#676/#678 conventions; keep lib/canary-rollout.sh pure.
Insight
For a canary you don't need the right answer — you need to detect the candidate answers differently than the incumbent on comparable traffic. Both #668 pathologies (dispatching when it shouldn't; never dispatching) are distribution pathologies: dispatch-rate ≈100% or ≈0% vs a stable-tier baseline. Detect gross always/never/gross shifts — precisely a canary's job.
Part A — Emission (pr-auto-review-reusable.yml)
Mark the decision as a no-op step per outcome branch so the taken branch's step name encodes the decision:
decision: dispatched, decision: skip-draft, decision: skip-checks-pending, decision: skip-changes-requested, decision: skip-unresolved-threads (mirror the existing readiness branches — one decision: <class> step on each early-exit + the dispatch path). Only the taken branch runs; the rest are skipped. No artifacts, no state store, no consumer-stub change — the gate already fetches gh run view --json jobs (memoized in _RUN_SIG_CACHE); decision steps ride that same surface. The reusable edit itself soaks through the rings like any candidate (self-hosting).
Part B — Consumption (scripts/canary-rollout.sh + scripts/lib/canary-rollout.sh + standards/canary-rings.json)
Registry (opt-in, pr-auto-review only initially):
"gate": {
"correctness": {
"decision_step_prefix": "decision: ",
"min_candidate_sample": 10,
"min_baseline_sample": 20,
"max_shift_permille": 400
}
}Engine (orchestrator, in _frontier_state): when gate.correctness is present, sample up to K=15 candidate runs on the source tier and K baseline runs on the stable tier (prior version, trailing window); tally decision-class shares from step names (prefix-matched). Pure core gains decide_decision_shift <cand_counts_json> <base_counts_json> <knobs> → OK | SHIFT | INSUFFICIENT:
INSUFFICIENT(either sample below its min, or unparseable/absent decision steps) → no effect — degrades to today's behavior; low-traffic agents never stall on a signal they can't produce.SHIFT(any decision class's share moves ≥max_shift_permille) → hold with triageSUSPECT(correctness variant), andsync-issuesrenders a decision-mix table (candidate vs baseline shares) in the blocker issue.OK→ no effect.
Never rolls back, never auto-acts — SHIFT only holds (worst case: a few hours of human latency). Composition drift (e.g. a draft-PR burst) is the main false-positive; the mix table makes it a ~30-second human dismissal.
Tests (tests/canary_rollout.bats)
decide_decision_shift: OK (shares within threshold); SHIFT (one class ≥ 40pp move); INSUFFICIENT (candidate<min, baseline<min, empty/unparseable steps).- share tally from mixed step names (prefix match; ignore non-
decision:steps). - SHIFT surfaces SUSPECT + mix table; INSUFFICIENT is a no-op (agent promotes normally on reliability).
- agent without
gate.correctness→ unchanged.
Out of scope
- Extending telemetry to dependabot-automerge / auto-rebase (later; pr-auto-review first).
- Statistical rigor beyond gross-shift (n=10–25 detects always/never/gross only — by design).
- Layer 0 pre-merge decision cores (increment 5); SUSPECT→PRE_EXISTING auto-downgrade (increment 6).
Acceptance
lib/pure; all schema additions default-off; every agent withoutgate.correctnessbyte-identical.- pr-auto-review runs emit
decision: <class>steps; engine tallies +decide_decision_shiftgates SHIFT→hold(SUSPECT)+mix table, INSUFFICIENT→no-op. - bats green; lint + ShellCheck clean.
Activity
don-petry commentedon Jul 12, 2026
Dev-Lead Implementation Plan
Issue: #683 — canary #668 increment 4: decision telemetry (Layer 2) — decision-step emission + decide_decision_shift
Scope
Additive, default-off decision telemetry: pr-auto-review runs emit
decision: <class>no-op steps; the canary engine tallies candidate-vs-baseline decision-class shares and a new pure coredecide_decision_shiftgates a gross distribution shift → hold(SUSPECT)+mix table. Agents withoutgate.correctnessstay byte-identical.Implementation Checklist
.github/workflows/pr-auto-review-reusable.yml: emit adecisionoutput on each outcome branch (draft/checks-pending/changes-requested/unresolved-threads/dispatched) + add 5 no-opdecision: <class>steps gated byif:so only the taken branch runs.standards/canary-rings.json: add opt-ingate.correctnessblock (decision_step_prefix,min_candidate_sample: 10,min_baseline_sample: 20,max_shift_permille: 400) +_correctness_noteto pr-auto-review only.scripts/lib/canary-rollout.sh: adddecide_decision_shift <cand_counts_json> <base_counts_json> <knobs_json>→OK | SHIFT | INSUFFICIENT(lib stays pure).scripts/canary-rollout.sh: helpers to tally candidate (source-tier, per-candidate window) + baseline (stable-tier, trailing window) decision counts fromgh run view --json jobsstep names; in_frontier_state, whengate.correctnesspresent + reliability PROMOTE, overlaySHIFT → BLOCKEDwith triageSUSPECT(correctness variant); propagate amix_shiftfield.INSUFFICIENT/OK→ no effect.cmd_evaluate/cmd_promote/cmd_sync_issuesfield readers.Tests to Write (
tests/canary_rollout.bats)decide_decision_shift: OK (within threshold); SHIFT (one class ≥ 40pp move); INSUFFICIENT (candidate<min, baseline<min, empty/unparseable).decision:steps).gate.correctness; every other agent has none (default-off).Test Command
./node_modules/.bin/bats tests/canary_rollout.batsandbash .dev-lead/scripts/dev-lead-lint.shfeat: implement issue #683 — canary #668 increment 4: decision teleme…
fix(canary #683): address review — single-pass jq counts, empty-argjs…
feat: implement issue #683 — canary #668 increment 4: decision teleme…