You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Enhancement: canary gate should assess correctness, not just run-reliability #668
The #548 canary gate currently promotes a candidate on run reliability — dwell + sample + cumulative run health (did the workflow exit success vs failure). It does not assess correctness — whether the agent made the right decision on a green run.
For a decision-making workflow this leaves a gap: a candidate can soak green (every run exits 0) while behaving wrong — e.g. pr-auto-review dispatching a review it shouldn't (or failing to dispatch one it should), dev-lead applying a wrong fix, a classifier mislabeling. Reliability-gating auto-promotes it anyway.
Scope note
This is not specific to pr-auto-review — it applies to every agentic workflow under canary (dev-lead, feature-ideation, idea-triage, …). Run-success is a proxy for "it worked," not "it was right."
Possible directions (for discussion)
Correctness smoke signal: a per-agent fixture assertion (like the deprecated *-canary.yml idea, but a correctness check, not synthetic traffic) whose pass/fail feeds the gate — e.g. "on this fixture PR, did pr-auto-review reach the expected dispatch decision?"
Human confirmation at a ring boundary: require a manual go/no-go for ring1 → stable on agents flagged correctness_sensitive: true, so reliability clears the inner rings but a human clears the outer.
Outcome telemetry: track a downstream correctness proxy (e.g. review-decisions later overturned, fixes later reverted) and gate on it.
Why logged now
Surfaced while onboarding pr-auto-review to canary — the team agreed it should be canary-managed on reliability now, and correctness-gating is a separate enhancement to design deliberately rather than block the onboarding on.
Design: a correctness-aware canary gate (research → review → recommendation)
Deep-dive against the live engine (scripts/canary-rollout.sh, scripts/lib/canary-rollout.sh, standards/canary-rings.json @ 0dee71d), the gate standard #548, and the incident record (#864, #664, the #1010 rejection). Opinionated recommendation at the end; a first increment is already up as draft PR #672.
1. Problem statement — two gaps, one root cause
The #548 gate decides on exactly one signal: workflow-run conclusion (decide_graduated: dwell + sample + cum_fail/cum_startup == 0). Run conclusion is both:
Too coarse (green ≠ right). A decision-making agent can exit 0 while deciding wrong: pr-auto-review dispatching a review it shouldn't (or silently never dispatching), dev-lead applying a wrong fix, idea-triage mislabeling. Every green run of a systematically wrong candidate counts toward promotion. Nothing in the gate can even represent "ran fine, decided wrong."
Too credulous when red (red ≠ candidate-caused). The failure-triage layer has one guardrail — benign_failure_classes applies only when the candidate's reusable is byte-identical to the prior channel (differs=0, enforced in _frontier_state via _reusable_differs, fail-closed). Sound in isolation, but for an agent under active development differs=1 is the normal state, so the allowlist is effectively permanently disabled for exactly the agents that need it most. Evidence:
.github-private#864 — Dependabot-context runs fail at secrets: evaluation before any candidate code executes. Version-independent by mechanism, yet counted as REGRESSION at differs=1; dev-lead sat blocked ~6 days.
These are two faces of one root cause: the gate consumes a 1-bit proxy for behavior. The enhancement should widen the signal in both directions — classify red runs instead of merely counting them, and characterize green runs instead of merely trusting them.
A crucial asymmetry (from #664) shapes everything below: some failure classes are provably version-independent (they fire before/independent of candidate code — Dependabot secrets), while others are only probably environmental (a timeout is usually workload, but a candidate can regress performance). Any design that treats these the same is either unsafe (blanket-allowlisting timeouts) or noisy (blocking on Dependabot secrets). They need different machinery.
2. Critical review of the three candidate directions
2a. Correctness smoke signal (fixture assertion) — reject at canary time; adopt the reframed pre-merge version
The idea: a per-agent fixture (a designated PR/issue with a known expected decision), asserted per candidate, feeding the gate.
Problems as a gate input:
It is a manufactured run wearing a different hat. The fixture needs a harness workflow invoking the candidate against fixture state — operationally the same shape as the rejected *-canary.yml (Org Status — 2026-08-25 #1010), even if its result feeds a correctness bit instead of the sample count. The settled principle is organic traffic drives rollout.
Fixture rot is a standing tax. Fixture PRs go stale (branch protection changes, CI checks change, the fixture merges/closes), and a rotten fixture fails ambiguously — you can't tell "candidate wrong" from "fixture stale" without human triage, which recreates the Canary blocker: dev-lead ring1->stable (cum_fail=1, PRE_EXISTING) #664 problem in a new place. 14 agents × fixtures is real maintenance.
One point in decision space. A fixture asserts a single input→output pair. pr-auto-review's interesting failures are conditional (wrong on CHANGES_REQUESTED + unresolved-threads combinations), which a single "happy fixture" never exercises. False confidence is worse than no signal.
For LLM-judgment agents it's an eval harness, not a smoke check. "Did dev-lead apply the right fix on this fixture issue" is nondeterministic, costs a full engine run per candidate, and needs a grader. Out of scope for a promotion gate.
The reframe that survives review: for agents whose decision core is deterministic (pr-auto-review's readiness logic is pure bash over API state; same for dependabot-automerge, auto-rebase eligibility, etc.), the assertion belongs pre-merge, in CI on the PR that changes the reusable — extract the decision logic into a pure script + bats decision-matrix, exactly the house pattern this engine itself uses (lib/canary-rollout.sh is a side-effect-free core precisely so tests/canary_rollout.bats can pin its behavior). That catches correctness bugs before a cut exists, costs nothing at canary time, and can cover the whole decision matrix instead of one fixture point. Today pr-auto-review's logic is inline YAML bash — untestable. This is the single highest-ROI correctness investment available and needs zero gate changes.
Limits acknowledged: pre-merge tests can't see environment/API interactions or LLM judgment. Those are covered by layers 2–3 below.
2b. Human confirmation at a ring boundary — adopt, opt-in, evidence-carrying
The honest position: for judgment agents there is no machine-checkable ground truth, at any price, under any architecture. A human look at the outer boundary is not a cop-out; it's the only signal that actually measures the thing.
Tension to resolve: #548 settled "human role = override only, not a required gate." That decision was made for the reliability gate (and stands — nothing below adds a human to reliability). A correctness hold is a different question on an axis the gate currently doesn't evaluate at all, and #668 explicitly reopens it. The resolution: default stays fully autonomous; require_confirmation is opt-in per agent, at exactly one transition (ring1->stable), for agents flagged correctness-sensitive. Reliability still clears next/ring0/ring1 unattended.
Design requirements that fall out of reviewing it critically:
Never a hanging job. GitHub Environments + required reviewers would suspend the scheduled run (and pile up 4-hourly waiting runs). Instead: a gate state (AWAITING_CONFIRMATION) that promote-all treats as SOAKING-like (no advance), cleared by the already-existing manual workflow_dispatch promote path with a --confirm flag. The confirmation is the dispatch — a deliberate, Actions-audit-logged human act. No new auth surface, no comment-parsing, no state store (consistent with "channel tags ARE the state").
Evidence or it's rubber-stamping. A bare "approve?" decays into reflex-clicking within weeks. The confirmation request must be filed by sync-issues (reusing the blocker-issue machinery, marker <!-- canary-confirm:<agent> -->) carrying: the compare/<prior>...<cand> diff link, per-tier run stats, suspect-class matches (layer 1), and the decision-mix table (layer 2) once available. The human confirms a briefing, not a checkbox.
Keep the flagged set minimal — dev-lead only, initially. Cost: human latency at one boundary for one agent; "slow is fine" is already the org's stated rollout posture.
2c. Outcome telemetry (reverts/overturns) — reject as a gate input; optional observability later
Lag kills it.ring1->stable gates at 12h dwell; a wrong fix gets reverted days later. To gate on reverts you'd hold candidates for a revert-window of days-to-weeks — turning the outer boundary into a parking lot and still catching almost nothing at this org's volume (a handful of dev-lead merges per window; base revert rate ~0 means one noise event = catastrophic false-block).
Attribution is genuinely hard GitHub-native. Mapping "revert commit R" → "dev-lead run under candidate X" requires marker archaeology across repos; overturned pr-auto-review decisions aren't recorded as structured events at all.
One principle threads through: the gate never gains an oracle; it gains classification (of red) and characterization (of green), plus a human at the one place judgment is irreducible. All engine changes keep lib/canary-rollout.sh pure and unit-testable; all schema changes are additive with defaults preserving today's behavior.
Layer 1 — differs-aware failure classes (fixes the #864/#664 pain) — increment 1 shipped as draft PR #672
1a. version_independent: true on a benign class (PR #672). A class may declare that its failure mechanism fires before/independent of the candidate's own code, so it can never be candidate-caused — it stays excluded from cum_fail even at differs=1. Implementation: _benign_patterns <agent> <differs> filters classes (select($d=="0" or .version_independent==true)); _cumulative_health takes differs; pure core untouched. The bar, documented in _benign_note: provable from the failure mechanism, not merely likely.
dev-lead dependabot-context-dispatch → flagged (fails at secrets: evaluation; airtight).
dev-lead fix-review-git-push-permission → deliberately not flagged (a candidate could change push behavior).
Bonus cleanup: ci-failure-analyst carried a verbatim clone of dev-lead's classes (inert — workflow: "Dev-Lead Agent" can never match its runs); emptied.
1b. suspect_failure_classes — routed judgment for the ambiguous middle (increment 2). The #664 timeout class can't be allowlisted (a timeout can be a perf regression) but shouldn't present as a generic REGRESSION either. New optional per-agent list:
"suspect_failure_classes": [
{
"id": "workload-timeout",
"workflow": "Dev-Lead Agent",
"step": "[Tt]imeout|exit 124|[Bb]udget",
"guidance": "Check elapsed vs budget in the run log. Workload-shaped (huge issue / rebase-conflict, elapsed≈budget) → environmental: promote --confirm. Elapsed regression vs prior version on similar workloads → rollback.",
"reason": "#664 — exit-124 is usually workload, but a candidate CAN regress perf; needs judgment, not an allowlist."
}
]
Semantics: matching in-window failures still count in cum_fail (state stays BLOCKED — never auto-advance), but when every non-benign in-window failure matches a suspect class, triage refines REGRESSION → SUSPECT. The blocker issue then leads with the per-class guidance and the matched evidence instead of the generic "HALT + rollback" scare. classify_failure in the pure core gains the third verdict; sync-issues renders it. What #664 required — a human ad-hoc investigating "is this timeout the candidate's fault?" — becomes a pre-diagnosed, routed decision with the discriminating question already asked.
Optional 1c (later, data-driven): auto-downgrade SUSPECT→PRE_EXISTING when the candidate's suspect-class rate ≤ the trailing-window rate of the same class on prior versions (the gate already computes per-day baselines in _baseline_daily; this extends classification to the baseline window). Adds real API cost and rate-statistics-at-tiny-n risk — only worth building if SUSPECT confirmations become frequent enough to be a toil source. Not in the initial plan.
Known limitation to carry in the docs: benign/suspect matching keys off failed step names (_run_signature; empty signature fails closed). A failure with no failed step (some startup shapes) can never match a class. Fine today; if it bites, extend the signature to job names — do not weaken fail-closed.
Layer 2 — decision telemetry on organic runs (the actual correctness signal)
The insight that makes correctness measurable without ground truth: for a canary, you don't need to know the right answer — you need to detect that the candidate answers differently than the incumbent on comparable traffic. Both pathologies named in this issue (dispatching when it shouldn't; never dispatching) are distribution pathologies: dispatch-rate ≈100% or ≈0% against a stable-tier baseline of ~40–60%.
Emission — zero new channels. The reusable marks its decision as a no-op step per outcome branch: decision: dispatched, decision: skip-draft, decision: skip-checks-pending, … Only the taken branch's step executes; the others are skipped. The gate already fetches gh run view --json jobs (memoized in _RUN_SIG_CACHE) for failure signatures — decision steps ride the same API surface. No artifacts, no state store, no dashboards, no consumer-stub changes; the reusable edit itself rolls out through the canary (pleasingly self-hosting).
"gate": {
"correctness": {
"decision_step_prefix": "decision: ",
"min_candidate_sample": 10,
"min_baseline_sample": 20,
"max_shift_permille": 400// 40pp absolute shift in any decision class share
}
}
Engine: at the frontier, sample up to K (=15) candidate runs on the source tier and K baseline runs on the stable tier (prior version, trailing window); tally decision-class shares from step names; pure-core decide_decision_shift <cand_counts> <base_counts> <knobs> → OK | SHIFT | INSUFFICIENT. SHIFT → hold with triage SUSPECT (correctness variant) + a mix table in the blocker issue. INSUFFICIENT → no effect (low-traffic agents never stall on a signal they can't produce).
Statistical honesty: at n=10–25 this detects always/never/gross pathologies only — which is precisely a canary's job. A 40pp threshold with min-samples keeps the false-SHIFT rate low against ordinary traffic-mix drift; and because SHIFT holds (never rolls back, never auto-acts), its worst case is a few hours of human latency, not a bad automated action. Composition drift (e.g. a burst of draft PRs legitimately shifting skip-rate) is the main false-positive source — the mix table in the issue makes that a 30-second human dismissal.
Opt-in order:pr-auto-review first (binary decision, ~50 exec/day on busy tiers, decision already computed at one obvious point in the reusable). Then dependabot-automerge / auto-rebase (merge/skip, rebase/skip). Not dev-lead — its "decision" space (what code to write) doesn't reduce to classes; dev-lead's correctness story is Layer 1b + Layer 3.
Layer 3 — opt-in human confirmation at ring1->stable
Orchestrator overlay in _frontier_state (pure core unchanged): when the reliability verdict is PROMOTE and the frontier transition requires confirmation and no confirmation is present → state AWAITING_CONFIRMATION. Scheduled promote-all does not advance it; workflow_dispatch promote <agent> --confirm does (and only from a state that is otherwise PROMOTE — --confirm is not--override and cannot bypass reliability). sync-issues files/updates the evidence-carrying confirmation issue (marker canary-confirm:<agent>, auto-closed on promote or candidate change). Dashboard learns the new state.
Opt-in: dev-lead only initially. Everything else keeps lights-out promotion. If confirmation latency proves painful, the knob is per-agent and reversible in one registry line.
Layer 0 — standard, not engine: pure decision cores + pre-merge decision-matrix tests
Per §2a: extract deterministic decision logic (starting with pr-auto-review's readiness check) into scripts/ + bats, gate it in CI on the PR that changes the reusable, and codify in AGENTS.md/standards/agent-standards.md: a decision-making reusable keeps its decision logic in a pure, unit-tested script; the workflow is I/O glue. The engine itself is the exemplar. This is where most correctness bugs get caught — before a version exists.
No outcome/revert gating (lag + attribution noise; see §2c) — revisit only as fleet-monitor observability.
No LLM-as-judge grading of agent runs — cost/nondeterminism/who-grades-the-grader; the human at ring1→stable with evidence is strictly better at this scale.
No change to reliability gating: dwell/sample/cum-health and the differs=1 fail-closed default all stand.
4. Migration, risks, failure modes
Migration: every schema addition is optional-with-default-off; agents without new keys behave byte-identically (13/14 agents in increment 1). Engine changes deploy by merging (the engine runs from main; it isn't channel-tagged). The only consumer-visible change is Layer 2's decision steps inside pr-auto-review-reusable.yml — which itself soaks through the rings like any candidate.
Failure modes and mitigations:
risk
mitigation
version_independent misuse masks a real regression
documented bar ("provable from the failure mechanism"); registry changes are PR-reviewed; the flag is per-class, never per-agent; tests pin the filter semantics
suspect classes over-match (broad ERE) and dilute REGRESSION urgency
SUSPECT still BLOCKS + still needs a human; guidance text carries the discriminating question; start with one class (dev-lead timeout)
decision-step rename breaks telemetry
decide_decision_shift returns INSUFFICIENT on unparseable/absent steps — degrades to today's behavior, never false-blocks; pre-merge tests (Layer 0) pin the step names
traffic-composition drift → false SHIFT
high threshold + min samples; SHIFT holds rather than acts; mix table makes dismissal cheap
confirmation fatigue → rubber-stamping
one agent, one boundary; evidence-carrying issue; revisit if >2–3 confirms/week
API budget (extra gh run view calls)
telemetry only for frontier agents with correctness config, capped at 2K runs/agent/tick, memoized via _RUN_SIG_CACHE
empty failure signature can't match any class
fail-closed retained (safe direction); extend signature to job names only if it recurs
Activity
feat: onboard pr-auto-review to canary — unmanaged block now empty (#669
fix(canary): route blocker issues to dev-lead + fix broken needs-huma…
don-petry commentedon Jul 12, 2026
Design: a correctness-aware canary gate (research → review → recommendation)
Deep-dive against the live engine (
scripts/canary-rollout.sh,scripts/lib/canary-rollout.sh,standards/canary-rings.json@0dee71d), the gate standard #548, and the incident record (#864, #664, the #1010 rejection). Opinionated recommendation at the end; a first increment is already up as draft PR #672.1. Problem statement — two gaps, one root cause
The #548 gate decides on exactly one signal: workflow-run conclusion (
decide_graduated: dwell + sample +cum_fail/cum_startup== 0). Run conclusion is both:Too coarse (green ≠ right). A decision-making agent can exit 0 while deciding wrong:
pr-auto-reviewdispatching a review it shouldn't (or silently never dispatching), dev-lead applying a wrong fix,idea-triagemislabeling. Every green run of a systematically wrong candidate counts toward promotion. Nothing in the gate can even represent "ran fine, decided wrong."Too credulous when red (red ≠ candidate-caused). The failure-triage layer has one guardrail —
benign_failure_classesapplies only when the candidate's reusable is byte-identical to the prior channel (differs=0, enforced in_frontier_statevia_reusable_differs, fail-closed). Sound in isolation, but for an agent under active developmentdiffers=1is the normal state, so the allowlist is effectively permanently disabled for exactly the agents that need it most. Evidence:secrets:evaluation before any candidate code executes. Version-independent by mechanism, yet counted as REGRESSION at differs=1; dev-lead sat blocked ~6 days.These are two faces of one root cause: the gate consumes a 1-bit proxy for behavior. The enhancement should widen the signal in both directions — classify red runs instead of merely counting them, and characterize green runs instead of merely trusting them.
A crucial asymmetry (from #664) shapes everything below: some failure classes are provably version-independent (they fire before/independent of candidate code — Dependabot secrets), while others are only probably environmental (a timeout is usually workload, but a candidate can regress performance). Any design that treats these the same is either unsafe (blanket-allowlisting timeouts) or noisy (blocking on Dependabot secrets). They need different machinery.
2. Critical review of the three candidate directions
2a. Correctness smoke signal (fixture assertion) — reject at canary time; adopt the reframed pre-merge version
The idea: a per-agent fixture (a designated PR/issue with a known expected decision), asserted per candidate, feeding the gate.
Problems as a gate input:
*-canary.yml(Org Status — 2026-08-25 #1010), even if its result feeds a correctness bit instead of the sample count. The settled principle is organic traffic drives rollout.pr-auto-review's interesting failures are conditional (wrong on CHANGES_REQUESTED + unresolved-threads combinations), which a single "happy fixture" never exercises. False confidence is worse than no signal.The reframe that survives review: for agents whose decision core is deterministic (pr-auto-review's readiness logic is pure bash over API state; same for dependabot-automerge, auto-rebase eligibility, etc.), the assertion belongs pre-merge, in CI on the PR that changes the reusable — extract the decision logic into a pure script + bats decision-matrix, exactly the house pattern this engine itself uses (
lib/canary-rollout.shis a side-effect-free core precisely sotests/canary_rollout.batscan pin its behavior). That catches correctness bugs before a cut exists, costs nothing at canary time, and can cover the whole decision matrix instead of one fixture point. Today pr-auto-review's logic is inline YAML bash — untestable. This is the single highest-ROI correctness investment available and needs zero gate changes.Limits acknowledged: pre-merge tests can't see environment/API interactions or LLM judgment. Those are covered by layers 2–3 below.
2b. Human confirmation at a ring boundary — adopt, opt-in, evidence-carrying
The honest position: for judgment agents there is no machine-checkable ground truth, at any price, under any architecture. A human look at the outer boundary is not a cop-out; it's the only signal that actually measures the thing.
Tension to resolve: #548 settled "human role = override only, not a required gate." That decision was made for the reliability gate (and stands — nothing below adds a human to reliability). A correctness hold is a different question on an axis the gate currently doesn't evaluate at all, and #668 explicitly reopens it. The resolution: default stays fully autonomous;
require_confirmationis opt-in per agent, at exactly one transition (ring1->stable), for agents flagged correctness-sensitive. Reliability still clears next/ring0/ring1 unattended.Design requirements that fall out of reviewing it critically:
AWAITING_CONFIRMATION) thatpromote-alltreats as SOAKING-like (no advance), cleared by the already-existing manualworkflow_dispatch promotepath with a--confirmflag. The confirmation is the dispatch — a deliberate, Actions-audit-logged human act. No new auth surface, no comment-parsing, no state store (consistent with "channel tags ARE the state").sync-issues(reusing the blocker-issue machinery, marker<!-- canary-confirm:<agent> -->) carrying: thecompare/<prior>...<cand>diff link, per-tier run stats, suspect-class matches (layer 1), and the decision-mix table (layer 2) once available. The human confirms a briefing, not a checkbox.2c. Outcome telemetry (reverts/overturns) — reject as a gate input; optional observability later
ring1->stablegates at 12h dwell; a wrong fix gets reverted days later. To gate on reverts you'd hold candidates for a revert-window of days-to-weeks — turning the outer boundary into a parking lot and still catching almost nothing at this org's volume (a handful of dev-lead merges per window; base revert rate ~0 means one noise event = catastrophic false-block).<!-- dev-lead-issue N status=... reason=... -->, consumed by fleet-monitor per Org Status — 2026-07-24 #901/feat: implement issue #1023 — canary autocut: scope bump signals to watched-path commits, paginate the range, and persist failed promotions #1024) — it belongs in fleet-monitor reporting, not in the promotion gate.3. Recommended design — four layers
One principle threads through: the gate never gains an oracle; it gains classification (of red) and characterization (of green), plus a human at the one place judgment is irreducible. All engine changes keep
lib/canary-rollout.shpure and unit-testable; all schema changes are additive with defaults preserving today's behavior.Layer 1 — differs-aware failure classes (fixes the #864/#664 pain) — increment 1 shipped as draft PR #672
1a.
version_independent: trueon a benign class (PR #672). A class may declare that its failure mechanism fires before/independent of the candidate's own code, so it can never be candidate-caused — it stays excluded fromcum_faileven atdiffers=1. Implementation:_benign_patterns <agent> <differs>filters classes (select($d=="0" or .version_independent==true));_cumulative_healthtakesdiffers; pure core untouched. The bar, documented in_benign_note: provable from the failure mechanism, not merely likely.dependabot-context-dispatch→ flagged (fails atsecrets:evaluation; airtight).fix-review-git-push-permission→ deliberately not flagged (a candidate could change push behavior).workflow: "Dev-Lead Agent"can never match its runs); emptied.1b.
suspect_failure_classes— routed judgment for the ambiguous middle (increment 2). The #664 timeout class can't be allowlisted (a timeout can be a perf regression) but shouldn't present as a generic REGRESSION either. New optional per-agent list:Semantics: matching in-window failures still count in
cum_fail(state stays BLOCKED — never auto-advance), but when every non-benign in-window failure matches a suspect class, triage refinesREGRESSION → SUSPECT. The blocker issue then leads with the per-classguidanceand the matched evidence instead of the generic "HALT + rollback" scare.classify_failurein the pure core gains the third verdict;sync-issuesrenders it. What #664 required — a human ad-hoc investigating "is this timeout the candidate's fault?" — becomes a pre-diagnosed, routed decision with the discriminating question already asked.Optional 1c (later, data-driven): auto-downgrade SUSPECT→PRE_EXISTING when the candidate's suspect-class rate ≤ the trailing-window rate of the same class on prior versions (the gate already computes per-day baselines in
_baseline_daily; this extends classification to the baseline window). Adds real API cost and rate-statistics-at-tiny-n risk — only worth building if SUSPECT confirmations become frequent enough to be a toil source. Not in the initial plan.Known limitation to carry in the docs: benign/suspect matching keys off failed step names (
_run_signature; empty signature fails closed). A failure with no failed step (some startup shapes) can never match a class. Fine today; if it bites, extend the signature to job names — do not weaken fail-closed.Layer 2 — decision telemetry on organic runs (the actual correctness signal)
The insight that makes correctness measurable without ground truth: for a canary, you don't need to know the right answer — you need to detect that the candidate answers differently than the incumbent on comparable traffic. Both pathologies named in this issue (dispatching when it shouldn't; never dispatching) are distribution pathologies: dispatch-rate ≈100% or ≈0% against a stable-tier baseline of ~40–60%.
Emission — zero new channels. The reusable marks its decision as a no-op step per outcome branch:
decision: dispatched,decision: skip-draft,decision: skip-checks-pending, … Only the taken branch's step executes; the others are skipped. The gate already fetchesgh run view --json jobs(memoized in_RUN_SIG_CACHE) for failure signatures — decision steps ride the same API surface. No artifacts, no state store, no dashboards, no consumer-stub changes; the reusable edit itself rolls out through the canary (pleasingly self-hosting).Consumption — gross-shift detection, honestly scoped. Registry:
Engine: at the frontier, sample up to K (=15) candidate runs on the source tier and K baseline runs on the stable tier (prior version, trailing window); tally decision-class shares from step names; pure-core
decide_decision_shift <cand_counts> <base_counts> <knobs>→OK | SHIFT | INSUFFICIENT.SHIFT→ hold with triageSUSPECT(correctness variant) + a mix table in the blocker issue.INSUFFICIENT→ no effect (low-traffic agents never stall on a signal they can't produce).Statistical honesty: at n=10–25 this detects always/never/gross pathologies only — which is precisely a canary's job. A 40pp threshold with min-samples keeps the false-SHIFT rate low against ordinary traffic-mix drift; and because SHIFT holds (never rolls back, never auto-acts), its worst case is a few hours of human latency, not a bad automated action. Composition drift (e.g. a burst of draft PRs legitimately shifting skip-rate) is the main false-positive source — the mix table in the issue makes that a 30-second human dismissal.
Opt-in order:
pr-auto-reviewfirst (binary decision, ~50 exec/day on busy tiers, decision already computed at one obvious point in the reusable). Thendependabot-automerge/auto-rebase(merge/skip, rebase/skip). Not dev-lead — its "decision" space (what code to write) doesn't reduce to classes; dev-lead's correctness story is Layer 1b + Layer 3.Layer 3 — opt-in human confirmation at
ring1->stablePer §2b. Registry:
Orchestrator overlay in
_frontier_state(pure core unchanged): when the reliability verdict is PROMOTE and the frontier transition requires confirmation and no confirmation is present → stateAWAITING_CONFIRMATION. Scheduledpromote-alldoes not advance it;workflow_dispatch promote <agent> --confirmdoes (and only from a state that is otherwise PROMOTE —--confirmis not--overrideand cannot bypass reliability).sync-issuesfiles/updates the evidence-carrying confirmation issue (markercanary-confirm:<agent>, auto-closed on promote or candidate change). Dashboard learns the new state.Opt-in: dev-lead only initially. Everything else keeps lights-out promotion. If confirmation latency proves painful, the knob is per-agent and reversible in one registry line.
Layer 0 — standard, not engine: pure decision cores + pre-merge decision-matrix tests
Per §2a: extract deterministic decision logic (starting with pr-auto-review's readiness check) into
scripts/+ bats, gate it in CI on the PR that changes the reusable, and codify inAGENTS.md/standards/agent-standards.md: a decision-making reusable keeps its decision logic in a pure, unit-tested script; the workflow is I/O glue. The engine itself is the exemplar. This is where most correctness bugs get caught — before a version exists.What this does NOT do (deliberate)
*-canary.ymlrevival (Org Status — 2026-08-25 #1010 stays dead).4. Migration, risks, failure modes
Migration: every schema addition is optional-with-default-off; agents without new keys behave byte-identically (13/14 agents in increment 1). Engine changes deploy by merging (the engine runs from main; it isn't channel-tagged). The only consumer-visible change is Layer 2's decision steps inside
pr-auto-review-reusable.yml— which itself soaks through the rings like any candidate.Failure modes and mitigations:
version_independentmisuse masks a real regressiondecide_decision_shiftreturns INSUFFICIENT on unparseable/absent steps — degrades to today's behavior, never false-blocks; pre-merge tests (Layer 0) pin the step namesgh run viewcalls)correctnessconfig, capped at 2K runs/agent/tick, memoized via_RUN_SIG_CACHE5. Incremental delivery plan
version_independentbenign classes + registry + 11 tests (106/106 green). Smallest safe slice; on its own it retires the [#850] standards-deploy driver can't write to consumer repos — wrong token (ORG_SCORECARD_TOKEN) + opaque put-failed #864-shape false-block permanently.suspect_failure_classes+classify_failurethird verdict + blocker-issue guidance + dev-leadworkload-timeoutclass. Retires the Canary blocker: dev-lead ring1->stable (cum_fail=1, PRE_EXISTING) #664-shape toil (pre-diagnosed instead of ad-hoc).require_confirmation+AWAITING_CONFIRMATION+promote --confirm+ confirmation issue, dev-lead opted in. Engine-only; independent of 2 (order swappable).decide_decision_shift+ orchestrator sampling + registry knobs (engine PR), then decision steps inpr-auto-review-reusable.yml(reusable PR, soaks via canary).Each step is independently shippable, independently revertible, and leaves the fleet on default behavior unless an agent opts in.
Files/functions relied on:
scripts/lib/canary-rollout.sh(decide_graduated,classify_failure,benign_match,robust_sample_target),scripts/canary-rollout.sh(_frontier_state,_cumulative_health,_benign_patterns,_reusable_differs,_run_signature/_RUN_SIG_CACHE,cmd_promote,cmd_sync_issues),standards/canary-rings.json(gate{}per agent),.github/workflows/canary-rollout.yml,tests/canary_rollout.bats; issues #548 (gate standard + "override-only" decision), .github-private#864, #664, #1010 (synthetic-canary rejection), #1025 P2 (benign allowlist), #901/#1024 (timeout observability).feat(canary): differs-aware benign classes (version_independent) — #668…
feat: implement issue #675 — canary #668 increment 2: SUSPECT triage …
feat: implement issue #677 — canary #668 increment 3: opt-in human co…
feat: implement issue #683 — canary #668 increment 4: decision teleme…
feat: implement issue #683 — canary #668 increment 4: decision teleme…
ci: add canary-rollout-tests.yml — enforce the engine's bats suite (C…
feat: implement issue #686 — canary #668 increment 5: Layer 0 — pure …
feat(canary): differs-aware benign classes (version_independent) — #668…
feat(canary): differs-aware benign classes (version_independent) — #668…
feat(canary): differs-aware benign classes (version_independent) — #668…
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state
fix: prevent --override from advancing AWAITING_CONFIRMATION state