Skip to content

canary-rollout: close the remaining fail-open gaps in ingress attribution (#1244 items 3, 11, 12, 10) #1246

Closed

Description

@don-petry

Summary

The ADR-0007 ingress attribution shipped in #1238 fails closed almost everywhere. These four items are the places where unreadable or unattributable evidence can still read as healthy (or as "no caller"). They are the highest-priority follow-ups from the #1238 review (tracked in #1244).

Design principle (from the maintainer): expected outcomes must not be failures. A quiet ingress, or a repo that legitimately does not run a role, is "no caller" and must not hold the gate. Only unreadable evidence or a genuine misconfiguration should.

Items

  • Item 3 — _run_decision_class swallows jobs-read failures. It does _run_jobs_json … || echo '{}', so a sustained jobs outage returns an empty class, which _sample_decision_counts silently skips. Mix-shift sampling degrades to "insufficient" and is never recorded as UNRESOLVED, while the same condition elsewhere is. A failed read is also cached in _RUN_DECISION_CACHE. Fix: return a distinct non-zero status for a read error (vs "no decision step"), surface it through _sample_decision_counts, and map it to UNRESOLVED / a hold in _correctness_verdict. Do not cache the failure. Confirmed still open on main.
  • Item 11 — the ingress run list can silently truncate. The list is fetched with -L ${CANARY_INGRESS_RUN_LIMIT:-5000}; the jobs cap fails closed when hit but the list cap does not, so a very busy ingress could look quieter than it is. Fix: detect a list that returned exactly its limit and treat it like the jobs cap (UNRESOLVED, or the truncation-aware baseline).
  • Item 12 — a job-less cancelled run is dropped as benign. {skipped:"cancelled"} is decided before the UNATTRIBUTED branch, so a run whose jobs are merely not recorded yet (jobs API lag) is discarded instead of flagged. Fix: distinguish "cancelled before any job started" from "jobs not recorded yet" (e.g. require the run to be older than a settle window, or to report a cancelled conclusion with zero jobs on a re-read).
  • Item 10 — an empty ingress window never validates ingress_job. With the per-role workflow absent, the ingress present and no completed run in the window, _ingress_agent_runs returns [] without checking that the role exists, so a misspelled ingress_job looks like "no caller" and waive_sample_if_no_caller can allow a dwell-only promotion. Mitigated today: as soon as any run appears without the role's job it is flagged NOROLE/UNRESOLVED. Approach is a maintainer decision (see below).

Item 10 — options (keeping expected outcomes from failing)

A runtime check cannot fail on "role absent from this member's ingress", because that is also what a legitimate non-consumer looks like. So:

  1. Static registry lint (recommended). Validate every registered ingress_job against the canonical agent-ingress.yml template in CI. Runtime keeps treating an empty window as "no caller" (expected, not a failure); a typo in the registry is caught before it ships. Dependency: standards/workflows/agent-ingress.yml does not exist in this repo yet ([Phase 3] Collapse the first repo's Class-1 event-driven stubs into agent-ingress.yml and rename the required checks .github-private#1729 AC11(a) is to publish it). The registry's ingress.file currently points at the path inside each member repo.
  2. Warn-only at runtime. Emit a ::warning:: for "ingress present, zero runs, role never seen" with no effect on the gate. Cheap, but does not close the fail-open.
  3. Runtime read of the live ingress file. Rejected as the primary approach: extra read per tick, and it cannot tell a registry typo from a non-consumer unless it compares across members.
  4. Accept as is.

Acceptance criteria

Refs #1244, #1238.

Activity

  1. don-petry commented on Oct 4, 2026

    @don-petry
    ContributorAuthor

    PR A is up: #1250 covers items 3, 11 and 12 here (and closes #1248, item 13).

    • Item 3: fixed. A transient jobs-read failure during decision-mix sampling is now UNRESOLVED and never memoized; 404 stays a no-op. Confirmed the old behaviour on main first.
    • Item 11: fixed. A run list that hit CANARY_INGRESS_RUN_LIMIT without reaching the window start is UNRESOLVED (baseline: newest days only).
    • Item 12: closed as an accepted risk, with a test. A job-less cancelled run during jobs-API lag is indistinguishable from an in-flight run, which the gate already tolerates, and the jobs are re-read each sweep, so it is attributed on the next tick. A hold would false-block on every concurrency-superseded run. If you would rather hold, say so and I will add it.
    • Item 10: still open pending a maintainer decision on the approach (see above). This issue stays open until it is done.

    Generated by Claude Code

  2. don-petry commented on Oct 4, 2026

    @don-petry
    ContributorAuthor

    PR A is merged: #1250 (squashed into main at 03:20Z; pr-review approved 97f1dbd).

    • Item 3 — fixed: a transient jobs-read failure during decision-mix sampling is recorded UNRESOLVED and never memoized; 404 stays a no-op.
    • Item 11 — fixed: an ingress run list that hit CANARY_INGRESS_RUN_LIMIT without reaching the window start is UNRESOLVED (baseline: newest days only); a non-array list is UNRESOLVED; the limit is normalized once.
    • Item 12 — closed as an accepted risk with a test (a job-less cancelled run during jobs-API lag is treated like an in-flight run and attributed on the next sweep).
    • Item 10 — still open, awaiting the maintainer's approach decision (recommended: a static registry lint against the canonical agent-ingress.yml, which depends on that template landing in this repo; interim option: a warn-only annotation).

    Leaving this issue open for item 10.


    Generated by Claude Code

  3. don-petry commented on Oct 10, 2026

    @don-petry
    ContributorAuthor

    Item 10 (static registry lint, the accepted approach) is implemented in #1276, pending review/merge:

    • standards/workflows/agent-ingress.yml published as the canonical ADR-0007 ingress (one job per ring-registered role; the pilot's two defective pins corrected to pr-review/v1-stable and ci-failure-analyst/v1-stable). Published as a standard only, not deployed.
    • CI lint: every registered ingress_job must be a job of that template, plus plain/unique job-key checks; canary-rollout-tests.yml now triggers on registry edits (it previously did not).
    • Runtime behavior is unchanged: an empty window is still "no caller".

    Items 3, 11, 12 and 13 shipped earlier in #1250.


    Generated by Claude Code

  4. don-petry commented on Oct 21, 2026

    @don-petry
    ContributorAuthor

    Closing after the 2026-10-21 backlog triage.

    Why: All four items shipped.
    Evidence: PRs #1250, #1276 merged

    Reopen if this still reproduces on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions