Skip to content

Canary promotion gate counts a fleet-paused feature-ideation run as a healthy sample #1310

Open

Description

@don-petry

Problem

scripts/canary-rollout.sh (_tier_sample, lines 1258 to 1266 at b1387cbd) counts whole workflow runs that ended success or failure as health samples for a release channel. feature-ideation is in standards/canary-rings.json with no ingress_job, so its runs are sampled whole.

Once "feature-ideation reusable calls the model but ignores AGENTS_PAUSED" (#1295) lands, a run made while the fleet is paused skips both working jobs and still ends success, because the caller stub's own prep job always succeeds (and the notice job succeeds too). That run did nothing, and it counts as a healthy sample of the release it pinned. ring1 -> stable needs sample_min: 1, so a single paused Friday run in a ring1 repository can satisfy the gate and promote a release nobody exercised.

petry-projects/.github-private avoids this for the workflows it samples by giving them no notice job, so a paused run ends skipped. That cannot be done from inside this reusable: the stub's prep job is what makes the run green, and the stubs are frozen.

This is not new with #1295 in one respect: before it, a paused run was not paused at all. It becomes reachable when the gate reaches a channel.

Scope

Make the sampler recognise a paused run and leave it out. The run's own record is the fleet-paused job concluding success; the variable's value at run time is not in the API afterwards. petry-projects/.github-private has the same rule for its monitors ("Three monitors misread a paused run", combsmith/.github-private#2266, scripts/lib/fleet-pause.sh).

Acceptance criteria

  1. A sampled run whose fleet-paused job concluded success is not counted as a success or a failure sample.
  2. A test with a paused run as the only run in the window: the gate reports insufficient samples, not a pass.
  3. Unpaused runs are counted exactly as before.
  4. The cost of the extra read is stated (jobs are needed only for runs that ended success of workflows whose reusable carries the notice job).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reports

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions