Repository navigation
💡 Canary Synthetic Exercise Runs to Replace Sample Waivers with Evidence for Idle Rings #1518
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
Replace the growing set of canary sample waivers (waive_sample, waive_sample_if_no_caller, and today's waive_sample_if_idle_after_hours) with synthetic exercise runs. Add a dedicated fixture repo, a ring member tagged
synthetic, that dispatches each agent reusable at the candidate tag against canned issue/PR fixtures in DRY_RUN mode. Idle rings then promote on observed executions, not on elapsed time.Market Signal
Progressive-delivery tools solved the same 'no traffic, no evidence' problem with synthetic load during analysis. Flagger's load tester is invoked by a rollout webhook 'for workloads that are not receiving constant traffic', and Argo Rollouts' analysis and Kayenta-style canary analysis use the same pattern. Google SRE practice calls these probers. The industry consensus is to generate the minimum traffic needed for a verdict rather than promote on silence.
User Signal
Canary sample starvation is the single most recurrent bug theme in this repo. #1513/#1514 (merged today): pr-review ring1->stable never cleared because bmad-bgreat-suite was idle; the fix was an idle waiver. #1270: ci-failure-analyst stable has been frozen on v1.0.0 since 2026-07-10 because ring1 has no sample. #1121: dwell/sample count only candidate-executing runs, and NO_COVERAGE is reported when there are none. #1310: a fleet-paused feature-ideation run was counted as a healthy sample. #1324: two candidates in flight, a BLOCKED gate can't be advanced. Each fix either waives evidence or tightens what counts, which tends toward more waivers.
Technical Opportunity
scripts/canary-rollout.sh already has the needed seams. standards/canary-rings.json
member_tokensand per-agent ring lists can name a fixture repo._tier_sampleand_sample_decision_countscount executed runs per member, and_tier_inflighttreats in-flight runs conservatively. Every agent reusable already honours DRY_RUN (mutations go to a JSONL log, as in feature-ideation's discussion-mutations.sh). A scheduledcanary-exercise.ymlin the fixture repo would (1) read the candidate tag per agent, (2) open or reuse fixture issues and PRs, (3) invoke the reusable at that tag with DRY_RUN=1, and (4) assert on its step summary or decision step. The gate counts these runs in a separatesyntheticbucket, requiring for example at least 2 synthetic successes in place of any idle or no-caller waiver, and never letting synthetic runs satisfy the organic-sample floor where organic traffic exists.Assessment
Adversarial Review
Strongest objection: Synthetic fixtures don't exercise real-world input diversity, so they give false confidence. They also cost LLM tokens and Actions minutes, and add a repo to maintain. The idle waiver merged today already solves #1513.
Rebuttal: The comparison isn't synthetic versus organic evidence; it's synthetic evidence versus none. Waivers promote a candidate that has never executed on that tier, so a startup crash, a missing secret mapping, a broken workflow_call interface or a permissions regression ships straight to stable. A DRY_RUN fixture run catches exactly that class, which is what #1123 workload-timeout and the startup-failure classes look for. The cost is bounded: idle rings only, one or two runs per candidate, DRY_RUN, and a cheap model family. The idle waiver can stay as the fallback when synthetic runs are disabled. This builds on #1475 (offline property simulation of the gate) by feeding the live gate real executions.
Suggested Next Step
Prototype for one agent (ci-failure-analyst, stuck since 2026-07-10 per #1270): create the fixture repo, add it to that agent's ring1 in canary-rings.json with a
synthetic: trueflag, write canary-exercise.yml dispatching the reusable at the ring1 candidate tag with DRY_RUN=1, and teach_tier_sampleto report synthetic counts separately in the gate notice. Success means ci-failure-analyst reaches stable on evidence within one dwell window.Sources
_tier_sample,_tier_inflight,waive_sample_if_idle_after_hours)Generated by the weekly feature-ideation run (Mary, BMAD Analyst).
All reactions