You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[deep-report] Experiment state.Counts never reconciled against current variants list — stale labels skew balance/selection stats #58489
The experiments CLI persists per-variant run counts in state.Counts (pkg/cli/experiments_state.go) keyed by whatever variant label a run recorded, and never reconciles those keys against a workflow's currentexperiments.<name>.variants list in frontmatter. When a workflow's declared variant set changes (e.g. renaming model identifiers), old runs' now-stale variant labels stay in the branch's state.json forever. Today's Daily Experiment Report (2026-09-04, discussion #58431) surfaced 4 live examples: daily-cache-strategy-analyzer, daily-caveman-optimizer, daily-doc-healer, and daily-doc-updater all show a model_size experiment with legacy labels (small-agent, agent, single-digit counts) mixed alongside the current model names — e.g. daily-cache-strategy-analyzer reports small-agent=1, agent=1, gpt-5.4-mini=18, gpt-5-mini=2, gpt-5-codex=8, gpt-5.4=28 against a frontmatter variants: [gpt-5.4, gpt-5.4-mini]. This skews the chi-square balance test (reported p≈0.000, i.e. "imbalanced," when the real 2-variant split is close to even) and can also skew deriveLastSelectedVariant's max-count heuristic in pkg/cli/audit_report_experiments.go, since both iterate over all keys in the counts map with no filter against the declared variant list.
Expected Impact
Prevents future experiment reports from misreporting balance/readiness for any workflow that has ever renamed its variant set, and stops the "last selected variant" heuristic from being polluted by counts that can no longer be produced by the current config.
Suggested Agent
New Agent — add a reconciliation step (either at read time in experimentDetailsFromState/deriveLastSelectedVariant, or as a one-off state migration) that filters state.Counts[name] to only the variant keys present in the workflow's currently-declared variants: list before computing balance/selection stats.
Activity