Skip to content

[deep-report] Experiment state.Counts never reconciled against current variants list — stale labels skew balance/selection stats #58489

Closed

Description

@github-actions

Description

The experiments CLI persists per-variant run counts in state.Counts (pkg/cli/experiments_state.go) keyed by whatever variant label a run recorded, and never reconciles those keys against a workflow's current experiments.<name>.variants list in frontmatter. When a workflow's declared variant set changes (e.g. renaming model identifiers), old runs' now-stale variant labels stay in the branch's state.json forever. Today's Daily Experiment Report (2026-09-04, discussion #58431) surfaced 4 live examples: daily-cache-strategy-analyzer, daily-caveman-optimizer, daily-doc-healer, and daily-doc-updater all show a model_size experiment with legacy labels (small-agent, agent, single-digit counts) mixed alongside the current model names — e.g. daily-cache-strategy-analyzer reports small-agent=1, agent=1, gpt-5.4-mini=18, gpt-5-mini=2, gpt-5-codex=8, gpt-5.4=28 against a frontmatter variants: [gpt-5.4, gpt-5.4-mini]. This skews the chi-square balance test (reported p≈0.000, i.e. "imbalanced," when the real 2-variant split is close to even) and can also skew deriveLastSelectedVariant's max-count heuristic in pkg/cli/audit_report_experiments.go, since both iterate over all keys in the counts map with no filter against the declared variant list.

Expected Impact

Prevents future experiment reports from misreporting balance/readiness for any workflow that has ever renamed its variant set, and stops the "last selected variant" heuristic from being polluted by counts that can no longer be produced by the current config.

Suggested Agent

New Agent — add a reconciliation step (either at read time in experimentDetailsFromState/deriveLastSelectedVariant, or as a one-off state migration) that filters state.Counts[name] to only the variant keys present in the workflow's currently-declared variants: list before computing balance/selection stats.

Estimated Effort

Medium (1-4 hours)

Data Source

DeepReport Intelligence analysis, 2026-09-04 cycle, derived from Daily Experiment Report discussion #58431.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 🔬 Deep Report · claude · agent · 231.7 AIC · ⌖ 10.9 AIC · ⊞ 12.4K · ◷

  • expires on Sep 6, 2026, 4:48 AM UTC-08:00

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions