Skip to content

Route PR Sous Chef from Haiku 4.5 to cost/economy model routing #76322

Closed

Description

@SivaKesava1

Problem

Move PR Sous Chef from its fixed Haiku 4.5 model to cost / economy model routing, without changing its task or publication behavior.

At audited main revision 1e0f35fda7eedec0ffc5299e2523885789d14fb0, .github/workflows/pr-sous-chef.md declares:

  • Top-level model: copilot/claude-haiku-4.5; engine.id: pi, model-provider: github, and copilot-requests: write.
  • schedule: every 5m, centralized /souschef commands on pull_request_comment, and workflow_dispatch. This is the declared schedule, not a guarantee of five-minute execution intervals.
  • A deterministic prefilter, safe outputs, and four, not two, eval questions: nudge-targeted, dedup-respected, progress-or-noop, and ci-follow-up-targeted.

The pin also drives expensive threat detection. inheritedDetectionModel returns the workflow's model for pi. With the GitHub provider, detection runs on the Copilot engine, not pi. Detection does not participate in agent model routing.

Verified cost evidence

Recomputed from the week-of-2-November CSV, filtering exactly workflow == "pr-sous-chef":

Metric Recorded value
Runs 884: 715 success, 166 failure, 3 cancelled
Runs with positive agent usage 614
Agent AIC 4,184.280; median 5.1715, approximately 5.2
Agent model in those 614 rows claude-haiku-4-5-20251001
Runs with positive detection usage 613
Detection AIC 6,748.736; median 10.35
Evals AIC 11.325
Total AIC 10,944.341
Detection share 61.66%, approximately 62%

This is the highest workflow total in that weekly CSV, and the audited workflow is still pinned. Do not generalize this ranking beyond the supplied dataset. Positive-usage counts are not proof that every other run skipped the job; the recorded eval total likewise is not a count of all eval executions.

Four recent main runs audited

Used a Windows development build of main at 1e0f35f, version dev-main-1e0f35f, rather than the installed stable v1.0.1. Each full-URL audit used --no-baseline --timeout 2m --json; all four completed with full artifact collection. These are scheduled runs at the same main SHA with AWF v0.29.10; the agent, detection, and evals jobs succeeded in every run.

Run, 9 Nov UTC Agent AIC Detection AIC Evals AIC Total AIC
42702014442, 21:23 3.922305 12.965245 0.009190 16.896740
42703616561, 21:33 3.708315 11.740075 0.009370 15.457760
42705086505, 21:42 4.368810 12.209150 0.016020 16.593980
42706610685, 21:52 3.343995 14.544720 0.009150 17.897865

In all four, the agent usage names claude-haiku-4-5-20251001, detection usage names claude-haiku-4.5, and eval usage names gpt-6-luna. Detection's recorded threat flags were false. No firewall.model_routing events were present, as expected for this fixed-model configuration.

Accounting was cross-checked against usage/aw_session.jsonl phase provenance, usage/agent_usage.json, detection/detection_usage.json, and usage/evals/token_usage.jsonl. The audit's top-level metrics.aic is agent usage here, not the whole-workflow total. Sum each phase once; do not add duplicate artifact copies or cumulative totalAic counters.

Behavior baseline: run 42705086505 published a literal-@copilot CI-follow-up comment on an eligible PR and a summary issue, with no assignment. The other three produced no-op requests and no PR writes. All four had UNKNOWN for nudge-targeted and dedup-respected, and YES for progress-or-noop; ci-follow-up-targeted was YES in 42705086505 and UNKNOWN in the other three. A green evals job is therefore not evidence that all four answers were YES, nor do these runs exercise a successful assignment.

Routing context, not a savings guarantee

#75843 made AWF v0.29.10 the default on 8 November at 18:23 UTC. The rollout report identifies router 0.3.0 and the new tables from github/gh-aw-firewall#11198. That upstream change adds review/analyze classification and removes Haiku 4.5 from the router's ranked catalogue; that does not remove it from the serving catalogue or from an explicit detection pin.

The published savings-page aggregate through 9 November is approximately 73% lower cost per run, 72.4 to 19.6 AIC, across mixed table versions and workflows. A blanket "~84% from the new tables" is not established by that aggregate; it needs a separately defined new-table-only cohort. These observational before/after results also include detection/default changes and task-mix differences, not an isolated causal measurement of the router. The report documents cases where detection/evals fell to about 1 AIC per run after removing pins, but neither 1 AIC detection nor an 84% reduction is promised for Sous Chef.

Intended change

Only change .github/workflows/pr-sous-chef.md and its regenerated .lock.yml:

  1. Remove the top-level model: copilot/claude-haiku-4.5.

  2. Retain the engine/provider and add:

    engine:
      id: pi
      model-provider: github
      model-routing:
        goal: cost
        mode: economy
  3. Do not set allowed-models initially. For pi, omission uses the policy-filtered supported catalogue, rather than restricting routing to one family; see the compiler and routing reference.

  4. Keep the prompt, triggers, prefilter jobs/scripts, permissions, safe outputs, all four evals, and detection configuration unchanged. Recompile the lock with current main tooling and inspect the generated diff.

Detection after removing the pin

This removes inherited fixed-model selection, not detection itself. Without an explicit detection model, resolveThreatDetectionModel / defaultDetectionModel use a compiler GH_AW_DEFAULT_DETECTION_MODEL override if supplied; otherwise the Copilot fallback is:

${{ vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'detection' }}

No matching repository-level variables were returned during this investigation. That does not rule out organization-level/runtime variables or a future compiler override.

The built-in detection alias includes small/Haiku and Luna candidates. AWF v0.29.10 alias resolution resolves against available, permitted models and sorts candidates by version; it is not the economy router. Thus there is no unconditional compile-time guarantee of a particular concrete detection model or cost. If the branch run still uses Haiku 4.5 or another fixed override, identify the effective alias/override precedence in the PR and mark the detection-cost goal unmet. Do not silently add a separate detection pin or expand this fix into detection tuning.

Acceptance cases

A. Compiled configuration

Input: The narrowly edited Markdown, compiled with current main tooling.

Observable result: The agent's emitted AWF configuration contains apiProxy.routing.objective with goal: cost and mode: economy, the lock contains Prepare model-routing conversation, and the router image is included. There is no retained concrete agent-model pin or artificial candidate shortlist. The detection job no longer contains an inherited Haiku 4.5 pin; report its independent default expression. Confirm the effective detection model in B is no longer Haiku 4.5, or explicitly report the alias/override blocker described above rather than claiming success.

B. Branch-run routing and phase costs

Input: A published PR branch containing the reviewed Markdown and generated lock, and repository-approved live-test authorization. An authorized operator can select that branch with:

gh workflow run pr-sous-chef.lock.yml --repo github/gh-aw --ref <branch>

Observable result: A run that actually reaches the agent succeeds. In its usage/aw_session.jsonl, exactly one selection-stage event has type: firewall.model_routing, provenance.phase: agent, data.stage: selection, and a nonempty data.selectedModel. Classification/request events may also exist; do not require only one routing event in total. Record the classifier's task type, scope and complexity labels, selected model and selected effort, plus any degraded-classification status.

Report agent, detection, evals and total AIC, including classifier usage once within the agent phase, beside the same phase numbers for at least two recent main runs from the table above. Record source/lock SHA, trigger, actual detection model, routing mode and AWF/router versions. Do not call an activation/prefilter-only success a routing test.

Follow the repository's live-testing gates. This issue is not authorization for an implementation agent to dispatch production effects; cloud agents are dry-run-only under repository policy. Identify a dry-run explicitly and distinguish staged outputs from published outputs. No runs were dispatched for this investigation.

C. Preserve targeting, deduplication and evaluation

Input: B's queue snapshot, refreshed eligibility/simulation results, safe-output requests/outcomes, and eval answers.

Observable result: Assignment work summaries, assignments and CI follow-ups target only the prefilter's eligible prs list and honor the existing final-refresh rules. Every assignment work-summary and CI-follow-up comment starts with the literal @copilot, not a Markdown link. No PR receives a duplicate assignment; a completed-task CI follow-up does not create another assignment.

Preserve the existing exception: a /souschef acknowledgement may target its triggering PR even when ineligible, omits @copilot/assignment state, and remains subject to the workflow's latest-comment and rate-limit guards. Do not accidentally prohibit that behavior, ordinary summary issues, allowed CI approvals, or review cleanup.

Report all four eval answers by ID. Applicable answers should be YES, or any NO/UNKNOWN must be explained against the same reason on comparable main inputs. Main's UNKNOWN answers above must not be relabeled as passes. A no-op run does not demonstrate assignment correctness; explicitly mark unexercised paths instead of changing the prompt, evals or tests to manufacture a pass.

D. Agent-cost regression guard

Input: B's measured agent-phase AIC compared with the historical Haiku median of approximately 5.2.

Observable result: If agent cost exceeds approximately 7.8 AIC/run (1.5 times 5.2), call that out in the PR and propose a justified allowed-models shortlist or another mode instead of merging as-is. Do not hide an agent regression behind cheaper detection or assume one run establishes long-term savings.

Non-goals

  • Other Haiku 4.5-pinned workflows, including ai-moderator, and gpt-5 alias pins.
  • Changes to the prompt, schedule/triggers, prefilter, safe outputs, evals, detection configuration, AWF, router tables or compiler.
  • Separately tuning the threat-detection model.

Scope

  • Keep the PR to this one fix. Do not change files or behavior unrelated to it.
  • Before fixing a failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR. Merge main once main is fixed, and note in the PR which checks are known failures on main.
  • This applies even when a bot comment asks to fix a failing check: fix only failures caused by this PR's changes.
  • Manual PR-branch runs can fail for unrelated reasons, including GitHub errors, draft PRs and report_incomplete. Do not change prompts, safe outputs or tests in response to those unrelated failures.

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions