Skip to content

Route PR Sous Chef to cost/economy model routing instead of the Haiku 4.5 pin - #76325

Merged
SivaKesava1 merged 2 commits into
mainfrom
copilot/route-pr-sous-chef-cost-economy
Nov 9, 2026
Merged

SivaKesava1 merged 2 commits into
mainfrom
copilot/route-pr-sous-chef-cost-economy

Conversation

Copilot AI commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

PR Sous Chef was pinned to copilot/claude-haiku-4.5. Detection inherits that pin, and it was about 62% of the workflow's weekly AIC. It was also the highest-cost workflow in the week-of-2-November CSV. This PR switches the agent to cost/economy routing. The prompt, triggers, prefilter, permissions, safe outputs, evals and detection configuration stay the same.

Change (pr-sous-chef.md)

-model: copilot/claude-haiku-4.5
 engine:
   id: pi
   model-provider: github
+  model-routing:
+    goal: cost
+    mode: economy

There is no allowed-models, so pi routes over the policy-filtered supported catalogue.

Acceptance case A in the regenerated lock

  • Agent AWF config: apiProxy.routing.objective is {"goal":"cost","mode":"economy"}, with experimental.modelRouting: true and task.conversationFile=/tmp/gh-aw/routing-conversation.json.
  • Routing setup: the Prepare model-routing conversation step is added, and the router image ghcr.io/github/gh-aw-firewall/router:0.29.10 is in the manifest and image download.
  • No agent pin:
    • GH_AW_PI_MODEL and agent_model are gone.
    • GH_AW_INFO_MODEL is ${{ vars.GH_AW_MODEL_AGENT_CUSTOM || 'agent' }}.
    • Downstream GH_AW_ENGINE_MODEL now reads needs.agent.outputs.model.
    • The allowed-models list is empty, so there is no candidate shortlist.
    • No haiku string remains in the lock.

Detection model after the change

Detection no longer inherits Haiku 4.5. It now uses the independent default chain:

COPILOT_MODEL: ${{ vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'detection' }}
GH_AW_MODEL_FALLBACK: ${{ vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }}

The detection alias is resolved by AWF at runtime, not by the economy router. The compiled lock alone doesn't guarantee which model detection ends up on. A branch run (case B) has to confirm it isn't Haiku 4.5.

Still open (maintainers)

  • Cases B–D:
    • Branch run with a selection-stage firewall.model_routing event.
    • Agent, detection, evals and total AIC, compared with main runs.
    • All four eval answers by ID.
    • Agent-cost guard: if agent AIC is above ~7.8/run, propose an allowed-models shortlist or another mode before merging.
  • Not dispatched: the workflow hasn't been run from this branch.
  • make agent-finish fails (exit 2) on two timing-sensitive pkg/cli logs-download tests: TestDownloadWorkflowLogsForTargetsConcurrentAndResilient and TestDownloadWorkflowLogsForTargetsUsesOneWallClockTimeout. Both pass when run alone, with and without this change, and this PR doesn't touch that code. I haven't compared against the same check on main.
  • CodeQL scan: it timed out, so there's no security-scan result for this change.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Copilot AI changed the title [WIP] Move PR Sous Chef to cost/economy model routing Route PR Sous Chef to cost/economy model routing instead of the Haiku 4.5 pin Nov 9, 2026
Copilot AI requested a review from SivaKesava1 November 9, 2026 22:59
@SivaKesava1
SivaKesava1 marked this pull request as ready for review November 9, 2026 23:03
Copilot AI dynamic review requested due to automatic review settings November 9, 2026 23:03
Comment thread .github/workflows/pr-sous-chef.lock.yml Resolved
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

🚀 PR Specification Reviewer has started processing this pull request

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 8.5K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt61 medium · ~4 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 181 · ◷

Generated by Ponytail Reviewer for #76325

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

PR Code Quality Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Code Quality Reviewer · gpt56 · ~3 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Maintainer validation is needed because branch-runtime evidence for routing selection, the effective detection model, all four evaluation answers, and phase costs against the ~7.8 agent-AIC guard is missing.

No open issues 🎉

What changed in this PR

Updates PR Sous Chef’s model selection to address #76322’s cost-reduction goal without changing its task, triggers, permissions, safe outputs, or four eval questions.

Changes:

  • Replaces the Haiku 4.5 pin with cost/economy routing, without an allowed-models shortlist.
  • Regenerates the lock with router setup, runtime model reporting, and routing artifacts.
  • Removes detection’s inherited pin so it uses the independent default model chain.

Runtime model selection, cost savings, and eval outcomes remain unverified; the PR reports no branch run.

File Description
.github/​workflows/​pr-sous-chef.md Enables cost/economy routing while preserving workflow behavior and controls.
.github/​workflows/​pr-sous-chef.lock.yml Adds generated routing support and metadata, and restores detection’s default model selection.

Review effort: Balanced (auto)

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-09T23:10:00Z
review_event: COMMENT
top_themes: [unvalidated economy model routing without allowed-models shortlist]
files_reviewed: [.github/workflows/pr-sous-chef.md, .github/workflows/pr-sous-chef.lock.yml]
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The routing change is directionally consistent, but the new economy policy has no model shortlist and is not yet validated by the acceptance runs described in the PR. I posted one inline comment requesting a tested shortlist or completion of those validations before relying on arbitrary catalogue selection.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

Comment thread .github/workflows/pr-sous-chef.md Resolved

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd to the changed config. Requesting changes on the detection model claim and missing regression guard.

📋 Key Themes & Highlights

Key Themes

  • Detection may still use Haiku: the detection alias chain (small → mini → haiku) can resolve to claude-haiku-4.5, so the "no longer inherits Haiku" claim is unverified.
  • No regression guard: nothing asserts the compiled routing config or prevents a future Haiku pin.
  • Unbounded routing cost: economy routing has no allowed-models shortlist, so the ~7.8 AIC/run guard is not enforced.
  • Prompt injection (existing comment): the routing conversation writes untrusted PR text verbatim (lock line 3539). The existing security-bot comment covers this; I did not duplicate it.

Positive Highlights

  • ✅ Removing the agent-level Haiku pin in favour of declarative model-routing is a clean, schema-valid change.
  • ✅ Routing config is consistently wired into the agent AWF config.

Limitations

  • The local origin/main diff was unavailable, so line anchors were checked against HEAD (8f7b803), not the PR diff.
  • Whether needs.agent.outputs.model is populated under routing was not verified.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~9 AIC · ⌖ ~2 AIC · ⊞ 6.3K
Comment /matt to run again

Comment thread .github/workflows/pr-sous-chef.lock.yml Resolved
Comment thread .github/workflows/pr-sous-chef.md Resolved
Comment thread .github/workflows/pr-sous-chef.md Resolved
@SivaKesava1

Copy link
Copy Markdown
Collaborator

Branch run: case B–D evidence

Dispatched pr-sous-chef.lock.yml on copilot/route-pr-sous-chef-cost-economy at 8f7b803: run 42718933073, all jobs success (prefilter, agent, detection, evals, safe_outputs). Audited with a dev build of main.

Routing (case B): one agent-phase firewall.model_routing selection: github-copilot/claude-haiku-5.5, effort high, mode economy. Classifier labels chore / subsystem / easy; the classifier call ran on Haiku 5.5. All 9 routed agent requests returned 200.

Cost by phase (from firewall.token_usage by provenance.phase, plus evals usage.report):

Run Agent (incl. classifier) Detection Evals Total
This branch run (routed) 1.20 1.25 (Haiku 5.5) 0.03 2.49
main 42702014442 3.92 12.97 (Haiku 4.5) 0.01 16.90
main 42705086505 4.37 12.21 (Haiku 4.5) 0.02 16.59

Case D guard (agent ≤ about 7.8 AIC): met. Detection no longer inherits Haiku 4.5; it resolved to Haiku 5.5 through the detection default.

Behavior (case C): this run did real work, which the four main baselines in the issue didn't exercise. It posted two assignment work summaries and assigned Copilot on #76331 and #76332, then created its summary issue. Both comments start with the literal @copilot, list a branch update plus failing checks or unresolved review threads with links, and keep the structured data envelope. The footer shows routed: haiku55 high.

Evals:

Eval This run main, last two weeks (127 runs)
nudge-targeted NO NO 31, YES 20, UNKNOWN 76
dedup-respected UNKNOWN YES 11, UNKNOWN 116
progress-or-noop YES YES 124, NO 2, UNKNOWN 1
ci-follow-up-targeted UNKNOWN YES 3, NO 2, UNKNOWN 2

The judge gives no reason for nudge-targeted. On main it answers NO in about 60% of the runs that assign work (31 of 51), so a NO here is within the Haiku 4.5 baseline, not a regression we can attribute to routing. Main eval counts are from the router-rollout eval data for the weeks of 2 and 9 Nov.

Code-scanning alert about routing-conversation.json: answered as out of scope (it applies to every routed workflow).

One run isn't a long-term measurement; we'll compare the first scheduled routed runs after merge with the rollout tracker.

@SivaKesava1
SivaKesava1 dismissed github-actions[bot]’s stale review November 9, 2026 23:34

Points answered on the threads: the branch run shows detection on Haiku 5.5 (not 4.5) at 1.25 AIC, a compile test is out of scope for #76322, and agent cost is 1.20 AIC (limit about 7.8). Evidence: #76325 (comment)

@SivaKesava1
SivaKesava1 merged commit 34f5e69 into main Nov 9, 2026
134 of 135 checks passed
@SivaKesava1
SivaKesava1 deleted the copilot/route-pr-sous-chef-cost-economy branch November 9, 2026 23:39
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Route PR Sous Chef from Haiku 4.5 to cost/economy model routing

4 participants