Repository navigation
Route PR Sous Chef to cost/economy model routing instead of the Haiku 4.5 pin - #76325
Conversation
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
|
🚀 PR Specification Reviewer has started processing this pull request |
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Code Quality Reviewer completed the code quality review. PR Code Quality ReviewerOutcome: The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
There was a problem hiding this comment.
🔵 Needs a closer look
Maintainer validation is needed because branch-runtime evidence for routing selection, the effective detection model, all four evaluation answers, and phase costs against the ~7.8 agent-AIC guard is missing.
No open issues 🎉
What changed in this PR
Updates PR Sous Chef’s model selection to address #76322’s cost-reduction goal without changing its task, triggers, permissions, safe outputs, or four eval questions.
Changes:
- Replaces the Haiku 4.5 pin with cost/economy routing, without an
allowed-modelsshortlist. - Regenerates the lock with router setup, runtime model reporting, and routing artifacts.
- Removes detection’s inherited pin so it uses the independent default model chain.
Runtime model selection, cost savings, and eval outcomes remain unverified; the PR reports no branch run.
| File | Description |
|---|---|
.github/workflows/pr-sous-chef.md |
Enables cost/economy routing while preserving workflow behavior and controls. |
.github/workflows/pr-sous-chef.lock.yml |
Adds generated routing support and metadata, and restores detection’s default model selection. |
Review effort: Balanced (auto)
Comment MemoryPeek at saved memory (pr-code-quality-reviewer)Note This comment is managed by comment memory.Expand the saved memory block to view or edit the persistent context for this thread.
|
There was a problem hiding this comment.
The routing change is directionally consistent, but the new economy policy has no model shortlist and is not yet validated by the acceptance runs described in the PR. I posted one inline comment requesting a tested shortlist or completion of those validations before relying on arbitrary catalogue selection.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnosing-bugs and /tdd to the changed config. Requesting changes on the detection model claim and missing regression guard.
📋 Key Themes & Highlights
Key Themes
- Detection may still use Haiku: the
detectionalias chain (small → mini → haiku) can resolve toclaude-haiku-4.5, so the "no longer inherits Haiku" claim is unverified. - No regression guard: nothing asserts the compiled routing config or prevents a future Haiku pin.
- Unbounded routing cost: economy routing has no
allowed-modelsshortlist, so the ~7.8 AIC/run guard is not enforced. - Prompt injection (existing comment): the routing conversation writes untrusted PR text verbatim (lock line 3539). The existing security-bot comment covers this; I did not duplicate it.
Positive Highlights
- ✅ Removing the agent-level Haiku pin in favour of declarative
model-routingis a clean, schema-valid change. - ✅ Routing config is consistently wired into the agent AWF config.
Limitations
- The local
origin/maindiff was unavailable, so line anchors were checked against HEAD (8f7b803), not the PR diff. - Whether
needs.agent.outputs.modelis populated under routing was not verified.
@copilot please address the review comments above.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~9 AIC · ⌖ ~2 AIC · ⊞ 6.3K
Comment /matt to run again
Branch run: case B–D evidenceDispatched Routing (case B): one agent-phase Cost by phase (from
Case D guard (agent ≤ about 7.8 AIC): met. Detection no longer inherits Haiku 4.5; it resolved to Haiku 5.5 through the Behavior (case C): this run did real work, which the four main baselines in the issue didn't exercise. It posted two assignment work summaries and assigned Copilot on #76331 and #76332, then created its summary issue. Both comments start with the literal Evals:
The judge gives no reason for Code-scanning alert about One run isn't a long-term measurement; we'll compare the first scheduled routed runs after merge with the rollout tracker. |
Points answered on the threads: the branch run shows detection on Haiku 5.5 (not 4.5) at 1.25 AIC, a compile test is out of scope for #76322, and agent cost is 1.20 AIC (limit about 7.8). Evidence: #76325 (comment)
|
🎉 This pull request is included in a new release. Release: |
PR Sous Chef was pinned to
copilot/claude-haiku-4.5. Detection inherits that pin, and it was about 62% of the workflow's weekly AIC. It was also the highest-cost workflow in the week-of-2-November CSV. This PR switches the agent to cost/economy routing. The prompt, triggers, prefilter, permissions, safe outputs, evals and detection configuration stay the same.Change (
pr-sous-chef.md)There is no
allowed-models, so pi routes over the policy-filtered supported catalogue.Acceptance case A in the regenerated lock
apiProxy.routing.objectiveis{"goal":"cost","mode":"economy"}, withexperimental.modelRouting: trueandtask.conversationFile=/tmp/gh-aw/routing-conversation.json.ghcr.io/github/gh-aw-firewall/router:0.29.10is in the manifest and image download.GH_AW_PI_MODELandagent_modelare gone.GH_AW_INFO_MODELis${{ vars.GH_AW_MODEL_AGENT_CUSTOM || 'agent' }}.GH_AW_ENGINE_MODELnow readsneeds.agent.outputs.model.haikustring remains in the lock.Detection model after the change
Detection no longer inherits Haiku 4.5. It now uses the independent default chain:
The
detectionalias is resolved by AWF at runtime, not by the economy router. The compiled lock alone doesn't guarantee which model detection ends up on. A branch run (case B) has to confirm it isn't Haiku 4.5.Still open (maintainers)
firewall.model_routingevent.allowed-modelsshortlist or another mode before merging.make agent-finishfails (exit 2) on two timing-sensitivepkg/clilogs-download tests:TestDownloadWorkflowLogsForTargetsConcurrentAndResilientandTestDownloadWorkflowLogsForTargetsUsesOneWallClockTimeout. Both pass when run alone, with and without this change, and this PR doesn't touch that code. I haven't compared against the same check on main.