Repository navigation
Route Matt Pocock Skills Reviewer with model routing instead of pinning claude-sonnet-5 - #73836
Conversation
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Code Quality Reviewer completed the code quality review.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🔬 Test Quality Sentinel is analyzing test quality on this pull request... |
There was a problem hiding this comment.
Review: no additional actionable issues
The routing change is consistent with the workflow configuration. I did not identify a separate correctness or reliability issue beyond the prompt-injection concerns already raised in the existing review comments.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.7K
Comment /review to run again
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the cost-focused model routing change.
No open issues 🎉
What changed in this PR
Switches Matt Pocock Skills Reviewer to cost-focused model routing, implementing #73823’s plan to reduce spending on advisory PR reviews.
Changes:
- Replaces the
claude-sonnet-5pin withgoal: costandmode: economy, preserving thepr-triagesub-agent’s model. - Regenerates the lock file with a digest-pinned router image and routed model, reasoning effort, and status reporting.
- Moves threat detection from the inherited model pin to configured defaults.
| File | Description |
|---|---|
.github/workflows/mattpocock-skills-reviewer.md |
Enables economy routing and removes the fixed model. |
.github/workflows/mattpocock-skills-reviewer.lock.yml |
Adds compiled routing configuration, downstream reporting, and detection model defaults. |
Review effort: Balanced (auto)
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd and /codebase-design to this config-only change. No blocking issues; two non-blocking items below.
📋 Findings
1. Rollback steps in the PR description are incomplete
Restoring model: claude-sonnet-5 alone will not revert the change. engine.model-routing overrides fixed model settings (the schema says routing "ignores fixed model and effort settings"), so a rollback must also remove the model-routing block and run make recompile. Update the "To roll back" line to match.
2. Threat-detection model change is not covered by the post-merge checks
The base lock pins the detection job to claude-sonnet-5 via COPILOT_MODEL. This PR changes it to vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'detection'. Unless one of those repo variables is set, the value is the literal detection, which I could not confirm is a valid model ID. The same 'detection' fallback appears in 185 lock files, so it may be intentional, but threat detection is a security gate. Confirm the resolved model on a routed run, or set GH_AW_MODEL_DETECTION_COPILOT explicitly, and add this to the post-merge list.
Already covered
I agree with the existing prompt-injection findings on the routing prompt and have not duplicated them.
Method
Triage was classified directly from the diff (pr-triage was not invoked). The change is config-only, so the review focused on the frontmatter and the compiled lock-file deltas.
Positive highlights
- ✅ Keeping
pr-triageon Haiku while routing the main agent is a clean split.
@copilot please update the rollback wording and confirm the detection model.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~6 AIC · ⌖ ~1 AIC · ⊞ 5.4K
Comment /matt to run again
| id: copilot | ||
| copilot-sdk: true | ||
| max-continuations: 6 | ||
| model-routing: |
There was a problem hiding this comment.
[/codebase-design] Rollback is incomplete. Restoring model: claude-sonnet-5 alone is overridden while model-routing is set. To roll back, remove this block and run make recompile.
@copilot please address this.
| COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode | ||
| COPILOT_GITHUB_TOKEN: ${{ github.token }} | ||
| COPILOT_MODEL: claude-sonnet-5 | ||
| COPILOT_MODEL: ${{ vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'detection' }} |
There was a problem hiding this comment.
[/tdd] Threat-detection model is no longer pinned to claude-sonnet-5. If neither repo variable is set, it resolves to the literal detection, which may not be a valid model ID. Set GH_AW_MODEL_DETECTION_COPILOT explicitly, or confirm the resolved model on a routed run.
@copilot please address this.
|
🎉 This pull request is included in a new release. Release: |
Matt Pocock Skills Reviewer is the most expensive gh-aw workflow that isn't routed yet: about 230 runs a week at about 111 AIC each, all pinned to
claude-sonnet-5. This PR switches it to model routing withgoal: cost,mode: economy. The issue estimates this saves about 23,100 AIC a week. Its reviews are advisory comments, so the risk from a cheaper model is thinner reviews, not broken code.Changes
mattpocock-skills-reviewer.mdmodel: claude-sonnet-5.model-routingto theengineblock:pr-triagesub-agent stays onclaude-haiku-4.5. Noallowed-models, blocked models ormodel-providerare set.mattpocock-skills-reviewer.lock.yml(regenerated bymake recompile)gh-aw-firewall/router:0.29.0.claude-sonnet-5.GH_AW_MODEL_DETECTION_COPILOT, falling back toGH_AW_DEFAULT_MODEL_COPILOT, then to thedetectiondefault.After merge
gh aw audit <run>on the first routed runs (Model Routing section) and compare AIC per run with the current ~111.mode: balanced. To roll back, restoremodel: claude-sonnet-5.