Skip to content

Route Matt Pocock Skills Reviewer with model routing instead of pinning claude-sonnet-5 - #73836

Merged
SivaKesava1 merged 2 commits into
mainfrom
copilot/route-matt-pocock-skills-reviewer
Nov 1, 2026
Merged

SivaKesava1 merged 2 commits into
mainfrom
copilot/route-matt-pocock-skills-reviewer

Conversation

Copilot AI commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Matt Pocock Skills Reviewer is the most expensive gh-aw workflow that isn't routed yet: about 230 runs a week at about 111 AIC each, all pinned to claude-sonnet-5. This PR switches it to model routing with goal: cost, mode: economy. The issue estimates this saves about 23,100 AIC a week. Its reviews are advisory comments, so the risk from a cheaper model is thinner reviews, not broken code.

Changes

  • mattpocock-skills-reviewer.md
    • Removes model: claude-sonnet-5.
    • Adds model-routing to the engine block:
      engine:
        id: copilot
        copilot-sdk: true
        max-continuations: 6
        model-routing:
          goal: cost
          mode: economy
    • The pr-triage sub-agent stays on claude-haiku-4.5. No allowed-models, blocked models or model-provider are set.
  • mattpocock-skills-reviewer.lock.yml (regenerated by make recompile)
    • Turns routing on, sets the cost goal and economy mode, and pins the AWF router image gh-aw-firewall/router:0.29.0.
    • The agent job now exports the routed model, reasoning effort and routing status. The downstream jobs read these instead of the hardcoded claude-sonnet-5.
    • Threat detection no longer inherits the pin. It now uses GH_AW_MODEL_DETECTION_COPILOT, falling back to GH_AW_DEFAULT_MODEL_COPILOT, then to the detection default.

After merge

  • Run gh aw audit <run> on the first routed runs (Model Routing section) and compare AIC per run with the current ~111.
  • Compare a sample of the first week's reviews with recent ones. If they're thinner, switch to mode: balanced. To roll back, restore model: claude-sonnet-5.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Copilot AI changed the title [WIP] Route Matt Pocock Skills Reviewer using model routing Route Matt Pocock Skills Reviewer with model routing instead of pinning claude-sonnet-5 Nov 1, 2026
Copilot AI requested a review from SivaKesava1 November 1, 2026 17:58
@SivaKesava1
SivaKesava1 marked this pull request as ready for review November 1, 2026 18:01
Copilot AI dynamic review requested due to automatic review settings November 1, 2026 18:01
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · ~8 AIC · ⌖ ~6 AIC · ⊞ 15.2K · ◷

Generated by Ponytail Reviewer for #73836

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: gpt56 max · ~1 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 7.7K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

🔬 Test Quality Sentinel is analyzing test quality on this pull request...

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: no additional actionable issues

The routing change is consistent with the workflow configuration. I did not identify a separate correctness or reliability issue beyond the prompt-injection concerns already raised in the existing review comments.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.7K
Comment /review to run again

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the cost-focused model routing change.

No open issues 🎉

What changed in this PR

Switches Matt Pocock Skills Reviewer to cost-focused model routing, implementing #73823’s plan to reduce spending on advisory PR reviews.

Changes:

  • Replaces the claude-sonnet-5 pin with goal: cost and mode: economy, preserving the pr-triage sub-agent’s model.
  • Regenerates the lock file with a digest-pinned router image and routed model, reasoning effort, and status reporting.
  • Moves threat detection from the inherited model pin to configured defaults.
File Description
.github/​workflows/​mattpocock-skills-reviewer.md Enables economy routing and removes the fixed model.
.github/​workflows/​mattpocock-skills-reviewer.lock.yml Adds compiled routing configuration, downstream reporting, and detection model defaults.

Review effort: Balanced (auto)

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /tdd and /codebase-design to this config-only change. No blocking issues; two non-blocking items below.

📋 Findings

1. Rollback steps in the PR description are incomplete

Restoring model: claude-sonnet-5 alone will not revert the change. engine.model-routing overrides fixed model settings (the schema says routing "ignores fixed model and effort settings"), so a rollback must also remove the model-routing block and run make recompile. Update the "To roll back" line to match.

2. Threat-detection model change is not covered by the post-merge checks

The base lock pins the detection job to claude-sonnet-5 via COPILOT_MODEL. This PR changes it to vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'detection'. Unless one of those repo variables is set, the value is the literal detection, which I could not confirm is a valid model ID. The same 'detection' fallback appears in 185 lock files, so it may be intentional, but threat detection is a security gate. Confirm the resolved model on a routed run, or set GH_AW_MODEL_DETECTION_COPILOT explicitly, and add this to the post-merge list.

Already covered

I agree with the existing prompt-injection findings on the routing prompt and have not duplicated them.

Method

Triage was classified directly from the diff (pr-triage was not invoked). The change is config-only, so the review focused on the frontmatter and the compiled lock-file deltas.

Positive highlights

  • ✅ Keeping pr-triage on Haiku while routing the main agent is a clean split.

@copilot please update the rollback wording and confirm the detection model.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~6 AIC · ⌖ ~1 AIC · ⊞ 5.4K
Comment /matt to run again

id: copilot
copilot-sdk: true
max-continuations: 6
model-routing:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] Rollback is incomplete. Restoring model: claude-sonnet-5 alone is overridden while model-routing is set. To roll back, remove this block and run make recompile.

@copilot please address this.

COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode
COPILOT_GITHUB_TOKEN: ${{ github.token }}
COPILOT_MODEL: claude-sonnet-5
COPILOT_MODEL: ${{ vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'detection' }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] Threat-detection model is no longer pinned to claude-sonnet-5. If neither repo variable is set, it resolves to the literal detection, which may not be a valid model ID. Set GH_AW_MODEL_DETECTION_COPILOT explicitly, or confirm the resolved model on a routed run.

@copilot please address this.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model routing: route Matt Pocock Skills Reviewer instead of pinning claude-sonnet-5

4 participants