Skip to content

Route Contribution Check through model routing (cost/economy) - #73837

Merged
SivaKesava1 merged 2 commits into
mainfrom
copilot/model-routing-route-contribution-check
Nov 1, 2026
Merged

SivaKesava1 merged 2 commits into
mainfrom
copilot/model-routing-route-contribution-check

Conversation

Copilot AI commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Contribution Check runs every 4 hours (about 42 runs a week) with no model pin, so Copilot auto-selects claude-sonnet-5.5 at about 33 AIC a run. The routed orchestrator only fans out PRs and compiles verdicts. Economy routing should cut this by up to about 870 AIC a week.

Change

  • .github/workflows/contribution-check.md: adds model-routing to the engine block.

    engine:
      id: copilot
      copilot-sdk: true
      agent: contribution-checker
      max-continuations: 25
      model-routing:
        goal: cost
        mode: economy
  • .github/workflows/contribution-check.lock.yml: regenerated with make recompile. The diff:

    • turns routing on (GH_AW_MODEL_ROUTING, experimental.modelRouting in the AWF config, routing.enabled: true in GH_AW_INFO_MODEL_CONFIG)
    • pins ghcr.io/github/gh-aw-firewall/router:0.29.0
    • adds the routing-conversation prep step
    • passes model_effort and model_routing_status from the agent job to downstream jobs

Unchanged

  • Only the orchestrator is routed, so the saving is an upper bound. Per-PR checks stay on claude-haiku-4.5, the prompt's model: claude-haiku-4.5 instruction is kept, and report-formatter and comment-dispatcher stay on gpt-6-luna.
  • No allowed-models, blocked models or model-provider.
  • make recompile also rewrote .github/skills/agentic-workflows/SKILL.md, dropping its .github/aw/formal-methods.md entry. That's outside this change, so I reverted it.

After merge

  • Check the first routed runs with gh aw audit <run> (Model Routing section) and compare AIC per run with about 33.
  • Confirm it still dispatches every PR and posts the same kind of report.
  • If the reports get worse, switch to mode: balanced. To roll back, remove model-routing.

Copilot AI linked an issue Nov 1, 2026 that may be closed by this pull request
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Copilot AI changed the title [WIP] Route Contribution Check with model routing Route Contribution Check through model routing (cost/economy) Nov 1, 2026
Copilot AI requested a review from SivaKesava1 November 1, 2026 17:59
@SivaKesava1
SivaKesava1 marked this pull request as ready for review November 1, 2026 18:00
Copilot AI dynamic review requested due to automatic review settings November 1, 2026 18:01
Comment thread .github/workflows/contribution-check.md Resolved
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: opus55 low · ~12 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 7.7K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · sonnet50 · ~16 AIC · ⌖ ~16 AIC · ◇ ~8 AIC · ⊞ 12.7K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · ~7 AIC · ⌖ ~7 AIC · ⊞ 15.2K · ◷

Generated by Ponytail Reviewer for #73837

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

No additional actionable issues found in the reviewed source hunk. The generated lock-file changes were not present in the pre-fetched patch, so this review is limited to the workflow frontmatter addition; existing automated comments already cover the routing prompt-injection concern.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~2 AIC · ⌖ ~1 AIC · ⊞ 1.7K
Comment /review to run again

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the orchestrator’s cost-focused economy routing change.

No open issues 🎉

What changed in this PR

Enables cost-focused economy model routing for the Contribution Check orchestrator, addressing #73824 while leaving sub-agent model selections unchanged.

Changes:

  • Adds goal: cost and mode: economy to the workflow’s engine configuration.
  • Regenerates the lock file with a pinned router image, routing prompt preparation, and runtime configuration.
  • Exposes the selected model, reasoning effort, and routing status to downstream reporting.
File Description
.github/​workflows/​contribution-check.md Enables cost/economy routing for the orchestrator.
.github/​workflows/​contribution-check.lock.yml Adds generated routing infrastructure and reporting metadata.

Review effort: Balanced (auto)

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /codebase-design — this is a scoped, low-risk config change (3-line frontmatter addition + regenerated lock file), so no blocking issues. Approving with context on the pre-existing security-bot flag.

📋 Key Themes & Highlights

Key Themes

  • GHAS prompt-injection flag is a known, accepted model-routing limitation, not new risk: the docs (docs/src/content/docs/reference/model-routing.md, "How routing works") explicitly state the classifier "reads the workflow's prompt... and treats that text as data, not instructions." This is the same design already shipped on 20+ routed workflows in this repo (avenger, duplicate-code-detector, design-decision-gate, several smoke tests). Contribution Check is not a special case needing a different mitigation before merge.
  • The routed surface here is narrower than the automated comment implies: per the PR description, only the orchestrator is routed — it reads pr-filter-results.json (PR numbers/counts) and contributing-guidelines-truncated.md, not raw PR diffs/bodies. The contribution-checker subagent that actually processes untrusted PR patches stays pinned to claude-haiku-4.5 and is not routed. That mitigation is real, but it isn't spelled out next to the GHAS finding, so a reviewer skimming the diff could assume untrusted PR content flows straight into the classifier. Worth a one-line reply on that thread pointing to the "Unchanged" section of the PR description.
  • Reversion of the unrelated SKILL.md recompile diff is confirmed correct — pr-meta.json shows only the two intended files changed (contribution-check.md +3/-0, contribution-check.lock.yml +38/-13), matching the PR description's claim.

Positive Highlights

  • ✅ Clear, auditable rollback path stated in the PR description (gh aw audit, compare AIC, revert model-routing block if reports regress).
  • ✅ Consistent with the existing repo-wide pattern for goal: cost / mode: economy on low-risk, repetitive triage-style workflows.
  • ✅ No fixed-model conflicts introduced — model: claude-haiku-4.5 for the per-PR subagent dispatch prompt is correctly left untouched, since only the orchestrator engine is routed.

Note: sandboxed git remote access failed with a certificate error, so I could not independently re-diff the lock file against the prior committed revision; this review relies on the pre-fetched diff/PR description for the lock-file summary.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · ~115 AIC · ⌖ ~20 AIC · ⊞ 10.7K
Comment /matt to run again

@SivaKesava1
SivaKesava1 merged commit f4bfda4 into main Nov 1, 2026
83 checks passed
@SivaKesava1
SivaKesava1 deleted the copilot/model-routing-route-contribution-check branch November 1, 2026 18:43
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model routing: route Contribution Check

4 participants