Skip to content

docs: lead model-choice guidance with default routing and model: router - #76985

Merged
SivaKesava1 merged 7 commits into
mainfrom
copilot/docs-lead-model-choice-guidance
Nov 12, 2026
Merged

SivaKesava1 merged 7 commits into
mainfrom
copilot/docs-lead-model-choice-guidance

Conversation

Copilot AI commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Several docs and authoring-guide passages still taught model choice as if every workflow must pin a model and model: router didn't exist. This PR leads with routing when a task has no specific model requirement, and, at @rabeckett's request (comment), rewrites the Model Routing page around model: router. Explicit model selection stays as the alternative. Docs only.

Changes

  • reference/model-routing.md: rewritten around router values, about half the previous length (4,743 → 2,266 words):
    • Get started has one model: router example, explains that the router picks economy, balanced or robust per task (balanced when it can't classify), and notes that model: auto is the engine's own selector, not gh-aw routing.
    • Setting the level yourself covers router/economy, router/balanced and router/robust, merged with the old mode table.
    • Default routing for workflows without a model is a short paragraph (model variables override it at run time; engine.model-routing: false or a pinned model opts out), with the full eligibility list collapsed. The #default-routing-for-workflows-without-a-model anchor is kept.
    • Limiting which models it can pick, Engines and providers, Sub-agents and Check which model was used use router values in every example.
    • Advanced: engine model routing holds what still needs the block: the equivalence table, goal: cost-speed, allowed-models shortlists, a policy combined with a shortlist, shared routing settings, and the router value restrictions (moved out of Get started).
    • Stale facts removed: the native-metadata note no longer names specific models, and the Claude reasoning-effort limitation is gone (AWF v0.29.12 ranks only the efforts each model's provider metadata lists).
  • reference/cost-management.md
    • Choose a Cheaper Model starts with eligible default routing (a workflow that omits model already behaves like model: router) and model: router/economy, which sets the lowest success target, not a spending cap. Explicit selection follows: an alias such as small can resolve to a different model over time, and an exact ID is how to require one specific model.
    • The nightly-triage example uses model: router/economy instead of gpt-4.1-mini.
    • AI Credits with Dynamic Model Selectors keeps the model: auto example and says auto is the engine or provider's selector, not gh-aw's router.
    • Roll Out Shared Defaults drops the default_model_* entries from the sample (a sample model default would opt eligible workflows out of routing) and says those variables are explicit overrides that opt out of default routing and must not hold router or router/* values.
    • No gpt-4.1-mini or claude-haiku-4-5 examples remain.
  • reference/engines/codex.md: "no default model" applies only to the native OpenAI provider, where model: router routes on OpenAI; a separate paragraph says eligible Codex workflows on model-provider: github route by default.
  • reference/frontmatter.md: the engine.model-routing paragraph is one sentence pointing to the advanced section.
  • .github/aw/syntax-agentic.md: the top-level model: entry lists the four router values, links syntax-engine.md, and says: unless the task needs a specific model, use model: router or a router/ level; eligible workflows with no model already route the same way (with a link to the eligibility rules).

Acceptance cases and validation

  • docs: lead model-choice guidance with default routing and model: router #76983 A–D: covered by the cost-management, Codex, authoring-guide and model-routing changes above.
  • Ryan's "done when": every example that doesn't need a shortlist or goal: cost-speed uses a router value; engine.model-routing appears only in the advanced section apart from the false opt-out; Get started has one example and no compiler-rejection list; anchors linked from other pages (#default-routing-for-workflows-without-a-model, #native-anthropic-and-openai-routing, #advanced-engine-model-routing) resolve.
  • E: the docs build check passes in CI. Routing examples compile with gh-aw built from main. make test-guidance fails test_debug_playbook_agent_consumption_budget (debug guide word count 2870 > 2800) the same way on main, so it's left alone.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Copilot AI changed the title [WIP] Update documentation for model-choice guidance with routing docs: lead model-choice guidance with default routing and model: router Nov 11, 2026
Copilot AI requested a review from SivaKesava1 November 11, 2026 22:41
@github-actions

Copy link
Copy Markdown
Contributor

Category: docs | Risk: low | Score: 29/100

  • Impact 8 + urgency 7 + quality 14
  • Recommended action: auto_merge
  • Reason: docs-only change with passing CI and no active blockers; it is a narrow improvement to model-choice guidance and does not affect runtime behavior.

Generated by 🔧 PR Triage Agent · copilot · mai11 · ~4 AIC · ⌖ ~1 AIC · ⊞ 15K · ◷

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
@SivaKesava1
SivaKesava1 marked this pull request as ready for review November 12, 2026 05:39
@SivaKesava1

Copy link
Copy Markdown
Collaborator

Verified against #76983:

  • A (cost-management.md): the cheaper-model section now leads with eligible default routing and model: router/economy, and links Model Routing. Aliases are described as resolving at run time, not as fixed pins. The gpt-4.1-mini and claude-haiku-4-5 examples, including the shared-defaults sample and the nightly-triage example, are gone. The section on dynamic model selectors separates model: auto from model: router, and the default_model_* note says these variables can't hold router values.
  • B (codex.md): "no default model" is now limited to the native OpenAI provider and links the default-routing section; the #native-anthropic-and-openai-routing anchor exists.
  • C (syntax-agentic.md): the top-level model: entry lists all four router values, links syntax-engine.md, and tells the authoring agent to omit model or use a router value unless a specific model is needed. Eligibility stays qualified.
  • D (model-routing.md): a note near Get started separates model: auto, model: router and routing's mode: auto, and keeps the balanced fallback.
  • E: the docs build check passes. make test-guidance's test_debug_playbook_agent_consumption_budget fails the same way on main, so it's not caused by this PR.

Only the four files change. Round 1 (a6786ca) fixed the sentence that introduced the model: small example as an exact model ID.

@SivaKesava1
SivaKesava1 requested review from pelikhan and a dynamic review from Copilot and removed request for Copilot November 12, 2026 05:39
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt61 medium · ~3 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 181 · ◷

Generated by Ponytail Reviewer for #76985

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

PR Specification Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Specification Reviewer · gpt56 · ~2 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

Generated by PR Specification Reviewer for #76985

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /grill-with-docs and /codebase-design to the routing docs. The PR is docs-only, so the review checked factual claims against the implementation rather than test coverage. Four inline comments follow, all clarity or consistency items. No blocking issues.

📋 Verification and key themes

Verified

  • ✅ Internal links and anchors in the four edited pages resolve.
  • ✅ The small alias resolves on both the Copilot and Claude harnesses.
  • ✅ The default_model_claude and default_model_copilot keys are valid in pkg/cli/env_command.go.
  • ✅ The Codex no-default-model wording matches the compiler fallback, which is an empty string.
  • ✅ The nightly-triage example compiles with gh aw compile --dry-run built from this checkout, and it does not need copilot-requests: write.

Key themes

  • Router-value placement: the nightly example YAML shape is inconsistent with the rest of the page.
  • Duplicated eligibility rules: syntax-agentic.md restates model-routing.md, so the two can drift.
  • Readability: the Codex sentence and the default_model_claude example each carry more than one rule.

Positive highlights

  • ✅ Cross-linking to model-routing.md for default routing is consistent across the pages.
  • ✅ The distinction between model: auto and model: router is now stated explicitly, which removes a real ambiguity.
  • ✅ The default_model_* note correctly warns that a router value there stops the run before the agent starts.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~8 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

Comment thread .github/aw/syntax-agentic.md Resolved
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (docs/src/content/docs/reference/cost-management.md:512): address unresolved feedback - Review comment
  3. Review (docs/src/content/docs/reference/cost-management.md:447): address unresolved feedback - Review comment
  4. Review (.github/aw/syntax-agentic.md:25): address unresolved feedback - Review comment
  5. Review (docs/src/content/docs/reference/engines/codex.md:12): address unresolved feedback - Review comment

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "a6786ca15c459fe5e0916e5a173a8edf3608d8e3",
  "kind": "requested",
  "schema_version": "1",
  "state": "bf258de9e580c4d765163df09ab00afc90c143f9b97af231cc1259ccf8e4f071",
  "work": [
    "0eb5d7bece38aa29fe8d95f673eca03eb90aedfe12ac3bb915da342aa64e57e6",
    "1532e6a8e05e23c3eba143eaa78d4f8afad6896cc4c53adcd25dbba2a68b2819",
    "294361954328758b00a8a3e7c8c4597acd573db17f62d110000413b2907cb944",
    "2bac51e4011afa6070e6c12a465861345d7345f29e772adbcc023efccc5c46aa"
  ],
  "workflow": "pr-sous-chef"
}

Generated by 👨🍳 PR Sous Chef · pi · routed: haiku55 xhigh · ~2 AIC · ⌖ ~2 AIC · ⊞ 6.9K · ◷
Comment /souschef to run again

Copilot AI and others added 2 commits November 12, 2026 06:32
…el-choice-guidance

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI dynamic review requested due to automatic review settings November 12, 2026 06:36

Copilot AI commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:...

Merged latest main in caad819 and addressed the four threads in 7c28f96. Guidance tests and the scoped final gate pass; full recompilation stalled on an npm registry lookup.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The shared-defaults documentation still removes the explicit model alias instead of updating it to the required non-stale example, leaving the documented override behavior incomplete.

1 open issue
What changed in this PR

This documentation PR updates model-selection guidance to prefer eligible gh-aw default routing or explicit router values when a workflow does not require a specific model, while clarifying provider-specific behavior and preserving explicit model selection.

Changes:

  • Updates cost-management guidance for default routing, economy routing, dynamic selectors, and shared model defaults.
  • Clarifies native OpenAI versus GitHub Copilot routing behavior for Codex.
  • Adds router values and routing recommendations to the authoring syntax reference.
  • Distinguishes provider-level model: auto from gh-aw model: router.
  • The shared-defaults sample removes default_model_claude instead of replacing its stale value with the requested small alias.
File Description
docs/​src/​content/​docs/​reference/​model-routing.md Distinguishes provider or engine model: auto selection from gh-aw model: router behavior.
docs/​src/​content/​docs/​reference/​engines/​codex.md Clarifies native OpenAI model behavior and GitHub Copilot default routing.
docs/​src/​content/​docs/​reference/​cost-management.md Updates cost guidance, routing examples, dynamic selectors, and shared-default behavior.
.github/​aw/​syntax-agentic.md Documents router values and advises authors when to omit or explicitly set model.

Review effort: Lite (auto)

@github-actions

Copy link
Copy Markdown
Contributor

Category: docs
Risk: low
Score: 32/100 (impact 12 + urgency 6 + quality 14)
Recommended action: defer
Reason: documentation-only change with low operational impact; it is lower priority than the automation and security-oriented PRs in the current queue.

Generated by 🔧 PR Triage Agent · copilot · mai11 · ~6 AIC · ⌖ ~2 AIC · ⊞ 15.9K · ◷

@rabeckett

Copy link
Copy Markdown
Collaborator

@copilot Please widen this PR to also simplify the Model Routing page. #76983 said to leave the rest of model-routing.md alone, but now that model: router exists (#76617), the page should be rewritten around it, and that rewrite is in scope here.

Why

Routing a workflow now takes one line. With model: router, the router reads each task, decides whether it needs the economy, balanced or robust level (balanced when it can't classify the task), and picks the cheapest model and effort likely to succeed at that level. router/economy, router/balanced and router/robust set the level yourself. The page shows this at the top, but the rest still teaches engine.model-routing: 10 of its 12 frontmatter examples use a block with goal and mode, the page is about 4,700 words, and Get started follows the router table with a paragraph of compiler rejections and then a second way to route.

Router values cover nearly every example on the page. With gh-aw built from main (838595e), each of these compiles to a routed workflow with the expected mode, provider and candidates:

  • model: router/robust with models.blocked: ["claude-opus-*"]
  • model: router with models.allowed: ["gpt-*"]
  • engine: claude with model: router/economy (native Anthropic, claude-* candidates)
  • engine: codex with model: router (native OpenAI, gpt-* candidates)
  • Claude with model-provider: github and model: router/balanced
  • Copilot with copilot-sdk: true, model: router and a Claude sub-agent

Only a shortlist in allowed-models and goal: cost-speed still need the block.

model-routing.md

Aim for roughly half the current length, in this order:

  1. Why use model routing, trimmed.
  2. Get started: one model: router example and a plain explanation that the router chooses the level for each task. Keep this PR's note on model: auto, but word it without goal and mode.
  3. Setting the level yourself: the three router/ values and when to use each. Merge in the mode table from "Choose a goal and mode" instead of repeating it.
  4. Workflows without a model: a short paragraph saying eligible workflows already behave like model: router, that GH_AW_MODEL_AGENT_<ENGINE> and GH_AW_DEFAULT_MODEL_<ENGINE> override that at run time, and that engine.model-routing: false or a pinned model opts out. Keep the full eligibility list lower down or collapsed. Keep the #default-routing-for-workflows-without-a-model anchor, since several pages link to it.
  5. Limiting which models it can pick: models.allowed and models.blocked with router values, using the GPT-only and no-Opus examples above.
  6. Engines and providers: the engine table, router-value examples for Claude and Codex, and a short list of native routing requirements.
  7. Sub-agents: keep the Copilot CLI-mode family constraint and its table, with the example rewritten to use model: router.
  8. Checking which model was used: unchanged.
  9. Advanced, engine.model-routing: the equivalence table, goal: cost-speed, shortlists in allowed-models (exact names and aliases), a policy combined with a shortlist, and shared routing blocks. Summarize alias expansion and policy narrowing in a few sentences and link the model alias specification for the full rules.
  10. Known limitations and troubleshooting, trimmed to what still applies. Rows that are only about allowed-models move to the advanced section.

The list of rejected router values and placements (unknown router/... values, ?effort= parameters, router values in model lists, aliases, sub-agents, threat detection or evals, combining with engine.model-routing or engine.fallback-models, expressions that resolve to a router value at run time) should move out of Get started into a short reference subsection or troubleshooting. The compiler errors already explain each case.

Stale facts on the page

Checked against AWF v0.29.12, which gh-aw now pins by default:

  • Native routing lists gpt-5.4-mini among the models skipped for lack of routing metadata. containers/api-proxy/model-api-mapping.json in v0.29.12 has routing metadata for gpt-5.4-mini and gpt-5.4, while the Haiku models still have none. Correct the list, or describe the rule without naming models so it doesn't go stale.
  • "Reasoning effort with Claude": AWF now resolves each model's supported efforts from provider metadata (model-reasoning.js, used by routing-catalogue.js), and the native Claude entries list only low to max. Check whether the router can still choose none or minimal for Claude in v0.29.12, and remove or narrow the limitation if it can't.

Other pages

  • cost-management.md: replace "already routed with goal: cost and mode: auto" with "already behaves like model: router". Readers of that page shouldn't need to learn goal and mode.
  • engines/codex.md: instead of "configure model routing", say "set model: router", which routes natively on OpenAI.
  • .github/aw/syntax-agentic.md: the new model: sentence is hard to follow. Simplify it to: unless the task needs a specific model, use model: router or a router/ level, and eligible workflows with no model already route the same way. Keep the eligibility link.
  • frontmatter.md: the engine.model-routing paragraph says it applies "when they use GitHub Copilot inference", but Claude and Codex also route natively on Anthropic and OpenAI. Cut it to one sentence that points to the advanced section.

Done when

  • Every example that doesn't need a shortlist or goal: cost-speed uses a router value. Apart from engine.model-routing: false for opting out, engine.model-routing appears only in the advanced section.
  • Get started has one example, no list of compiler rejections, and no mention of engine.model-routing.
  • Nothing true is lost. Anything taken out of the main flow is in the advanced section, in troubleshooting, or on a linked page.
  • The #default-routing-for-workflows-without-a-model anchor still resolves, and the docs build passes with no broken links.
  • Still docs only: no compiler, schema, runtime, workflow or lock file changes, and the same rule as docs: lead model-choice guidance with default routing and model: router #76983 for checks that also fail on main.

Copilot AI and others added 2 commits November 12, 2026 17:38
…el-choice-guidance

Co-authored-by: rabeckett <5445735+rabeckett@users.noreply.github.com>
Co-authored-by: rabeckett <5445735+rabeckett@users.noreply.github.com>

Copilot AI commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

@copilot Please widen this PR to also simplify the Model Routing page. #76983 said to leave the rest of model-routing.md alone, but now that model: router exists (#76617), the page should be rewri...

Updated in 1149308. Guidance tests, the scoped gate, and routing example compilation pass. The docs build remains blocked by DNS failures for the configured npm registry and approved 1ES feed.

Copilot AI requested a review from rabeckett November 12, 2026 17:43
@SivaKesava1
SivaKesava1 merged commit 741ff0c into main Nov 12, 2026
46 checks passed
@SivaKesava1
SivaKesava1 deleted the copilot/docs-lead-model-choice-guidance branch November 12, 2026 18:02
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: lead model-choice guidance with default routing and model: router

5 participants