Skip to content

feat(router)!: update the bundled routing tables to schema version 2 - #11198

Merged
rabeckett merged 6 commits into
mainfrom
router-tables-2026-11-06
Nov 6, 2026
Merged

rabeckett merged 6 commits into
mainfrom
router-tables-2026-11-06

Conversation

@rabeckett

@rabeckett rabeckett commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #11197

Replaces the six routing tables in containers/router/routing/ with newly trained tables in routing-table schema version 2, and bumps the router version to 0.3.0 because the table format and the HTTP contract change.

Schema version 2

  • Adds schemas/routing-table.v2.schema.json, which is version 1 with only the version, the label pattern and the cell count changed. Version 1 stays frozen and pinned. The loader accepts only schema_version 2.
  • Replaces the explain task type with analyze and review in contract.go, openapi.yaml and the api-proxy's classifier-output schema in routing-contract.js.
  • Replaces ClassificationContract with a new classification contract. It defines the version 2 task types, revises the guidance for scope, complexity and mode, and adds guidance for long agentic-workflow prompts. SystemPrompt, the output shape and the mode values are unchanged. ClassificationLabels is now analyze/local/trivial.
  • The heuristic fallback labels its explanation cues analyze instead of explain. The cues are unchanged.
  • TestBundledProfilesRouteAndClassify no longer requires the classifier order to equal the balanced table's ranking of the classifier cell, because the tables now carry their own classifier order. Loading still requires every table to share it.

Fixture changes

  • testdata/parity/labels.json: the 69 expectations that were explain are now analyze. No other expectation changes.
  • testdata/parity/classification.json, testdata/routing-contract/classify-response.json and the api-proxy's copy in fixtures/routing-contract/cases.json: the new contract text. The conversation data is unchanged.
  • testdata/routing-contract/errors.json: the recorded detail lists the new task types.

Smoke workflow

test-model-routing.yml offered models that these tables no longer rank. The routed Copilot steps now use claude-haiku-5.5 instead of claude-haiku-4.5, and the native OpenAI job uses openai/gpt-5.6-luna instead of openai/gpt-5.4. Every model in the new catalogue has reasoning efforts, so the two-candidate step sends the routed effort for the selected model and low for the model it deviates to. The cross-endpoint step doesn't route and is unchanged.

Models added

  • GPT-6.1 Terra (Copilot and OpenAI)

Models removed

  • Claude Haiku 4.5 (Copilot and Anthropic)
  • Claude Opus 4.8 (Copilot and Anthropic)
  • Claude Opus 5 (Copilot and Anthropic)
  • GPT-5.4 (Copilot and OpenAI)
  • GPT-5.4 mini (Copilot and OpenAI)
  • GPT-5.5 (Copilot and OpenAI)
  • Grok 4.5 (Copilot)

Testing

  • gofmt -l ., go vet ./... and go test ./... in containers/router.
  • The container tests against a locally built linux/arm64 image (GH_AW_ROUTER_TEST_IMAGE=... go test -run '^TestContainer' .).
  • npx jest in containers/api-proxy. On Windows, 23 tests fail on main as well (symlinks and POSIX file permissions); no other test fails.
  • Served the bundled tables locally and checked /healthz, /capabilities, /classify and /route for all six profiles. A live classifier call on the /classify plan returned version 2 labels, which the api-proxy's validation accepted and the router routed.

Replace the six routing tables in containers/router/routing/ with newly
trained tables, and adopt routing-table schema version 2, which replaces
the explain task type with analyze and review (240 label cells instead
of 210).

- Publish schemas/routing-table.v2.schema.json; v1 stays frozen. The
  loader now accepts only schema_version 2.
- Classifier labels, the classification contract, the heuristic labels
  and the api-proxy's classifier-output schema use the new task types.
- Bump the router to 0.3.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rabeckett rabeckett added the model-routing Model routing selection, compatibility, and diagnostics label Nov 6, 2026
Copilot AI dynamic review requested due to automatic review settings November 6, 2026 16:11
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Routing smoke tests still reference removed models and fail with 422 no_route, while classification rule ordering can bypass the balanced fallback for unintelligible requests.

2 open issues
What changed in this PR

Migrates AWF’s router to routing-table schema v2 and release 0.3.0, keeping the router and API proxy aligned on the new task labels.

Changes:

  • Adds the frozen v2 schema and updates table loading for 240 label cells.
  • Replaces explain with analyze and review, and revises classifier guidance.
  • Synchronizes version metadata, documentation, tests and contract fixtures.
File Description
containers/​router/​testdata/​routing-contract/​errors.json Updates accepted task types in error expectations.
containers/​router/​testdata/​routing-contract/​discovery.json Expects router version 0.3.0.
containers/​router/​testdata/​routing-contract/​classify-response.json Refreshes the classifier prompt fixture.
containers/​router/​testdata/​parity/​labels.json Changes explanation expectations to analyze.
containers/​router/​testdata/​parity/​classification.json Synchronizes classification-contract text.
containers/​router/​table.go Accepts schema version 2 instead of 1.
containers/​router/​table_test.go Updates version checks and cell-count expectations.
containers/​router/​serve_test.go Updates logged cell and model counts.
containers/​router/​schemas/​routing-table.v2.schema.json Defines the v2 table format and task labels.
containers/​router/​schema_test.go Pins the v2 schema hash and checks runtime alignment.
containers/​router/​route_test.go Uses new labels and table-provided classifier ordering.
containers/​router/​README.md Documents v2 support and migration incompatibility.
containers/​router/​openapi.yaml Updates API version and task-type enumeration.
containers/​router/​main.go Bumps the router release version.
containers/​router/​heuristics.go Maps existing explanation cues to analyze.
containers/​router/​heuristics_test.go Updates heuristic label expectations.
containers/​router/​Dockerfile Bumps container version metadata.
containers/​router/​contract.go Defines analyze and review task types.
containers/​router/​cli_test.go Updates validation-summary counts.
containers/​router/​classify.go Revises classifier guidance and classifier-request labels.
containers/​api-proxy/​routing-controller.test.js Uses analyze in classifier-response coverage.
containers/​api-proxy/​routing-contract.test.js Tests new labels and rejection of explain.
containers/​api-proxy/​routing-contract.js Aligns classifier-output validation with v2 labels.
containers/​api-proxy/​fixtures/​routing-contract/​cases.json Synchronizes the proxy’s classifier-prompt fixture.

Review effort: Max (auto)

Comment thread containers/router/cli_test.go Resolved
Comment thread containers/router/classify.go Resolved
@lpcox

lpcox commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

@copilot merge main, resolve conflicts, and address all review feedback

Copilot AI and others added 2 commits November 6, 2026 16:49
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
The new routing tables drop github-copilot/claude-haiku-4.5 and
openai/gpt-5.4, so the routed smoke steps that offered only them would
get no_route. Use claude-haiku-5.5 for the routed Copilot steps and
openai/gpt-5.6-luna for the native OpenAI job. Every model in the new
catalogue has reasoning efforts, so the two-candidate step now sends the
routed effort for either model, and low for the model it deviates to.
The cross-endpoint step doesn't route and keeps claude-haiku-4.5.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@rabeckett Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@rabeckett

Copy link
Copy Markdown
Collaborator Author

@copilot The merge from main was all this needed. Both review threads are already handled, so please don't change any code for them:

  • The smoke workflow fix is in c01b710.
  • The classification contract in containers/router/classify.go stays exactly as it is. See the reply on that thread.

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>

Copilot AI commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

@copilot merge main, resolve conflicts, and address all review feedback

No further changes are needed: 1bd6441 already merges main at 691410a (the current origin/main), and the branch head c01b710 includes that merge. The smoke-workflow update is in c01b710; Router Tests and Build Verification pass on that head.

…outer-tables-2026-11-06

# Conflicts:
#	.github/workflows/test-model-routing.yml

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 93.76% 93.76% ➡️ +0.00%
Statements 92.30% 92.30% ➡️ +0.00%
Functions 93.08% 93.08% ➡️ +0.00%
Branches 86.58% 86.58% ➡️ +0.00%
📁 Per-file Coverage Changes (1 files)
File Lines (Before → After) Statements (Before → After)
src/enclave/host-executor-server.ts 91.2% → 91.2% (+0.00%) 90.1% → 89.8% (-0.29%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Move "unknown = no intelligible request." back after the mode rules,
where it is on every other axis, and drop the test that pinned it first.
The contract is used exactly as it was evaluated, and in that evaluation
unintelligible requests got unknown for mode on every call, so the
reordering changed a tested prompt without re-testing it. The review
thread on classify.go has the details.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rabeckett

Copy link
Copy Markdown
Collaborator Author

8115315 restores the classification contract to the evaluated text and removes the test that pinned unknown first. A Copilot session that started before my reply on the classify.go thread had applied that suggestion. The thread explains why we're keeping the contract as evaluated. The rest of that session's work stays: the merge from main and the workflow conflict resolution, which kept c01b710's version.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✅ Security Guard completed successfully!

Security Guard

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Security Guard · sonnet55 · ~21 AIC · ⊞ 8.2K · ◷

Generated by Security Guard for #11198

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🪪 Smoke Copilot BYOK AOAI (Entra) is testing Azure OpenAI BYOK (Entra / GitHub OIDC) mode on this pull request...

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Claude passed

Generated by Smoke Claude for #11198

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Smoke test: FAIL

  • chore: refresh canonical model sources
  • docs: sync schemas and specs with source changes
  • GitHub merged PRs ✅
  • PR detail lookup ✅
  • Playwright title check ❌
  • Temp file + cat ✅
  • Build (npm ci && npm run build) ✅

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude Results

  • API status: ✅ PASS
  • GitHub check: ✅ PASS
  • File status: ✅ PASS

Overall result: PASS

Generated by Smoke Claude for #11198 · claude · haiku45 · ~35 AIC · ⊞ 6.1K · ◷
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Cloud Hypervisor + Copilot smoke test

  1. Merged PR lookup: PASS
  2. github.com connectivity: PASS (HTTP 200)
  3. File write/read: PASS
  4. example.com blocked: PASS (HTTP 000)

Overall: PASS

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

EGRESS_RESULT allow=pass deny=pass

  • ✅ Allowed domain (api.github.com → 200)
  • ✅ Blocked domain (example.com blocked)

Overall: PASS — @rabeckett

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia ✅ 1/1 passed ✅ PASS
Bun hono ✅ 1/1 passed ✅ PASS
C++ fmt ✅ N/A ✅ PASS
C++ json ✅ N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world ✅ N/A ✅ PASS
.NET json-parse ✅ N/A ✅ PASS
Go color ✅ pass ✅ PASS
Go env ✅ pass ✅ PASS
Go uuid ✅ pass ✅ PASS
Java gson ❌ 0 run ❌ FAIL
Java caffeine ❌ 0 run ❌ FAIL
Node.js clsx ✅ pass ✅ PASS
Node.js execa ✅ pass ✅ PASS
Node.js p-limit ✅ pass ✅ PASS
Rust fd ✅ 1/1 passed ✅ PASS
Rust zoxide ✅ 1/1 passed ✅ PASS

Overall: 7/8 ecosystems passed — FAIL

Java failure details:

  • ~/.m2 is root-owned, so the default local repository could not be created. I used -Dmaven.repo.local=/tmp/gh-aw/agent/m2 as a workaround.
  • With that workaround, Maven failed to download plugins from Maven Central: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty. The JVM truststore is empty or unusable in this environment.
  • Since Java failed, the build-test label was not added.

.NET note: the first attempt crashed because /run/user/1001 was missing. After setting XDG_RUNTIME_DIR, it passed.

Generated by Build Test Suite for #11198 · copilot · auto · ~26 AIC · ⊞ 5.7K · ◷
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode ✅

Test Result
MCP Connectivity ✅
HTTP Code (github.com) ✅ 200
File Write/Read ✅
BYOK Inference ✅

Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com

Status: PASS

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

chore: refresh canonical model sources
docs: sync schemas and specs with source changes
GitHub MCP Testing: ✅
GitHub.com Connectivity: ✅
File Write/Read Test: ✅
BYOK Inference Test: ✅
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)
Overall status: PASS
@rabeckett

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot: PRs: "feat(router)!: update the bundled routing tables to schema version 2"

  • GitHub MCP ✅
  • GitHub.com (200) ✅
  • File write/read ✅

Overall: PASS — @rabeckett

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Chroot Version Comparison

Runtime Host Version Chroot Version Match?
Python Python 3.12.16 Python 3.12.16 ✅ YES
Node.js v24.22.2 v2.98.0 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Result: Not all tests passed (Node.js mismatch), so the smoke-chroot label was not added.

Tested by Smoke Chroot
Add label ready-for-aw to run again

This branch had an error being deployed

1 failed deployment
aoai-model — 8115315f Deployed Nov 6, 2026 by rabeckett via conclusion #2270
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Router: update the bundled routing tables to schema version 2

4 participants