Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# ADR-68081: Document Known Legacy Routing Discrepancies as Per-Case Allowances in Model-Routing Goldens

**Date**: 2026-10-12
**Status**: Draft
**Proposed deciders**: PR author (SivaKesava1), gh-aw maintainers

---

### Context

`pkg/cli/model_routing_golden_test.go` replays recorded run artifacts through the audit CLI in two variants — "downloaded" (full artifact set, including `usage/aw_session.jsonl` and `agent/aw_info.json`) and "legacy" (the reduced artifact set older runs produce) — and asserts that the *raw routing facts* (status, selected model/effort, endpoints, routed/outcome counts, deviations) agree between the two. The four new fixtures added for issue #68076 cover routed **failure** paths (budget exhaustion with steering, provider unavailable) and routed paths where some artifacts carry no `model_routing` block at all. For two of those fixtures the two variants genuinely disagree: `parse_token_usage.cjs` relabels a budget failure as `harness_selection_mismatch` only when the session file is present, and `applyAwInfoModelRouting` skips `agent/aw_info.json`, so the legacy variant loses `effective_endpoint`/`selected_endpoint`. These are real product discrepancies, not fixture defects, so the existing strict cross-variant equality assertion cannot hold as written.

### Decision

We will record each known downloaded-vs-legacy discrepancy as an explicit, per-case `legacyRoutingAllowance map[string]string` on `modelRoutingGoldenCase`, keyed by routing-identity field name with the reason as the value. During cross-variant comparison the test asserts that each allowed field **still differs**, fails with a "remove the allowance and its README entry" message if it no longer does, then zeroes that field before the remaining identity comparison stays strict. The projection is also widened with `failure`, `degraded_classification` and `degraded_reason`, and the failure code joins the cross-variant identity so routed-failure fixtures assert on the failure itself rather than only on status. The primary driver is that tolerating a discrepancy silently (or loosening the assertion globally) would let an unrelated regression slip through; a self-expiring, documented allowance keeps the guard tight and forces the fixture and README to be updated when the underlying bug is fixed.

### Alternatives Considered

These alternatives are inferred from the diff and PR description; they do not record maintainer approval or rejection.

#### Alternative 1: Fix the underlying discrepancies in `parse_token_usage.cjs` / `applyAwInfoModelRouting` in this PR

Making the legacy path read `agent/aw_info.json` and keeping the proxy failure status consistent would make the fixtures pass under the existing strict assertion with no new test machinery. It was not chosen because this PR's scope is capturing evidence (golden fixtures regenerated from real runs) ahead of behavioural change; fixing the audit behaviour in the same change would mean the goldens never record the current, shipped behaviour, and the fix would land without a regression baseline.

#### Alternative 2: Drop the divergent fields from the cross-variant identity for all cases

Removing `EffectiveEndpoint`, `SelectedEndpoint` and `Status` from `routingIdentity` is a one-line change that makes all cases pass. It was rejected because those fields are exactly the routing facts the golden test exists to protect; weakening them repo-wide to accommodate two fixtures would disable the check for the other cases and leave no record of why.

#### Alternative 3: Record the divergence only in `expected.legacy.json` and the README

The per-variant expected files already capture the differing values, so the discrepancy is visible in review. This was insufficient on its own: the cross-variant assertion would still fail, and a prose-only note never expires — once the product bug is fixed nothing would prompt removal of the stale documentation.

### Consequences

#### Positive
- Routed failure paths and `model_routing`-less artifact layouts now have golden coverage, including the failure code, degraded-classification flag and degraded reason.
- Each tolerated discrepancy is machine-checked and self-expiring: fixing the product bug turns the test red until the allowance and its README row are deleted, so documentation cannot drift from behaviour.
- The remaining routing-identity comparison stays strict for every other field and every other case; the escape hatch is scoped to named fields on named cases.

#### Negative
- The allowance mechanism uses `reflect.FieldByName` and zeroes fields on a local copy, which is more machinery (and more failure modes, e.g. a typo'd field name) than a plain `reflect.DeepEqual`.
- Known product bugs are now encoded as "expected" in test data; a reader who skips the allowance reason may mistake the legacy output (missing endpoints, divergent status) for intended behaviour.
- Fixture volume grows substantially (≈3k added lines of recorded artifacts and expected outputs), increasing repository size and the cost of regenerating goldens.

#### Neutral
- `make fmt` (prettier) reformats the new fixtures' `aw_info.json`; the change is whitespace-only but required by `fmt-check-json`, so recorded artifacts are not byte-identical to the downloaded originals.
- `codex-routed-provider-unavailable` projects identically in both variants, so it carries no `expected.legacy.json` and no `legacyDifference`.
- The README gains rows for the four fixtures plus a "known discrepancies" section, making it the human-facing counterpart to the in-code allowances.

---

*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*
63 changes: 57 additions & 6 deletions pkg/cli/model_routing_golden_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,29 @@ var modelRoutingGoldenCases = []modelRoutingGoldenCase{
unifiedClassifierAIC: new(0.03908),
legacyClassifierAIC: new(0.03908),
},
{
name: "copilot-budget-exhausted-steering",
legacyDifference: "the session workflow.info routing status (rejected) and steering notices are read only from usage/aw_session.jsonl; the legacy variant reports the proxy status (failed) and lists no notices",
unifiedClassifierAIC: new(0.6796),
legacyClassifierAIC: new(0.6796),
},
{
name: "codex-routed-provider-unavailable",
unifiedClassifierAIC: new(0.03344),
legacyClassifierAIC: new(0.03344),
},
{
name: "copilot-sdk-claude-subagent-completions",
legacyDifference: "routing endpoints come only from agent/aw_info.json and the session workflow.info; the root aw_info.json has no model_routing, so the legacy variant loses effective_endpoint and selected_endpoint and reconciles per-agent credits against an unknown endpoint",
unifiedClassifierAIC: new(0.06362),
legacyClassifierAIC: new(0.06362),
},
{
name: "copilot-cli-builtin-subagents",
legacyDifference: "routing endpoints come only from agent/aw_info.json and the session workflow.info; the root aw_info.json has no model_routing, so the legacy variant loses effective_endpoint and selected_endpoint and reconciles per-agent credits against an unknown endpoint",
unifiedClassifierAIC: new(0.6116),
legacyClassifierAIC: new(0.6116),
},
}

type modelRoutingGoldenProjection struct {
Expand All @@ -65,6 +88,7 @@ type modelRoutingGoldenProjection struct {

type modelRoutingGoldenRouting struct {
Status string `json:"status"`
Failure *ModelRoutingFailure `json:"failure,omitempty"`
SelectedModel string `json:"selected_model,omitempty"`
SelectedEffort string `json:"selected_effort,omitempty"`
Endpoint string `json:"endpoint,omitempty"`
Expand Down Expand Up @@ -144,7 +168,7 @@ func TestModelRoutingGoldenAudit(t *testing.T) {

downloaded := results[testCase.name]["downloaded"]
legacy := results[testCase.name]["legacy"]
assertModelRoutingGoldenVariantAgreement(t, downloaded.projection, legacy.projection)
assertModelRoutingGoldenVariantAgreement(t, testCase, downloaded.projection, legacy.projection)
downloadedPath := filepath.Join("testdata", "model_routing_golden", testCase.name, "expected.json")
legacyPath := filepath.Join("testdata", "model_routing_golden", testCase.name, "expected.legacy.json")
if update {
Expand Down Expand Up @@ -255,7 +279,8 @@ func projectModelRoutingGoldenAudit(routing *ModelRoutingSummary, usage *TokenUs
projection := modelRoutingGoldenProjection{}
if routing != nil {
projected := &modelRoutingGoldenRouting{
Status: routing.Status, SelectedModel: routing.SelectedModel, SelectedEffort: routing.SelectedEffort,
Status: routing.Status,
SelectedModel: routing.SelectedModel, SelectedEffort: routing.SelectedEffort,
Endpoint: routing.Endpoint, EffectiveEndpoint: routing.EffectiveEndpoint, SelectedEndpoint: routing.SelectedEndpoint,
RoutedCounts: routing.RoutedCounts, OutcomeCounts: routing.OutcomeCounts,
Deviations: slices.Clone(routing.Deviations), ClassifierCost: roundedModelRoutingCost(routing.ClassifierCost),
Expand All @@ -265,6 +290,10 @@ func projectModelRoutingGoldenAudit(routing *ModelRoutingSummary, usage *TokenUs
SubagentCosts: slices.Clone(routing.SubagentCosts),
EndpointOnlyDeviationNormalized: routing.EndpointOnlyDeviationNormalized,
}
if routing.Failure != nil {
failure := *routing.Failure
projected.Failure = &failure
}
slices.SortFunc(projected.Deviations, func(left, right ModelRoutingDeviation) int {
return strings.Compare(
strings.Join([]string{left.RequestedModel, left.RequestedEffort, left.SelectedModel, left.SelectedEffort, left.Deviation}, "\x00"),
Expand Down Expand Up @@ -436,14 +465,15 @@ func assertModelRoutingGoldenFile(t *testing.T, path string, actual []byte) {
}
}

func assertModelRoutingGoldenVariantAgreement(t *testing.T, downloaded, legacy modelRoutingGoldenProjection) {
func assertModelRoutingGoldenVariantAgreement(t *testing.T, testCase modelRoutingGoldenCase, downloaded, legacy modelRoutingGoldenProjection) {
t.Helper()
if (downloaded.ModelRouting == nil) != (legacy.ModelRouting == nil) {
t.Fatalf("legacy fallback changed whether routing was available: downloaded=%+v legacy=%+v", downloaded.ModelRouting, legacy.ModelRouting)
}
if downloaded.ModelRouting != nil {
type routingIdentity struct {
Status string
FailureCode string
SelectedModel string
SelectedEffort string
Endpoint string
Expand All @@ -454,14 +484,35 @@ func assertModelRoutingGoldenVariantAgreement(t *testing.T, downloaded, legacy m
Deviations []ModelRoutingDeviation
}
identity := func(routing *modelRoutingGoldenRouting) routingIdentity {
failureCode := ""
if routing.Failure != nil {
failureCode = routing.Failure.Code
}
return routingIdentity{
Status: routing.Status, SelectedModel: routing.SelectedModel, SelectedEffort: routing.SelectedEffort,
Status: routing.Status, FailureCode: failureCode, SelectedModel: routing.SelectedModel, SelectedEffort: routing.SelectedEffort,
Endpoint: routing.Endpoint, EffectiveEndpoint: routing.EffectiveEndpoint, SelectedEndpoint: routing.SelectedEndpoint,
RoutedCounts: routing.RoutedCounts, OutcomeCounts: routing.OutcomeCounts, Deviations: routing.Deviations,
}
}
if !reflect.DeepEqual(identity(downloaded.ModelRouting), identity(legacy.ModelRouting)) {
t.Fatalf("raw routing facts should agree across variants: downloaded=%+v legacy=%+v", identity(downloaded.ModelRouting), identity(legacy.ModelRouting))
downloadedIdentity, legacyIdentity := identity(downloaded.ModelRouting), identity(legacy.ModelRouting)
switch testCase.name {
case "copilot-budget-exhausted-steering":
if downloadedIdentity.Status == legacyIdentity.Status {
t.Fatalf("documented status discrepancy for %s no longer occurs; remove its README entry", testCase.name)
}
downloadedIdentity.Status = legacyIdentity.Status
case "copilot-sdk-claude-subagent-completions", "copilot-cli-builtin-subagents":
if downloadedIdentity.EffectiveEndpoint == legacyIdentity.EffectiveEndpoint {
t.Fatalf("documented effective-endpoint discrepancy for %s no longer occurs; remove its README entry", testCase.name)
}
if downloadedIdentity.SelectedEndpoint == legacyIdentity.SelectedEndpoint {
t.Fatalf("documented selected-endpoint discrepancy for %s no longer occurs; remove its README entry", testCase.name)
}
downloadedIdentity.EffectiveEndpoint = legacyIdentity.EffectiveEndpoint
downloadedIdentity.SelectedEndpoint = legacyIdentity.SelectedEndpoint
}
if !reflect.DeepEqual(downloadedIdentity, legacyIdentity) {
t.Fatalf("raw routing facts should agree across variants: downloaded=%+v legacy=%+v", downloadedIdentity, legacyIdentity)
}
}
downloadedUsage, legacyUsage := downloaded.FirewallTokenUsage, legacy.FirewallTokenUsage
Expand Down
45 changes: 40 additions & 5 deletions pkg/cli/testdata/model_routing_golden/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,44 @@ listed workflow runs; they retain only the inputs needed by those analyses.
| --- | ---: | --- |
| `copilot-routed-gpt-responses` | `37886511291` | Copilot routing through `/responses`, classifier credits, and legacy raw-token fallback |
| `claude-awf-selected-messages` | `37971713032` | Claude selected through `/v1/messages`, including the legacy endpoint normalization |
| `claude-awf-steering-notices` | `37971713032` | Synthetic copy of `claude-awf-selected-messages` with hand-added AWF `steering` fields (agent `ai_credit` 80% and `timeout` 90%, detection `token` 95%); the legacy variant lists no notices |
| `claude-awf-steering-notices` | `37971713032` | Edge case only: synthetic copy of `claude-awf-selected-messages` with hand-added AWF `steering` fields (agent `ai_credit` 80% and `timeout` 90%, detection `token` 95%); the legacy variant lists no notices. `copilot-budget-exhausted-steering` covers real `ai_credit` notices; this case remains because no real run has a `timeout` notice or a detection-phase `token` notice |
| `pi-claude-two-subagents` | `37971726317` | Pi session with two Claude subagents and tool-execution correlation |
| `copilot-subagent-failed-and-alias` | `37971738805` | Failed and aliased Copilot subagent routing, cost merging, and dated served model IDs |
| `legacy-no-session-routing` | `37505153056` | Legacy run without unified routing events; routing and classifier costs come from proxy logs |
| `copilot-budget-exhausted-steering` | `38174216426` | Copilot CLI routed to gpt-5.6-sol with `max-ai-credits: 30`: real AWF `ai_credit` steering notices at 80/90/95%, a proxy `ai_credits_limit_exceeded` failure, three requests rejected with HTTP 403, and 30.1812 AIC including a 0.6796 AIC classifier |
| `codex-routed-provider-unavailable` | `37728959715` | Codex routed to gpt-5.6-luna (high) on `/responses`: one aborted request, a proxy `provider_unavailable` failure, and an upstream 400; predates `workflow.info` routing, so the status comes from the proxy log alone |
| `copilot-sdk-claude-subagent-completions` | `38181851405` | Copilot SDK mode with the main session on gpt-5.6-luna (xhigh); sub-agent `file-summarizer` on claude-haiku-4.5 through `/chat/completions` and `quick-checker` on gpt-5.6-luna |
| `copilot-cli-builtin-subagents` | `38164386960` | Copilot CLI routed to gpt-5.6-sol (medium) with the built-in `explore`, `research` and `task` sub-agents on gpt-5.6-luna and `code-review` and `general-purpose` on gpt-5.6-sol in the unified session; audit names their rows by display name |

Audit does not read the conclusion job's failure issue, `agent-stdio.log`,
`awf-config.json`, or `agent_execution_exit_code.txt`, so the goldens cannot pin
them. They are recorded here for the routed-failure cases:

| Case | Harness exit code | Failure issue title |
| --- | ---: | --- |
| `copilot-budget-exhausted-steering` | 78 | `[aw] Task exceeded max AI credits` |
| `codex-routed-provider-unavailable` | 78 | `[aw] Task reported incomplete result` (does not mention routing) |

## Known discrepancies pinned by the goldens

These are current audit behavior, not intended behavior. Each is allowed by the
case's `legacyRoutingAllowance` in `model_routing_golden_test.go`, and the test
fails once the discrepancy no longer occurs. When fixing one, remove the
allowance and this entry and review the golden diff.

- `copilot-budget-exhausted-steering` reports `status: rejected` with the
session (`usage/aw_session.jsonl`) and `status: failed` without it; both
variants report `failure.code: ai_credits_limit_exceeded`. The `rejected`
status comes from `aw_info.json` and the session `workflow.info`, where
`resolveModelRoutingOutcome` in `actions/setup/js/parse_token_usage.cjs`
compared the final budget failure record, which has no `wire_model`, with the
advisory `awf-routing-outcome.json` and relabeled the outcome as
`harness_selection_mismatch`. That code never appears in the audit report.
- `copilot-sdk-claude-subagent-completions` and `copilot-cli-builtin-subagents`
lose `effective_endpoint` and `selected_endpoint` in the legacy variant. In
these runs only `agent/aw_info.json` carries `model_routing`, and
`applyAwInfoModelRouting` in `pkg/cli/model_routing.go` skips that file. The
older fixtures still carry `model_routing` in the root `aw_info.json`.

## Verify and refresh expected output

Expand All @@ -25,8 +59,8 @@ Review every changed expected file in the diff; regeneration is not a substitute
for checking that the changed routing, credit, token, warning, and subagent
values are correct.

The five existing fixtures predate the capture tool, so the first recapture of
an existing case may rewrite its fixture files without changing its analysis.
The first six fixtures in the table predate the capture tool, so the first
recapture of one of them may rewrite its fixture files without changing its analysis.
For example, a recapture uses `example-org/routing-fixture` instead of the
committed fixtures' `example-org/routing-sandbox`, retains a different set of
fields, redacts `task_complexity`, and omits `usage/aw_info.json` for
Expand Down Expand Up @@ -58,8 +92,9 @@ input under `testdata/model_routing_golden_capture/` exercises this path.

To add a case, capture it, add its description and expected costs to
`modelRoutingGoldenCases` in `model_routing_golden_test.go`, then run
`make update-model-routing-golden`. Review both the sanitized fixture and the
generated golden diff. Capture redacts content, local paths, conversation
`make update-model-routing-golden`. Run `make fmt` as well: `make lint` checks
the fixture JSON with prettier, which only changes whitespace. Review both the
sanitized fixture and the generated golden diff. Capture redacts content, local paths, conversation
hashes, signatures, encrypted fields, email addresses, and repository identities;
it intentionally retains AWF credit values, model IDs, endpoints, efforts,
token counts, timestamps, and request IDs because the analysis depends on them.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{"type":"session.init","thread_id":"01a119d6-9f80-7521-af04-b29be1cd7079","data":{"sourceEngine":"codex","sessionId":"01a119d6-9f80-7521-af04-b29be1cd7079"}}
{"type":"session.result","message":"[redacted]","data":{"sourceType":"error","errors":["{\"error\":{\"message\":\"Invalid 'input[8].id': 'ctco_01a119d6-ab2a-73c0-be36-4329d931c4e8'. Expected an ID that begins with 'fc'.\",\"code\":\"invalid_request_body\"}}\n"]}}
{"type":"session.result","error":{"message":"[redacted]"},"data":{"sourceType":"turn.failed","status":"failed","errors":[{"message":"[redacted]"}]}}
{"type":"agent.execution","data":{"categories":[],"errorCodes":["invalid_request_body"],"errorTypes":[],"exitCode":78}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"input_tokens": 9544,
"output_tokens": 298,
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"ambient_context": 802,
"ai_credits": 0.22664,
"primary_model": "gpt-5.6-luna"
}
Loading
Loading