Skip to content

Route nine pinned Codex workflows through GitHub Copilot model routing #76320

Closed

Description

@SivaKesava1

Summary

Move these nine Codex workflows from fixed OpenAI API model pins to GitHub Copilot model routing. OpenAI native routing in AWF v0.29.10/v0.29.11 has routing-effort metadata only for the gpt-5.4 and gpt-5.4-mini families; the remaining families used here are not routable through today's OpenAI mapping. The open github/gh-aw-firewall#10030 tracks the missing native-provider metadata. The recent migration pattern is #74866/PR #75461 and #75469/PR #75470, which moved pinned Codex workflows to Copilot routing.

Workflows and cost evidence

The weekly figures are from githubnext/gh-aw-router-training, router-rollout/data/runs/2026-11-02.csv. Agent AIC/run is median agent-phase AIC; All-phase AIC/week is the weekly total including other phases. Modes follow prior batches: code-changing workflows use balanced; analysis/report/triage workflows use economy.

Workflow Current pin Purpose/output on main Runs/week Agent AIC/run (median) All-phase AIC/week Mode Supplied successful main run
daily-arxiv-researcher openai/gpt-5.6-sol Research discussion or noop 7 196.3 1,351 economy 42586270957
daily-storify openai/gpt-6-sol Daily episode-story discussion or noop 7 37.3 331 economy 42582084571
spec-enforcer openai/gpt-5.6-sol Spec-driven Go tests PR or noop 7 101.4 325 balanced 42608675651
gpclean openai/gpt-6.1-sol GPL dependency issue or noop 7 12.7 162 economy 42555618059
issue-arborist openai/gpt-6-sol Related sub-issue links and discussion, or noop 7 15.8 129 economy 42566193701
ab-testing-advisor openai/gpt-6.1-sol Experiment campaign PR or noop 7 16.1 76 balanced 40086220770
daily-fact openai/gpt-5.6-terra Verse comment in the Daily facts discussion #4750 5 10.1 77 economy 42632684024
windows-grower openai/gpt-5.6-terra Windows support issue or noop 6 49.3 55 economy 42592446751
feature-grower openai/gpt-5.6-terra Next feature child issue or noop 5 6.0 45 economy 42202785516

The supplied ab-testing-advisor reference run is from 24 Oct 2026, older than the other baselines. If a newer comparable successful run exists on main when testing, use that instead and record why. The Terra pins are not being migrated because the OpenAI API is retiring them: the 16 Nov 2026 retirement applies to Copilot, not the OpenAI API. This migration still includes them as part of the batch. The separate #76318 covers Copilot-provider Terra pins, not these OpenAI-pinned workflows.

Change

For each of the nine workflow sources:

  • Keep engine.id: codex, remove the top-level fixed model: pin, set engine.model-provider: github, and add engine.model-routing with goal: cost and the table's mode.
  • Add permissions: copilot-requests: write where missing. On current main this is missing from spec-enforcer, gpclean, and daily-fact; retain existing permissions in the other workflows.
  • Do not set allowed-models.
  • Keep prompts, triggers, tools, safe outputs, and evals unchanged. Recompile and include exactly these nine generated lock files; do not hand-edit locks.
  • The nine Markdown sources currently have no explicit OPENAI_API_KEY reference or api.openai.com network entry. The current locks contain compiler-generated Codex/OpenAI plumbing. Inspect the regenerated locks and remove OpenAI credential/host use that exists only for this provider, only when nothing else uses it; state exactly what disappeared. Do not remove unrelated secrets or network access.

Acceptance criteria

A. Compiled configuration

Each of the nine lock files has Copilot/GitHub provider routing with apiProxy.routing.objective set to goal cost and the table's mode; includes the Prepare model-routing conversation step and router image; has no fixed agent model; and does not pass or use an OpenAI API key for agent inference. Verify that inference requests use the Copilot provider. Report any remaining generated OpenAI-related plumbing and why it is not used by the agent, if applicable.

B. PR-branch run evidence

Dispatch one run of each workflow from the PR branch and audit with a development build of main (gh aw audit; the stable release is too old on Windows). Each should succeed. In each usage/aw_session.jsonl, verify exactly one agent-phase firewall.model_routing event with stage: selection and a GPT-family selectedModel; verify the requests went through the Copilot provider. In the PR, report the routing label, selected model and effort, routed run's AIC, and the audited AIC of the latest comparable successful run on main beside its run link. Do not treat the weekly estimates above as per-run baselines.

C. Output parity

Verify each workflow still produces its existing kind of output: discussions, issue/child-issue creation, sub-issue links, PRs, the comment on Daily facts #4750, or a noop with a reason when no action is needed. If a routed run costs clearly more than its comparable main run, document the difference and propose a different mode or allowed-models for that workflow; do not merge it unchanged.

Non-goals

  • Codex workflows already using the GitHub provider, smoke workflows such as smoke-drive on openai/gpt-6-astra, and workflows on the OpenAI key with engines other than Codex.
  • Native OpenAI routing coverage in github/gh-aw-firewall#10030.
  • Changes to prompts, schedules/triggers, tools, safe outputs, evals, AWF/router tables, or the compiler.

Scope

  • Keep the PR to this one fix. Do not change files or behavior unrelated to it.
  • Before fixing a failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR; merge main once main is fixed, and note in the PR which checks are known failures on main.
  • This applies even if a bot comment asks to fix a failing check: fix only failures caused by this PR's changes.
  • Manual test runs on the PR branch can fail for reasons unrelated to this change (GitHub errors, draft PRs, report_incomplete); don't change prompts, safe outputs, or tests in response to them. Fix only failures caused by this PR.

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions