Repository navigation
Replace agent-based evals with batched Decisions requests - #73918
Conversation
Add a reusable predicate/choice/score client and independent BinEval execution. Preserve isolated credentials, accounting, and failure artifacts; regenerate compiled workflows. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove AWF installation, sandbox/proxy configuration, and log collection from evals. Capture token usage and cache-aware AIC/USD estimates directly from the response, including malformed answer batches, for summaries and daily accounting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Verify predicate, choice, and score batching against the live gpt-6-luna Decisions API and retain provider-specific wire model selection. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Malformed JSON errors can expose private provider response bodies in logs, and an unresolved model alias causes make lint to fail.
No open issues 🎉
What changed in this PR
Replaces gh-aw’s agent-based BinEval execution with one batched Decisions request, separating evaluations from the main agent runtime.
Changes:
- Adds an SDK-free Decisions client with named-answer validation and explicit failures.
- Updates evals configuration, provider authentication, and usage-accounting outputs.
- Refreshes model aliases, schema, documentation, tests, and generated workflow locks.
| File | Description |
|---|---|
pkg/workflow/token_broker_test.go |
Checks brokered credentials reach the Decisions step. |
pkg/workflow/threat_detection_isolation_test.go |
Separates evals from sandbox and sidecar expectations. |
pkg/workflow/release_integrity_validation_test.go |
Removes evals engine-installation expectations. |
pkg/workflow/model_aliases_test.go |
Checks the new evals model default. |
pkg/workflow/evals_job.go |
Updates provider permissions and accounting outputs. |
pkg/workflow/evals_job_test.go |
Checks the new accounting step wiring. |
pkg/workflow/evals_config.go |
Rejects question-level models and limits batch size. |
pkg/workflow/evals_config_test.go |
Covers model restrictions and question limits. |
pkg/workflow/engine_fallback_models_test.go |
Checks engine-independent evals fallback. |
pkg/workflow/engine_cooldown_test.go |
Ensures evals omit agent cooldown controls. |
pkg/workflow/data/model_aliases.json |
Changes the built-in evals alias. |
pkg/workflow/compiler_admission_test.go |
Removes evals admission-budget expectations. |
pkg/parser/schemas/main_workflow_schema.json |
Defines single-model batches of 1–400 questions. |
docs/src/content/docs/reference/frontmatter-full.md |
Updates the generated evals reference. |
docs/src/content/docs/experimental/experiments.md |
Explains batching, credentials, and failure semantics. |
actions/setup/README.md |
Documents the reusable Decisions client. |
actions/setup/js/render_evals_summary.test.cjs |
Updates expected judge identity to Decisions. |
actions/setup/js/model_aliases.json |
Synchronizes the runtime evals alias. |
actions/setup/js/decisions_client.test.cjs |
Adds batching and validation tests. |
actions/setup/js/decisions_client.cjs |
Implements batched requests and response validation. |
.github/workflows/smoke-work-queue-worker.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-work-queue-dispatcher.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-pi-auto.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-github-codex.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-codex-bare-auto.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-codex-auto.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-claude-sub-agents.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-claude-copilot.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-claude-copilot-auto.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-claude-auto.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-azure-pi.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/smoke-azure-codex.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/notion-issue-summary.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/example-permissions-warning.lock.yml |
Refreshes embedded model aliases. |
.github/workflows/codex-github-remote-mcp-test.lock.yml |
Refreshes embedded model aliases. |
.github/aw/evals.md |
Documents models, credentials, limits, and accounting. |
.changeset/evals-decisions.md |
Records the backend and compatibility changes. |
Review effort: Max (auto)
Regenerate workflow locks with the merged compiler to retain Decisions evals alongside upstream AWF and model-routing updates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve Decisions-based evals while incorporating CI contract repairs and stable safe-output updates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Default to copilot/gpt-6-luna-decisions and use openai/gpt-6-luna explicitly for OpenAI. Pass model IDs through without a Copilot-specific rename. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
✅ Smoke OTEL completed successfully!
|
|
✅ Smoke Copilot Auto completed successfully!
|
|
🥧 Pi initializing... Smoke Pi begins on this pull request... |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟
|
|
🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "update.googleapis.com"See Network Configuration for more information.
|
|
🚀 Smoke Gemini MISSION COMPLETE! Gemini has spoken. ✨
|
|
✅ All tools validated successfully! Agent Container Smoke Test confirms agent container is ready.
|
|
🌿 Evals shed their weight One call, no sandbox, Inspired by the PR replacing agent-based BinEval with a single direct Decisions call.
|
Codex smoke: FAIL✅ GitHub MCP; ✅ Serena; ❌ Playwright (policy blocked); ❌ native web fetch (tool error).
7 passed, 2 failed/blocked, 1 skipped.
|
Comment MemoryPeek at saved memory (smoke-codex)Note This comment is managed by comment memory.Expand the saved memory block to view or edit the persistent context for this thread.
|
Smoke Test Results (Gemini)
|
|
Smoke (partial): 1✅ 2✅ 3❌ 4❌ 5✅ 6✅ 7✅ 8✅ 9✅ 10✅ 11✅ 12❌ 13❌ 14✅ 15✅ 16❌ — Overall FAIL. Author @pelikhan, no assignees.
|
|
Me see this. Good talk. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "update.googleapis.com"See Network Configuration for more information.
|
|
Overall: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "update.googleapis.com"See Network Configuration for more information.
|
Comment MemoryPeek at saved memory (smoke_haiku)Note This comment is managed by comment memory.Expand the saved memory block to view or edit the persistent context for this thread. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "update.googleapis.com"See Network Configuration for more information.
|
There was a problem hiding this comment.
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
update.googleapis.com
To allow these domains, add them to the network.allowed list in your workflow frontmatter:
network:
allowed:
- defaults
- "update.googleapis.com"See Network Configuration for more information.
📰 BREAKING: Report filed by Smoke Copilot - AOAI (Entra) · copilot · o40mini · ~47 AIC · ⌖ ~3 AIC · ⊞ 22.7K
Comment /smoke-copilot-aoai-entra to run again
Add label smoke to run again
|
🎉 This pull request is included in a new release. Release: |
Summary
BinEval evaluations should not need a full agent runtime or sandbox. This replaces the previous agent-based implementation with a direct OpenAI-compatible Decisions call that batches all questions and shared evidence into one POST.
Compatibility
Evals default to
copilot/gpt-6-luna-decisions, subject toGH_AW_DEFAULT_EVALS_MODEL; theevalsalias resolves without a proxy. Unqualified model IDs use Copilot; anopenai/prefix selects OpenAI credentials. Copilot supports a configured token,copilot-requests: write, or a brokered inference credential.Configure the actual provider model ID:
copilot/gpt-6-luna-decisionsfor Copilot oropenai/gpt-6-lunafor OpenAI. Both use/v1/decisions; the provider prefix selects credentials and is removed from the request model, with no provider-specific name mapping.A batch supports 1-400 questions. Per-question model overrides are rejected because the entire batch shares one model and request. This is a full replacement, not a fallback to agent execution. AWF credit limits and sandbox target overrides no longer apply to evals. Missing response usage or model pricing is reported as unavailable, not zero; reported costs are catalog-derived estimates, not provider billing totals.
Validation
Actual-model-ID follow-up: compiler/alias tests, JavaScript type checking and 117 focused JavaScript tests passed. All 353 workflows were regenerated, and
make agent-report-progresspassed. Regression coverage verifies both provider-qualified IDs and that Copilot no longer silently renamesgpt-6-luna.Formatting, builds, targeted compiler tests, and all 353 workflow compilations passed.
make agent-report-progresspassed impacted tests; after encountering a sharedgolangci-lintlock, its remaining validation passed viamake agent-report-progress-no-testwith an isolated temporary directory.go test ./pkg/workflow -run 'Evals|TokenBrokerSupportsEvals' -count=1passed.JavaScript type checking and 147 targeted tests passed across the Decisions client, evals runtime, summary rendering, and daily AIC component accounting. Coverage includes a real local HTTP batching test, direct authentication, cache/reasoning cost calculations, zero and unavailable costs, malformed answers, stale accounting cleanup, and provider-specific alias resolution.
Live Copilot test: one POST batched predicate, choice, and score questions using synthetic basket/apple evidence. Results were probability
1, choicethree, and score2; all assertions passed. The response reported modelgpt-6-luna, 395 input tokens, zero output tokens, and estimated cost of 0.00395 AIC ($0.0000395).The initial live call returned HTTP 404. Inspecting the authenticated provider catalog identified the required
/v1/decisionsendpoint andgpt-6-luna-decisionswire ID; both are configured explicitly with regression coverage, without model-name remapping.No workflow dispatches were performed.