Skip to content

Replace agent-based evals with batched Decisions requests - #73918

Merged
pelikhan merged 6 commits into
mainfrom
pelikhan/evals-decisions-batching
Nov 2, 2026
Merged

pelikhan merged 6 commits into
mainfrom
pelikhan/evals-decisions-batching

Conversation

@pelikhan

@pelikhan pelikhan commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

BinEval evaluations should not need a full agent runtime or sandbox. This replaces the previous agent-based implementation with a direct OpenAI-compatible Decisions call that batches all questions and shared evidence into one POST.

  • Add a reusable, SDK-free JavaScript client supporting predicate, choice, and score questions, with named-answer validation and explicit transport/protocol failures.
  • Keep evals as named YES/NO/UNKNOWN choices, independent of the main agent's engine, model, tools, or harness.
  • Run a trusted JavaScript step against the controlled provider endpoint directly. Evals do not install AWF, an agent CLI, or Node, and do not start Docker, a sandbox, or an API proxy.
  • Collect response token usage, including cache and reasoning tokens, and compute AIC/USD cost estimates using the existing pricing catalog. Preserve accounting artifacts, job outputs, and daily accounting. Capture billed usage before answer validation so malformed answer batches do not lose their cost.
  • Preserve experiment assignments, redaction, result artifacts, and not-evaluated failure markers. Simplify compilation, update documentation and schema, and regenerate workflow locks. Most changed files are generated locks, which are substantially smaller.

Compatibility

Evals default to copilot/gpt-6-luna-decisions, subject to GH_AW_DEFAULT_EVALS_MODEL; the evals alias resolves without a proxy. Unqualified model IDs use Copilot; an openai/ prefix selects OpenAI credentials. Copilot supports a configured token, copilot-requests: write, or a brokered inference credential.

Configure the actual provider model ID: copilot/gpt-6-luna-decisions for Copilot or openai/gpt-6-luna for OpenAI. Both use /v1/decisions; the provider prefix selects credentials and is removed from the request model, with no provider-specific name mapping.

A batch supports 1-400 questions. Per-question model overrides are rejected because the entire batch shares one model and request. This is a full replacement, not a fallback to agent execution. AWF credit limits and sandbox target overrides no longer apply to evals. Missing response usage or model pricing is reported as unavailable, not zero; reported costs are catalog-derived estimates, not provider billing totals.

Validation

  • Actual-model-ID follow-up: compiler/alias tests, JavaScript type checking and 117 focused JavaScript tests passed. All 353 workflows were regenerated, and make agent-report-progress passed. Regression coverage verifies both provider-qualified IDs and that Copilot no longer silently renames gpt-6-luna.

  • Formatting, builds, targeted compiler tests, and all 353 workflow compilations passed. make agent-report-progress passed impacted tests; after encountering a shared golangci-lint lock, its remaining validation passed via make agent-report-progress-no-test with an isolated temporary directory.

  • go test ./pkg/workflow -run 'Evals|TokenBrokerSupportsEvals' -count=1 passed.

  • JavaScript type checking and 147 targeted tests passed across the Decisions client, evals runtime, summary rendering, and daily AIC component accounting. Coverage includes a real local HTTP batching test, direct authentication, cache/reasoning cost calculations, zero and unavailable costs, malformed answers, stale accounting cleanup, and provider-specific alias resolution.

  • Live Copilot test: one POST batched predicate, choice, and score questions using synthetic basket/apple evidence. Results were probability 1, choice three, and score 2; all assertions passed. The response reported model gpt-6-luna, 395 input tokens, zero output tokens, and estimated cost of 0.00395 AIC ($0.0000395).

  • The initial live call returned HTTP 404. Inspecting the authenticated provider catalog identified the required /v1/decisions endpoint and gpt-6-luna-decisions wire ID; both are configured explicitly with regression coverage, without model-name remapping.

No workflow dispatches were performed.

Add a reusable predicate/choice/score client and independent BinEval execution. Preserve isolated credentials, accounting, and failure artifacts; regenerate compiled workflows.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread actions/setup/js/decisions_client.cjs Resolved
Remove AWF installation, sandbox/proxy configuration, and log collection from evals. Capture token usage and cache-aware AIC/USD estimates directly from the response, including malformed answer batches, for summaries and daily accounting.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review November 2, 2026 01:12
Copilot AI dynamic review requested due to automatic review settings November 2, 2026 01:12
Comment thread actions/setup/js/decisions_client.cjs Resolved
Verify predicate, choice, and score batching against the live gpt-6-luna Decisions API and retain provider-specific wire model selection.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Malformed JSON errors can expose private provider response bodies in logs, and an unresolved model alias causes make lint to fail.

No open issues 🎉

What changed in this PR

Replaces gh-aw’s agent-based BinEval execution with one batched Decisions request, separating evaluations from the main agent runtime.

Changes:

  • Adds an SDK-free Decisions client with named-answer validation and explicit failures.
  • Updates evals configuration, provider authentication, and usage-accounting outputs.
  • Refreshes model aliases, schema, documentation, tests, and generated workflow locks.
File Description
pkg/​workflow/​token_broker_test.go Checks brokered credentials reach the Decisions step.
pkg/​workflow/​threat_detection_isolation_test.go Separates evals from sandbox and sidecar expectations.
pkg/​workflow/​release_integrity_validation_test.go Removes evals engine-installation expectations.
pkg/​workflow/​model_aliases_test.go Checks the new evals model default.
pkg/​workflow/​evals_job.go Updates provider permissions and accounting outputs.
pkg/​workflow/​evals_job_test.go Checks the new accounting step wiring.
pkg/​workflow/​evals_config.go Rejects question-level models and limits batch size.
pkg/​workflow/​evals_config_test.go Covers model restrictions and question limits.
pkg/​workflow/​engine_fallback_models_test.go Checks engine-independent evals fallback.
pkg/​workflow/​engine_cooldown_test.go Ensures evals omit agent cooldown controls.
pkg/​workflow/​data/​model_aliases.json Changes the built-in evals alias.
pkg/​workflow/​compiler_admission_test.go Removes evals admission-budget expectations.
pkg/​parser/​schemas/​main_workflow_schema.json Defines single-model batches of 1–400 questions.
docs/​src/​content/​docs/​reference/​frontmatter-full.md Updates the generated evals reference.
docs/​src/​content/​docs/​experimental/​experiments.md Explains batching, credentials, and failure semantics.
actions/​setup/​README.md Documents the reusable Decisions client.
actions/​setup/​js/​render_evals_summary.test.cjs Updates expected judge identity to Decisions.
actions/​setup/​js/​model_aliases.json Synchronizes the runtime evals alias.
actions/​setup/​js/​decisions_client.test.cjs Adds batching and validation tests.
actions/​setup/​js/​decisions_client.cjs Implements batched requests and response validation.
.github/​workflows/​smoke-work-queue-worker.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-work-queue-dispatcher.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-pi-auto.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-github-codex.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-codex-bare-auto.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-codex-auto.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-claude-sub-agents.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-claude-copilot.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-claude-copilot-auto.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-claude-auto.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-azure-pi.lock.yml Refreshes embedded model aliases.
.github/​workflows/​smoke-azure-codex.lock.yml Refreshes embedded model aliases.
.github/​workflows/​notion-issue-summary.lock.yml Refreshes embedded model aliases.
.github/​workflows/​example-permissions-warning.lock.yml Refreshes embedded model aliases.
.github/​workflows/​codex-github-remote-mcp-test.lock.yml Refreshes embedded model aliases.
.github/​aw/​evals.md Documents models, credentials, limits, and accounting.
.changeset/​evals-decisions.md Records the backend and compatibility changes.

Review effort: Max (auto)

Regenerate workflow locks with the merged compiler to retain Decisions evals alongside upstream AWF and model-routing updates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
pelikhan and others added 2 commits November 1, 2026 18:13
Preserve Decisions-based evals while incorporating CI contract repairs and stable safe-output updates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Default to copilot/gpt-6-luna-decisions and use openai/gpt-6-luna explicitly for OpenAI. Pass model IDs through without a Copilot-specific rename.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan pelikhan added the smoke label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke OTEL completed successfully!

Generated by Smoke OTEL

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Copilot Auto completed successfully!

Generated by Smoke Copilot Auto for #73918

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

🥧 Pi initializing... Smoke Pi begins on this pull request...

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • update.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "update.googleapis.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Smoke Gemini MISSION COMPLETE! Gemini has spoken. ✨

✨ Smoke Gemini — Powered by Gemini

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

✅ All tools validated successfully! Agent Container Smoke Test confirms agent container is ready.

🔧 Tool validation by Agent Container Smoke Test

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

🌿 Evals shed their weight

One call, no sandbox,
answers batch into one—
light as morning dew.

Inspired by the PR replacing agent-based BinEval with a single direct Decisions call.

Generated by 🌸 Smoke Copilot Auto for #73918 · copilot · auto · ~11 AIC · ⌖ ~7 AIC · ⊞ 7.1K · ◷
Add label smoke to run again

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Codex smoke: FAIL

✅ GitHub MCP; ✅ Serena; ❌ Playwright (policy blocked); ❌ native web fetch (tool error).
✅ File writing; ✅ Bash; ✅ Build; ✅ Comment memory; ✅ Cache memory; ⏭️ Issue field (discovery unavailable).
❌ Additional title validation blocked by loopback policy.
PR titles:

  • fix: repair CI contracts and regenerate workflow locks
  • Mark four GitHub safe outputs as stable

7 passed, 2 failed/blocked, 1 skipped.
§41377713618

🔮 The oracle has spoken through Smoke Codex · codex · ~16 AIC · ⌖ ~1 AIC · ⊞ 21.7K · ◷
Comment /smoke-codex to run again

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (smoke-codex)
Quiet branches sway
Small green lights wake in the dusk
Fresh code greets the dawn

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔮 The oracle has spoken through Smoke Codex · codex · ~16 AIC · ⌖ ~1 AIC · ⊞ 21.7K · ◷
Comment /smoke-codex to run again

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Test Results (Gemini)

  • Playwright title validation: ❌
    1. GitHub MCP Test: ✅
    1. Web Fetch Test: ✅
    1. File Write Test: ✅
    1. Bash Test: ❌
    1. Build Test: ❌
  • Overall Status: FAIL

✨ Smoke Gemini — Powered by Gemini · gemini · agent · ~85 AIC · ⌖ ~1 AIC · ⊞ 12.4K · ◷
Comment /smoke-gemini to run again

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Smoke (partial): 1✅ 2✅ 3❌ 4❌ 5✅ 6✅ 7✅ 8✅ 9✅ 10✅ 11✅ 12❌ 13❌ 14✅ 15✅ 16❌ — Overall FAIL. Author @pelikhan, no assignees.

📰 BREAKING: Report filed by Smoke Copilot · copilot · auto · ~33 AIC · ⌖ ~1 AIC · ⊞ 7.6K · ◷
Comment /smoke-copilot to run again
Add label smoke to run again

@pelikhan
pelikhan merged commit 0261414 into main Nov 2, 2026
491 of 492 checks passed
@pelikhan
pelikhan deleted the pelikhan/evals-decisions-batching branch November 2, 2026 02:39
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Me see this. Good talk.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • update.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "update.googleapis.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot - AOAI (Entra) · copilot · o40mini · ~47 AIC · ⌖ ~3 AIC · ⊞ 22.7K · ◷
Comment /smoke-copilot-aoai-entra to run again
Add label smoke to run again

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

@pelikhan Smoke tests:

  1. gh PR list ✅; mcpscripts ❌; serena ❌; playwright ✅
  2. curl ✅; file test ✅; az ✅; discussion ✅
  3. build ✅; artifact ❌; create discussion ✅; haiku_printer ✅
  4. review ✅; memory ✅; summarizer ❌; checkrun ✅

Overall: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • update.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "update.googleapis.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot - AOAI (Entra) · copilot · o40mini · ~47 AIC · ⌖ ~3 AIC · ⊞ 22.7K · ◷
Comment /smoke-copilot-aoai-entra to run again
Add label smoke to run again

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (smoke_haiku)
Morning mist drifts
Code emerges in light
Success breaks the dawn

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • update.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "update.googleapis.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot - AOAI (Entra) · copilot · o40mini · ~47 AIC · ⌖ ~3 AIC · ⊞ 22.7K · ◷
Comment /smoke-copilot-aoai-entra to run again
Add label smoke to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • update.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "update.googleapis.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot - AOAI (Entra) · copilot · o40mini · ~47 AIC · ⌖ ~3 AIC · ⊞ 22.7K
Comment /smoke-copilot-aoai-entra to run again
Add label smoke to run again

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.0

This branch was previously deployed

1 inactive deployment
aoai-model — af8e10ed Deployed Nov 2, 2026 by github-actions[bot] via conclusion #257
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants