Skip to content

Add local agent plugin benchmark (v1) - #973

Merged
Nikola Metulev (nmetulev) merged 4 commits into
mainfrom
nmetulev-local-agent-benchmark
Oct 6, 2026
Merged

Nikola Metulev (nmetulev) merged 4 commits into
mainfrom
nmetulev-local-agent-benchmark

Conversation

@nmetulev

@nmetulev Nikola Metulev (nmetulev) commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Description

Adds benchmarks/agents/, a small local benchmark. It shows which skills GitHub Copilot CLI loads for realistic developer requests, and how many tokens that costs, with four plugin setups: none, winapp (this repo's plugins/winapp), winui (published microsoft/win-dev-skills v0.7.1), and both. We'll use it to:

  • record today's behavior,
  • confirm the WinUI plugin routes the same after it moves into this repo,
  • compare candidate skill restructurings.

How a run works:

  • Each run gets a fresh, isolated Copilot home and a workspace in %TEMP%, outside any repo.
  • The plugins for that setup are installed, then copilot skill list --json must show exactly the expected plugin skills.
  • The prompt runs with copilot -p with shell, file writes, and URLs denied.
  • The runner reads the session's saved event log line by line and records:
    • skills the agent invoked
    • approximate skill context size
    • tokens, AI credits, turns
    • tool calls, including denied ones
    • duration and exit status
  • Results go to the gitignored results/<timestamp>/ as runs.jsonl, summary.md, and run-info.json.

Usage Example

pwsh benchmarks\agents\run.ps1 -Plan                                          # dry run: 378 sessions with defaults
pwsh benchmarks\agents\run.ps1 -Model claude-sonnet-5.5 -Iterations 1         # quick baseline: 42 sessions
pwsh benchmarks\agents\run.ps1                                                # full matrix: sonnet-5.5, opus-5.5, gpt-6.1-sol x 3
pwsh benchmarks\agents\run.ps1 -Configuration winui,both -WinUIPlugin <path>  # compare a candidate plugin
pwsh benchmarks\agents\run.ps1 -Rescore benchmarks\agents\results\<timestamp> # re-check expectations, no model calls

-Rescore re-evaluates recorded pass/fail runs against the current scenario expectations. It writes runs.rescored.jsonl and summary.rescored.md next to the originals and leaves them unchanged.

Prerequisites: PowerShell 7.4, git, and Copilot CLI authenticated through GH_TOKEN/COPILOT_GITHUB_TOKEN. See benchmarks/agents/README.md.

Scenarios: please review the prompts

Prompts are written as natural requests and never name a skill. Expect lists the skills that count as a pass; at least one must be invoked.

id Setups Prompt (short) Expect
electron-notifications winapp, both Native toast with an "open file" button when the export job in main.js finishes winapp-identity / winapp-frameworks / winapp-setup; forbid winui-*
winforms-package-sign winapp, both Package my WinForms app as an MSIX for testers, signed with a dev cert winapp-package / winapp-signing / winapp-frameworks
sign-existing-msix winapp, both Sign the pipeline-built MSIX with the company .pfx, then verify winapp-signing; forbid winui-*, winapp-setup
msix-install-fails winapp, both Installing dist\ContosoApp.msix fails with 0x800B0109. Figure out why and fix it. winapp-troubleshoot / winapp-signing
manifest-alias-icons winapp, both Add a contoso alias and replace all icons with Assets\logo-new.png winapp-manifest
restore-cloned-cpp winapp, both Cloned C++ project can't find winrt/Microsoft.UI.Xaml.h; restore without changing pinned versions winapp-setup / winapp-troubleshoot
maui-windows-msix winapp, both Signed, sideloadable MSIX of this MAUI app using CI-friendly commands winapp-maui / winapp-package
sandbox-run-test winapp, both Run this app in Windows Sandbox and tell me whether it starts cleanly without my machine's dependencies winapp-sandbox
new-winui-app winui, both Create a C# WinUI 3 app called PhotoTagger and run it winapp-setup / winui-dev-workflow / winui-setup
winui-settings-page winui, both Add a Settings page with a light/dark toggle to the NavigationView winui-design / winapp-find-ui
winui-code-review winui, both Review OrdersPage for bugs, binding, a11y, theming, perf before I commit winui-code-review / winui-design
winui-feature-ui-test winui, both Write and run UI tests that verify adding an item on MainPage updates the counter winui-ui-testing; forbid winapp-signing
wpf-to-winui winui, both Migrate this WPF app to WinUI 3, starting with MainWindow winui-wpf-migration
wpf-add-notifications winapp, winui, both WPF app that references the Windows App SDK: toast when the CSV import finishes. Tests that Windows App SDK ≠ WinUI. winapp-identity / winapp-frameworks / winapp-setup / winapp-find-api; forbid winui-*
find-winrt-api-battery winapp, winui, both Show battery % and plugged-in state in this C++/WinRT WinUI app, kept updated winapp-find-api
xaml-unknown-member winapp, winui, both WMC0011 Unknown member 'Watermark' on 'TextBox'. Fix it. winapp-find-api / winui-design / winapp-troubleshoot / winui-dev-workflow
click-button-running-app winapp, winui, both Click Save in the running Contoso Notes app and tell me the status bar text winapp-ui-automation / winui-ui-testing
console-arg-parsing (near miss) none, winapp, winui, both Add --input/--verbose parsing to a .NET console tool no Windows skill (maxSkills 0)

Every scenario that installs WinUI also forbids winui-session-report. An expected skill that isn't installed in a setup is ignored for that setup.

Findings so far

All runs: Copilot CLI 1.0.87-0, claude-sonnet-5.5, one iteration.

Scenario / setup Result Skills invoked Input (incl. cached) Skill context Output Time
electron-notifications / winapp pass winapp-frameworks 115.0k not measured* 1.7k 23s
winui-settings-page / both (×2) fail none 94k–118k 0 3.3k–4.5k 27–34s
manifest-alias-icons / winapp pass winapp-manifest 142.9k ~2.0k 1.5k 20s
wpf-add-notifications / both pass winapp-identity (no WinUI skill) 177.2k ~2.5k 3.0k 30s

* That run happened before skill context measurement was added.

  • Routing gap: for the settings-page request, Sonnet went straight to writing XAML without loading winui-design. Its write and shell attempts were denied and the workspace stayed unchanged. One iteration only, so treat this as a lead, not a conclusion.
  • Input tokens are mostly Copilot's own prompt. System prompt, tool definitions, and conversation dominate. The skill context column is what to watch when skills change size.

Where Copilot CLI behaved differently than expected

  • Pinned CLI version: with COPILOT_AUTO_UPDATE=false (or --no-auto-update) alone, the launcher runs its bundled build, 1.0.71-0, not the newest one installed. The runner finds the newest installed build once, passes --prefer-version <v> on every call, and checks --version agrees. run-info.json records the version.
  • Saved log events:
    • The skill-context record is skill.context_delivered_ref. It holds only a hash plus wrapper text, so the runner estimates size as wrapper + the skill body from skill.invoked, divided by 4 for tokens. It's labeled approximate.
    • session.skills_loaded never appeared.
    • Token usage comes from session.shutdown.modelMetrics[*].usage. A run killed by timeout has no shutdown event, so its tokens are null with a reason, never 0.
  • Token meaning: usage.inputTokens is the whole prompt including cached tokens. The top-level tokenDetails.input is uncached input only. The summary shows input including cached, plus cache read separately.
  • Environment cleanup: an agent host sets many COPILOT_* variables, such as COPILOT_CLI_RESOLVED_DIST_DIR. The child process gets all of them removed except COPILOT_GITHUB_TOKEN. Workspaces live outside any repo because skill list picks up .github/skills from the working directory. Preflight also checks that every plugin skill path is inside the isolated home.
  • Plugin install: copilot plugin install <local path> prints a deprecation warning for direct installs. That's fine for v1.

Type of Change

  • ✨ New feature
  • 🧪 Test update

Checklist

  • New tests added for new functionality: benchmarks/agents/tests (Pester, 13 tests) covers the event parser with a completed log, a log with no shutdown event, and a missing log, plus expectations, rescoring, the workspace snapshot, the summary, and scenario validation.
  • Tested locally on Windows

Additional Notes

Limits:

  • Local only, Copilot CLI only, sequential runs, no CI.
  • Shell, file writes, and URLs are denied, so this measures routing and skill loading, not whether the agent completes the task.
  • Results vary run to run; use several iterations before drawing conclusions.

scripts/validate-plugin-package.ps1 is untouched.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

⏳ Build in progress — metrics below are from a previous commit and will update when the current build finishes.

Build Metrics Report

Validation passed. All required build and validation jobs succeeded.

Binary Sizes

Artifact Baseline Current Delta
CLI (ARM64) 57.36 MB 57.36 MB ✅ 0.0 KB (0.00%)
CLI (x64) 57.40 MB 57.40 MB ✅ 0.0 KB (0.00%)
MSIX (ARM64) 23.83 MB 23.83 MB 📈 +0.0 KB (+0.00%)
MSIX (x64) 25.29 MB 25.29 MB 📈 +0.0 KB (+0.00%)
NPM Package 49.70 MB 49.70 MB 📈 +0.4 KB (+0.00%)
NuGet Package 49.81 MB 49.81 MB 📉 -0.5 KB (-0.00%)

.NET Test Results (TRX reports)

Other suites are reflected in the overall validation status above.

✅ 7981 passed, 37 skipped out of 8018 tests in 1193.9s (-53.5s vs. baseline)

Test Coverage

✅ 86.4% line coverage, 81% branch coverage · ✅ no change vs. baseline

CLI Startup Time

60ms median (x64, winapp --version) · ✅ +9ms vs. baseline

Try This Build

Installs the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing.

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 973
Switching between builds often?

Put the tool on your PATH once:

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPath

Then this build is just:

winapp-pr 973

Run winapp-pr with no arguments to pick from a list of open PRs.


Updated 2026-10-03 02:52:47 UTC · commit e5d444b · workflow run

Runs realistic prompts through Copilot CLI in isolated homes with each plugin
configuration (none, winapp, winui, both), and records which skills load,
tokens, and AI credits from the persisted session log.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… pwsh -File

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev
Nikola Metulev (nmetulev) force-pushed the nmetulev-local-agent-benchmark branch from e5d444b to 575907e Compare October 6, 2026 21:07
@nmetulev
Nikola Metulev (nmetulev) marked this pull request as ready for review October 6, 2026 21:19
Copilot AI balanced review requested due to automatic review settings October 6, 2026 21:19
@nmetulev
Nikola Metulev (nmetulev) merged commit f61c1b7 into main Oct 6, 2026
35 of 36 checks passed
@nmetulev
Nikola Metulev (nmetulev) deleted the nmetulev-local-agent-benchmark branch October 6, 2026 21:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Process isolation and tool restrictions still need end-to-end validation, and correctness fixes remain unresolved.

Review effort: Balanced
Findings: 1 High severity · 5 Medium severity

Open (6)
What changed in this PR

Adds a local benchmark for comparing Copilot CLI skill routing and token usage across plugin configurations, supporting future WinUI migration and skill restructuring.

Changes:

  • Adds 18 developer scenarios with routing expectations and fixtures.
  • Implements isolated runs, usage reporting, and offline rescoring.
  • Adds Pester coverage, usage documentation, and ignored result storage.
File Description
benchmarks/​agents/​tests/​events-no-shutdown.jsonl Incomplete-session parser fixture.
benchmarks/​agents/​tests/​events-complete.jsonl Completed-session parser fixture.
benchmarks/​agents/​tests/​Benchmark.Tests.ps1 Tests parsing, scoring, summaries, and validation.
benchmarks/​agents/​scenarios/​xaml-unknown-member/​scenario.json Defines XAML troubleshooting expectations.
benchmarks/​agents/​scenarios/​xaml-unknown-member/​fixture/​MainWindow.xaml Seeds an invalid XAML property.
benchmarks/​agents/​scenarios/​xaml-unknown-member/​fixture/​ContosoSearch.csproj Supplies WinUI project context.
benchmarks/​agents/​scenarios/​wpf-to-winui/​scenario.json Defines migration routing expectations.
benchmarks/​agents/​scenarios/​wpf-to-winui/​fixture/​MainWindow.xaml.cs Supplies WPF migration behavior.
benchmarks/​agents/​scenarios/​wpf-to-winui/​fixture/​MainWindow.xaml Supplies WPF migration UI.
benchmarks/​agents/​scenarios/​wpf-to-winui/​fixture/​ContosoInventory.csproj Identifies the WPF project.
benchmarks/​agents/​scenarios/​wpf-add-notifications/​scenario.json Tests notification routing without WinUI.
benchmarks/​agents/​scenarios/​wpf-add-notifications/​fixture/​MainWindow.xaml.cs Seeds CSV import behavior.
benchmarks/​agents/​scenarios/​wpf-add-notifications/​fixture/​MainWindow.xaml Supplies importer UI context.
benchmarks/​agents/​scenarios/​wpf-add-notifications/​fixture/​ContosoImporter.csproj Establishes WPF with Windows App SDK.
benchmarks/​agents/​scenarios/​winui-settings-page/​scenario.json Defines settings-page routing expectations.
benchmarks/​agents/​scenarios/​winui-settings-page/​fixture/​MainWindow.xaml.cs Seeds navigation behavior.
benchmarks/​agents/​scenarios/​winui-settings-page/​fixture/​MainWindow.xaml Supplies NavigationView layout.
benchmarks/​agents/​scenarios/​winui-settings-page/​fixture/​ContosoReader.csproj Supplies WinUI project context.
benchmarks/​agents/​scenarios/​winui-feature-ui-test/​scenario.json Defines UI-testing routing expectations.
benchmarks/​agents/​scenarios/​winui-feature-ui-test/​fixture/​MainPage.xaml.cs Seeds list and counter behavior.
benchmarks/​agents/​scenarios/​winui-feature-ui-test/​fixture/​MainPage.xaml Supplies controls for testing.
benchmarks/​agents/​scenarios/​winui-feature-ui-test/​fixture/​ContosoTasks.csproj Supplies WinUI project context.
benchmarks/​agents/​scenarios/​winui-code-review/​scenario.json Defines code-review routing expectations.
benchmarks/​agents/​scenarios/​winui-code-review/​fixture/​OrdersPage.xaml.cs Supplies code-review targets.
benchmarks/​agents/​scenarios/​winui-code-review/​fixture/​OrdersPage.xaml Supplies UI-review targets.
benchmarks/​agents/​scenarios/​winui-code-review/​fixture/​ContosoOrders.csproj Supplies WinUI project context.
benchmarks/​agents/​scenarios/​winforms-package-sign/​scenario.json Defines packaging and signing expectations.
benchmarks/​agents/​scenarios/​winforms-package-sign/​fixture/​Program.cs Seeds a WinForms application.
benchmarks/​agents/​scenarios/​winforms-package-sign/​fixture/​ContosoTimer.csproj Supplies WinForms project context.
benchmarks/​agents/​scenarios/​sign-existing-msix/​scenario.json Defines signing-only expectations.
benchmarks/​agents/​scenarios/​sandbox-run-test/​scenario.json Defines Sandbox routing expectations.
benchmarks/​agents/​scenarios/​sandbox-run-test/​fixture/​Program.cs Seeds an app with SQLite dependency.
benchmarks/​agents/​scenarios/​sandbox-run-test/​fixture/​ContosoClock.csproj Declares Sandbox fixture dependencies.
benchmarks/​agents/​scenarios/​restore-cloned-cpp/​scenario.json Defines pinned-SDK restoration expectations.
benchmarks/​agents/​scenarios/​restore-cloned-cpp/​fixture/​winapp.yaml Supplies pinned package versions.
benchmarks/​agents/​scenarios/​restore-cloned-cpp/​fixture/​main.cpp References missing generated headers.
benchmarks/​agents/​scenarios/​restore-cloned-cpp/​fixture/​CMakeLists.txt Supplies C++ build context.
benchmarks/​agents/​scenarios/​restore-cloned-cpp/​fixture/​.gitignore Excludes generated fixture outputs.
benchmarks/​agents/​scenarios/​new-winui-app/​scenario.json Defines app-creation routing expectations.
benchmarks/​agents/​scenarios/​msix-install-fails/​scenario.json Defines certificate troubleshooting expectations.
benchmarks/​agents/​scenarios/​maui-windows-msix/​scenario.json Defines MAUI packaging expectations.
benchmarks/​agents/​scenarios/​maui-windows-msix/​fixture/​ContosoMobile.csproj Supplies MAUI project context.
benchmarks/​agents/​scenarios/​manifest-alias-icons/​scenario.json Defines manifest-editing expectations.
benchmarks/​agents/​scenarios/​manifest-alias-icons/​fixture/​Package.appxmanifest Seeds package identity and visuals.
benchmarks/​agents/​scenarios/​find-winrt-api-battery/​scenario.json Defines battery API routing expectations.
benchmarks/​agents/​scenarios/​find-winrt-api-battery/​fixture/​MainWindow.xaml.cpp Seeds battery-display implementation.
benchmarks/​agents/​scenarios/​find-winrt-api-battery/​fixture/​MainWindow.xaml Supplies battery-display UI.
benchmarks/​agents/​scenarios/​find-winrt-api-battery/​fixture/​BatteryWidget.vcxproj Supplies C++/WinRT project context.
benchmarks/​agents/​scenarios/​electron-notifications/​scenario.json Defines Electron notification expectations.
benchmarks/​agents/​scenarios/​electron-notifications/​fixture/​package.json Supplies Electron package context.
benchmarks/​agents/​scenarios/​electron-notifications/​fixture/​main.js Seeds export notification behavior.
benchmarks/​agents/​scenarios/​console-arg-parsing/​scenario.json Defines the no-Windows-skill control.
benchmarks/​agents/​scenarios/​console-arg-parsing/​fixture/​Program.cs Seeds console argument handling.
benchmarks/​agents/​scenarios/​console-arg-parsing/​fixture/​LogTrimmer.csproj Supplies plain .NET project context.
benchmarks/​agents/​scenarios/​click-button-running-app/​scenario.json Defines running-app automation expectations.
benchmarks/​agents/​run.ps1 Orchestrates planning, isolated runs, and reporting.
benchmarks/​agents/​README.md Documents benchmark usage and interpretation.
benchmarks/​agents/​lib/​Benchmark.psm1 Implements parsing, scoring, and reporting helpers.
benchmarks/​agents/​config.json Sets default models, iterations, and plugins.
AGENTS.md Links contributors to the benchmark.
.gitignore Excludes local benchmark results.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread benchmarks/agents/run.ps1
}
$skillList = @(Get-Content -Raw (Join-Path $logs 'skill-list.out') | ConvertFrom-Json)
$nonBuiltin = @($skillList | Where-Object { $_.source -ne 'builtin' })
$unexpected = @($nonBuiltin | Where-Object { $_.name -notin $installed -or $_.source -ne 'plugin' -or -not $_.path.StartsWith($copilotHome, [StringComparison]::OrdinalIgnoreCase) } | ForEach-Object { "$($_.name) ($($_.source))" })
Comment on lines +269 to +272
$usage = [ordered]@{ input = 0L; output = 0L; cacheRead = 0L; cacheWrite = 0L; reasoning = 0L }
foreach ($m in $data.modelMetrics.GetEnumerator()) {
$u = $m.Value.usage
if ($null -eq $u) { continue }
Comment on lines +421 to +427
$tokIn = ($rows | Where-Object { $_.tokens } | ForEach-Object { $_.tokens.input } | Measure-Object -Sum).Sum
$tokOut = ($rows | Where-Object { $_.tokens } | ForEach-Object { $_.tokens.output } | Measure-Object -Sum).Sum
$credits = ($rows | Where-Object { $null -ne $_.aiCredits } | ForEach-Object { $_.aiCredits } | Measure-Object -Sum).Sum
[void]$sb.AppendLine("- Tokens: $(Format-Count $tokIn) input, $(Format-Count $tokOut) output; AI credits: $(if ($null -ne $credits) { '{0:N1}' -f $credits } else { 'n/a' })")
$ctx = @($rows | Where-Object { $null -ne $_.skillContextTokensApprox })
$ctxSum = ($ctx | ForEach-Object { $_.skillContextTokensApprox } | Measure-Object -Sum).Sum
[void]$sb.AppendLine("- Skill context delivered: ~$(Format-Count $ctxSum) tokens (approximate, characters / 4; $($ctx.Count) of $($rows.Count) runs measured)")
$evaluated = @($runs | Where-Object { $_.status -in 'pass', 'fail' })
$skillText = 'n/a'
if ($evaluated) {
$top = $evaluated | ForEach-Object { if ($_.skillsLoaded) { ($_.skillsLoaded -join ', ') } else { '(none)' } } |
Comment thread benchmarks/agents/run.ps1
if ($p.Skills.Count -eq 0) { throw "Plugin '$($p.Name)' at $($p.Path) has no skills." }
}

$knownSkills = @($plugins.Values.Skills)
Comment thread benchmarks/agents/run.ps1
if (-not $OutDir) { $OutDir = Join-Path $PSScriptRoot "results\$(Get-Date -Format 'yyyyMMdd-HHmmss')" }
New-Item -ItemType Directory -Force -Path $OutDir | Out-Null
$OutDir = (Resolve-Path $OutDir).Path
$runsPath = Join-Path $OutDir 'runs.jsonl'
Nikola Metulev (nmetulev) added a commit that referenced this pull request Oct 6, 2026
Layer 2 of the plugin stack (base: #973). Imports the WinUI agent plugin
as-is so later layers can publish it from here, then unify and trim.

## What's imported

- Source:
[`microsoft/win-dev-skills`](https://github.com/microsoft/win-dev-skills)
tag `v0.7.1` = `2e8c902e9ba675e4c57550e066fc7e48b64c49b0`, folder
`plugins/winui/`.
- Copied to `plugins/winui/` with the same layout (`agent-plugin/`,
`.claude-plugin/`, `.codex-plugin/`, `agents/`, `openclaw.plugin.json`,
`package.json`, `index.js`), so host wrapper paths keep working.
Manifests keep their own version `0.7.1`.
- **24 files. Byte parity:** every path + SHA-256 matches the source
tree, git blob IDs match the source commit, and a fresh `git checkout`
of the folder still matches.
- `.gitattributes`: `plugins/winui/** text eol=lf` + `*.png binary`, so
this repo's `*.ps1 eol=crlf` rule and `text=auto` don't rewrite
`Analyze-Session.ps1` or other files on checkout.
- Licensing: both repos are Microsoft MIT. win-dev-skills
`THIRD_PARTY_NOTICES.md` lists no vendored third-party content for the
plugin and `cgmanifest.json` has no registrations, so nothing extra to
carry. No README added; provenance is recorded here.

## Intentionally not included

- No entry in `.github/plugin/marketplace.json` /
`.claude-plugin/marketplace.json` or README install docs: this PR must
not create a second live install source.
- No win-dev-skills `src/tools` (winui-analyzer, winmd-cli), CI
workflows, or release scripts. **Ownership of analyzer/tool source is a
separate decision.**

## Tooling

No fixes needed. `generate-llm-docs.ps1`, `validate-llm-docs.ps1`, and
`validate-plugin-package.ps1` only reference `plugins/winapp` paths
explicitly, so they neither stamp the CLI version into WinUI manifests
nor validate it.

Benchmark (`benchmarks/agents`): the default WinUI plugin is now the
local `plugins/winui/agent-plugin` (relative path, like winapp).
`-WinUIPlugin published` fetches the pinned `win-dev-skills@v0.7.1` for
comparison (config moved under `plugins.winui.published`); `-WinUIPlugin
<path>` is unchanged. README updated.

## Validation

- `scripts/validate-plugin-package.ps1` (this base): pass.
- `scripts/validate-llm-docs.ps1`: not run; it needs a built CLI
(`artifacts/cli/win-x64/winapp.exe`). It only reads winapp manifests, so
this PR can't affect it.
- Benchmark Pester tests: 16/16 pass. `-Plan` shows the local default
and `published`; the published fetch path was exercised and its
`agent-plugin` hashes identically to the local copy.

### #972 skill checks, run locally (not committed)

- **Discovery gap:** #972 finds plugins as `plugins/<name>/skills`, so
it **silently skips WinUI** (`plugins/winui/agent-plugin/skills`). Needs
fixing in #972 or a later layer.
- Forced over WinUI (`-PluginsRoot plugins/winui`):
- 4 errors: `winui-code-review:45`, `winui-design:106`,
`winui-dev-workflow:51`, `winui-wpf-migration:77` reference
`references/sourcegen-patterns.md`, which lives in `winui-packaging`.
The text says "see `winui-packaging`'s
`references/sourcegen-patterns.md`" (v0.7.1 deliberately replaced
cross-skill links with plain text), so the check resolves it against the
wrong skill. Decide in a later layer: relax the check or reword.
- 4 description-length warnings (>300 chars): `winui-design` 412,
`winui-wpf-migration` 424, `winui-ui-testing` 388, `winui-packaging`
352.
  - `winapp` command examples: no unknown commands.
- Size: metadata ~764 tokens always loaded; bodies ~19.7k; references
~21.9k.
- Content not changed.

### Benchmark smoke vs baseline (winui, claude-sonnet-5.5)

Preflight installed the local plugin (source `this repo`, v0.7.1, not
dirty) with the same 8 skills as published v0.7.1; no unexpected or
missing skills.

| Scenario | Baseline (3 runs) | This PR (1 run) | Skills (both) |
|---|---|---|---|
| click-button-running-app | 3/3 | pass | winui-ui-testing |
| console-arg-parsing | 3/3 | pass | none |
| find-winrt-api-battery | 3/3 | pass | none |
| new-winui-app | 3/3 | pass | winui-dev-workflow |
| winui-code-review | 3/3 | pass | winui-code-review |
| winui-feature-ui-test | 3/3 | pass | winui-ui-testing |
| winui-settings-page | 0/3 | fail | none |
| wpf-add-notifications | 3/3 | pass | none |
| wpf-to-winui | 3/3 | pass | winui-wpf-migration |
| xaml-unknown-member | 0/3 | fail | none |

All 10 match the baseline status and skill set. The two failures are
pre-existing routing gaps, not caused by the import.

## Next layers

1. Publishing pipeline for WinUI from this repo.
2. Flip: marketplace/catalog entries point here; win-dev-skills
redirects.
3. Unify/trim: shared validation (incl. #972 discovery), description
budgets, cross-skill references, overlap with winapp skills.

Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nikola Metulev (nmetulev) added a commit that referenced this pull request Oct 7, 2026
#981)

Layer 3 of the benchmark stack. #973 and #976 are merged, so this now
targets `main`. It adds the benchmark improvements tested during the
weekend experiments, plus reporting.

## Changes

**From the experiments branch (cherry-picked, no conflicts)**
- `-Agent <name>` runs every session with `--agent`. `-MaxCredits <n>`
stops launching sessions once the run has spent `n` credits. Each run
also records `winappCommands` (including `node <subcommand>` from the
npm wrapper) and `selectedAgent`.
- 24 reviewed scenarios. Expectations are tightened on six existing
ones, with no prompt or fixture changes. Packaging scenarios need a
packaging skill to load; signing alone does not pass. Review notes are
in `benchmarks/agents/scenarios-review.md`.
- Handles one skill name shipped by two plugins, which Copilot lists as
`<plugin>:<name>`. Preflight requires one enabled copy per plugin.

**New in this PR**
- **`n/a` status:** a run where nothing in the expectation applies to
the installed skills gets `n/a` and is left out of pass rates. "Nothing
applies" means no expected or forbidden skill is installed and there is
no `maxSkills`. The summary lists these runs separately.
- **Repeated deliveries:** each run records skills delivered more than
once in a session and the extra skill context they add. `summary.md`
shows them in the totals and per row, as `unknown` or `partial` when
their size is not known. It also counts launched runs that had no credit
count.
- **`run.ps1 -Compare <dirs> -Candidate <dirs>`** builds a Markdown
report by model and by cell (model, scenario, configuration). It shows
pass rate (only pass and fail are scored; other statuses are listed as
excluded, the same as in `summary.md`), skill context per run, input
tokens per run, credits per run, the change in each, and repeated
deliveries. Only cells present on both sides are compared. Both sides
are re-scored in memory against the current expectations, and the input
folders are not modified. A cell shows `check differs` when the
installed skills that the expectation checks differ by name between the
sides, for example when a candidate adds or removes an expected skill,
including removing all of them. It is then left out of the per-model
pass rate. `-Scenario` and `-Model` must name values present in the
compared results. Runs of scenarios that no longer exist are excluded as
`scenario_removed`.
- **`-MaxCredits` with unknown credits:** a session that ran but
reported no credits (for example a timeout) counts at the average of the
measured sessions, or at 35 before any is measured.
- **Every scenario must include `both`:** validation fails otherwise,
and there is a Pester test for it. All 42 scenarios comply.
- **`-Rescore`** also re-scores `n/a` runs and prints status
transitions. A mistyped `-Scenario` now fails before `-Rescore` or
`-Compare` runs.
- **README:** documents the new parameters and statuses, the `both`
rule, and the recommended workflow: a quick Sonnet screen, then 3 models
× 3 iterations on the affected scenarios plus the near-misses.

## Scenarios and plan totals

42 scenarios: 18 existing + 24 new.
- `-Plan`, full default matrix: **954** sessions (3 models × 3
iterations).
- Sonnet, 1 iteration: **106** sessions.

## Validation
- Pester: 46/46 pass (was 19).
- `-Compare` on the weekend result folders (read-only). This is the E4
candidate vs the current plugins; the numbers match the weekend report:

| Model | Cells | Pass (base → cand) | Δ pass | Skill context/run | Δ |
Input/run | Δ | AI credits/run | Δ |
|---|---|---|---|---|---|---|---|---|---|
| claude-opus-5.5 | 105 | 310/315 (98%) → 306/315 (97%) | -1 pp | 3.8k →
3.9k | +4% | 174.7k → 170.8k | -2% | 21.3 → 19.4 | -9% |
| claude-sonnet-5.5 | 105 | 194/315 (62%) → 227/315 (72%) | +10 pp |
1.8k → 2.1k | +16% | 122.4k → 122.2k | = | 8.3 → 7.9 | -6% |
| gpt-6.1-sol | 105 | 304/315 (97%) → 309/315 (98%) | +2 pp | 6.7k →
6.0k | -11% | 130.0k → 101.6k | -22% | 10.1 → 8.3 | -18% |

E4.2 shares `winapp-find-api` into the WinUI plugin. That changes what
15 of its 24 cells check (5 per model), so they show `check differs`.
The per-model pass rates use the other 9 cells: Opus 9/9 → 9/9, Sonnet
3/9 → 4/9, GPT 9/9 → 9/9. The E4 comparison has no `check differs`
cells.
- `-Rescore` on a copy of the original 297-run baseline: 3 runs went
pass → fail (from the tightened expectations), and 0 went to `n/a`.
Across all 954 baseline runs, no current cell is `n/a`. The status
exists so that candidates which drop or rename skills don't get free
passes.
- No new model runs.

## Notes
- **`n/a` is narrower than "no expected skill installed".** Under that
literal rule, clean runs in forbid-checked cells would become `n/a`
while overreach still fails. An example is `wpf-add-notifications /
winui`, where GPT WinUI overreach went 0/3 → 3/3 in E2. That rule would
hide that result, so a cell stays scored when a forbidden skill is
installed.
- **The `sign-existing-msix-fixture` control was dropped.** Its `.msix`
is ignored by the `dist` rule in `.gitignore`, so the committed copy is
not the fixture that was actually run. It also repeats
`sign-existing-msix`. Its finding (the empty folder does not explain the
skipped skill) is recorded in `scenarios-review.md`.
- **Empty fixtures in `sign-existing-msix` and `msix-install-fails` are
unchanged.** Filling them would change the scenario input and require a
new baseline.
- **Repeated deliveries can't be recovered from older runs.** Their
records lack per-delivery data, so the compare report shows `-` for
them.
- **More scenario changes may follow in this layer.** A follow-up
realism review will likely add or modify scenarios here.

---------

Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants