Repository navigation
Add local agent plugin benchmark (v1) - #973
Conversation
Build Metrics ReportValidation passed. All required build and validation jobs succeeded. Binary Sizes
.NET Test Results (TRX reports)Other suites are reflected in the overall validation status above. ✅ 7981 passed, 37 skipped out of 8018 tests in 1193.9s (-53.5s vs. baseline) Test Coverage✅ 86.4% line coverage, 81% branch coverage · ✅ no change vs. baseline CLI Startup Time60ms median (x64, Try This BuildInstalls the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing. & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 973Switching between builds often?Put the tool on your PATH once: & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPathThen this build is just: winapp-pr 973Run Updated 2026-10-03 02:52:47 UTC · commit |
Runs realistic prompts through Copilot CLI in isolated homes with each plugin configuration (none, winapp, winui, both), and records which skills load, tokens, and AI credits from the persisted session log. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… pwsh -File Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
e5d444b to
575907e
Compare
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Process isolation and tool restrictions still need end-to-end validation, and correctness fixes remain unresolved.
Review effort: Balanced
Findings: 1
Open (6)
Enforce home directory boundaries for skill paths · New Preserve unknown usage when model metrics are unavailable · New Preserve missing usage totals instead of reporting zero · New Group skill sets independently of invocation order · New Fix empty plugin collection handling in strict mode · New Reject reused output directories for separate benchmarks · New
What changed in this PR
Adds a local benchmark for comparing Copilot CLI skill routing and token usage across plugin configurations, supporting future WinUI migration and skill restructuring.
Changes:
- Adds 18 developer scenarios with routing expectations and fixtures.
- Implements isolated runs, usage reporting, and offline rescoring.
- Adds Pester coverage, usage documentation, and ignored result storage.
| File | Description |
|---|---|
| benchmarks/agents/tests/events-no-shutdown.jsonl | Incomplete-session parser fixture. |
| benchmarks/agents/tests/events-complete.jsonl | Completed-session parser fixture. |
| benchmarks/agents/tests/Benchmark.Tests.ps1 | Tests parsing, scoring, summaries, and validation. |
| benchmarks/agents/scenarios/xaml-unknown-member/scenario.json | Defines XAML troubleshooting expectations. |
| benchmarks/agents/scenarios/xaml-unknown-member/fixture/MainWindow.xaml | Seeds an invalid XAML property. |
| benchmarks/agents/scenarios/xaml-unknown-member/fixture/ContosoSearch.csproj | Supplies WinUI project context. |
| benchmarks/agents/scenarios/wpf-to-winui/scenario.json | Defines migration routing expectations. |
| benchmarks/agents/scenarios/wpf-to-winui/fixture/MainWindow.xaml.cs | Supplies WPF migration behavior. |
| benchmarks/agents/scenarios/wpf-to-winui/fixture/MainWindow.xaml | Supplies WPF migration UI. |
| benchmarks/agents/scenarios/wpf-to-winui/fixture/ContosoInventory.csproj | Identifies the WPF project. |
| benchmarks/agents/scenarios/wpf-add-notifications/scenario.json | Tests notification routing without WinUI. |
| benchmarks/agents/scenarios/wpf-add-notifications/fixture/MainWindow.xaml.cs | Seeds CSV import behavior. |
| benchmarks/agents/scenarios/wpf-add-notifications/fixture/MainWindow.xaml | Supplies importer UI context. |
| benchmarks/agents/scenarios/wpf-add-notifications/fixture/ContosoImporter.csproj | Establishes WPF with Windows App SDK. |
| benchmarks/agents/scenarios/winui-settings-page/scenario.json | Defines settings-page routing expectations. |
| benchmarks/agents/scenarios/winui-settings-page/fixture/MainWindow.xaml.cs | Seeds navigation behavior. |
| benchmarks/agents/scenarios/winui-settings-page/fixture/MainWindow.xaml | Supplies NavigationView layout. |
| benchmarks/agents/scenarios/winui-settings-page/fixture/ContosoReader.csproj | Supplies WinUI project context. |
| benchmarks/agents/scenarios/winui-feature-ui-test/scenario.json | Defines UI-testing routing expectations. |
| benchmarks/agents/scenarios/winui-feature-ui-test/fixture/MainPage.xaml.cs | Seeds list and counter behavior. |
| benchmarks/agents/scenarios/winui-feature-ui-test/fixture/MainPage.xaml | Supplies controls for testing. |
| benchmarks/agents/scenarios/winui-feature-ui-test/fixture/ContosoTasks.csproj | Supplies WinUI project context. |
| benchmarks/agents/scenarios/winui-code-review/scenario.json | Defines code-review routing expectations. |
| benchmarks/agents/scenarios/winui-code-review/fixture/OrdersPage.xaml.cs | Supplies code-review targets. |
| benchmarks/agents/scenarios/winui-code-review/fixture/OrdersPage.xaml | Supplies UI-review targets. |
| benchmarks/agents/scenarios/winui-code-review/fixture/ContosoOrders.csproj | Supplies WinUI project context. |
| benchmarks/agents/scenarios/winforms-package-sign/scenario.json | Defines packaging and signing expectations. |
| benchmarks/agents/scenarios/winforms-package-sign/fixture/Program.cs | Seeds a WinForms application. |
| benchmarks/agents/scenarios/winforms-package-sign/fixture/ContosoTimer.csproj | Supplies WinForms project context. |
| benchmarks/agents/scenarios/sign-existing-msix/scenario.json | Defines signing-only expectations. |
| benchmarks/agents/scenarios/sandbox-run-test/scenario.json | Defines Sandbox routing expectations. |
| benchmarks/agents/scenarios/sandbox-run-test/fixture/Program.cs | Seeds an app with SQLite dependency. |
| benchmarks/agents/scenarios/sandbox-run-test/fixture/ContosoClock.csproj | Declares Sandbox fixture dependencies. |
| benchmarks/agents/scenarios/restore-cloned-cpp/scenario.json | Defines pinned-SDK restoration expectations. |
| benchmarks/agents/scenarios/restore-cloned-cpp/fixture/winapp.yaml | Supplies pinned package versions. |
| benchmarks/agents/scenarios/restore-cloned-cpp/fixture/main.cpp | References missing generated headers. |
| benchmarks/agents/scenarios/restore-cloned-cpp/fixture/CMakeLists.txt | Supplies C++ build context. |
| benchmarks/agents/scenarios/restore-cloned-cpp/fixture/.gitignore | Excludes generated fixture outputs. |
| benchmarks/agents/scenarios/new-winui-app/scenario.json | Defines app-creation routing expectations. |
| benchmarks/agents/scenarios/msix-install-fails/scenario.json | Defines certificate troubleshooting expectations. |
| benchmarks/agents/scenarios/maui-windows-msix/scenario.json | Defines MAUI packaging expectations. |
| benchmarks/agents/scenarios/maui-windows-msix/fixture/ContosoMobile.csproj | Supplies MAUI project context. |
| benchmarks/agents/scenarios/manifest-alias-icons/scenario.json | Defines manifest-editing expectations. |
| benchmarks/agents/scenarios/manifest-alias-icons/fixture/Package.appxmanifest | Seeds package identity and visuals. |
| benchmarks/agents/scenarios/find-winrt-api-battery/scenario.json | Defines battery API routing expectations. |
| benchmarks/agents/scenarios/find-winrt-api-battery/fixture/MainWindow.xaml.cpp | Seeds battery-display implementation. |
| benchmarks/agents/scenarios/find-winrt-api-battery/fixture/MainWindow.xaml | Supplies battery-display UI. |
| benchmarks/agents/scenarios/find-winrt-api-battery/fixture/BatteryWidget.vcxproj | Supplies C++/WinRT project context. |
| benchmarks/agents/scenarios/electron-notifications/scenario.json | Defines Electron notification expectations. |
| benchmarks/agents/scenarios/electron-notifications/fixture/package.json | Supplies Electron package context. |
| benchmarks/agents/scenarios/electron-notifications/fixture/main.js | Seeds export notification behavior. |
| benchmarks/agents/scenarios/console-arg-parsing/scenario.json | Defines the no-Windows-skill control. |
| benchmarks/agents/scenarios/console-arg-parsing/fixture/Program.cs | Seeds console argument handling. |
| benchmarks/agents/scenarios/console-arg-parsing/fixture/LogTrimmer.csproj | Supplies plain .NET project context. |
| benchmarks/agents/scenarios/click-button-running-app/scenario.json | Defines running-app automation expectations. |
| benchmarks/agents/run.ps1 | Orchestrates planning, isolated runs, and reporting. |
| benchmarks/agents/README.md | Documents benchmark usage and interpretation. |
| benchmarks/agents/lib/Benchmark.psm1 | Implements parsing, scoring, and reporting helpers. |
| benchmarks/agents/config.json | Sets default models, iterations, and plugins. |
| AGENTS.md | Links contributors to the benchmark. |
| .gitignore | Excludes local benchmark results. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| } | ||
| $skillList = @(Get-Content -Raw (Join-Path $logs 'skill-list.out') | ConvertFrom-Json) | ||
| $nonBuiltin = @($skillList | Where-Object { $_.source -ne 'builtin' }) | ||
| $unexpected = @($nonBuiltin | Where-Object { $_.name -notin $installed -or $_.source -ne 'plugin' -or -not $_.path.StartsWith($copilotHome, [StringComparison]::OrdinalIgnoreCase) } | ForEach-Object { "$($_.name) ($($_.source))" }) |
| $usage = [ordered]@{ input = 0L; output = 0L; cacheRead = 0L; cacheWrite = 0L; reasoning = 0L } | ||
| foreach ($m in $data.modelMetrics.GetEnumerator()) { | ||
| $u = $m.Value.usage | ||
| if ($null -eq $u) { continue } |
| $tokIn = ($rows | Where-Object { $_.tokens } | ForEach-Object { $_.tokens.input } | Measure-Object -Sum).Sum | ||
| $tokOut = ($rows | Where-Object { $_.tokens } | ForEach-Object { $_.tokens.output } | Measure-Object -Sum).Sum | ||
| $credits = ($rows | Where-Object { $null -ne $_.aiCredits } | ForEach-Object { $_.aiCredits } | Measure-Object -Sum).Sum | ||
| [void]$sb.AppendLine("- Tokens: $(Format-Count $tokIn) input, $(Format-Count $tokOut) output; AI credits: $(if ($null -ne $credits) { '{0:N1}' -f $credits } else { 'n/a' })") | ||
| $ctx = @($rows | Where-Object { $null -ne $_.skillContextTokensApprox }) | ||
| $ctxSum = ($ctx | ForEach-Object { $_.skillContextTokensApprox } | Measure-Object -Sum).Sum | ||
| [void]$sb.AppendLine("- Skill context delivered: ~$(Format-Count $ctxSum) tokens (approximate, characters / 4; $($ctx.Count) of $($rows.Count) runs measured)") |
| $evaluated = @($runs | Where-Object { $_.status -in 'pass', 'fail' }) | ||
| $skillText = 'n/a' | ||
| if ($evaluated) { | ||
| $top = $evaluated | ForEach-Object { if ($_.skillsLoaded) { ($_.skillsLoaded -join ', ') } else { '(none)' } } | |
| if ($p.Skills.Count -eq 0) { throw "Plugin '$($p.Name)' at $($p.Path) has no skills." } | ||
| } | ||
|
|
||
| $knownSkills = @($plugins.Values.Skills) |
| if (-not $OutDir) { $OutDir = Join-Path $PSScriptRoot "results\$(Get-Date -Format 'yyyyMMdd-HHmmss')" } | ||
| New-Item -ItemType Directory -Force -Path $OutDir | Out-Null | ||
| $OutDir = (Resolve-Path $OutDir).Path | ||
| $runsPath = Join-Path $OutDir 'runs.jsonl' |
Layer 2 of the plugin stack (base: #973). Imports the WinUI agent plugin as-is so later layers can publish it from here, then unify and trim. ## What's imported - Source: [`microsoft/win-dev-skills`](https://github.com/microsoft/win-dev-skills) tag `v0.7.1` = `2e8c902e9ba675e4c57550e066fc7e48b64c49b0`, folder `plugins/winui/`. - Copied to `plugins/winui/` with the same layout (`agent-plugin/`, `.claude-plugin/`, `.codex-plugin/`, `agents/`, `openclaw.plugin.json`, `package.json`, `index.js`), so host wrapper paths keep working. Manifests keep their own version `0.7.1`. - **24 files. Byte parity:** every path + SHA-256 matches the source tree, git blob IDs match the source commit, and a fresh `git checkout` of the folder still matches. - `.gitattributes`: `plugins/winui/** text eol=lf` + `*.png binary`, so this repo's `*.ps1 eol=crlf` rule and `text=auto` don't rewrite `Analyze-Session.ps1` or other files on checkout. - Licensing: both repos are Microsoft MIT. win-dev-skills `THIRD_PARTY_NOTICES.md` lists no vendored third-party content for the plugin and `cgmanifest.json` has no registrations, so nothing extra to carry. No README added; provenance is recorded here. ## Intentionally not included - No entry in `.github/plugin/marketplace.json` / `.claude-plugin/marketplace.json` or README install docs: this PR must not create a second live install source. - No win-dev-skills `src/tools` (winui-analyzer, winmd-cli), CI workflows, or release scripts. **Ownership of analyzer/tool source is a separate decision.** ## Tooling No fixes needed. `generate-llm-docs.ps1`, `validate-llm-docs.ps1`, and `validate-plugin-package.ps1` only reference `plugins/winapp` paths explicitly, so they neither stamp the CLI version into WinUI manifests nor validate it. Benchmark (`benchmarks/agents`): the default WinUI plugin is now the local `plugins/winui/agent-plugin` (relative path, like winapp). `-WinUIPlugin published` fetches the pinned `win-dev-skills@v0.7.1` for comparison (config moved under `plugins.winui.published`); `-WinUIPlugin <path>` is unchanged. README updated. ## Validation - `scripts/validate-plugin-package.ps1` (this base): pass. - `scripts/validate-llm-docs.ps1`: not run; it needs a built CLI (`artifacts/cli/win-x64/winapp.exe`). It only reads winapp manifests, so this PR can't affect it. - Benchmark Pester tests: 16/16 pass. `-Plan` shows the local default and `published`; the published fetch path was exercised and its `agent-plugin` hashes identically to the local copy. ### #972 skill checks, run locally (not committed) - **Discovery gap:** #972 finds plugins as `plugins/<name>/skills`, so it **silently skips WinUI** (`plugins/winui/agent-plugin/skills`). Needs fixing in #972 or a later layer. - Forced over WinUI (`-PluginsRoot plugins/winui`): - 4 errors: `winui-code-review:45`, `winui-design:106`, `winui-dev-workflow:51`, `winui-wpf-migration:77` reference `references/sourcegen-patterns.md`, which lives in `winui-packaging`. The text says "see `winui-packaging`'s `references/sourcegen-patterns.md`" (v0.7.1 deliberately replaced cross-skill links with plain text), so the check resolves it against the wrong skill. Decide in a later layer: relax the check or reword. - 4 description-length warnings (>300 chars): `winui-design` 412, `winui-wpf-migration` 424, `winui-ui-testing` 388, `winui-packaging` 352. - `winapp` command examples: no unknown commands. - Size: metadata ~764 tokens always loaded; bodies ~19.7k; references ~21.9k. - Content not changed. ### Benchmark smoke vs baseline (winui, claude-sonnet-5.5) Preflight installed the local plugin (source `this repo`, v0.7.1, not dirty) with the same 8 skills as published v0.7.1; no unexpected or missing skills. | Scenario | Baseline (3 runs) | This PR (1 run) | Skills (both) | |---|---|---|---| | click-button-running-app | 3/3 | pass | winui-ui-testing | | console-arg-parsing | 3/3 | pass | none | | find-winrt-api-battery | 3/3 | pass | none | | new-winui-app | 3/3 | pass | winui-dev-workflow | | winui-code-review | 3/3 | pass | winui-code-review | | winui-feature-ui-test | 3/3 | pass | winui-ui-testing | | winui-settings-page | 0/3 | fail | none | | wpf-add-notifications | 3/3 | pass | none | | wpf-to-winui | 3/3 | pass | winui-wpf-migration | | xaml-unknown-member | 0/3 | fail | none | All 10 match the baseline status and skill set. The two failures are pre-existing routing gaps, not caused by the import. ## Next layers 1. Publishing pipeline for WinUI from this repo. 2. Flip: marketplace/catalog entries point here; win-dev-skills redirects. 3. Unify/trim: shared validation (incl. #972 discovery), description budgets, cross-skill references, overlap with winapp skills. Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
#981) Layer 3 of the benchmark stack. #973 and #976 are merged, so this now targets `main`. It adds the benchmark improvements tested during the weekend experiments, plus reporting. ## Changes **From the experiments branch (cherry-picked, no conflicts)** - `-Agent <name>` runs every session with `--agent`. `-MaxCredits <n>` stops launching sessions once the run has spent `n` credits. Each run also records `winappCommands` (including `node <subcommand>` from the npm wrapper) and `selectedAgent`. - 24 reviewed scenarios. Expectations are tightened on six existing ones, with no prompt or fixture changes. Packaging scenarios need a packaging skill to load; signing alone does not pass. Review notes are in `benchmarks/agents/scenarios-review.md`. - Handles one skill name shipped by two plugins, which Copilot lists as `<plugin>:<name>`. Preflight requires one enabled copy per plugin. **New in this PR** - **`n/a` status:** a run where nothing in the expectation applies to the installed skills gets `n/a` and is left out of pass rates. "Nothing applies" means no expected or forbidden skill is installed and there is no `maxSkills`. The summary lists these runs separately. - **Repeated deliveries:** each run records skills delivered more than once in a session and the extra skill context they add. `summary.md` shows them in the totals and per row, as `unknown` or `partial` when their size is not known. It also counts launched runs that had no credit count. - **`run.ps1 -Compare <dirs> -Candidate <dirs>`** builds a Markdown report by model and by cell (model, scenario, configuration). It shows pass rate (only pass and fail are scored; other statuses are listed as excluded, the same as in `summary.md`), skill context per run, input tokens per run, credits per run, the change in each, and repeated deliveries. Only cells present on both sides are compared. Both sides are re-scored in memory against the current expectations, and the input folders are not modified. A cell shows `check differs` when the installed skills that the expectation checks differ by name between the sides, for example when a candidate adds or removes an expected skill, including removing all of them. It is then left out of the per-model pass rate. `-Scenario` and `-Model` must name values present in the compared results. Runs of scenarios that no longer exist are excluded as `scenario_removed`. - **`-MaxCredits` with unknown credits:** a session that ran but reported no credits (for example a timeout) counts at the average of the measured sessions, or at 35 before any is measured. - **Every scenario must include `both`:** validation fails otherwise, and there is a Pester test for it. All 42 scenarios comply. - **`-Rescore`** also re-scores `n/a` runs and prints status transitions. A mistyped `-Scenario` now fails before `-Rescore` or `-Compare` runs. - **README:** documents the new parameters and statuses, the `both` rule, and the recommended workflow: a quick Sonnet screen, then 3 models × 3 iterations on the affected scenarios plus the near-misses. ## Scenarios and plan totals 42 scenarios: 18 existing + 24 new. - `-Plan`, full default matrix: **954** sessions (3 models × 3 iterations). - Sonnet, 1 iteration: **106** sessions. ## Validation - Pester: 46/46 pass (was 19). - `-Compare` on the weekend result folders (read-only). This is the E4 candidate vs the current plugins; the numbers match the weekend report: | Model | Cells | Pass (base → cand) | Δ pass | Skill context/run | Δ | Input/run | Δ | AI credits/run | Δ | |---|---|---|---|---|---|---|---|---|---| | claude-opus-5.5 | 105 | 310/315 (98%) → 306/315 (97%) | -1 pp | 3.8k → 3.9k | +4% | 174.7k → 170.8k | -2% | 21.3 → 19.4 | -9% | | claude-sonnet-5.5 | 105 | 194/315 (62%) → 227/315 (72%) | +10 pp | 1.8k → 2.1k | +16% | 122.4k → 122.2k | = | 8.3 → 7.9 | -6% | | gpt-6.1-sol | 105 | 304/315 (97%) → 309/315 (98%) | +2 pp | 6.7k → 6.0k | -11% | 130.0k → 101.6k | -22% | 10.1 → 8.3 | -18% | E4.2 shares `winapp-find-api` into the WinUI plugin. That changes what 15 of its 24 cells check (5 per model), so they show `check differs`. The per-model pass rates use the other 9 cells: Opus 9/9 → 9/9, Sonnet 3/9 → 4/9, GPT 9/9 → 9/9. The E4 comparison has no `check differs` cells. - `-Rescore` on a copy of the original 297-run baseline: 3 runs went pass → fail (from the tightened expectations), and 0 went to `n/a`. Across all 954 baseline runs, no current cell is `n/a`. The status exists so that candidates which drop or rename skills don't get free passes. - No new model runs. ## Notes - **`n/a` is narrower than "no expected skill installed".** Under that literal rule, clean runs in forbid-checked cells would become `n/a` while overreach still fails. An example is `wpf-add-notifications / winui`, where GPT WinUI overreach went 0/3 → 3/3 in E2. That rule would hide that result, so a cell stays scored when a forbidden skill is installed. - **The `sign-existing-msix-fixture` control was dropped.** Its `.msix` is ignored by the `dist` rule in `.gitignore`, so the committed copy is not the fixture that was actually run. It also repeats `sign-existing-msix`. Its finding (the empty folder does not explain the skipped skill) is recorded in `scenarios-review.md`. - **Empty fixtures in `sign-existing-msix` and `msix-install-fails` are unchanged.** Filling them would change the scenario input and require a new baseline. - **Repeated deliveries can't be recovered from older runs.** Their records lack per-delivery data, so the compare report shows `-` for them. - **More scenario changes may follow in this layer.** A follow-up realism review will likely add or modify scenarios here. --------- Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>


Description
Adds
benchmarks/agents/, a small local benchmark. It shows which skills GitHub Copilot CLI loads for realistic developer requests, and how many tokens that costs, with four plugin setups:none,winapp(this repo'splugins/winapp),winui(publishedmicrosoft/win-dev-skillsv0.7.1), andboth. We'll use it to:How a run works:
%TEMP%, outside any repo.copilot skill list --jsonmust show exactly the expected plugin skills.copilot -pwith shell, file writes, and URLs denied.results/<timestamp>/asruns.jsonl,summary.md, andrun-info.json.Usage Example
-Rescorere-evaluates recordedpass/failruns against the current scenario expectations. It writesruns.rescored.jsonlandsummary.rescored.mdnext to the originals and leaves them unchanged.Prerequisites: PowerShell 7.4, git, and Copilot CLI authenticated through
GH_TOKEN/COPILOT_GITHUB_TOKEN. Seebenchmarks/agents/README.md.Scenarios: please review the prompts
Prompts are written as natural requests and never name a skill. Expect lists the skills that count as a pass; at least one must be invoked.
contosoalias and replace all icons with Assets\logo-new.pngwinrt/Microsoft.UI.Xaml.h; restore without changing pinned versions--input/--verboseparsing to a .NET console toolEvery scenario that installs WinUI also forbids
winui-session-report. An expected skill that isn't installed in a setup is ignored for that setup.Findings so far
All runs: Copilot CLI 1.0.87-0,
claude-sonnet-5.5, one iteration.* That run happened before skill context measurement was added.
winui-design. Its write and shell attempts were denied and the workspace stayed unchanged. One iteration only, so treat this as a lead, not a conclusion.Where Copilot CLI behaved differently than expected
COPILOT_AUTO_UPDATE=false(or--no-auto-update) alone, the launcher runs its bundled build, 1.0.71-0, not the newest one installed. The runner finds the newest installed build once, passes--prefer-version <v>on every call, and checks--versionagrees.run-info.jsonrecords the version.skill.context_delivered_ref. It holds only a hash plus wrapper text, so the runner estimates size as wrapper + the skill body fromskill.invoked, divided by 4 for tokens. It's labeled approximate.session.skills_loadednever appeared.session.shutdown.modelMetrics[*].usage. A run killed by timeout has no shutdown event, so its tokens arenullwith a reason, never 0.usage.inputTokensis the whole prompt including cached tokens. The top-leveltokenDetails.inputis uncached input only. The summary shows input including cached, plus cache read separately.COPILOT_*variables, such asCOPILOT_CLI_RESOLVED_DIST_DIR. The child process gets all of them removed exceptCOPILOT_GITHUB_TOKEN. Workspaces live outside any repo becauseskill listpicks up.github/skillsfrom the working directory. Preflight also checks that every plugin skill path is inside the isolated home.copilot plugin install <local path>prints a deprecation warning for direct installs. That's fine for v1.Type of Change
Checklist
benchmarks/agents/tests(Pester, 13 tests) covers the event parser with a completed log, a log with no shutdown event, and a missing log, plus expectations, rescoring, the workspace snapshot, the summary, and scenario validation.Additional Notes
Limits:
scripts/validate-plugin-package.ps1is untouched.