Skip to content

Import WinUI agent plugin from win-dev-skills (unchanged) - #976

Merged
Nikola Metulev (nmetulev) merged 1 commit into
mainfrom
nmetulev-import-winui-plugin
Oct 6, 2026
Merged

Nikola Metulev (nmetulev) merged 1 commit into
mainfrom
nmetulev-import-winui-plugin

Conversation

@nmetulev

Copy link
Copy Markdown
Member

Layer 2 of the plugin stack (base: #973). Imports the WinUI agent plugin as-is so later layers can publish it from here, then unify and trim.

What's imported

  • Source: microsoft/win-dev-skills tag v0.7.1 = 2e8c902e9ba675e4c57550e066fc7e48b64c49b0, folder plugins/winui/.
  • Copied to plugins/winui/ with the same layout (agent-plugin/, .claude-plugin/, .codex-plugin/, agents/, openclaw.plugin.json, package.json, index.js), so host wrapper paths keep working. Manifests keep their own version 0.7.1.
  • 24 files. Byte parity: every path + SHA-256 matches the source tree, git blob IDs match the source commit, and a fresh git checkout of the folder still matches.
  • .gitattributes: plugins/winui/** text eol=lf + *.png binary, so this repo's *.ps1 eol=crlf rule and text=auto don't rewrite Analyze-Session.ps1 or other files on checkout.
  • Licensing: both repos are Microsoft MIT. win-dev-skills THIRD_PARTY_NOTICES.md lists no vendored third-party content for the plugin and cgmanifest.json has no registrations, so nothing extra to carry. No README added; provenance is recorded here.

Intentionally not included

  • No entry in .github/plugin/marketplace.json / .claude-plugin/marketplace.json or README install docs: this PR must not create a second live install source.
  • No win-dev-skills src/tools (winui-analyzer, winmd-cli), CI workflows, or release scripts. Ownership of analyzer/tool source is a separate decision.

Tooling

No fixes needed. generate-llm-docs.ps1, validate-llm-docs.ps1, and validate-plugin-package.ps1 only reference plugins/winapp paths explicitly, so they neither stamp the CLI version into WinUI manifests nor validate it.

Benchmark (benchmarks/agents): the default WinUI plugin is now the local plugins/winui/agent-plugin (relative path, like winapp). -WinUIPlugin published fetches the pinned win-dev-skills@v0.7.1 for comparison (config moved under plugins.winui.published); -WinUIPlugin <path> is unchanged. README updated.

Validation

  • scripts/validate-plugin-package.ps1 (this base): pass.
  • scripts/validate-llm-docs.ps1: not run; it needs a built CLI (artifacts/cli/win-x64/winapp.exe). It only reads winapp manifests, so this PR can't affect it.
  • Benchmark Pester tests: 16/16 pass. -Plan shows the local default and published; the published fetch path was exercised and its agent-plugin hashes identically to the local copy.

#972 skill checks, run locally (not committed)

  • Discovery gap: Add CI checks for agent skill packages #972 finds plugins as plugins/<name>/skills, so it silently skips WinUI (plugins/winui/agent-plugin/skills). Needs fixing in Add CI checks for agent skill packages #972 or a later layer.
  • Forced over WinUI (-PluginsRoot plugins/winui):
    • 4 errors: winui-code-review:45, winui-design:106, winui-dev-workflow:51, winui-wpf-migration:77 reference references/sourcegen-patterns.md, which lives in winui-packaging. The text says "see winui-packaging's references/sourcegen-patterns.md" (v0.7.1 deliberately replaced cross-skill links with plain text), so the check resolves it against the wrong skill. Decide in a later layer: relax the check or reword.
    • 4 description-length warnings (>300 chars): winui-design 412, winui-wpf-migration 424, winui-ui-testing 388, winui-packaging 352.
    • winapp command examples: no unknown commands.
    • Size: metadata ~764 tokens always loaded; bodies ~19.7k; references ~21.9k.
  • Content not changed.

Benchmark smoke vs baseline (winui, claude-sonnet-5.5)

Preflight installed the local plugin (source this repo, v0.7.1, not dirty) with the same 8 skills as published v0.7.1; no unexpected or missing skills.

Scenario Baseline (3 runs) This PR (1 run) Skills (both)
click-button-running-app 3/3 pass winui-ui-testing
console-arg-parsing 3/3 pass none
find-winrt-api-battery 3/3 pass none
new-winui-app 3/3 pass winui-dev-workflow
winui-code-review 3/3 pass winui-code-review
winui-feature-ui-test 3/3 pass winui-ui-testing
winui-settings-page 0/3 fail none
wpf-add-notifications 3/3 pass none
wpf-to-winui 3/3 pass winui-wpf-migration
xaml-unknown-member 0/3 fail none

All 10 match the baseline status and skill set. The two failures are pre-existing routing gaps, not caused by the import.

Next layers

  1. Publishing pipeline for WinUI from this repo.
  2. Flip: marketplace/catalog entries point here; win-dev-skills redirects.
  3. Unify/trim: shared validation (incl. Add CI checks for agent skill packages #972 discovery), description budgets, cross-skill references, overlap with winapp skills.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Build Metrics Report

Validation passed. All required build and validation jobs succeeded.

Binary Sizes

Artifact Baseline Current Delta
CLI (ARM64) 57.42 MB 57.42 MB 📈 +3.0 KB (+0.01%)
CLI (x64) 57.46 MB 57.46 MB 📈 +3.0 KB (+0.01%)
MSIX (ARM64) 23.85 MB 23.86 MB 📈 +3.1 KB (+0.01%)
MSIX (x64) 25.32 MB 25.32 MB 📈 +0.6 KB (+0.00%)
NPM Package 49.75 MB 49.76 MB 📈 +3.0 KB (+0.01%)
NuGet Package 49.85 MB 49.86 MB 📈 +6.7 KB (+0.01%)

.NET Test Results (TRX reports)

Other suites are reflected in the overall validation status above.

✅ 8062 passed, 37 skipped out of 8099 tests in 1305.3s (+10 tests, +116.2s vs. baseline)

Test Coverage

✅ 86.4% line coverage, 81% branch coverage · ✅ no change vs. baseline

CLI Startup Time

66ms median (x64, winapp --version) · ✅ no change vs. baseline

Try This Build

Installs the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing.

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 976
Switching between builds often?

Put the tool on your PATH once:

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPath

Then this build is just:

winapp-pr 976

Run winapp-pr with no arguments to pick from a list of open PRs.


Updated 2026-10-06 21:59:29 UTC · commit daff58b · workflow run

Nikola Metulev (nmetulev) added a commit that referenced this pull request Oct 5, 2026
## Description

Adds a fast, model-free check that catches common skill and plugin
problems before review. It extends `scripts/validate-plugin-package.ps1`
and checks every **plugin root** under `plugins/`: any folder that holds
`plugin.json` and an immediate `skills/` folder. That covers
`plugins/winapp` and the WinUI layout from #976
(`plugins/winui/agent-plugin`, with host wrappers in `plugins/winui`)
with no per-plugin setup.

**Errors (fail the check):**
- `skills/<id>/SKILL.md` is missing, or its frontmatter is missing or
unterminated.
- `name` or `description` is missing. `name` must match the folder name
and the Agent Skills naming rules. `description` can be at most 1024
characters.
- A `SKILL.md` under `plugins/` is not at `<plugin
root>/skills/<id>/SKILL.md`, for example because it is nested too deep
or has no `plugin.json` above it. New layouts can't be skipped silently.
- A relative Markdown link, a reference definition, or an inline-code
`references/`, `scripts/` or `assets/` path points to a missing file or
outside the plugin. URLs and in-page anchors are skipped.
- A cross-skill mention such as ``see `winui-packaging`'s
`references/sourcegen-patterns.md` `` (or "the winui-packaging skill's
…") is resolved against that sibling skill.
- It is an error only if neither the current skill nor the named sibling
has the file.
- A `winapp …` line in a fenced code block uses an unknown command. This
covers skills, `com.github.copilot/agents/*.agent.md`, and host wrapper
agents in outer `agents/` folders.
- Valid commands come from `docs/cli-schema.json` (including aliases)
plus the npm wrapper's `node` subcommands, read from `NODE_SUBCOMMANDS`
in `src/winapp-npm/src/cli.ts`.
- Placeholders, comments, paths and positional arguments end the command
path, so they are never flagged.

**Warnings (shown, but the check still passes):** descriptions over 300
characters. Change the limit with `$DescriptionWarnChars`.

**Size report:** an approximate token table for each plugin root,
printed in the log and appended to the GitHub Actions job summary. A
host wrapper with the same file name as the inner agent is listed but
left out of the totals, so the agent isn't counted twice.

**Early PR check:** the new `Plugin Check` workflow
(`.github/workflows/plugin-check.yml`) runs on every PR to `main` on
`ubuntu-latest` with pwsh, read-only `contents` permission, and no Node
or .NET setup.
- The trigger has no path filter, so the job always reports a status and
is safe to mark as a required check.
- The job checks out the PR merge commit with `fetch-depth: 2` and runs
`git diff --name-only HEAD^1 HEAD` to list the files the PR changes.
- If none of `plugins/**`, `scripts/validate-plugin-package.ps1`,
`docs/cli-schema.json`, `src/winapp-npm/src/cli.ts` or the workflow file
changed, it prints `No skill-related changes; skipping.` to the log and
job summary and passes.
- Otherwise it runs the validator, and the job fails if the validator
reports an error.

The post-build `validate-docs` job is unchanged. The validator's path
handling is cross-platform so it can run on ubuntu.

To run the check locally, or against a copy of the plugins folder:

```powershell
.\scripts\validate-plugin-package.ps1
.\scripts\validate-plugin-package.ps1 -PluginsRoot <path-to-plugins-folder>
```

## Current results

- **main:** 0 errors and 7 description warnings: find-api 873, find-ui
834, manifest 440, ui-automation 345, package 322, setup 314, maui 307.
- **With the WinUI import (#976, branch
`nmetulev-import-winui-plugin`):** 0 errors. WinUI adds 4 description
warnings: winui-wpf-migration 424, winui-design 412, winui-ui-testing
388, winui-packaging 352.

### Plugin size: winapp

| Component | Description chars | Metadata ~tokens | Body ~tokens |
Other skill files ~tokens |
|---|--:|--:|--:|--:|
| winapp-find-api | 873 | 223 | 6,405 | 0 |
| winapp-find-ui | 834 | 213 | 2,607 | 0 |
| winapp-frameworks | 244 | 66 | 2,633 | 0 |
| winapp-identity | 267 | 71 | 2,564 | 0 |
| winapp-manifest | 440 | 115 | 2,059 | 0 |
| winapp-maui | 307 | 80 | 2,979 | 0 |
| winapp-package | 322 | 85 | 2,837 | 0 |
| winapp-sandbox | 282 | 75 | 2,301 | 0 |
| winapp-setup | 314 | 82 | 6,089 | 0 |
| winapp-signing | 186 | 51 | 2,308 | 0 |
| winapp-troubleshoot | 192 | 54 | 2,608 | 0 |
| winapp-ui-automation | 345 | 92 | 8,892 | 2,868 |
| agent: winapp.agent.md | 865 | 219 | 14,547 | |
| **Total** | 5,471 | **1,426** | 58,829 | 2,868 |

### Plugin size: winui/agent-plugin (from #976)

| Component | Description chars | Metadata ~tokens | Body ~tokens |
Other skill files ~tokens |
|---|--:|--:|--:|--:|
| winui-code-review | 207 | 57 | 1,620 | 3,156 |
| winui-design | 412 | 107 | 3,329 | 4,319 |
| winui-dev-workflow | 295 | 80 | 1,714 | 0 |
| winui-packaging | 352 | 93 | 1,718 | 1,392 |
| winui-session-report | 289 | 78 | 1,594 | 13,028 |
| winui-setup | 291 | 77 | 2,102 | 0 |
| winui-ui-testing | 388 | 102 | 5,611 | 0 |
| winui-wpf-migration | 424 | 116 | 1,486 | 0 |
| agent: winui-dev.agent.md | 203 | 54 | 505 | |
| agent (host wrapper): winui/agents/winui-dev.agent.md (not in totals)
| 203 | 54 | 504 | |
| **Total** | 2,861 | **764** | 19,679 | 21,895 |

The tables are approximate: ~tokens = UTF-8 bytes / 4. Metadata (`name:
description`) loads in every session, so the metadata total is each
plugin's always-loaded cost: ~1,426 tokens for winapp and ~764 for
WinUI. Other skill files (references, scripts, assets) load only when
read. Most of WinUI's 13,028 is `Analyze-Session.ps1`.

## Type of Change

- 🔧 Config/build

## Checklist

- [x] Tested locally on Windows

## Additional Notes

- **No skill content changes.** All existing links and command examples
in both plugins pass, so nothing needed an allowlist.
- **Error checks were tested on scratch copies** of `plugins/` from main
and from the WinUI branch.
  - These failed the run as expected:
- name mismatch, invalid name, missing description, 1100-character
description, missing `SKILL.md`
    - an unrooted `SKILL.md` and a too-deep `SKILL.md`
- a link escaping the plugin, a root-relative link, a missing link
target, a missing inline reference
- a cross-skill path whose named sibling lacks the file, and a
cross-skill path naming a skill that doesn't exist
- unknown commands in a skill, the inner agent, and a host wrapper
agent, plus a wrapper link escaping its folder
- These did not fail: valid cross-skill mentions in both the backtick
and "the … skill's" forms, `winapp node create-addon`, the `pack` and
`run-buildtool` aliases, positional arguments, `<placeholders>` and
comments.
- **Frontmatter parsing matches a real YAML parser.** The description
lengths match PyYAML exactly.
- **The generic placement check replaces the winapp-only
nested-`SKILL.md` check**, which reported the same problem.
- **Command options are not validated.** Aliases, placeholders and `--`
would cause false failures.
- **Inline-code checks are limited to `references/`, `scripts/` and
`assets/` paths.** Paths like `./<name>` usually describe the user's
project, not the plugin.
- **The 300-character warnings are noisy** until the descriptions are
trimmed: 7 of 12 winapp skills and 4 of 8 WinUI skills are over the
limit.
- **Verified on ubuntu:** a `Plugin Check` run on this PR detected the
changed files, ran the validator and passed, with the same size totals
as on Windows. The skip branch hasn't run in CI because this PR always
changes relevant files; its file-matching pattern was tested locally.

---------

Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev
Nikola Metulev (nmetulev) marked this pull request as ready for review October 6, 2026 21:19
Copilot AI balanced review requested due to automatic review settings October 6, 2026 21:19
Base automatically changed from nmetulev-local-agent-benchmark to main October 6, 2026 21:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The session analyzer misreads current Copilot logs and can report successful builds as failures.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
What changed in this PR

Imports the WinUI 0.7.1 agent plugin unchanged, preserves source byte parity, and makes the local copy the benchmark default without publishing it.

Changes:

  • Adds the 24-file multi-host WinUI plugin.
  • Preserves LF and binary assets through .gitattributes.
  • Updates benchmarks to compare local and published plugins.
File Description
.gitattributes Preserves imported file bytes.
benchmarks/​agents/​README.md Documents local and published modes.
benchmarks/​agents/​config.json Makes the local plugin default.
benchmarks/​agents/​run.ps1 Resolves local or published WinUI plugins.
plugins/​winui/​package.json Defines the OpenClaw package.
plugins/​winui/​openclaw.plugin.json Declares OpenClaw skill discovery.
plugins/​winui/​index.js Adds the OpenClaw entry point.
plugins/​winui/​agents/​winui-dev.agent.md Adds the host agent wrapper.
plugins/​winui/​agent-plugin/​plugin.json Defines the portable plugin.
plugins/​winui/​agent-plugin/​com.github.copilot/​agents/​winui-dev.agent.md Adds the Copilot agent.
plugins/​winui/​agent-plugin/​assets/​logo.svg Adds vector branding.
plugins/​winui/​agent-plugin/​assets/​logo-512.png Adds raster branding.
plugins/​winui/​agent-plugin/​skills/​winui-code-review/​SKILL.md Adds review guidance.
plugins/​winui/​agent-plugin/​skills/​winui-code-review/​references/​quality-rules.md Adds detailed quality rules.
plugins/​winui/​agent-plugin/​skills/​winui-design/​SKILL.md Adds WinUI design guidance.
plugins/​winui/​agent-plugin/​skills/​winui-design/​references/​brushes-and-icons.md Adds brush and icon reference.
plugins/​winui/​agent-plugin/​skills/​winui-design/​references/​layout-review.md Adds responsive-layout guidance.
plugins/​winui/​agent-plugin/​skills/​winui-design/​references/​theme-accessibility.md Adds theming guidance.
plugins/​winui/​agent-plugin/​skills/​winui-dev-workflow/​SKILL.md Adds development workflow guidance.
plugins/​winui/​agent-plugin/​skills/​winui-packaging/​SKILL.md Adds packaging guidance.
plugins/​winui/​agent-plugin/​skills/​winui-packaging/​references/​sourcegen-patterns.md Adds AOT/source-generator patterns.
plugins/​winui/​agent-plugin/​skills/​winui-session-report/​SKILL.md Adds session-report workflow.
plugins/​winui/​agent-plugin/​skills/​winui-session-report/​Analyze-Session.ps1 Implements session analysis.
plugins/​winui/​agent-plugin/​skills/​winui-setup/​SKILL.md Adds prerequisite setup guidance.
plugins/​winui/​agent-plugin/​skills/​winui-ui-testing/​SKILL.md Adds UI-testing workflow.
plugins/​winui/​agent-plugin/​skills/​winui-wpf-migration/​SKILL.md Adds WPF migration guidance.
plugins/​winui/​.codex-plugin/​plugin.json Adds Codex metadata.
plugins/​winui/​.claude-plugin/​plugin.json Adds Claude metadata.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copy microsoft/win-dev-skills plugins/winui at v0.7.1
(2e8c902e9ba675e4c57550e066fc7e48b64c49b0) byte-for-byte into plugins/winui
(24 files; git blob ids and SHA-256 match the source). Pin LF line endings and
binary PNGs for the folder in .gitattributes so checkouts stay byte-exact.

Point the benchmark's default WinUI plugin at the local
plugins/winui/agent-plugin; -WinUIPlugin published still fetches the pinned
win-dev-skills release for comparison.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev
Nikola Metulev (nmetulev) merged commit 5a064a4 into main Oct 6, 2026
38 checks passed
@nmetulev
Nikola Metulev (nmetulev) deleted the nmetulev-import-winui-plugin branch October 6, 2026 23:18
Nikola Metulev (nmetulev) added a commit that referenced this pull request Oct 7, 2026
…tests (#1007)

Part of moving the WinUI plugin's source of truth from
microsoft/win-dev-skills to this repo (follow-up to #976). No file under
`plugins/` changes.

## What changed
- **`.claude-plugin/marketplace.json`**: adds a `winui` entry with
`source: ./plugins/winui`. OpenClaw reads this file, and remote OpenClaw
catalogs accept only relative paths inside the same repo. The entry has
no `version`, so `generate-llm-docs.ps1`/`validate-llm-docs.ps1` (which
stamp/check every `"version"` in the file) leave it alone and hosts read
`0.7.1` from `plugins/winui/.claude-plugin/plugin.json`.
- **`scripts/tests/`**: copies `Test-SessionBuildClassification.ps1`,
`Test-WinuiUiTestingSandbox.ps1`, `Test-SetupVersionDetection.ps1` from
win-dev-skills@2e8c902e unchanged. The relative layout (`scripts/tests`
-> `plugins/winui`) is the same, so no path edits were needed. They
aren't `*.Tests.ps1`, so the existing Pester run doesn't pick them up.
- **`plugin-check.yml`**: new `winui-tests` job on `windows-latest` that
runs the three scripts when `plugins/winui/**`, those tests or the
workflow change. Otherwise it skips quickly.

## OpenClaw evidence (source-level)
From openclaw `v2026.9.8` `src/plugins/marketplace.ts`:
- `--marketplace owner/repo` shallow-clones the repo, then reads
`.claude-plugin/marketplace.json` (falling back to a root
`marketplace.json`).
- Remote catalogs reject the **whole file** if any entry is not a
relative path inside the repo (`remote marketplaces may not use github
plugin sources`).
- Entries are matched by `name`. `version` is metadata only.

`plugins/winui` is byte-identical to win-dev-skills v0.7.1, where
`index.js` + `openclaw.plugin.json` + `package.json` already installed
as `Format: openclaw`. Install command after merge:
```
openclaw plugins install winui --marketplace microsoft/winappCli
```
I couldn't run the real install here: Microsoft's npm feed blocks the
`openclaw` package by policy (`PolicyOverride`), so I didn't work around
it.

## Which hosts see `winui` in this repo's catalogs
- **Claude Code**: yes, as `winui@winappcli` (reads
`.claude-plugin/marketplace.json`). This adds a second install route
next to `winui@win-dev-skills`.
- **OpenClaw**: yes (reads the same file).
- **Copilot CLI**: no. It reads `.github/plugin/marketplace.json`, which
is unchanged. I checked this with an isolated `COPILOT_HOME`:
`marketplace browse winappcli` lists only `winappcli`.
- **Codex**: no (this repo has no `.agents/plugins/marketplace.json`).

## Validation
- `scripts/validate-plugin-package.ps1`: passes (only the existing
description-length warnings).
- All three WinUI tests pass locally on both LF and CRLF checkouts.

Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nikola Metulev (nmetulev) added a commit that referenced this pull request Oct 7, 2026
#981)

Layer 3 of the benchmark stack. #973 and #976 are merged, so this now
targets `main`. It adds the benchmark improvements tested during the
weekend experiments, plus reporting.

## Changes

**From the experiments branch (cherry-picked, no conflicts)**
- `-Agent <name>` runs every session with `--agent`. `-MaxCredits <n>`
stops launching sessions once the run has spent `n` credits. Each run
also records `winappCommands` (including `node <subcommand>` from the
npm wrapper) and `selectedAgent`.
- 24 reviewed scenarios. Expectations are tightened on six existing
ones, with no prompt or fixture changes. Packaging scenarios need a
packaging skill to load; signing alone does not pass. Review notes are
in `benchmarks/agents/scenarios-review.md`.
- Handles one skill name shipped by two plugins, which Copilot lists as
`<plugin>:<name>`. Preflight requires one enabled copy per plugin.

**New in this PR**
- **`n/a` status:** a run where nothing in the expectation applies to
the installed skills gets `n/a` and is left out of pass rates. "Nothing
applies" means no expected or forbidden skill is installed and there is
no `maxSkills`. The summary lists these runs separately.
- **Repeated deliveries:** each run records skills delivered more than
once in a session and the extra skill context they add. `summary.md`
shows them in the totals and per row, as `unknown` or `partial` when
their size is not known. It also counts launched runs that had no credit
count.
- **`run.ps1 -Compare <dirs> -Candidate <dirs>`** builds a Markdown
report by model and by cell (model, scenario, configuration). It shows
pass rate (only pass and fail are scored; other statuses are listed as
excluded, the same as in `summary.md`), skill context per run, input
tokens per run, credits per run, the change in each, and repeated
deliveries. Only cells present on both sides are compared. Both sides
are re-scored in memory against the current expectations, and the input
folders are not modified. A cell shows `check differs` when the
installed skills that the expectation checks differ by name between the
sides, for example when a candidate adds or removes an expected skill,
including removing all of them. It is then left out of the per-model
pass rate. `-Scenario` and `-Model` must name values present in the
compared results. Runs of scenarios that no longer exist are excluded as
`scenario_removed`.
- **`-MaxCredits` with unknown credits:** a session that ran but
reported no credits (for example a timeout) counts at the average of the
measured sessions, or at 35 before any is measured.
- **Every scenario must include `both`:** validation fails otherwise,
and there is a Pester test for it. All 42 scenarios comply.
- **`-Rescore`** also re-scores `n/a` runs and prints status
transitions. A mistyped `-Scenario` now fails before `-Rescore` or
`-Compare` runs.
- **README:** documents the new parameters and statuses, the `both`
rule, and the recommended workflow: a quick Sonnet screen, then 3 models
× 3 iterations on the affected scenarios plus the near-misses.

## Scenarios and plan totals

42 scenarios: 18 existing + 24 new.
- `-Plan`, full default matrix: **954** sessions (3 models × 3
iterations).
- Sonnet, 1 iteration: **106** sessions.

## Validation
- Pester: 46/46 pass (was 19).
- `-Compare` on the weekend result folders (read-only). This is the E4
candidate vs the current plugins; the numbers match the weekend report:

| Model | Cells | Pass (base → cand) | Δ pass | Skill context/run | Δ |
Input/run | Δ | AI credits/run | Δ |
|---|---|---|---|---|---|---|---|---|---|
| claude-opus-5.5 | 105 | 310/315 (98%) → 306/315 (97%) | -1 pp | 3.8k →
3.9k | +4% | 174.7k → 170.8k | -2% | 21.3 → 19.4 | -9% |
| claude-sonnet-5.5 | 105 | 194/315 (62%) → 227/315 (72%) | +10 pp |
1.8k → 2.1k | +16% | 122.4k → 122.2k | = | 8.3 → 7.9 | -6% |
| gpt-6.1-sol | 105 | 304/315 (97%) → 309/315 (98%) | +2 pp | 6.7k →
6.0k | -11% | 130.0k → 101.6k | -22% | 10.1 → 8.3 | -18% |

E4.2 shares `winapp-find-api` into the WinUI plugin. That changes what
15 of its 24 cells check (5 per model), so they show `check differs`.
The per-model pass rates use the other 9 cells: Opus 9/9 → 9/9, Sonnet
3/9 → 4/9, GPT 9/9 → 9/9. The E4 comparison has no `check differs`
cells.
- `-Rescore` on a copy of the original 297-run baseline: 3 runs went
pass → fail (from the tightened expectations), and 0 went to `n/a`.
Across all 954 baseline runs, no current cell is `n/a`. The status
exists so that candidates which drop or rename skills don't get free
passes.
- No new model runs.

## Notes
- **`n/a` is narrower than "no expected skill installed".** Under that
literal rule, clean runs in forbid-checked cells would become `n/a`
while overreach still fails. An example is `wpf-add-notifications /
winui`, where GPT WinUI overreach went 0/3 → 3/3 in E2. That rule would
hide that result, so a cell stays scored when a forbidden skill is
installed.
- **The `sign-existing-msix-fixture` control was dropped.** Its `.msix`
is ignored by the `dist` rule in `.gitignore`, so the committed copy is
not the fixture that was actually run. It also repeats
`sign-existing-msix`. Its finding (the empty folder does not explain the
skipped skill) is recorded in `scenarios-review.md`.
- **Empty fixtures in `sign-existing-msix` and `msix-install-fails` are
unchanged.** Filling them would change the scenario input and require a
new baseline.
- **Repeated deliveries can't be recovered from older runs.** Their
records lack per-delivery data, so the compare report shows `-` for
them.
- **More scenario changes may follow in this layer.** A follow-up
realism review will likely add or modify scenarios here.

---------

Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nikola Metulev (nmetulev) added a commit that referenced this pull request Oct 8, 2026
…982)

Follows #981 (now in main). Implements the scenario realism review:
expectations are now capabilities instead of skill names, scenarios are
grouped into cohorts, there is a gated held-out set authored blind by a
different model family, and a lint catches prompts that echo skill
descriptions. Each run is scored on routing (did the right skill load)
and, separately, on answer quality (does the final response name what a
correct answer must). Includes a new baseline for the current plugins.

## What changed

**Capability expectations** (`benchmarks/agents/capabilities.json`)
- 22 capabilities (`msix.sign`, `api.lookup`, `identity`,
`winui.review`, …), each with a plain-language definition and the
`winapp` command patterns that serve it.
- Per-plugin maps from skills to capabilities, identified by a skill-set
hash. A map applies when all its skills are installed, so a candidate
that renames or merges skills gets its own map and stays scorable. Tests
fail if the repo plugins change skill names without a map update.
- Scenario `expect.capabilities`: `primary` (alternatives; a group must
load together), `acceptable`, `forbid` (patterns like `winui.*`, `*`),
optional `budgetTokens`, plus `commands` regexes.
- Statuses: `pass`, new `partial` (only acceptable, or part of a group),
`fail`, `n/a` (#981 semantics). A primary capability only reachable
through a skill carrying a forbidden capability counts as not installed.
- "Right `winapp` command named without loading a primary skill" is
reported separately.
- The old skill-name format still loads and scores, so old result
folders can be rescored. Runs now record a prompt hash; rescoring skips
runs whose prompt changed.

**Cohorts and sets**
- `cohort`: `implicit`, `error`, `vague`, `followup` (reserved),
`explicit-command`, `near-miss`, `trap`. `summary.md` and `-Compare`
report by set and cohort. Explicit-command and the `none` control are
left out of the headline routing rate. The `none` control has its own
table.
- `set`: `dev` (default) or `heldout`; `-Set dev|heldout|all`, `-Variant
base|novice|terse`. Paraphrases are declared once and expand to
`<id>.<variant>`.

**Lint** (`run.ps1 -Lint`, `lib/ScenarioLint.psm1`, Pester)
- Leakage against every skill and agent description of both plugins:
distinctive shared bigrams, and content-word Jaccard > 0.10. Fixture
file names count as prompt text. Framework and product names ("WinUI
app", "MSIX", "Microsoft Store") are neutral. Errors for held-out,
warnings for dev.
- Held-out: no "Contoso", no naming the tooling. Fixture realism: an
empty fixture file the prompt names is an error unless `routingSnapshot:
true`; a named file the fixture lacks is a warning.

**Dev set (67 = 42 + 25)**
- 12 leaky or over-polished prompts reworded per the review
(`winui-code-review`, `restore-cloned-cpp`, `winui-infobar-enum-lookup`,
`winui-card-grid-control`, `winui-control-sample-request`,
`winui-repair-prerequisites`, `electron-desktop-ui-test`,
`winui-generic-csharp-bug` with real tests, `flutter-trust-certificate`,
`electron-notifications`, `clean-machine-ui-smoke`,
`explicit-session-diagnostic`).
- Expectation fixes: `xaml-unknown-member`, `store-submission`,
`msi-app-sparse-identity`, `electron-notifications` (identity not
mandated), `clean-machine-ui-smoke` (Sandbox and UI automation
together).
- 25 new scenarios from the reviews: 9 error-first, 3 vague, 3 journeys,
10 near-misses/traps.
- Cohorts: implicit 32, error 15, trap 9, near-miss 6, vague 3,
explicit-command 2. Near-miss + trap = 22%.

**Held-out set (40 base × 3 prompts = 120)**
- Written by GPT-6.1 Sol given only capability definitions, personas,
public sources (GitHub issues, Stack Overflow, Learn), fixture
conventions, and lint rules. It saw no skill or agent descriptions. Two
revision rounds: one for realism (no invented euphemisms, no routing
hints), one for lint failures.
- Cohorts: error 12, implicit 10, vague 7, trap 7, near-miss 4 (27.5%
near-miss/trap); 14 conflicting-signal cases; every capability primary ≥
2 times; randomized names.
- Configurations: `none` control + `both` + the single plugin that
provides the primary capability.
- After the baseline, 5 held-out forbids that contradicted the
capability map were narrowed; prompts were not touched. The mapping
choices that affect scores are in the README (Capabilities).
- README: gated, for release and decision checks, not iteration.

**Answer metric (deterministic, next to routing)**
- Each capability in `capabilities.json` can carry small answer signals:
`require` groups of regexes (one per group must appear in the final
response) and optional `forbid` regexes. Signals are the `winapp`
command where a capability has one (for example `winapp
(sign|az-sign|cert)`), and a key term for a few that don't
(`Microsoft.UI.Xaml` for a WinUI port, a `--version` check for
prerequisites, manifest extension elements). Capabilities whose right
answer is scenario-specific (`winui.design`, `winui.review`,
`winui.build`, `troubleshoot`, `framework.guidance`, `session.report`)
have none and score `n/a`. No LLM judge.
- Every run gets `answer` = pass / partial / fail / n/a next to the
unchanged routing `status`, plus per-primary-capability routing and
answer results. Near-misses pass when the response names no `winapp`
command.
- Runs now keep `finalResponse`. Runs recorded without it are scored
from their recorded `winapp` commands (`answerBasis: commands`); only
signal groups made entirely of `winapp <command>` patterns are checked,
the rest are `n/a`.
- `summary.md` and `-Compare` show routing and answer side by side per
model, set/cohort, and primary capability, plus a 2 × 2 (routed +
answered, routed only, answered only, neither).
- Lint: `leak-answer` flags a prompt that already contains an answer
signal of its own expected capabilities (error for held-out; held-out is
clean).

**Integration with #981 (main)**
- Pass rates use #981's single format: `pass`, `partial`, and `fail` are
scored (`partial` counts against the rate), every other status is listed
as excluded, e.g. `1/2 (50%), 1 partial; excluded: 1 n/a`.
- Capability scoring feeds #981's check-set comparison: a run's applied
checks are the installed primary alternatives, acceptable (`~`), and
forbidden (`!`) capabilities, so `check differs` fires when a candidate
adds or removes an expected capability, but not when it only renames or
merges skills that the capability map covers. Skill-name expectations
keep #981's skill check sets. Removed scenarios stay `scenario_removed`.
- #981's stricter packaging expectations carried over:
`explicit-winapp-cli-package` and `wpf-winappsdk-msix-trap` now have
`msix.package` primary and `msix.sign` acceptable (signing alone is
`partial`, not a pass); `winui-release-msix` already did.
- The internal review notes file was removed; durable rationale lives in
the README.

**Fix:** `none`-only runs crashed in run.ps1 (pre-existing: empty plugin
list under StrictMode).

## Validation
- `Invoke-Pester benchmarks\agents\tests`: 108 passed.
- `run.ps1 -Lint`: 187 prompts, 0 errors (held-out clean), 28 dev
warnings (dev keeps a few leaky originals as regression sentinels; 2 are
`leak-answer`: `explicit-find-api-numberbox` names `winapp find-api`,
`winui-repair-prerequisites` quotes `dotnet --list-sdks`).
- Gaming check: always loading the three most-loaded skills passes 15/65
dev and 15/120 held-out prompts (it passed 35% of the old set).

## New baseline (current plugins)
1,335 new sessions, **13,193 AI credits** (cap 15,000). Held-out: 3
models × 1 run per prompt. Dev: new/reworded 3 models × 2 iterations;
unchanged scenarios are the prior 3 × 3 runs rescored with capabilities.

Routing and answer pass rates (both leave out `none` and
explicit-command; answer is over runs where it could be scored):

| Model | Dev routing | Dev answer | Held-out routing | Held-out answer
| Held-out routing base / novice / terse |
|---|---|---|---|---|---|
| claude-sonnet-5.5 | 57% | 79% (172/218) | 32% | 68% (80/118) | 35% /
28% / 34% |
| claude-opus-5.5 | 92% | 85% (191/226) | 78% | 79% (95/121) | 76% / 81%
/ 76% |
| gpt-6.1-sol | 94% | 77% (163/213) | 84% | 70% (85/122) | 86% / 80% /
85% |

Routing × answer (runs where both are scored):

| Set | Model | Routed + answered | Routed only | Answered only |
Neither |
|---|---|---|---|---|---|
| held-out | claude-sonnet-5.5 | 52 | 9 | 28 | 29 |
| held-out | claude-opus-5.5 | 94 | 14 | 1 | 12 |
| held-out | gpt-6.1-sol | 84 | 24 | 1 | 13 |
| held-out | all | 230 | 47 | 30 | 54 |
| dev | claude-sonnet-5.5 | 142 | 10 | 27 | 36 |
| dev | claude-opus-5.5 | 183 | 28 | 5 | 7 |
| dev | gpt-6.1-sol | 154 | 37 | 7 | 12 |
| dev | all | 479 | 75 | 39 | 55 |

By cohort (all models), routing / answer: implicit dev 83% / 70%,
held-out 66% / 72%; error 65% / 70%, 50% / 64%; vague 52% / 39%, 79% /
59%; near-miss 96% / 98%, 100% / 100%; trap 89% / 98%, 60% / 92%;
explicit-command routing 87% (dev only).

Answer scores for this baseline come from recorded `winapp` commands,
because these runs predate `finalResponse`: 0 of 2,019 baseline runs
have the response text; 1,693 have recorded commands and were scored
from them; 324 older dev runs (prior copies recorded before commands
were captured) and 2 held-out timeouts have no answer score. No sessions
were re-run. On held-out, 277 of 638 routing runs are answer `n/a` (no
objective signal, or the signal needs the response text).

No-plugin control (held-out base prompts): models named the right
`winapp` command in 4 of 71 runs; answer rate on implicit/error/vague
base prompts is 8% (1/13) for each model vs 67–73% with both plugins on
the same prompts; with no plugins, sessions cost 66–82% of the `both`
cost for the same prompts (sonnet 6.0 vs 7.3, opus 12.0 vs 18.1, gpt 5.4
vs 8.1 credits).

Weakest capabilities on held-out (`both`, routing): `winui.migrate` 17%,
`framework.guidance` 22%, `winui.review` 28%, `project.setup` 33%,
`store.publish` 33% (8 partial). Per-capability answer on held-out:
`store.publish` 0% (no run named `winapp store`), `project.setup` 39%,
`winui.package` 44%, `msix.sign` 56%; `identity` 100%, `api.lookup` 92%.

## Notes
- Results are gitignored. Compare-ready folders are in my worktree:
`results/baseline-v2-heldout`, `results/baseline-v2-dev` (and the
per-lane `v2-*` folders); `runs.rescored.jsonl` / `summary.rescored.md`
there carry the answer scores.
- The prior baseline mixed published win-dev-skills v0.7.1 and the repo
import (#976); the import is unchanged from v0.7.1, so I treated them as
the same plugin.
- Judgment calls: `winui-packaging` maps to `winui.package` (not
`msix.package`), so non-WinUI scenarios that forbid `winui.*` fail it;
`winui-design` provides `api.lookup`. The failing "WPF but
Fluent-looking" traps expect `framework.guidance`, which no current
skill really covers, so treat them as a coverage gap as much as a
routing miss.
- Answer judgment calls: signals require the `winapp` command where one
exists, not platform equivalents (`signtool`, `MakeAppx`), so the metric
measures "gave the winapp answer". `winui.package` and `maui.package`
accept `winapp package`/`pack` (and `sign` for MAUI). A primary
alternative without checkable signals makes a miss `n/a`, not `fail`; a
group alternative with an uncheckable part earns at most `partial`. No
`forbid` answer patterns are set yet; none was objective enough.
- Held-out was run once per prompt; the paraphrases act as repetitions.
Read per-cell numbers as screens; per-model and per-cohort rows are the
signal.

---------

Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants