Repository navigation
Import WinUI agent plugin from win-dev-skills (unchanged) - #976
Conversation
Build Metrics ReportValidation passed. All required build and validation jobs succeeded. Binary Sizes
.NET Test Results (TRX reports)Other suites are reflected in the overall validation status above. ✅ 8062 passed, 37 skipped out of 8099 tests in 1305.3s (+10 tests, +116.2s vs. baseline) Test Coverage✅ 86.4% line coverage, 81% branch coverage · ✅ no change vs. baseline CLI Startup Time66ms median (x64, Try This BuildInstalls the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing. & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 976Switching between builds often?Put the tool on your PATH once: & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPathThen this build is just: winapp-pr 976Run Updated 2026-10-06 21:59:29 UTC · commit |
## Description Adds a fast, model-free check that catches common skill and plugin problems before review. It extends `scripts/validate-plugin-package.ps1` and checks every **plugin root** under `plugins/`: any folder that holds `plugin.json` and an immediate `skills/` folder. That covers `plugins/winapp` and the WinUI layout from #976 (`plugins/winui/agent-plugin`, with host wrappers in `plugins/winui`) with no per-plugin setup. **Errors (fail the check):** - `skills/<id>/SKILL.md` is missing, or its frontmatter is missing or unterminated. - `name` or `description` is missing. `name` must match the folder name and the Agent Skills naming rules. `description` can be at most 1024 characters. - A `SKILL.md` under `plugins/` is not at `<plugin root>/skills/<id>/SKILL.md`, for example because it is nested too deep or has no `plugin.json` above it. New layouts can't be skipped silently. - A relative Markdown link, a reference definition, or an inline-code `references/`, `scripts/` or `assets/` path points to a missing file or outside the plugin. URLs and in-page anchors are skipped. - A cross-skill mention such as ``see `winui-packaging`'s `references/sourcegen-patterns.md` `` (or "the winui-packaging skill's …") is resolved against that sibling skill. - It is an error only if neither the current skill nor the named sibling has the file. - A `winapp …` line in a fenced code block uses an unknown command. This covers skills, `com.github.copilot/agents/*.agent.md`, and host wrapper agents in outer `agents/` folders. - Valid commands come from `docs/cli-schema.json` (including aliases) plus the npm wrapper's `node` subcommands, read from `NODE_SUBCOMMANDS` in `src/winapp-npm/src/cli.ts`. - Placeholders, comments, paths and positional arguments end the command path, so they are never flagged. **Warnings (shown, but the check still passes):** descriptions over 300 characters. Change the limit with `$DescriptionWarnChars`. **Size report:** an approximate token table for each plugin root, printed in the log and appended to the GitHub Actions job summary. A host wrapper with the same file name as the inner agent is listed but left out of the totals, so the agent isn't counted twice. **Early PR check:** the new `Plugin Check` workflow (`.github/workflows/plugin-check.yml`) runs on every PR to `main` on `ubuntu-latest` with pwsh, read-only `contents` permission, and no Node or .NET setup. - The trigger has no path filter, so the job always reports a status and is safe to mark as a required check. - The job checks out the PR merge commit with `fetch-depth: 2` and runs `git diff --name-only HEAD^1 HEAD` to list the files the PR changes. - If none of `plugins/**`, `scripts/validate-plugin-package.ps1`, `docs/cli-schema.json`, `src/winapp-npm/src/cli.ts` or the workflow file changed, it prints `No skill-related changes; skipping.` to the log and job summary and passes. - Otherwise it runs the validator, and the job fails if the validator reports an error. The post-build `validate-docs` job is unchanged. The validator's path handling is cross-platform so it can run on ubuntu. To run the check locally, or against a copy of the plugins folder: ```powershell .\scripts\validate-plugin-package.ps1 .\scripts\validate-plugin-package.ps1 -PluginsRoot <path-to-plugins-folder> ``` ## Current results - **main:** 0 errors and 7 description warnings: find-api 873, find-ui 834, manifest 440, ui-automation 345, package 322, setup 314, maui 307. - **With the WinUI import (#976, branch `nmetulev-import-winui-plugin`):** 0 errors. WinUI adds 4 description warnings: winui-wpf-migration 424, winui-design 412, winui-ui-testing 388, winui-packaging 352. ### Plugin size: winapp | Component | Description chars | Metadata ~tokens | Body ~tokens | Other skill files ~tokens | |---|--:|--:|--:|--:| | winapp-find-api | 873 | 223 | 6,405 | 0 | | winapp-find-ui | 834 | 213 | 2,607 | 0 | | winapp-frameworks | 244 | 66 | 2,633 | 0 | | winapp-identity | 267 | 71 | 2,564 | 0 | | winapp-manifest | 440 | 115 | 2,059 | 0 | | winapp-maui | 307 | 80 | 2,979 | 0 | | winapp-package | 322 | 85 | 2,837 | 0 | | winapp-sandbox | 282 | 75 | 2,301 | 0 | | winapp-setup | 314 | 82 | 6,089 | 0 | | winapp-signing | 186 | 51 | 2,308 | 0 | | winapp-troubleshoot | 192 | 54 | 2,608 | 0 | | winapp-ui-automation | 345 | 92 | 8,892 | 2,868 | | agent: winapp.agent.md | 865 | 219 | 14,547 | | | **Total** | 5,471 | **1,426** | 58,829 | 2,868 | ### Plugin size: winui/agent-plugin (from #976) | Component | Description chars | Metadata ~tokens | Body ~tokens | Other skill files ~tokens | |---|--:|--:|--:|--:| | winui-code-review | 207 | 57 | 1,620 | 3,156 | | winui-design | 412 | 107 | 3,329 | 4,319 | | winui-dev-workflow | 295 | 80 | 1,714 | 0 | | winui-packaging | 352 | 93 | 1,718 | 1,392 | | winui-session-report | 289 | 78 | 1,594 | 13,028 | | winui-setup | 291 | 77 | 2,102 | 0 | | winui-ui-testing | 388 | 102 | 5,611 | 0 | | winui-wpf-migration | 424 | 116 | 1,486 | 0 | | agent: winui-dev.agent.md | 203 | 54 | 505 | | | agent (host wrapper): winui/agents/winui-dev.agent.md (not in totals) | 203 | 54 | 504 | | | **Total** | 2,861 | **764** | 19,679 | 21,895 | The tables are approximate: ~tokens = UTF-8 bytes / 4. Metadata (`name: description`) loads in every session, so the metadata total is each plugin's always-loaded cost: ~1,426 tokens for winapp and ~764 for WinUI. Other skill files (references, scripts, assets) load only when read. Most of WinUI's 13,028 is `Analyze-Session.ps1`. ## Type of Change - 🔧 Config/build ## Checklist - [x] Tested locally on Windows ## Additional Notes - **No skill content changes.** All existing links and command examples in both plugins pass, so nothing needed an allowlist. - **Error checks were tested on scratch copies** of `plugins/` from main and from the WinUI branch. - These failed the run as expected: - name mismatch, invalid name, missing description, 1100-character description, missing `SKILL.md` - an unrooted `SKILL.md` and a too-deep `SKILL.md` - a link escaping the plugin, a root-relative link, a missing link target, a missing inline reference - a cross-skill path whose named sibling lacks the file, and a cross-skill path naming a skill that doesn't exist - unknown commands in a skill, the inner agent, and a host wrapper agent, plus a wrapper link escaping its folder - These did not fail: valid cross-skill mentions in both the backtick and "the … skill's" forms, `winapp node create-addon`, the `pack` and `run-buildtool` aliases, positional arguments, `<placeholders>` and comments. - **Frontmatter parsing matches a real YAML parser.** The description lengths match PyYAML exactly. - **The generic placement check replaces the winapp-only nested-`SKILL.md` check**, which reported the same problem. - **Command options are not validated.** Aliases, placeholders and `--` would cause false failures. - **Inline-code checks are limited to `references/`, `scripts/` and `assets/` paths.** Paths like `./<name>` usually describe the user's project, not the plugin. - **The 300-character warnings are noisy** until the descriptions are trimmed: 7 of 12 winapp skills and 4 of 8 WinUI skills are over the limit. - **Verified on ubuntu:** a `Plugin Check` run on this PR detected the changed files, ran the validator and passed, with the same size totals as on Windows. The skip branch hasn't run in CI because this PR always changes relevant files; its file-matching pattern was tested locally. --------- Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1d782e1 to
9422b53
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The session analyzer misreads current Copilot logs and can report successful builds as failures.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Imports the WinUI 0.7.1 agent plugin unchanged, preserves source byte parity, and makes the local copy the benchmark default without publishing it.
Changes:
- Adds the 24-file multi-host WinUI plugin.
- Preserves LF and binary assets through
.gitattributes. - Updates benchmarks to compare local and published plugins.
| File | Description |
|---|---|
.gitattributes |
Preserves imported file bytes. |
benchmarks/agents/README.md |
Documents local and published modes. |
benchmarks/agents/config.json |
Makes the local plugin default. |
benchmarks/agents/run.ps1 |
Resolves local or published WinUI plugins. |
plugins/winui/package.json |
Defines the OpenClaw package. |
plugins/winui/openclaw.plugin.json |
Declares OpenClaw skill discovery. |
plugins/winui/index.js |
Adds the OpenClaw entry point. |
plugins/winui/agents/winui-dev.agent.md |
Adds the host agent wrapper. |
plugins/winui/agent-plugin/plugin.json |
Defines the portable plugin. |
plugins/winui/agent-plugin/com.github.copilot/agents/winui-dev.agent.md |
Adds the Copilot agent. |
plugins/winui/agent-plugin/assets/logo.svg |
Adds vector branding. |
plugins/winui/agent-plugin/assets/logo-512.png |
Adds raster branding. |
plugins/winui/agent-plugin/skills/winui-code-review/SKILL.md |
Adds review guidance. |
plugins/winui/agent-plugin/skills/winui-code-review/references/quality-rules.md |
Adds detailed quality rules. |
plugins/winui/agent-plugin/skills/winui-design/SKILL.md |
Adds WinUI design guidance. |
plugins/winui/agent-plugin/skills/winui-design/references/brushes-and-icons.md |
Adds brush and icon reference. |
plugins/winui/agent-plugin/skills/winui-design/references/layout-review.md |
Adds responsive-layout guidance. |
plugins/winui/agent-plugin/skills/winui-design/references/theme-accessibility.md |
Adds theming guidance. |
plugins/winui/agent-plugin/skills/winui-dev-workflow/SKILL.md |
Adds development workflow guidance. |
plugins/winui/agent-plugin/skills/winui-packaging/SKILL.md |
Adds packaging guidance. |
plugins/winui/agent-plugin/skills/winui-packaging/references/sourcegen-patterns.md |
Adds AOT/source-generator patterns. |
plugins/winui/agent-plugin/skills/winui-session-report/SKILL.md |
Adds session-report workflow. |
plugins/winui/agent-plugin/skills/winui-session-report/Analyze-Session.ps1 |
Implements session analysis. |
plugins/winui/agent-plugin/skills/winui-setup/SKILL.md |
Adds prerequisite setup guidance. |
plugins/winui/agent-plugin/skills/winui-ui-testing/SKILL.md |
Adds UI-testing workflow. |
plugins/winui/agent-plugin/skills/winui-wpf-migration/SKILL.md |
Adds WPF migration guidance. |
plugins/winui/.codex-plugin/plugin.json |
Adds Codex metadata. |
plugins/winui/.claude-plugin/plugin.json |
Adds Claude metadata. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Copy microsoft/win-dev-skills plugins/winui at v0.7.1 (2e8c902e9ba675e4c57550e066fc7e48b64c49b0) byte-for-byte into plugins/winui (24 files; git blob ids and SHA-256 match the source). Pin LF line endings and binary PNGs for the folder in .gitattributes so checkouts stay byte-exact. Point the benchmark's default WinUI plugin at the local plugins/winui/agent-plugin; -WinUIPlugin published still fetches the pinned win-dev-skills release for comparison. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
9422b53 to
daff58b
Compare
…tests (#1007) Part of moving the WinUI plugin's source of truth from microsoft/win-dev-skills to this repo (follow-up to #976). No file under `plugins/` changes. ## What changed - **`.claude-plugin/marketplace.json`**: adds a `winui` entry with `source: ./plugins/winui`. OpenClaw reads this file, and remote OpenClaw catalogs accept only relative paths inside the same repo. The entry has no `version`, so `generate-llm-docs.ps1`/`validate-llm-docs.ps1` (which stamp/check every `"version"` in the file) leave it alone and hosts read `0.7.1` from `plugins/winui/.claude-plugin/plugin.json`. - **`scripts/tests/`**: copies `Test-SessionBuildClassification.ps1`, `Test-WinuiUiTestingSandbox.ps1`, `Test-SetupVersionDetection.ps1` from win-dev-skills@2e8c902e unchanged. The relative layout (`scripts/tests` -> `plugins/winui`) is the same, so no path edits were needed. They aren't `*.Tests.ps1`, so the existing Pester run doesn't pick them up. - **`plugin-check.yml`**: new `winui-tests` job on `windows-latest` that runs the three scripts when `plugins/winui/**`, those tests or the workflow change. Otherwise it skips quickly. ## OpenClaw evidence (source-level) From openclaw `v2026.9.8` `src/plugins/marketplace.ts`: - `--marketplace owner/repo` shallow-clones the repo, then reads `.claude-plugin/marketplace.json` (falling back to a root `marketplace.json`). - Remote catalogs reject the **whole file** if any entry is not a relative path inside the repo (`remote marketplaces may not use github plugin sources`). - Entries are matched by `name`. `version` is metadata only. `plugins/winui` is byte-identical to win-dev-skills v0.7.1, where `index.js` + `openclaw.plugin.json` + `package.json` already installed as `Format: openclaw`. Install command after merge: ``` openclaw plugins install winui --marketplace microsoft/winappCli ``` I couldn't run the real install here: Microsoft's npm feed blocks the `openclaw` package by policy (`PolicyOverride`), so I didn't work around it. ## Which hosts see `winui` in this repo's catalogs - **Claude Code**: yes, as `winui@winappcli` (reads `.claude-plugin/marketplace.json`). This adds a second install route next to `winui@win-dev-skills`. - **OpenClaw**: yes (reads the same file). - **Copilot CLI**: no. It reads `.github/plugin/marketplace.json`, which is unchanged. I checked this with an isolated `COPILOT_HOME`: `marketplace browse winappcli` lists only `winappcli`. - **Codex**: no (this repo has no `.agents/plugins/marketplace.json`). ## Validation - `scripts/validate-plugin-package.ps1`: passes (only the existing description-length warnings). - All three WinUI tests pass locally on both LF and CRLF checkouts. Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
#981) Layer 3 of the benchmark stack. #973 and #976 are merged, so this now targets `main`. It adds the benchmark improvements tested during the weekend experiments, plus reporting. ## Changes **From the experiments branch (cherry-picked, no conflicts)** - `-Agent <name>` runs every session with `--agent`. `-MaxCredits <n>` stops launching sessions once the run has spent `n` credits. Each run also records `winappCommands` (including `node <subcommand>` from the npm wrapper) and `selectedAgent`. - 24 reviewed scenarios. Expectations are tightened on six existing ones, with no prompt or fixture changes. Packaging scenarios need a packaging skill to load; signing alone does not pass. Review notes are in `benchmarks/agents/scenarios-review.md`. - Handles one skill name shipped by two plugins, which Copilot lists as `<plugin>:<name>`. Preflight requires one enabled copy per plugin. **New in this PR** - **`n/a` status:** a run where nothing in the expectation applies to the installed skills gets `n/a` and is left out of pass rates. "Nothing applies" means no expected or forbidden skill is installed and there is no `maxSkills`. The summary lists these runs separately. - **Repeated deliveries:** each run records skills delivered more than once in a session and the extra skill context they add. `summary.md` shows them in the totals and per row, as `unknown` or `partial` when their size is not known. It also counts launched runs that had no credit count. - **`run.ps1 -Compare <dirs> -Candidate <dirs>`** builds a Markdown report by model and by cell (model, scenario, configuration). It shows pass rate (only pass and fail are scored; other statuses are listed as excluded, the same as in `summary.md`), skill context per run, input tokens per run, credits per run, the change in each, and repeated deliveries. Only cells present on both sides are compared. Both sides are re-scored in memory against the current expectations, and the input folders are not modified. A cell shows `check differs` when the installed skills that the expectation checks differ by name between the sides, for example when a candidate adds or removes an expected skill, including removing all of them. It is then left out of the per-model pass rate. `-Scenario` and `-Model` must name values present in the compared results. Runs of scenarios that no longer exist are excluded as `scenario_removed`. - **`-MaxCredits` with unknown credits:** a session that ran but reported no credits (for example a timeout) counts at the average of the measured sessions, or at 35 before any is measured. - **Every scenario must include `both`:** validation fails otherwise, and there is a Pester test for it. All 42 scenarios comply. - **`-Rescore`** also re-scores `n/a` runs and prints status transitions. A mistyped `-Scenario` now fails before `-Rescore` or `-Compare` runs. - **README:** documents the new parameters and statuses, the `both` rule, and the recommended workflow: a quick Sonnet screen, then 3 models × 3 iterations on the affected scenarios plus the near-misses. ## Scenarios and plan totals 42 scenarios: 18 existing + 24 new. - `-Plan`, full default matrix: **954** sessions (3 models × 3 iterations). - Sonnet, 1 iteration: **106** sessions. ## Validation - Pester: 46/46 pass (was 19). - `-Compare` on the weekend result folders (read-only). This is the E4 candidate vs the current plugins; the numbers match the weekend report: | Model | Cells | Pass (base → cand) | Δ pass | Skill context/run | Δ | Input/run | Δ | AI credits/run | Δ | |---|---|---|---|---|---|---|---|---|---| | claude-opus-5.5 | 105 | 310/315 (98%) → 306/315 (97%) | -1 pp | 3.8k → 3.9k | +4% | 174.7k → 170.8k | -2% | 21.3 → 19.4 | -9% | | claude-sonnet-5.5 | 105 | 194/315 (62%) → 227/315 (72%) | +10 pp | 1.8k → 2.1k | +16% | 122.4k → 122.2k | = | 8.3 → 7.9 | -6% | | gpt-6.1-sol | 105 | 304/315 (97%) → 309/315 (98%) | +2 pp | 6.7k → 6.0k | -11% | 130.0k → 101.6k | -22% | 10.1 → 8.3 | -18% | E4.2 shares `winapp-find-api` into the WinUI plugin. That changes what 15 of its 24 cells check (5 per model), so they show `check differs`. The per-model pass rates use the other 9 cells: Opus 9/9 → 9/9, Sonnet 3/9 → 4/9, GPT 9/9 → 9/9. The E4 comparison has no `check differs` cells. - `-Rescore` on a copy of the original 297-run baseline: 3 runs went pass → fail (from the tightened expectations), and 0 went to `n/a`. Across all 954 baseline runs, no current cell is `n/a`. The status exists so that candidates which drop or rename skills don't get free passes. - No new model runs. ## Notes - **`n/a` is narrower than "no expected skill installed".** Under that literal rule, clean runs in forbid-checked cells would become `n/a` while overreach still fails. An example is `wpf-add-notifications / winui`, where GPT WinUI overreach went 0/3 → 3/3 in E2. That rule would hide that result, so a cell stays scored when a forbidden skill is installed. - **The `sign-existing-msix-fixture` control was dropped.** Its `.msix` is ignored by the `dist` rule in `.gitignore`, so the committed copy is not the fixture that was actually run. It also repeats `sign-existing-msix`. Its finding (the empty folder does not explain the skipped skill) is recorded in `scenarios-review.md`. - **Empty fixtures in `sign-existing-msix` and `msix-install-fails` are unchanged.** Filling them would change the scenario input and require a new baseline. - **Repeated deliveries can't be recovered from older runs.** Their records lack per-delivery data, so the compare report shows `-` for them. - **More scenario changes may follow in this layer.** A follow-up realism review will likely add or modify scenarios here. --------- Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…982) Follows #981 (now in main). Implements the scenario realism review: expectations are now capabilities instead of skill names, scenarios are grouped into cohorts, there is a gated held-out set authored blind by a different model family, and a lint catches prompts that echo skill descriptions. Each run is scored on routing (did the right skill load) and, separately, on answer quality (does the final response name what a correct answer must). Includes a new baseline for the current plugins. ## What changed **Capability expectations** (`benchmarks/agents/capabilities.json`) - 22 capabilities (`msix.sign`, `api.lookup`, `identity`, `winui.review`, …), each with a plain-language definition and the `winapp` command patterns that serve it. - Per-plugin maps from skills to capabilities, identified by a skill-set hash. A map applies when all its skills are installed, so a candidate that renames or merges skills gets its own map and stays scorable. Tests fail if the repo plugins change skill names without a map update. - Scenario `expect.capabilities`: `primary` (alternatives; a group must load together), `acceptable`, `forbid` (patterns like `winui.*`, `*`), optional `budgetTokens`, plus `commands` regexes. - Statuses: `pass`, new `partial` (only acceptable, or part of a group), `fail`, `n/a` (#981 semantics). A primary capability only reachable through a skill carrying a forbidden capability counts as not installed. - "Right `winapp` command named without loading a primary skill" is reported separately. - The old skill-name format still loads and scores, so old result folders can be rescored. Runs now record a prompt hash; rescoring skips runs whose prompt changed. **Cohorts and sets** - `cohort`: `implicit`, `error`, `vague`, `followup` (reserved), `explicit-command`, `near-miss`, `trap`. `summary.md` and `-Compare` report by set and cohort. Explicit-command and the `none` control are left out of the headline routing rate. The `none` control has its own table. - `set`: `dev` (default) or `heldout`; `-Set dev|heldout|all`, `-Variant base|novice|terse`. Paraphrases are declared once and expand to `<id>.<variant>`. **Lint** (`run.ps1 -Lint`, `lib/ScenarioLint.psm1`, Pester) - Leakage against every skill and agent description of both plugins: distinctive shared bigrams, and content-word Jaccard > 0.10. Fixture file names count as prompt text. Framework and product names ("WinUI app", "MSIX", "Microsoft Store") are neutral. Errors for held-out, warnings for dev. - Held-out: no "Contoso", no naming the tooling. Fixture realism: an empty fixture file the prompt names is an error unless `routingSnapshot: true`; a named file the fixture lacks is a warning. **Dev set (67 = 42 + 25)** - 12 leaky or over-polished prompts reworded per the review (`winui-code-review`, `restore-cloned-cpp`, `winui-infobar-enum-lookup`, `winui-card-grid-control`, `winui-control-sample-request`, `winui-repair-prerequisites`, `electron-desktop-ui-test`, `winui-generic-csharp-bug` with real tests, `flutter-trust-certificate`, `electron-notifications`, `clean-machine-ui-smoke`, `explicit-session-diagnostic`). - Expectation fixes: `xaml-unknown-member`, `store-submission`, `msi-app-sparse-identity`, `electron-notifications` (identity not mandated), `clean-machine-ui-smoke` (Sandbox and UI automation together). - 25 new scenarios from the reviews: 9 error-first, 3 vague, 3 journeys, 10 near-misses/traps. - Cohorts: implicit 32, error 15, trap 9, near-miss 6, vague 3, explicit-command 2. Near-miss + trap = 22%. **Held-out set (40 base × 3 prompts = 120)** - Written by GPT-6.1 Sol given only capability definitions, personas, public sources (GitHub issues, Stack Overflow, Learn), fixture conventions, and lint rules. It saw no skill or agent descriptions. Two revision rounds: one for realism (no invented euphemisms, no routing hints), one for lint failures. - Cohorts: error 12, implicit 10, vague 7, trap 7, near-miss 4 (27.5% near-miss/trap); 14 conflicting-signal cases; every capability primary ≥ 2 times; randomized names. - Configurations: `none` control + `both` + the single plugin that provides the primary capability. - After the baseline, 5 held-out forbids that contradicted the capability map were narrowed; prompts were not touched. The mapping choices that affect scores are in the README (Capabilities). - README: gated, for release and decision checks, not iteration. **Answer metric (deterministic, next to routing)** - Each capability in `capabilities.json` can carry small answer signals: `require` groups of regexes (one per group must appear in the final response) and optional `forbid` regexes. Signals are the `winapp` command where a capability has one (for example `winapp (sign|az-sign|cert)`), and a key term for a few that don't (`Microsoft.UI.Xaml` for a WinUI port, a `--version` check for prerequisites, manifest extension elements). Capabilities whose right answer is scenario-specific (`winui.design`, `winui.review`, `winui.build`, `troubleshoot`, `framework.guidance`, `session.report`) have none and score `n/a`. No LLM judge. - Every run gets `answer` = pass / partial / fail / n/a next to the unchanged routing `status`, plus per-primary-capability routing and answer results. Near-misses pass when the response names no `winapp` command. - Runs now keep `finalResponse`. Runs recorded without it are scored from their recorded `winapp` commands (`answerBasis: commands`); only signal groups made entirely of `winapp <command>` patterns are checked, the rest are `n/a`. - `summary.md` and `-Compare` show routing and answer side by side per model, set/cohort, and primary capability, plus a 2 × 2 (routed + answered, routed only, answered only, neither). - Lint: `leak-answer` flags a prompt that already contains an answer signal of its own expected capabilities (error for held-out; held-out is clean). **Integration with #981 (main)** - Pass rates use #981's single format: `pass`, `partial`, and `fail` are scored (`partial` counts against the rate), every other status is listed as excluded, e.g. `1/2 (50%), 1 partial; excluded: 1 n/a`. - Capability scoring feeds #981's check-set comparison: a run's applied checks are the installed primary alternatives, acceptable (`~`), and forbidden (`!`) capabilities, so `check differs` fires when a candidate adds or removes an expected capability, but not when it only renames or merges skills that the capability map covers. Skill-name expectations keep #981's skill check sets. Removed scenarios stay `scenario_removed`. - #981's stricter packaging expectations carried over: `explicit-winapp-cli-package` and `wpf-winappsdk-msix-trap` now have `msix.package` primary and `msix.sign` acceptable (signing alone is `partial`, not a pass); `winui-release-msix` already did. - The internal review notes file was removed; durable rationale lives in the README. **Fix:** `none`-only runs crashed in run.ps1 (pre-existing: empty plugin list under StrictMode). ## Validation - `Invoke-Pester benchmarks\agents\tests`: 108 passed. - `run.ps1 -Lint`: 187 prompts, 0 errors (held-out clean), 28 dev warnings (dev keeps a few leaky originals as regression sentinels; 2 are `leak-answer`: `explicit-find-api-numberbox` names `winapp find-api`, `winui-repair-prerequisites` quotes `dotnet --list-sdks`). - Gaming check: always loading the three most-loaded skills passes 15/65 dev and 15/120 held-out prompts (it passed 35% of the old set). ## New baseline (current plugins) 1,335 new sessions, **13,193 AI credits** (cap 15,000). Held-out: 3 models × 1 run per prompt. Dev: new/reworded 3 models × 2 iterations; unchanged scenarios are the prior 3 × 3 runs rescored with capabilities. Routing and answer pass rates (both leave out `none` and explicit-command; answer is over runs where it could be scored): | Model | Dev routing | Dev answer | Held-out routing | Held-out answer | Held-out routing base / novice / terse | |---|---|---|---|---|---| | claude-sonnet-5.5 | 57% | 79% (172/218) | 32% | 68% (80/118) | 35% / 28% / 34% | | claude-opus-5.5 | 92% | 85% (191/226) | 78% | 79% (95/121) | 76% / 81% / 76% | | gpt-6.1-sol | 94% | 77% (163/213) | 84% | 70% (85/122) | 86% / 80% / 85% | Routing × answer (runs where both are scored): | Set | Model | Routed + answered | Routed only | Answered only | Neither | |---|---|---|---|---|---| | held-out | claude-sonnet-5.5 | 52 | 9 | 28 | 29 | | held-out | claude-opus-5.5 | 94 | 14 | 1 | 12 | | held-out | gpt-6.1-sol | 84 | 24 | 1 | 13 | | held-out | all | 230 | 47 | 30 | 54 | | dev | claude-sonnet-5.5 | 142 | 10 | 27 | 36 | | dev | claude-opus-5.5 | 183 | 28 | 5 | 7 | | dev | gpt-6.1-sol | 154 | 37 | 7 | 12 | | dev | all | 479 | 75 | 39 | 55 | By cohort (all models), routing / answer: implicit dev 83% / 70%, held-out 66% / 72%; error 65% / 70%, 50% / 64%; vague 52% / 39%, 79% / 59%; near-miss 96% / 98%, 100% / 100%; trap 89% / 98%, 60% / 92%; explicit-command routing 87% (dev only). Answer scores for this baseline come from recorded `winapp` commands, because these runs predate `finalResponse`: 0 of 2,019 baseline runs have the response text; 1,693 have recorded commands and were scored from them; 324 older dev runs (prior copies recorded before commands were captured) and 2 held-out timeouts have no answer score. No sessions were re-run. On held-out, 277 of 638 routing runs are answer `n/a` (no objective signal, or the signal needs the response text). No-plugin control (held-out base prompts): models named the right `winapp` command in 4 of 71 runs; answer rate on implicit/error/vague base prompts is 8% (1/13) for each model vs 67–73% with both plugins on the same prompts; with no plugins, sessions cost 66–82% of the `both` cost for the same prompts (sonnet 6.0 vs 7.3, opus 12.0 vs 18.1, gpt 5.4 vs 8.1 credits). Weakest capabilities on held-out (`both`, routing): `winui.migrate` 17%, `framework.guidance` 22%, `winui.review` 28%, `project.setup` 33%, `store.publish` 33% (8 partial). Per-capability answer on held-out: `store.publish` 0% (no run named `winapp store`), `project.setup` 39%, `winui.package` 44%, `msix.sign` 56%; `identity` 100%, `api.lookup` 92%. ## Notes - Results are gitignored. Compare-ready folders are in my worktree: `results/baseline-v2-heldout`, `results/baseline-v2-dev` (and the per-lane `v2-*` folders); `runs.rescored.jsonl` / `summary.rescored.md` there carry the answer scores. - The prior baseline mixed published win-dev-skills v0.7.1 and the repo import (#976); the import is unchanged from v0.7.1, so I treated them as the same plugin. - Judgment calls: `winui-packaging` maps to `winui.package` (not `msix.package`), so non-WinUI scenarios that forbid `winui.*` fail it; `winui-design` provides `api.lookup`. The failing "WPF but Fluent-looking" traps expect `framework.guidance`, which no current skill really covers, so treat them as a coverage gap as much as a routing miss. - Answer judgment calls: signals require the `winapp` command where one exists, not platform equivalents (`signtool`, `MakeAppx`), so the metric measures "gave the winapp answer". `winui.package` and `maui.package` accept `winapp package`/`pack` (and `sign` for MAUI). A primary alternative without checkable signals makes a miss `n/a`, not `fail`; a group alternative with an uncheckable part earns at most `partial`. No `forbid` answer patterns are set yet; none was objective enough. - Held-out was run once per prompt; the paraphrases act as repetitions. Read per-cell numbers as screens; per-model and per-cohort rows are the signal. --------- Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Layer 2 of the plugin stack (base: #973). Imports the WinUI agent plugin as-is so later layers can publish it from here, then unify and trim.
What's imported
microsoft/win-dev-skillstagv0.7.1=2e8c902e9ba675e4c57550e066fc7e48b64c49b0, folderplugins/winui/.plugins/winui/with the same layout (agent-plugin/,.claude-plugin/,.codex-plugin/,agents/,openclaw.plugin.json,package.json,index.js), so host wrapper paths keep working. Manifests keep their own version0.7.1.git checkoutof the folder still matches..gitattributes:plugins/winui/** text eol=lf+*.png binary, so this repo's*.ps1 eol=crlfrule andtext=autodon't rewriteAnalyze-Session.ps1or other files on checkout.THIRD_PARTY_NOTICES.mdlists no vendored third-party content for the plugin andcgmanifest.jsonhas no registrations, so nothing extra to carry. No README added; provenance is recorded here.Intentionally not included
.github/plugin/marketplace.json/.claude-plugin/marketplace.jsonor README install docs: this PR must not create a second live install source.src/tools(winui-analyzer, winmd-cli), CI workflows, or release scripts. Ownership of analyzer/tool source is a separate decision.Tooling
No fixes needed.
generate-llm-docs.ps1,validate-llm-docs.ps1, andvalidate-plugin-package.ps1only referenceplugins/winapppaths explicitly, so they neither stamp the CLI version into WinUI manifests nor validate it.Benchmark (
benchmarks/agents): the default WinUI plugin is now the localplugins/winui/agent-plugin(relative path, like winapp).-WinUIPlugin publishedfetches the pinnedwin-dev-skills@v0.7.1for comparison (config moved underplugins.winui.published);-WinUIPlugin <path>is unchanged. README updated.Validation
scripts/validate-plugin-package.ps1(this base): pass.scripts/validate-llm-docs.ps1: not run; it needs a built CLI (artifacts/cli/win-x64/winapp.exe). It only reads winapp manifests, so this PR can't affect it.-Planshows the local default andpublished; the published fetch path was exercised and itsagent-pluginhashes identically to the local copy.#972 skill checks, run locally (not committed)
plugins/<name>/skills, so it silently skips WinUI (plugins/winui/agent-plugin/skills). Needs fixing in Add CI checks for agent skill packages #972 or a later layer.-PluginsRoot plugins/winui):winui-code-review:45,winui-design:106,winui-dev-workflow:51,winui-wpf-migration:77referencereferences/sourcegen-patterns.md, which lives inwinui-packaging. The text says "seewinui-packaging'sreferences/sourcegen-patterns.md" (v0.7.1 deliberately replaced cross-skill links with plain text), so the check resolves it against the wrong skill. Decide in a later layer: relax the check or reword.winui-design412,winui-wpf-migration424,winui-ui-testing388,winui-packaging352.winappcommand examples: no unknown commands.Benchmark smoke vs baseline (winui, claude-sonnet-5.5)
Preflight installed the local plugin (source
this repo, v0.7.1, not dirty) with the same 8 skills as published v0.7.1; no unexpected or missing skills.All 10 match the baseline status and skill set. The two failures are pre-existing routing gaps, not caused by the import.
Next layers