You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Skill descriptions ≤300 chars, WinUI scope rule, and 'give the command' line - #985
Next layer of #977. #983 has merged, so this now targets main (rebased onto 9c319222). It targets default users (no agent selected), because #983 showed Copilot CLI almost never delegates to the plugin agent unprompted. Skill descriptions and in-skill scope rules decide routing for them.
What changed
Nine commits, measured separately. Commits 3 and 4 are follow-ups to the first round of results (see Follow-ups); commits 5 and 6 address the pr-review findings (see Review fixes); commit 7 ties the WinUI plugin version to the repo version; commits 8 and 9 fix Copilot review findings. The benchmark runs were recorded before the rebases; the plugin content of commits 1–3 is unchanged by them, and SHAs in the result tables are the ones measured.
1. 9b066551 Rewrite skill descriptions and add the WinUI scope rule
17 of 20 skill descriptions rewritten to ≤300 characters with "Use when…" and "Not for…" clauses. They start from the weekend E4.2 candidates, plus the known fixes: the framework list stays in winapp-package, the find-api check stays in winui-design, and winapp-find-api gets "Not for Win32 or registry code".
The Windows App SDK is not WinUI rule is in the descriptions of winui-dev-workflow, winui-design, and winui-packaging, and at the top of those 3 skills. These are the skills GPT loaded for WPF apps in the weekend and v2 baselines. The body line says how to tell: <UseWinUI>true</UseWinUI> or Microsoft.UI.Xaml.
winui-dev agent (both copies): it loads winui-dev-workflow and winui-design only when the task needs them, instead of always, and carries the same rule. The agent was not benchmarked in this PR (no agent selected).
winapp-find-api is not shared into the WinUI plugin; that's a separate later layer.
2. 9bdd6e9d winapp skills: give the command when it can't be run
One line at the top of each of the 12 winapp skills: if you can't run winapp yourself (no shell, or the command is denied), give the user the exact winapp command(s) for their project instead of only describing the steps.
3. 6d259093 winapp-frameworks: route 'add a Windows feature, keep my framework' again
Restores the clause commit 1 dropped from the winapp-frameworks description: "Use when adding Windows features to one of these apps, or for setup, packaging, or identity work on one." (231 → 264 chars).
New dev trap scenario winforms-hello-stays-winforms (base + novice prompts): a WinForms app that needs a Windows Hello check and is not moving to WinUI. Written from scratch, not from held-out wording; lint-clean.
4. 5d30fb62 Benchmark: score 'tried the command, shell denied' as a blocked answer
New answer status blocked: the response alone is partial or fail, but the winapp commands the agent tried in a denied shell call would have passed. It counts as scored but not passed, so answer pass rates don't change; it's reported as its own count, and the routing × answer table gains "routed + blocked" and "blocked only".
Runs now record winappCommandsDenied. Older runs derive it (recorded commands the final response doesn't name, when a shell call was denied), so -Rescore and -Compare apply it offline. Blocked counts as scored but not passed in Format-PassRate too. 8 new Pester tests.
winapp-package: names console and CLI apps again. winapp-manifest: "Create or edit", for any app type. winapp-identity: drops toast notifications from the identity-only list, because unpackaged apps can show toasts with AppNotificationManager. winapp-find-api: "Not for registry code or unreferenced APIs" (find-api does read referenced Win32 .winmd metadata).
winapp-signing: the body now shows how to verify a signature: winapp tool signtool.exe verify /pa <file>.
winui-dev agent (both copies): the scope rule covers only the WinUI design, build, and packaging skills; winui-ui-testing still applies to WPF, WinForms, and Win32 apps.
(This commit also set the WinUI plugin to 0.8.0; commit 7 replaces that.)
6. c9fe52e9 winapp-frameworks: route toast notifications here
"Use when adding Windows features such as toast notifications to one of these apps…". Toasts in WPF, WinForms, and Electron apps are framework guidance, not identity.
7. d75b5740 WinUI plugin version follows the repo version
WinUI version now follows the repo version (0.7.2) and is stamped on release. Its three versioned manifests (agent-plugin/plugin.json, .claude-plugin/plugin.json, .codex-plugin/plugin.json) go from 0.7.1 to 0.7.2, so installed copies see an update.
scripts/plugin-version-manifests.ps1 is now the single list of versioned manifests (winapp and WinUI). generate-llm-docs.ps1 stamps version.json onto all of them, and validate-llm-docs.ps1 fails CI when any drifts. The OpenClaw and npm manifests (openclaw.plugin.json, package.json) and the Claude marketplace's winui entry carry no version and are untouched; no SKILL.md has a version field.
Release path: start-release.ps1 writes version.json (for a -Version override on the release branch, and for the patch bump on main), then runs build-cli.ps1 -SkipTests -SkipNpm -SkipMsix, whose docs step runs generate-llm-docs.ps1; the commit stages everything. So setting or bumping the version updates the WinUI manifests in the same commit. New tests cover the stamping, WinUI drift failing validation, build-cli running the stamp with those flags, and start-release never skipping it.
8. b820a77c winapp-find-api: keep SDK-scope lookups in scope
"Not for unreferenced APIs" excluded find-api's no-project SDK scope (it answers from the installed Windows SDK and Windows App SDK before an app exists). The description now says APIs come "from a project's references or the installed SDKs" and keeps only "Not for registry code". Spot check, 2 runs each: the registry near-miss stays clean (Sonnet 2/2, GPT 2/2), and the Bluetooth API lookup holds (GPT 2/2; Sonnet 0/2, as on every earlier side). 64 AI credits.
9. 2e00cd47 winforms-hello-stays-winforms: make the fixture compile
Enables implicit usings and adds a Program.cs entry point; the fixture builds. Runs recorded above used the earlier fixture (agents can't build in the benchmark, so it read as a WinForms app either way).
Descriptions, before → after (characters, at the PR head)
Plugin
Skill
Chars
New description
winapp
winapp-find-api
873 → 298
Look up real Windows, WinRT, and Windows App SDK APIs (types, members, enums) from a project's references or the installed SDKs with winapp find-api instead of guessing. Use before coding against an unverified API, or to fix unknown-member, CS0117, CS1061, or WMC0011 errors. Not for registry code.
winapp
winapp-find-ui
834 → 294
Find the right WinUI 3 control and a real, compiling sample from the WinUI Gallery or Community Toolkit with winapp find-ui. Use when building WinUI UI and unsure which control fits (cards, tabs, swipe lists, inline messages) or when asked for a sample. Not for WPF, WinForms, or a running app.
winapp
winapp-frameworks
244 → 292
Framework-specific Windows steps for Electron, WPF, WinForms, C++, Rust, Flutter, and Tauri: build output, npm scripts, Cargo, CMake, and how winapp fits each. Use when adding Windows features such as toast notifications to one of these apps, or for setup, packaging, or identity work on one.
winapp
winapp-identity
267 → 283
Give a desktop app (WPF, WinForms, Electron, C++, Rust) package identity so identity-only Windows APIs work: push notifications, background tasks, share target, startup tasks. Use when adding such a feature, or running and debugging with identity (winapp run, create-debug-identity).
winapp
winapp-manifest
440 → 297
Create or edit Package.appxmanifest or appxmanifest.xml for any app type (GUI, console, service): execution aliases, file associations, protocols, capabilities, identity, app icons/tiles from one image. Use when changing what the manifest declares or replacing icons. Not for packaging or signing.
winapp
winapp-maui
307 → 288
Package and sign .NET MAUI Windows apps with winapp: publish the Windows head first, then point packaging at the resolved resizetizer manifest. Use for a MAUI MSIX or signed unpackaged build, MAUI packaging in CI, or 'manifest contains unresolved placeholders' errors from winapp package.
winapp
winapp-package
322 → 294
Build an MSIX installer for a desktop, console, or CLI app (Electron, .NET, C++, Rust, Flutter, Tauri) from build output or a .csproj with winapp package, optionally signed. Use for an installer testers can sideload, app distribution, or MSIX in scripts or CI. Not for signing an existing MSIX.
winapp
winapp-sandbox
282
(unchanged)
winapp
winapp-setup
314 → 273
Add Windows SDK and Windows App SDK support to an existing project (winapp init), restore pinned SDKs after cloning (winapp restore), update SDKs, or create a new WinUI app (winapp new). Use for Electron, .NET, C++, Rust, Flutter, or Tauri projects that need Windows setup.
winapp
winapp-signing
186 → 284
Sign an existing .msix or .exe and manage code-signing certificates: winapp sign, cert generate/install/info, Azure Trusted Signing. Use when signing with a .pfx, creating or trusting a dev certificate, verifying a signature, or fixing a publisher mismatch. Not for building the MSIX.
winapp
winapp-troubleshoot
192 → 283
Diagnose Windows app packaging, signing, install, identity, and SDK errors: untrusted or mismatched certificates, Add-AppxPackage and 0x80073CFx install failures, no package identity, C++/WinRT headers not generated. Use when an MSIX won't install or build, or an error code appears.
winapp
winapp-ui-automation
345 → 263
Inspect, drive, and verify a running Windows app UI with winapp ui (UI Automation): find elements, click, type, read text and values, wait for state, take screenshots. Works with WinUI, WPF, WinForms, Win32, and Electron. Use to check, test, or act on a live app.
winui
winui-code-review
207 → 284
Review WinUI 3 code or changes for quality: MVVM, x:Bind correctness, accessibility, theming, security, performance, and WinUI analyzer findings. Use when asked to review or audit a WinUI app or its pending changes, or before committing. Catches issues the compiler and UI tests miss.
winui
winui-design
412 → 276
Read before writing or changing WinUI 3 XAML: choosing controls, pages and navigation, layout, Fluent Design, theming, accessibility, x:Bind, real samples, and checking that APIs exist with winapp find-api. Not for WPF or WinForms UI, even if the app uses the Windows App SDK.
winui
winui-dev-workflow
295 → 299
Create, build, run, and fix build errors in WinUI 3 apps (C# or C++) with WinApp CLI 0.7+: winapp new, winapp run, Native AOT, crash diagnosis. Use for a new WinUI app or a WinUI project that fails to build or crashes. Not for WPF, WinForms, or Electron apps that only reference the Windows App SDK.
winui
winui-packaging
352 → 300
MSIX packaging, signing, and Microsoft Store publishing for WinUI 3 apps with WinApp CLI 0.7+: project packaging, Native AOT, certificates, self-contained deployment, CI/CD. Use when releasing or publishing a WinUI app. Not for WPF, WinForms, or Electron apps that only reference the Windows App SDK.
winui
winui-session-report
289
(unchanged)
winui
winui-setup
291
(unchanged)
winui
winui-ui-testing
388 → 284
Write and run automated UI tests for Windows desktop apps (WinUI 3, WPF, WinForms, Win32) with the winapp ui harness: one batch script, assertions, input, file pickers, dialogs, screenshots, locally or in Windows Sandbox. Use when asked for UI tests or to verify a feature end to end.
winui
winui-wpf-migration
424 → 289
Migrate or port a WPF app to WinUI 3: namespaces (System.Windows to Microsoft.UI.Xaml), control mapping (DataGrid, WrapPanel, TabControl), Dispatcher to DispatcherQueue, imaging, MVVM with CommunityToolkit.Mvvm, ThemeResource. Use when converting WPF code or fixing migration build errors.
Metadata that is always in context (skill descriptions, at the PR head): winapp 4,606 → 3,431 chars (−26%), WinUI 2,658 → 2,312 (−13%). Together 7,264 → 5,743 chars, about 1.8k → 1.4k tokens (chars/4). Commit 2 adds about 40 tokens to each winapp skill body, paid only when the skill loads.
How the descriptions were checked: iterated only on the dev set. run.ps1 -Lint ran after every change and gives 0 errors (36 dev warnings, against 28 on the base). The first draft tripped 6 held-out leakage errors: Jaccard on short descriptions (winapp-troubleshoot, winapp-maui, winapp-setup) and the bigram "fluent look" in winui-design. I reworded the descriptions, not the prompts; no held-out prompt was edited. At the PR head (rebased on main): scripts/tests Pester passes 261 of 261, the WinUI plugin script tests pass, scripts/validate-plugin-package.ps1 passes, also with -CliSchemaPath from a Debug CLI build, and scripts/validate-llm-docs.ps1 passes. Invoke-Pester benchmarks/agents/tests passes 117 of 117, and -Lint gives 189 prompts, 0 errors.
Results
Copilot CLI 1.0.87-0, configuration both unless noted, no agent selected, 1 iteration per cell. The baseline is the #983 head at the time (076d5613, the router agent that has since merged), run fresh on this harness in parallel with the candidate. Routing: did the expected skill load? Answer: does the final response name what a correct answer needs (scored from the response text)? Answer totals leave out n/a runs.
a. Held-out (the ship decision): 40 scenarios × base + novice = 80 cells per model
Commit 1 (descriptions + scope rule) against the base:
Model
Routing base → cand
Answer base → cand
Credits/run
Skill context/run
Sonnet 5.5
26/80 → 35/80 (+11 pp)
38/56 → 41/57
6.9 → 7.1
486 → 886
Opus 5.5
62/80 → 63/80
45/58 → 43/59
18.1 → 17.8
2.6k → 2.8k
GPT-6.1 Sol
66/80 → 66/80
33/56 → 35/56
7.8 → 7.7
5.2k → 5.2k
Commit 2 (give the command), measured on top of commit 1:
Model
Prompts
Routing c1 → c2
Answer c1 → c2
GPT-6.1 Sol
base + novice (80)
66/80 → 66/80
35/56 → 46/59 (63% → 78%)
Sonnet 5.5
base (40)
18/40 → 18/40
21/28 → 22/28
Opus 5.5
not run (budget)
What commit 2 really did (rescored offline with commit 4's blocked bucket, no model calls): GPT's +11 held-out answer passes break down as 6 runs that had already tried the right command in the denied shell and then reported "blocked" without naming it (blocked → pass; the line made GPT tell the user the command it tried), 4 runs that never tried the command and now name it, 3 runs that became scorable, and −2 on a trap (amberkite: GPT wrongly loads winapp-find-api on both sides and now also tells the user to run it). Counting "answered or blocked" (the agent knew the command), GPT goes 41/56 (73%) → 47/59 (80%): a real but modest gain, about half of what the raw answer metric suggests. On GPT dev, blocked goes 8 → 3 and answer 25/46 → 44/55. Sonnet held-out base prompts: blocked 1 → 1, answer 21/28 → 22/28.
Routing × answer (runs where both are scored; blocked = tried the right command in a denied shell, but the response doesn't name it):
Model
Side
Runs
Routed + answered
Routed + blocked
Routed only
Answered only
Blocked only
Neither
Sonnet
base
56
24
0
2
14
2
14
Sonnet
commit 1
57
30
1
4
11
1
10
Opus
base
58
43
1
5
2
0
7
Opus
commit 1
59
41
1
7
2
0
8
GPT
base
56
28
9
9
5
0
5
GPT
commit 1
56
31
6
12
4
0
3
GPT
commit 2
59
45
0
7
1
1
5
By cohort, base → commit 1, routing (answer):
Cohort
Sonnet
Opus
GPT
error
1/24 → 3/24 (4/12 → 6/12)
15/24 → 17/24 (7/12 → 8/13)
18/24 → 20/24 (4/12 → 4/12)
implicit
4/20 → 8/20 (11/16 → 11/17)
16/20 → 16/20 (13/18 → 12/18)
17/20 → 18/20 (9/16 → 10/16)
vague
5/14 → 8/14 (7/12 → 8/12)
13/14 → 13/14 (9/12 → 7/12)
12/14 → 13/14 (4/12 → 5/12)
trap
8/14 → 8/14 (8/8 → 8/8)
10/14 → 9/14 (8/8 → 8/8)
11/14 → 7/14 (8/8 → 8/8)
near-miss
8/8 → 8/8
8/8 → 8/8
8/8 → 8/8
By capability, held-out, base → commit 1, routing (answer)
Capability
Sonnet
Opus
GPT
api.lookup
1/4 → 1/4 (2/4 → 4/4)
4/4 → 4/4 (3/4 → 3/4)
4/4 → 4/4 (0/4 → 0/4)
framework.guidance
0/4 → 0/4
0/4 → 0/4
4/4 → 0/4
maui.package
1/4 → 2/4 (2/4 → 2/4)
2/4 → 2/4 (2/4 → 2/4)
2/4 → 2/4 (2/4 → 2/4)
msix.manifest
0/6 → 0/6 (3/4 → 3/4)
6/6 → 6/6 (4/4 → 4/4)
6/6 → 6/6 (3/4 → 3/4)
msix.package
2/6 → 3/6 (3/6 → 3/6)
6/6 → 6/6 (4/6 → 4/6)
5/6 → 6/6 (3/6 → 3/6)
msix.sign
0/6 → 1/6 (2/4 → 4/4)
6/6 → 6/6 (4/4 → 4/4)
6/6 → 6/6 (3/4 → 4/4)
project.scaffold
2/4 → 2/4 (2/4 → 2/4)
2/4 → 2/4 (3/4 → 2/4)
4/4 → 4/4 (1/4 → 1/4)
project.setup
0/4 → 1/4 (1/4 → 2/4)
2/4 → 2/4 (2/4 → 2/4)
2/4 → 3/4 (1/4 → 2/4)
sandbox.run
2/4 → 3/4 (2/4 → 1/4)
4/4 → 4/4 (1/4 → 0/4)
4/4 → 4/4 (0/4 → 0/4)
session.report
0/4 → 0/4
1/4 → 2/4
4/4 → 4/4
store.publish
2/4 → 2/4 (0/4 → 0/4)
2/4 → 2/4 (0/4 → 0/4)
1/4 → 2/4 (0/4 → 0/4)
toolchain.prereqs
0/4 → 1/4
4/4 → 3/4
1/4 → 2/4
ui.automate
2/4 → 3/4 (2/4 → 1/4)
3/4 → 4/4 (2/4 → 1/4)
4/4 → 4/4 (0/4 → 0/4)
ui.samples / winui.design
0/4 → 0/4
4/4 → 4/4
4/4 → 4/4
ui.test
0/4 → 2/4 (0/4 → 0/4)
3/4 → 4/4 (3/4 → 1/4)
4/4 → 4/4 (0/4 → 0/4)
winui.build
0/4 → 0/4
2/4 → 3/4
4/4 → 4/4
winui.migrate
0/4 → 0/4
0/4 → 0/4
2/4 → 2/4
winui.package
2/4 → 2/4
2/4 → 4/4
3/4 → 4/4
winui.review
0/4 → 0/4
2/4 → 2/4
2/4 → 2/4
Pooled over all 3 models: identity 9/12 → 12/12 (answer 12/12 → 12/12), troubleshoot 15/24 → 16/24.
Ship rule check. Held-out routing improves for Sonnet (+9 cells, 9 different scenarios across the error, implicit, and vague cohorts, with no Sonnet cell lost). No model's held-out routing or answer total regresses beyond noise: Opus answer is −2 runs, GPT routing is flat. One regression inside GPT's flat total (framework.guidance 4/4 → 0/4) was real; commit 3 fixes it (see Follow-ups).
b. Dev set (iteration set; regression and trap check)
Sonnet and GPT on all 67 dev scenarios, and Opus on 21 trap, near-miss, and WinUI scenarios. Base → commit 1:
Model
Cells
Routing
Answer
Credits/run
Sonnet 5.5
65
31/65 → 41/65
40/50 → 41/52
7.3 → 7.5
Opus 5.5
21
21/21 → 21/21
18/18 → 18/19
15.8 → 14.1
GPT-6.1 Sol
65
64/65 → 64/65
24/46 → 25/46
7.8 → 7.5
Trap and near-miss cohorts, all models: trap 25/27 → 26/27, near-miss 17/18 → 18/18 (GPT no longer loads winapp-find-api for Win32 registry code). Sonnet lost 2 cells on the single run (wpf-to-winui, explicit-find-api-numberbox); re-running both 3× gave 3/3 and 2/3, so that was noise.
c. Scope configurations (WPF traps and WinUI-only tasks in winapp-only and winui-only)
10 scenarios × 2 configurations, base → commit 1. Opus was cut.
Model
Routing
Answer
Sonnet 5.5
9/20 → 11/20
8/10 → 10/12
GPT-6.1 Sol
17/20 → 20/20
6/10 → 5/10
In the winui-only configuration, GPT stopped loading WinUI skills for WPF apps: wpf-toast-no-identity (since replaced in main by wpf-share-target-no-identity; numbers are as recorded) (base loaded winui-packaging + winui-dev-workflow), wpf-winappsdk-msix-trap (base: winui-packaging), and wpf-bluetooth-api-lookup (base: winui-design). All 3 went from fail to pass. Sonnet still loads winui-packaging for wpf-winappsdk-msix-trap in winui-only on both sides.
Credits and cuts
Follow-ups (commits 3 and 4): 1,099 AI credits (cap 1,500; -MaxCredits on every invocation), 112 sessions, no timeouts or harness errors: new dev scenario on base, commit 2, and commit 3 (355); dev framework and trap scenarios on commit 3 (359); held-out trap recheck on commit 3 (385). Commit 4's rescoring made no model calls. Dev "before" numbers for the framework and trap scenarios reuse the commit 1 runs (same descriptions apart from commit 3's clause).
Round 1 total: 9,902 AI credits (cap 10,000; -MaxCredits on every invocation; -Plan checked first). 1,063 sessions, no timeouts or harness errors.
Part
Credits
a. Held-out, base and commit 1, 3 models
5,240
b. Dev, base and commit 1 (plus a 6-run Sonnet recheck)
2,705
c. Scope configurations, Sonnet + GPT
582
Commit 2: GPT dev, GPT held-out, Sonnet held-out base
1,375
Cuts, in the requested order:
c: Opus not run. The scenarios were limited to the 10 WPF-trap and WinUI-only ones.
b: 1 iteration only. Opus ran 21 of the 67 dev scenarios (trap, near-miss, WinUI, WPF).
a: kept whole as specified (base + novice, all 3 models, both sides). Terse paraphrases and extra iterations were never in scope.
Commit 2: not measured on Opus. Sonnet ran on held-out base prompts only.
Follow-ups (commits 3 and 4)
Commit 3: the "keep my framework" regression
Order of work: write a dev scenario for the intent → restore the description clause → measure on dev → recheck only the affected held-out cells. The held-out recheck was driven by a held-out finding (the first round showed GPT framework.guidance 4/4 → 0/4); nothing was tuned on held-out, and no held-out prompt was touched.
New dev scenario winforms-hello-stays-winforms (base + novice, configuration both), routing:
Model
Base (076d5613)
Commit 2 (ca083df0)
Commit 3
GPT-6.1 Sol
2/2
1/4 (3 partial: winapp-find-api only)
4/4
Opus 5.5
1/2 (1 partial)
0/4 (4 partial: winapp-find-api only)
4/4
Sonnet 5.5
0/2
0/4
0/4 (loads nothing)
The scenario reproduces the regression on dev for both GPT and Opus, and commit 3 fixes it. Sonnet loads no skill for it on any side.
Dev framework and trap scenarios (6 framework.guidance scenarios + 9 traps; Opus ran the 9 traps + wpf-add-notifications), 1 run, commit 1 → commit 3: Sonnet 13/15 → 14/15, Opus 10/10 → 10/10, GPT 15/15 → 15/15. Trap cohort 26/27 → 26/27. No regression.
Held-out recheck, only the 7 trap scenarios (which include both framework.guidance scenarios), base + novice, all 3 models:
Model
Trap routing: base → commit 1 → commit 3
framework.guidance routing
Trap answer: commit 1 → commit 3
GPT-6.1 Sol
11/14 → 7/14 → 12/14
4/4 → 0/4 → 4/4
8/8 → 6/8
Opus 5.5
10/14 → 9/14 → 12/14
0/4 → 0/4 → 2/4
8/8 → 8/8
Sonnet 5.5
8/14 → 8/14 → 8/14
0/4 → 0/4 → 0/4
8/8 → 8/8
GPT is back above its baseline. The remaining GPT trap misses are amberkite (loads winapp-find-api for a blur effect, on every side). The GPT trap-answer drop (8/8 → 6/8) is the same amberkite pair, from commit 2's "give the command" line, not commit 3.
Commit 4: blocked answers
The re-statement of commit 2 above uses it. All #985 runs were rescored offline with -Compare; no model calls. Blocked counts on held-out: GPT base 9, commit 1 6, commit 2 1; Sonnet 2 → 2; Opus 1 → 1.
Review fixes (commits 5 and 6)
Re-check of the routing-relevant review changes: 3 models, configuration both, no agent, base prompts. Before = 5d30fb62, after = 416f0069 (commit 5). 1,145 AI credits (940 for the re-check, 205 for the toast fix), -MaxCredits on every run.
Dev, 11 scenarios covering package (incl. rust-cli-alias-msix), manifest, identity (incl. wpf-share-target-no-identity and the toast/notification asks), signing, the registry near-miss, an API lookup, and the framework traps. 1 iteration:
Model
Routing before → after
Answer before → after
Opus 5.5
11/11 → 11/11
9/10 → 8/9
GPT-6.1 Sol
11/11 → 10/11
6/8 → 6/9
Sonnet 5.5
6/11 → 4/11 (1 partial)
6/10 → 6/9
No run had a blocked answer. I re-ran the 3 changed cells twice more on each side: GPT win32-registry-read 2/3 → 2/3 (both sides' misses load winapp-frameworks, not find-api), Sonnet rust-cli-alias-msix 3/3 → 2/3 (noise), and Sonnet winforms-add-toast-vague 2/3 → 0/3: removing "toast" from winapp-identity left Sonnet with no skill for a vague toast ask.
Commit 6 moves "toast notifications" into winapp-frameworks. Toast and notification dev scenarios, after commit 6 with 3 iterations (Opus 1), against the before runs (5d30fb62-equivalent descriptions; 1–4 runs per cell):
Scenario
Sonnet before → after
GPT before → after
Opus before → after
winforms-add-toast-vague
3/4 → 2/3
2/2 → 3/3
1/1 → 1/1
wpf-add-notifications
1/1 → 2/3
1/1 → 3/3
-
wpf-notifications-without-winui
0/2 → 1/3
2/2 → 3/3
-
electron-notifications
1/1 → 0/3
1/1 → 3/3
-
Sonnet's toast cell recovered to its before level. Sonnet electron-notifications was 0/1 on the base and on commit 1 as well, so its 0/3 here is within Sonnet's usual miss rate for that prompt. Answers on these cells are mostly partial or n/a on both sides (the toast answer signal is winapp create-debug-identity, which a framework answer doesn't need).
Held-out, after side only, run once as a regression check (not tuned on); before = the recorded runs (commit 2 for Sonnet and GPT, commit 1 for Opus). 7 base cells for these capabilities (quillmark, sable, juniper, mossgate, oxalis, vireo, bracken):
Model
Routing before → after
Answer before → after
Opus 5.5
7/7 → 7/7
6/6 → 6/6
GPT-6.1 Sol
7/7 → 7/7
6/6 → 6/6
Sonnet 5.5
3/7 → 1/7 (1 partial)
6/6 → 5/6
Sonnet held-out caveat: single runs. Sonnet lost bracken (signing; its description didn't change in this round) and quillmark went partial (loaded winapp-package, not winapp-signing). Sonnet's held-out routing is 30–40% with this much run-to-run variance, so I read it as noise, but the evidence is thin.
Caveats
GPT stops loading winapp-frameworks on held-out "stay on my framework" tasks (4/4 → 0/4 after commit 1): fixed by commit 3, 4/4 again.
Opus WinUI-only API lookups: this PR doesn't share winapp-find-api into the WinUI plugin. In the both configuration, Opus held-out api.lookup stayed 4/4. The WinUI-only lookup gap from the weekend (Opus 20/24 without the shared skill) was not re-measured here; that's the find-api layer.
Commit 2's gain is partly the harness, which denies the shell; see "What commit 2 really did". Real sessions can usually run the command, so the gain there will be smaller still. The line matters for plan-only, sandboxed, or restricted sessions, and it makes a misroute visible (amberkite).
Weak capabilities still weak on Sonnet held-out: winui.migrate 0/4, winui.review 0/4, framework.guidance 0/4, msix.manifest 0/6. Sonnet still skips implicit WinUI design and API tasks; descriptions alone don't fix that.
1 iteration per cell: only per-model and per-cohort rows are signal. Single cells are screens.
The winui-dev agent change (conditional skill loading) is not measured, because this PR runs without an agent.
Other suites are reflected in the overall validation status above.
✅ 8243 passed, 37 skipped out of 8280 tests in 1270.1s (+71.8s vs. baseline)
Test Coverage
✅ 86.6% line coverage, 81.2% branch coverage · ✅ no change vs. baseline
CLI Startup Time
65ms median (x64, winapp --version) · ✅ no change vs. baseline
Try This Build
Installs the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing.
The reason will be displayed to describe this comment to others. Learn more.
🟢 Approval recommended
No blocking defects remain, and focused tests cover denied-command capture, blocked scoring, rescoring, and reporting.
0 open findings
What changed in this PR
Improves skill selection for users without a selected agent, building on #983’s router changes while distinguishing WinUI work from other Windows frameworks.
“Not for unreferenced APIs” conflicts with this skill’s supported SDK-scope workflow: the same file documents using find-api with no project, before scaffolding, against the installed Windows SDK (SKILL.md:135-137, 226-234). Because this description is routing metadata, a pre-project API lookup can now be excluded even though winapp find-api ... --project sdk supports it. Describe the project-or-SDK metadata boundary instead of excluding all unreferenced APIs.
- Rewrite 17 of 20 skill descriptions (winapp and WinUI plugins) to <=300
characters with 'Use when...' and 'Not for...' clauses, from the E4.2
candidates plus fixes: keep the framework list in winapp-package, keep
the find-api check in winui-design, and add 'Not for Win32 or registry
code' to winapp-find-api.
- winui-dev-workflow, winui-design, winui-packaging: state that the Windows
App SDK is not WinUI in the description and at the top of the skill.
- winui-dev agent: load winui-dev-workflow and winui-design only when the
task needs them, and carry the same scope rule.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add one line at the top of each winapp skill: if the agent can't run winapp
itself (no shell, or the command is denied), give the user the exact winapp
command(s) instead of only describing the steps.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…gain
The shortened description dropped the 'adding Windows features to ...' clause,
and GPT stopped loading winapp-frameworks for developers who want a Windows
feature without leaving their framework. Restore that clause (264 chars).
Add a dev trap scenario for this intent, winforms-hello-stays-winforms: a
WinForms app that needs Windows Hello and is not moving to WinUI (base and
novice prompts). README dev counts updated (68 scenarios).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The benchmark denies the shell, so an agent that runs the right winapp
command, gets denied, and then answers 'blocked' without repeating it looked
the same as one that never knew the command.
- Read-SessionEvents records winappCommandsDenied: winapp commands from shell
calls whose execution was denied.
- Answer status 'blocked': the response alone is partial or fail, but the
denied commands would have passed. It counts as scored, not passed, so
answer pass rates don't change; it is reported as its own count.
- Older runs derive denied commands (recorded commands the final response
doesn't name, when a shell call was denied), so -Rescore and -Compare
apply it offline.
- summary.md and -Compare: answer-blocked counts, and the routing x answer
table gains 'routed + blocked' and 'blocked only'.
- Pester tests for scoring, recording, derivation, stats, and reports.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ui 0.8.0
- winapp-package: restore console and CLI apps.
- winapp-manifest: 'Create or edit', for any app type (GUI, console, service).
- winapp-identity: drop toast notifications from the identity-only list;
unpackaged apps can use AppNotificationManager.
- winapp-signing: add the verify command (winapp tool signtool.exe verify /pa).
- winapp-find-api: narrow 'Not for' to registry code or unreferenced APIs;
find-api reads any referenced metadata, including Win32 .winmd files.
- winui-dev agent (both copies): limit the scope rule to the WinUI design,
build, and packaging skills; winui-ui-testing still applies to WPF,
WinForms, and Win32 apps.
- WinUI plugin content changed, so bump its manifests to 0.8.0.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Conflicts:
- RunCommand.ProjectMode.cs: keep main's requireAdministrator message (#1030)
and this branch's RunFailure.Describe for the other launch errors.
- FakeAppLauncherService.cs: keep main's LaunchExecutableThrows and this
branch's environment parameter.
- winapp-ui-automation SKILL.md: main's 300-char description plus the
winui-devtools pointer.
Also trims the winui-devtools description to the 300-char budget from #985.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
## Description
Removes the `winui-session-report` skill from the WinUI agent plugin.
Its only job was to run `Analyze-Session.ps1`, which gives wrong results
on current Copilot CLI sessions: it misreads the current log format and
counts successful builds that print `0 Error(s)` as failures (#1006).
Rather than fix a script few people use, we're dropping the skill.
**For users:** the skill is gone starting with the next release. Asking
the agent for a "session report" or to analyze a previous session no
longer loads this skill or runs the script; the agent handles the
request without it.
What changed:
- **Skill:** deleted
`plugins/winui/agent-plugin/skills/winui-session-report/` (`SKILL.md`,
`Analyze-Session.ps1`). WinUI now ships 7 skills. No agent, other skill,
manifest, README, or docs page referenced it.
- **Tests/CI:** deleted
`scripts/tests/Test-SessionBuildClassification.ps1` (it tested the
script) and removed it from the `winui-tests` job in `plugin-check.yml`.
That job still runs the other two WinUI script tests, and its path
filter now lists only those.
- **Benchmark (`benchmarks/agents`):**
- Removed the `session.report` capability and its skill mapping; updated
the WinUI `skillSetHash` for the 7-skill set.
- Deleted the dev scenario `explicit-session-diagnostic` and the
held-out scenarios `thistledown-explain-last-assistants-detour` and
`rookhaven-assistant-retried-the-wrong-surface`. All three existed only
to check that this skill loads. Their prompts were not edited.
- Dropped `session.report` from the forbid list of 84 other scenarios;
no other expectation changed.
- README: counts are now 66 dev and 38 held-out scenarios (114 held-out
prompts). A one-model, one-iteration dev baseline is now 170 sessions,
and the full dev matrix 1530. Also added a note on what to update when a
skill is removed.
- Old result folders still `-Compare` and `-Rescore`. Runs of the
deleted scenarios show as `scenario_removed`, and runs that had the
skill installed are scored by the 7-skill map. These three scenarios are
the only places old runs ever loaded the skill.
## Related Issue
Closes#1006
## Type of Change
- 💥 Breaking change (removes a shipped plugin skill)
- 🧪 Test update
## Checklist
- [x] Tested locally on Windows
Validation:
- `scripts/validate-plugin-package.ps1`: passes.
- `Test-WinuiUiTestingSandbox.ps1`, `Test-SetupVersionDetection.ps1`:
pass.
- `Invoke-Pester benchmarks/agents/tests`: 109 passed, 0 failed.
- `run.ps1 -Lint`: 180 prompts, 0 errors.
- `run.ps1 -Plan`: passes.
- `-Compare` runs against existing v2 dev/held-out result folders and an
older-format folder. `-Rescore` runs on a copy of a v2 dev folder. All
exit 0.
## Additional Notes
If #985 merges after this PR, its new scenario
`winforms-hello-stays-winforms` still lists `session.report` in
`forbid`. Remove that entry, or the benchmark test that checks every
forbid pattern matches a capability will fail. Both PRs also update the
scenario and session counts at the top of `benchmarks/agents/README.md`,
so whichever merges second needs to recount.
---------
Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Conflict was the find-ui skill description only. main (#985) put every
skill description under the 300-char budget; this branch still carried
the long one it had extended to mention Toolkit helpers and converters.
Took main's description. The Toolkit-helpers detail is not lost -- it
lives in the body bullet "The Toolkit contributes more than controls",
which merged cleanly, and the repo convention is to state a user-facing
fact once on its canonical surface rather than duplicate it into the
always-loaded metadata.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 415c898a-0d29-4429-b6f8-e6428d685e5c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Next layer of #977. #983 has merged, so this now targets
main(rebased onto9c319222). It targets default users (no agent selected), because #983 showed Copilot CLI almost never delegates to the plugin agent unprompted. Skill descriptions and in-skill scope rules decide routing for them.What changed
Nine commits, measured separately. Commits 3 and 4 are follow-ups to the first round of results (see Follow-ups); commits 5 and 6 address the pr-review findings (see Review fixes); commit 7 ties the WinUI plugin version to the repo version; commits 8 and 9 fix Copilot review findings. The benchmark runs were recorded before the rebases; the plugin content of commits 1–3 is unchanged by them, and SHAs in the result tables are the ones measured.
1.
9b066551Rewrite skill descriptions and add the WinUI scope rulewinapp-package, the find-api check stays inwinui-design, andwinapp-find-apigets "Not for Win32 or registry code".winui-dev-workflow,winui-design, andwinui-packaging, and at the top of those 3 skills. These are the skills GPT loaded for WPF apps in the weekend and v2 baselines. The body line says how to tell:<UseWinUI>true</UseWinUI>orMicrosoft.UI.Xaml.winui-devagent (both copies): it loadswinui-dev-workflowandwinui-designonly when the task needs them, instead of always, and carries the same rule. The agent was not benchmarked in this PR (no agent selected).winapp-find-apiis not shared into the WinUI plugin; that's a separate later layer.2.
9bdd6e9dwinapp skills: give the command when it can't be runwinappyourself (no shell, or the command is denied), give the user the exactwinappcommand(s) for their project instead of only describing the steps.3.
6d259093winapp-frameworks: route 'add a Windows feature, keep my framework' againwinapp-frameworksdescription: "Use when adding Windows features to one of these apps, or for setup, packaging, or identity work on one." (231 → 264 chars).winforms-hello-stays-winforms(base + novice prompts): a WinForms app that needs a Windows Hello check and is not moving to WinUI. Written from scratch, not from held-out wording; lint-clean.4.
5d30fb62Benchmark: score 'tried the command, shell denied' as a blocked answerblocked: the response alone is partial or fail, but thewinappcommands the agent tried in a denied shell call would have passed. It counts as scored but not passed, so answer pass rates don't change; it's reported as its own count, and the routing × answer table gains "routed + blocked" and "blocked only".winappCommandsDenied. Older runs derive it (recorded commands the final response doesn't name, when a shell call was denied), so-Rescoreand-Compareapply it offline. Blocked counts as scored but not passed inFormat-PassRatetoo. 8 new Pester tests.5.
416f0069Address review: description facts, signature verify, WinUI scopewinapp-package: names console and CLI apps again.winapp-manifest: "Create or edit", for any app type.winapp-identity: drops toast notifications from the identity-only list, because unpackaged apps can show toasts withAppNotificationManager.winapp-find-api: "Not for registry code or unreferenced APIs" (find-api does read referenced Win32.winmdmetadata).winapp-signing: the body now shows how to verify a signature:winapp tool signtool.exe verify /pa <file>.winui-devagent (both copies): the scope rule covers only the WinUI design, build, and packaging skills;winui-ui-testingstill applies to WPF, WinForms, and Win32 apps.6.
c9fe52e9winapp-frameworks: route toast notifications here7.
d75b5740WinUI plugin version follows the repo versionagent-plugin/plugin.json,.claude-plugin/plugin.json,.codex-plugin/plugin.json) go from 0.7.1 to 0.7.2, so installed copies see an update.scripts/plugin-version-manifests.ps1is now the single list of versioned manifests (winapp and WinUI).generate-llm-docs.ps1stampsversion.jsononto all of them, andvalidate-llm-docs.ps1fails CI when any drifts. The OpenClaw and npm manifests (openclaw.plugin.json,package.json) and the Claude marketplace'swinuientry carry no version and are untouched; no SKILL.md has a version field.start-release.ps1writesversion.json(for a-Versionoverride on the release branch, and for the patch bump onmain), then runsbuild-cli.ps1 -SkipTests -SkipNpm -SkipMsix, whose docs step runsgenerate-llm-docs.ps1; the commit stages everything. So setting or bumping the version updates the WinUI manifests in the same commit. New tests cover the stamping, WinUI drift failing validation, build-cli running the stamp with those flags, and start-release never skipping it.8.
b820a77cwinapp-find-api: keep SDK-scope lookups in scope9.
2e00cd47winforms-hello-stays-winforms: make the fixture compileProgram.csentry point; the fixture builds. Runs recorded above used the earlier fixture (agents can't build in the benchmark, so it read as a WinForms app either way).Descriptions, before → after (characters, at the PR head)
winapp-find-apiwinapp-find-uiwinapp-frameworkswinapp-identitywinapp-manifestwinapp-mauiwinapp-packagewinapp-sandboxwinapp-setupwinapp-signingwinapp-troubleshootwinapp-ui-automationwinui-code-reviewwinui-designwinui-dev-workflowwinui-packagingwinui-session-reportwinui-setupwinui-ui-testingwinui-wpf-migrationMetadata that is always in context (skill descriptions, at the PR head): winapp 4,606 → 3,431 chars (−26%), WinUI 2,658 → 2,312 (−13%). Together 7,264 → 5,743 chars, about 1.8k → 1.4k tokens (chars/4). Commit 2 adds about 40 tokens to each winapp skill body, paid only when the skill loads.
How the descriptions were checked: iterated only on the dev set.
run.ps1 -Lintran after every change and gives 0 errors (36 dev warnings, against 28 on the base). The first draft tripped 6 held-out leakage errors: Jaccard on short descriptions (winapp-troubleshoot,winapp-maui,winapp-setup) and the bigram "fluent look" inwinui-design. I reworded the descriptions, not the prompts; no held-out prompt was edited. At the PR head (rebased onmain):scripts/testsPester passes 261 of 261, the WinUI plugin script tests pass,scripts/validate-plugin-package.ps1passes, also with-CliSchemaPathfrom a Debug CLI build, andscripts/validate-llm-docs.ps1passes.Invoke-Pester benchmarks/agents/testspasses 117 of 117, and-Lintgives 189 prompts, 0 errors.Results
Copilot CLI 1.0.87-0, configuration
bothunless noted, no agent selected, 1 iteration per cell. The baseline is the #983 head at the time (076d5613, the router agent that has since merged), run fresh on this harness in parallel with the candidate. Routing: did the expected skill load? Answer: does the final response name what a correct answer needs (scored from the response text)? Answer totals leave outn/aruns.a. Held-out (the ship decision): 40 scenarios × base + novice = 80 cells per model
Commit 1 (descriptions + scope rule) against the base:
Commit 2 (give the command), measured on top of commit 1:
What commit 2 really did (rescored offline with commit 4's
blockedbucket, no model calls): GPT's +11 held-out answer passes break down as 6 runs that had already tried the right command in the denied shell and then reported "blocked" without naming it (blocked → pass; the line made GPT tell the user the command it tried), 4 runs that never tried the command and now name it, 3 runs that became scorable, and −2 on a trap (amberkite: GPT wrongly loadswinapp-find-apion both sides and now also tells the user to run it). Counting "answered or blocked" (the agent knew the command), GPT goes 41/56 (73%) → 47/59 (80%): a real but modest gain, about half of what the raw answer metric suggests. On GPT dev, blocked goes 8 → 3 and answer 25/46 → 44/55. Sonnet held-out base prompts: blocked 1 → 1, answer 21/28 → 22/28.Routing × answer (runs where both are scored; blocked = tried the right command in a denied shell, but the response doesn't name it):
By cohort, base → commit 1, routing (answer):
By capability, held-out, base → commit 1, routing (answer)
Pooled over all 3 models:
identity9/12 → 12/12 (answer 12/12 → 12/12),troubleshoot15/24 → 16/24.Ship rule check. Held-out routing improves for Sonnet (+9 cells, 9 different scenarios across the error, implicit, and vague cohorts, with no Sonnet cell lost). No model's held-out routing or answer total regresses beyond noise: Opus answer is −2 runs, GPT routing is flat. One regression inside GPT's flat total (
framework.guidance4/4 → 0/4) was real; commit 3 fixes it (see Follow-ups).b. Dev set (iteration set; regression and trap check)
Sonnet and GPT on all 67 dev scenarios, and Opus on 21 trap, near-miss, and WinUI scenarios. Base → commit 1:
Trap and near-miss cohorts, all models: trap 25/27 → 26/27, near-miss 17/18 → 18/18 (GPT no longer loads
winapp-find-apifor Win32 registry code). Sonnet lost 2 cells on the single run (wpf-to-winui,explicit-find-api-numberbox); re-running both 3× gave 3/3 and 2/3, so that was noise.Commit 2 on GPT dev: routing 64/65 → 64/65, answer 25/46 → 44/55.
c. Scope configurations (WPF traps and WinUI-only tasks in
winapp-only andwinui-only)10 scenarios × 2 configurations, base → commit 1. Opus was cut.
In the
winui-only configuration, GPT stopped loading WinUI skills for WPF apps:wpf-toast-no-identity(since replaced inmainbywpf-share-target-no-identity; numbers are as recorded) (base loadedwinui-packaging+winui-dev-workflow),wpf-winappsdk-msix-trap(base:winui-packaging), andwpf-bluetooth-api-lookup(base:winui-design). All 3 went from fail to pass. Sonnet still loadswinui-packagingforwpf-winappsdk-msix-trapinwinui-only on both sides.Credits and cuts
Follow-ups (commits 3 and 4): 1,099 AI credits (cap 1,500;
-MaxCreditson every invocation), 112 sessions, no timeouts or harness errors: new dev scenario on base, commit 2, and commit 3 (355); dev framework and trap scenarios on commit 3 (359); held-out trap recheck on commit 3 (385). Commit 4's rescoring made no model calls. Dev "before" numbers for the framework and trap scenarios reuse the commit 1 runs (same descriptions apart from commit 3's clause).Round 1 total: 9,902 AI credits (cap 10,000;
-MaxCreditson every invocation;-Planchecked first). 1,063 sessions, no timeouts or harness errors.Cuts, in the requested order:
Follow-ups (commits 3 and 4)
Commit 3: the "keep my framework" regression
Order of work: write a dev scenario for the intent → restore the description clause → measure on dev → recheck only the affected held-out cells. The held-out recheck was driven by a held-out finding (the first round showed GPT
framework.guidance4/4 → 0/4); nothing was tuned on held-out, and no held-out prompt was touched.New dev scenario
winforms-hello-stays-winforms(base + novice, configurationboth), routing:076d5613)ca083df0)winapp-find-apionly)winapp-find-apionly)The scenario reproduces the regression on dev for both GPT and Opus, and commit 3 fixes it. Sonnet loads no skill for it on any side.
Dev framework and trap scenarios (6
framework.guidancescenarios + 9 traps; Opus ran the 9 traps +wpf-add-notifications), 1 run, commit 1 → commit 3: Sonnet 13/15 → 14/15, Opus 10/10 → 10/10, GPT 15/15 → 15/15. Trap cohort 26/27 → 26/27. No regression.Held-out recheck, only the 7 trap scenarios (which include both
framework.guidancescenarios), base + novice, all 3 models:framework.guidanceroutingGPT is back above its baseline. The remaining GPT trap misses are
amberkite(loadswinapp-find-apifor a blur effect, on every side). The GPT trap-answer drop (8/8 → 6/8) is the sameamberkitepair, from commit 2's "give the command" line, not commit 3.Commit 4: blocked answers
The re-statement of commit 2 above uses it. All #985 runs were rescored offline with
-Compare; no model calls. Blocked counts on held-out: GPT base 9, commit 1 6, commit 2 1; Sonnet 2 → 2; Opus 1 → 1.Review fixes (commits 5 and 6)
Re-check of the routing-relevant review changes: 3 models, configuration
both, no agent, base prompts. Before =5d30fb62, after =416f0069(commit 5). 1,145 AI credits (940 for the re-check, 205 for the toast fix),-MaxCreditson every run.Dev, 11 scenarios covering package (incl.
rust-cli-alias-msix), manifest, identity (incl.wpf-share-target-no-identityand the toast/notification asks), signing, the registry near-miss, an API lookup, and the framework traps. 1 iteration:No run had a blocked answer. I re-ran the 3 changed cells twice more on each side: GPT
win32-registry-read2/3 → 2/3 (both sides' misses loadwinapp-frameworks, not find-api), Sonnetrust-cli-alias-msix3/3 → 2/3 (noise), and Sonnetwinforms-add-toast-vague2/3 → 0/3: removing "toast" fromwinapp-identityleft Sonnet with no skill for a vague toast ask.Commit 6 moves "toast notifications" into
winapp-frameworks. Toast and notification dev scenarios, after commit 6 with 3 iterations (Opus 1), against the before runs (5d30fb62-equivalent descriptions; 1–4 runs per cell):winforms-add-toast-vaguewpf-add-notificationswpf-notifications-without-winuielectron-notificationsSonnet's toast cell recovered to its before level. Sonnet
electron-notificationswas 0/1 on the base and on commit 1 as well, so its 0/3 here is within Sonnet's usual miss rate for that prompt. Answers on these cells are mostlypartialorn/aon both sides (the toast answer signal iswinapp create-debug-identity, which a framework answer doesn't need).Held-out, after side only, run once as a regression check (not tuned on); before = the recorded runs (commit 2 for Sonnet and GPT, commit 1 for Opus). 7 base cells for these capabilities (
quillmark,sable,juniper,mossgate,oxalis,vireo,bracken):Sonnet held-out caveat: single runs. Sonnet lost
bracken(signing; its description didn't change in this round) andquillmarkwent partial (loadedwinapp-package, notwinapp-signing). Sonnet's held-out routing is 30–40% with this much run-to-run variance, so I read it as noise, but the evidence is thin.Caveats
GPT stops loading(4/4 → 0/4 after commit 1): fixed by commit 3, 4/4 again.winapp-frameworkson held-out "stay on my framework" taskswinapp-find-apiinto the WinUI plugin. In thebothconfiguration, Opus held-outapi.lookupstayed 4/4. The WinUI-only lookup gap from the weekend (Opus 20/24 without the shared skill) was not re-measured here; that's the find-api layer.amberkite).winui.migrate0/4,winui.review0/4,framework.guidance0/4,msix.manifest0/6. Sonnet still skips implicit WinUI design and API tasks; descriptions alone don't fix that.winui-devagent change (conditional skill loading) is not measured, because this PR runs without an agent.