Repository navigation
Prefer an element named exactly the selector text in single-element ui commands - #1102
Conversation
Single-element ui verbs now act on the element whose Name equals the text selector (ignoring case) when the substring query matches several elements, after the existing exact-AutomationId preference. search still lists every substring match. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Unresolved lookup and polling defects need correction, and real-window validation remains incomplete.
2 open findings
What changed in this PR
Adds exact-name preference to single-element UI lookups, addressing #1090 while keeping search results unchanged.
Changes:
- Prefer exact Names after exact AutomationIds.
- Add coverage for filtered, ambiguous, and owned-window matches.
- Document selection precedence in API docs and the automation skill.
| File | Description |
|---|---|
| src/winapp-CLI/WinApp.UIAutomation/Services/UiAutomationService.Query.cs | Adds constrained exact-name selection. |
| src/winapp-CLI/WinApp.UIAutomation/Services/UiAutomationService.cs | Applies preference across single-element lookups. |
| src/winapp-CLI/WinApp.UIAutomation/Services/IUiAutomation.cs | Documents selection precedence. |
| src/winapp-CLI/WinApp.UIAutomation.Tests/RealUiAutomationTests.QueryExactName.cs | Tests exact-name resolution and ambiguity. |
| plugins/winapp/skills/winapp-ui-automation/SKILL.md | Updates targeting guidance and examples. |
| docs/ui-automation.md | Explains exact-match behavior. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
When several elements share the exact Name, keep the existing disambiguation over all matches, so a lookup that used to resolve to the only invokable element still does (and wait-for --gone keeps waiting). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Build Metrics ReportValidation passed. All required build and validation jobs succeeded. Binary Sizes
.NET Test Results (TRX reports)Other suites are reflected in the overall validation status above. ✅ 8708 passed, 39 skipped out of 8747 tests in 1731.7s (+11 tests, +115.9s vs. baseline) Test Coverage✅ 87.5% line coverage, 82% branch coverage · ✅ no change vs. baseline CLI Startup Time71ms median (x64, Try This BuildInstalls the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing. & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 1102Switching between builds often?Put the tool on your PATH once: & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPathThen this build is just: winapp-pr 1102Run Updated 2026-10-15 19:38:43 UTC · commit |
DecisionMerge. On Calculator, Must fixNone — mergeable as-is. Non-blockingA plain text label with the exact name now wins over the only clickable control
What was exercised
|
Jaylyn Barbee (Jaylyn-Barbee)
left a comment
There was a problem hiding this comment.
LGTM, one optional comment.
|
Not changing in this PR: an exact Name is an explicit signal and the only-invokable rule is a tiebreaker; swapping them would bring back #1090's own case (a text named exactly X losing to a "Welcome to X" row). \invoke\ still invokes the label's invokable parent when it has one. |
…enchmarks/ui (#1053) ## What and why Adds `benchmarks/ui/`, two repeatable benchmarks for `winapp ui`. Rerun them after a `winapp ui` change to see its effect. - **`run.ps1` (agent benchmark).** A Copilot CLI agent does seven natural-language tasks in inbox Calculator and a pinned WinUI 3 Gallery (`v2.9.3`, built once by `setup-gallery.ps1`). The tasks include real-world friction: the app isn't running, two instances are open, the window is minimized, a dialog was left open, and a cold start. The harness scores the final app state itself through UIA, the process count and the window state. It never trusts the agent's answer, and it flags collateral damage such as extra instances, the wrong window changed, or an app closed. The report shows the pass rate, wall time, turns, tool vs model time, tokens and credits. Methods: `winapp` (this repo's plugin and CLI) and `none` (no plugin, `winapp` hidden). - **`perf.ps1` (scripted, no agent, no credits).** It times single `winapp ui` commands (20 runs after a warm-up; median, p95 and max). It also runs fixed command sequences that mirror the agent tasks, scored with the same checks. Both benchmarks refuse to start when Calculator or the Gallery is already open (unless you pass `-Force`), close only the processes they launched (by PID), and restore your Calculator mode afterward. `run.ps1` resumes when you rerun it into the same `-OutDir`. Other tools can be benchmarked locally with `-Method <file>` or `-Adapter <file>`; the README documents both formats. `benchmarks/agents/lib/Benchmark.psm1` is reused, not copied. Its only change is additive: `Read-SessionEvents` now also returns tool time, API time and session duration, and `Format-Count` is exported. ## Before / after New scripts; nothing in `winapp` itself changes. Observed output of `perf.ps1` on current `main` (x64 Xeon, NativeAOT build of `901e4cde`, Calculator only; excerpt): ```text > .\benchmarks\ui\perf.ps1 -Operation startup,list-windows,search,inspect-calculator,get-property,get-value,invoke,wait-for,focus,screenshot -Sequence calc-add,calc-two-instances,calc-minimized-scientific startup winapp median 91 ms p95 93 ms errors 0 list-windows winapp median 102 ms p95 105 ms errors 0 search winapp median 259 ms p95 267 ms errors 0 get-value winapp median 144 ms p95 158 ms errors 0 invoke winapp median 177 ms p95 183 ms errors 0 focus winapp median 176 ms p95 182 ms errors 0 [1/3] calc-add | winapp ... pass | 1,914 ms | 10 commands [1/3] calc-two-instances | winapp ... pass | 1,383 ms | 7 commands [1/3] calc-minimized-scientific | winapp ... pass | 1,993 ms | 10 commands ``` Agent benchmark, first full run on 2026-10-09: branch `93b2cbb8` on main `a87a8304`, ARM64 Snapdragon X2 Elite, Copilot CLI 1.0.93, `claude-sonnet-5.5`, 5 iterations. **70/70 passed with no collateral damage**, in 32 minutes and 497.5 credits. | Method | Median wall time | Median turns | Credits (35 runs) | |---|---|---|---| | `winapp` | 19.4 s | 6 | 315.7 | | `none` | 21.8 s | 5 | 181.8 | `winapp` was 20–30% faster on the Gallery tasks, about even on simple Calculator tasks, and slower on the minimized-window task (36.9 s vs 15.0 s). It used about 1.7× the credits, mostly to load the UI-automation skill. ## Not tested / risks - The agent numbers above predate several `main` changes and were not re-run: minimized-window handling in `winapp ui` (#1054), the `winapp-performance` skill (#837), and selector changes (#1102, #1104). The `gallery-dialog-left-open` task also changed since then. It now asks to turn the switch off, because the switch starts on. The minimized-window result in particular is likely better now, because `perf.ps1`'s `calc-minimized-scientific` sequence, which failed at its first step on 10/9, passes 3/3 on current `main`. - I did not re-run the Gallery perf sequences here, because the Gallery is not installed on this machine. - Neither benchmark runs in CI: both drive the desktop, and the agent benchmark costs credits. Only the Pester suites are suitable for automation. ## Checklist - [x] Tests added or updated - [x] Tested locally on Windows <details><summary>Notes for reviewers</summary> Validated on the merged head: - `Invoke-Pester benchmarks\ui\tests`: 135/135 passed. - `Invoke-Pester benchmarks\tasks\tests`: 54/54 passed. - `Invoke-Pester benchmarks\agents\tests`: 157 passed, 2 failed. `main` has the same 2 failures from the `winapp-performance` skill, and #1060 fixes them. - `benchmarks\agents\run.ps1 -Lint` and `-Plan` give the same output as `main`: 1 error, which is the same known issue, and 1,584 sessions. - `benchmarks\ui\run.ps1 -Plan`, `run.ps1 -Lint` (0 errors) and `perf.ps1 -Plan` all succeed. </details> --------- Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Zach Teutsch <88554871+zateutsch@users.noreply.github.com> Copilot-Session: 0cf8e7b3-7171-42da-9482-89167c7668b8

What and why
A text selector in
winapp uimatches any element whose Name or AutomationId contains the text. When the text matched several elements, an exact AutomationId match already won, but an exact Name match did not. Sowinapp ui click "Allodia Mail & Calendar"clicked a row named "Welcome to Allodia Mail & Calendar" instead of the element named exactly "Allodia Mail & Calendar".Now commands that act on one element (
click,invoke,hover,set-value,get-value,get-property,wait-for,scroll,focus, and others) choose in this order when the text matches several elements:If several elements have the exact Name, nothing changes: the command still applies the existing ambiguity rules. This includes
--actionand--root/--type/--class-name, which still fail withambiguous_selector.searchstill lists every match. No new flag was needed.Closes #1090
Part of #1084
Before / after
Same window, same commands. The window has a Text "Allodia Mail & Calendar" and a Button "Welcome to Allodia Mail & Calendar".
Before (
main): the command acts on the button with the longer name.After: the command acts on the element named exactly the text;
searchis unchanged.On the same window,
get-property ... --root ExactNameDemoandinvoke ... --action invokeused to fail with "Selector matched multiple elements". Now they select the "Allodia Mail & Calendar" Text element.Not tested / risks
ui clicklocally, because the foreground check refused to click on a shared desktop.clickuses the same element lookup that the commands above use.Checklist
plugins/winapp/skills/updated if user-facing behavior changedNotes for reviewers
--action, and the filtered lookups (--type/--root/--class-name). Slug and--rootresolution are unchanged.--action, or any filter) count matches across all app windows, so there an exact Name in a popup wins over a partial match in the main window.searchkeeps its result limit.