Skip to content

Prefer an element named exactly the selector text in single-element ui commands - #1102

Merged
Nikola Metulev (nmetulev) merged 6 commits into
mainfrom
nmetulev-ui-exact-name-preference
Oct 15, 2026
Merged

Nikola Metulev (nmetulev) merged 6 commits into
mainfrom
nmetulev-ui-exact-name-preference

Conversation

@nmetulev

@nmetulev Nikola Metulev (nmetulev) commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

What and why

A text selector in winapp ui matches any element whose Name or AutomationId contains the text. When the text matched several elements, an exact AutomationId match already won, but an exact Name match did not. So winapp ui click "Allodia Mail & Calendar" clicked a row named "Welcome to Allodia Mail & Calendar" instead of the element named exactly "Allodia Mail & Calendar".

Now commands that act on one element (click, invoke, hover, set-value, get-value, get-property, wait-for, scroll, focus, and others) choose in this order when the text matches several elements:

  1. An element whose AutomationId is exactly the text.
  2. New: the only element whose Name is exactly the text, ignoring case.
  3. The existing rules: the only invokable element, or an ambiguity error that lists slugs.

If several elements have the exact Name, nothing changes: the command still applies the existing ambiguity rules. This includes --action and --root/--type/--class-name, which still fail with ambiguous_selector. search still lists every match. No new flag was needed.

Closes #1090
Part of #1084

Before / after

Same window, same commands. The window has a Text "Allodia Mail & Calendar" and a Button "Welcome to Allodia Mail & Calendar".

Before (main): the command acts on the button with the longer name.

before

After: the command acts on the element named exactly the text; search is unchanged.

after

On the same window, get-property ... --root ExactNameDemo and invoke ... --action invoke used to fail with "Selector matched multiple elements". Now they select the "Allodia Mail & Calendar" Text element.

Not tested / risks

  • I couldn't run a real ui click locally, because the foreground check refused to click on a shared desktop. click uses the same element lookup that the commands above use.
  • Behavior change: a script that relied on a partial match winning over an element named exactly the text will now hit the exact one. That was the bug this issue reports.

Checklist

  • Tests added or updated
  • Tested locally on Windows
  • Docs, samples, and plugins/winapp/skills/ updated if user-facing behavior changed
Notes for reviewers
  • The preference is applied in each single-element lookup path: the main window without filters, the fallback that searches the app's other windows, the strict lookup used by --action, and the filtered lookups (--type/--root/--class-name). Slug and --root resolution are unchanged.
  • The fallback that searches the app's other windows still returns the first window with a resolvable match, the same scope the existing exact-AutomationId preference has. Strict lookups (--action, or any filter) count matches across all app windows, so there an exact Name in a popup wins over a partial match in the main window.
  • Without filters, a lookup already collected every match. Filtered single-element lookups now do the same, so they can find an exact Name among them. search keeps its result limit.

Single-element ui verbs now act on the element whose Name equals the
text selector (ignoring case) when the substring query matches several
elements, after the existing exact-AutomationId preference. search still
lists every substring match.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 14, 2026 23:09
@nmetulev Nikola Metulev (nmetulev) added the agent-preparing Agent is addressing feedback or completing required validation and CI label Oct 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved lookup and polling defects need correction, and real-window validation remains incomplete.

2 open findings
What changed in this PR

Adds exact-name preference to single-element UI lookups, addressing #1090 while keeping search results unchanged.

Changes:

  • Prefer exact Names after exact AutomationIds.
  • Add coverage for filtered, ambiguous, and owned-window matches.
  • Document selection precedence in API docs and the automation skill.
File Description
src/​winapp-CLI/​WinApp.UIAutomation/​Services/​UiAutomationService.Query.cs Adds constrained exact-name selection.
src/​winapp-CLI/​WinApp.UIAutomation/​Services/​UiAutomationService.cs Applies preference across single-element lookups.
src/​winapp-CLI/​WinApp.UIAutomation/​Services/​IUiAutomation.cs Documents selection precedence.
src/​winapp-CLI/​WinApp.UIAutomation.Tests/​RealUiAutomationTests.QueryExactName.cs Tests exact-name resolution and ambiguity.
plugins/​winapp/​skills/​winapp-ui-automation/​SKILL.md Updates targeting guidance and examples.
docs/​ui-automation.md Explains exact-match behavior.

🧠 Review effort: Balanced


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

When several elements share the exact Name, keep the existing
disambiguation over all matches, so a lookup that used to resolve to the
only invokable element still does (and wait-for --gone keeps waiting).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Build Metrics Report

Validation passed. All required build and validation jobs succeeded.

Binary Sizes

Artifact Baseline Current Delta
CLI (ARM64) 59.50 MB 59.52 MB 📈 +24.5 KB (+0.04%)
CLI (x64) 59.48 MB 59.50 MB 📈 +22.5 KB (+0.04%)
MSIX (ARM64) 24.71 MB 24.71 MB 📈 +1.5 KB (+0.01%)
MSIX (x64) 26.21 MB 26.21 MB 📈 +0.6 KB (+0.00%)
NPM Package 51.54 MB 51.55 MB 📈 +10.2 KB (+0.02%)
NuGet Package 51.65 MB 51.66 MB 📈 +11.9 KB (+0.02%)

.NET Test Results (TRX reports)

Other suites are reflected in the overall validation status above.

✅ 8708 passed, 39 skipped out of 8747 tests in 1731.7s (+11 tests, +115.9s vs. baseline)

Test Coverage

✅ 87.5% line coverage, 82% branch coverage · ✅ no change vs. baseline

CLI Startup Time

71ms median (x64, winapp --version) · ✅ -9ms vs. baseline

Try This Build

Installs the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing.

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 1102
Switching between builds often?

Put the tool on your PATH once:

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPath

Then this build is just:

winapp-pr 1102

Run winapp-pr with no arguments to pick from a list of open PRs.


Updated 2026-10-15 19:38:43 UTC · commit adabd6f · workflow run

@nmetulev Nikola Metulev (nmetulev) added ready-for-review Agent work and technical checks complete; awaiting review or re-review, not approval or merge and removed agent-preparing Agent is addressing feedback or completing required validation and CI labels Oct 15, 2026
@Jaylyn-Barbee

Copy link
Copy Markdown
Contributor

🤖 AI-generated review (winappcli pr-review skill) — verify before acting.

Decision

Merge. On Calculator, winapp ui invoke "Clear" now picks the button named exactly "Clear" over "Clear entry" and "Clear all memory". This works with --action and --type too. One optional item is below.

Must fix

None — mergeable as-is.

Non-blocking

A plain text label with the exact name now wins over the only clickable control

  • What is wrong: The exact-name rule runs before the older "pick the only invokable element" rule. So a Text element named exactly "Submit" now wins over a Button named "Submit order".
  • Show me: winapp ui invoke Submit -a myapp. Before this PR, the button was invoked. After it, the label is selected and invoke fails. ui click Submit clicks the label, exits 0, and presses nothing. The PR's test Query_ExactNameWinsOverLongerNameMatches(false,false) asserts this behavior.
  • Why it matters: Scripts that relied on the documented "only invokable element" rule can break. For click, the failure is silent.
  • Smallest fix: Either keep this order on purpose (the docs already describe it), or skip the exact-name pick when the winner isn't invokable and exactly one other match is.
  • Location: src/winapp-CLI/WinApp.UIAutomation/Services/UiAutomationService.cs:774 and :1943

What was exercised

  • Release publish (win-arm64): succeeded. The binary was run against real Calculator. ui search "Clear" still lists all 3 matches. ui invoke "Clear" (also with --action invoke and --type Button) picked btn-clearbutton-d050 and exited 0.
  • Targeted exact-name tests: 11 of 11 passed against a real WinForms window.
  • Not exercised: the label-vs-button case on a real app (only the test fixture covers it), and a before/after run on main.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one optional comment.

@nmetulev

Copy link
Copy Markdown
Member Author

Not changing in this PR: an exact Name is an explicit signal and the only-invokable rule is a tiebreaker; swapping them would bring back #1090's own case (a text named exactly X losing to a "Welcome to X" row). \invoke\ still invokes the label's invokable parent when it has one.

@nmetulev Nikola Metulev (nmetulev) added agent-preparing Agent is addressing feedback or completing required validation and CI ready-for-review Agent work and technical checks complete; awaiting review or re-review, not approval or merge and removed ready-for-review Agent work and technical checks complete; awaiting review or re-review, not approval or merge agent-preparing Agent is addressing feedback or completing required validation and CI labels Oct 15, 2026
@nmetulev
Nikola Metulev (nmetulev) merged commit fb43195 into main Oct 15, 2026
73 of 77 checks passed
@nmetulev
Nikola Metulev (nmetulev) deleted the nmetulev-ui-exact-name-preference branch October 15, 2026 19:50
Nikola Metulev (nmetulev) added a commit that referenced this pull request Oct 16, 2026
…enchmarks/ui (#1053)

## What and why

Adds `benchmarks/ui/`, two repeatable benchmarks for `winapp ui`. Rerun
them after a `winapp ui` change to see its effect.

- **`run.ps1` (agent benchmark).** A Copilot CLI agent does seven
natural-language tasks in inbox Calculator and a pinned WinUI 3 Gallery
(`v2.9.3`, built once by `setup-gallery.ps1`). The tasks include
real-world friction: the app isn't running, two instances are open, the
window is minimized, a dialog was left open, and a cold start. The
harness scores the final app state itself through UIA, the process count
and the window state. It never trusts the agent's answer, and it flags
collateral damage such as extra instances, the wrong window changed, or
an app closed. The report shows the pass rate, wall time, turns, tool vs
model time, tokens and credits. Methods: `winapp` (this repo's plugin
and CLI) and `none` (no plugin, `winapp` hidden).
- **`perf.ps1` (scripted, no agent, no credits).** It times single
`winapp ui` commands (20 runs after a warm-up; median, p95 and max). It
also runs fixed command sequences that mirror the agent tasks, scored
with the same checks.

Both benchmarks refuse to start when Calculator or the Gallery is
already open (unless you pass `-Force`), close only the processes they
launched (by PID), and restore your Calculator mode afterward. `run.ps1`
resumes when you rerun it into the same `-OutDir`. Other tools can be
benchmarked locally with `-Method <file>` or `-Adapter <file>`; the
README documents both formats.

`benchmarks/agents/lib/Benchmark.psm1` is reused, not copied. Its only
change is additive: `Read-SessionEvents` now also returns tool time, API
time and session duration, and `Format-Count` is exported.

## Before / after

New scripts; nothing in `winapp` itself changes. Observed output of
`perf.ps1` on current `main` (x64 Xeon, NativeAOT build of `901e4cde`,
Calculator only; excerpt):

```text
> .\benchmarks\ui\perf.ps1 -Operation startup,list-windows,search,inspect-calculator,get-property,get-value,invoke,wait-for,focus,screenshot -Sequence calc-add,calc-two-instances,calc-minimized-scientific
  startup              winapp     median     91 ms  p95     93 ms  errors 0
  list-windows         winapp     median    102 ms  p95    105 ms  errors 0
  search               winapp     median    259 ms  p95    267 ms  errors 0
  get-value            winapp     median    144 ms  p95    158 ms  errors 0
  invoke               winapp     median    177 ms  p95    183 ms  errors 0
  focus                winapp     median    176 ms  p95    182 ms  errors 0
[1/3] calc-add | winapp ... pass | 1,914 ms | 10 commands
[1/3] calc-two-instances | winapp ... pass | 1,383 ms | 7 commands
[1/3] calc-minimized-scientific | winapp ... pass | 1,993 ms | 10 commands
```

Agent benchmark, first full run on 2026-10-09: branch `93b2cbb8` on main
`a87a8304`, ARM64 Snapdragon X2 Elite, Copilot CLI 1.0.93,
`claude-sonnet-5.5`, 5 iterations. **70/70 passed with no collateral
damage**, in 32 minutes and 497.5 credits.

| Method | Median wall time | Median turns | Credits (35 runs) |
|---|---|---|---|
| `winapp` | 19.4 s | 6 | 315.7 |
| `none` | 21.8 s | 5 | 181.8 |

`winapp` was 20–30% faster on the Gallery tasks, about even on simple
Calculator tasks, and slower on the minimized-window task (36.9 s vs
15.0 s). It used about 1.7× the credits, mostly to load the
UI-automation skill.

## Not tested / risks

- The agent numbers above predate several `main` changes and were not
re-run: minimized-window handling in `winapp ui` (#1054), the
`winapp-performance` skill (#837), and selector changes (#1102, #1104).
The `gallery-dialog-left-open` task also changed since then. It now asks
to turn the switch off, because the switch starts on. The
minimized-window result in particular is likely better now, because
`perf.ps1`'s `calc-minimized-scientific` sequence, which failed at its
first step on 10/9, passes 3/3 on current `main`.
- I did not re-run the Gallery perf sequences here, because the Gallery
is not installed on this machine.
- Neither benchmark runs in CI: both drive the desktop, and the agent
benchmark costs credits. Only the Pester suites are suitable for
automation.

## Checklist

- [x] Tests added or updated
- [x] Tested locally on Windows

<details><summary>Notes for reviewers</summary>

Validated on the merged head:
- `Invoke-Pester benchmarks\ui\tests`: 135/135 passed.
- `Invoke-Pester benchmarks\tasks\tests`: 54/54 passed.
- `Invoke-Pester benchmarks\agents\tests`: 157 passed, 2 failed. `main`
has the same 2 failures from the `winapp-performance` skill, and #1060
fixes them.
- `benchmarks\agents\run.ps1 -Lint` and `-Plan` give the same output as
`main`: 1 error, which is the same known issue, and 1,584 sessions.
- `benchmarks\ui\run.ps1 -Plan`, `run.ps1 -Lint` (0 errors) and
`perf.ps1 -Plan` all succeed.

</details>

---------

Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Zach Teutsch <88554871+zateutsch@users.noreply.github.com>
Copilot-Session: 0cf8e7b3-7171-42da-9482-89167c7668b8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review Agent work and technical checks complete; awaiting review or re-review, not approval or merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Exact-name targeting for ui verbs that take a selector

3 participants