Skip to content

[Feature]: Scoped, exact and point targets for the ui press verbs, plus a horizontal wheel and a pinch angle #1084

Closed
6 of 6 issues completed

Description

Is your feature request related to a problem? Please describe.

Following #798: with 0.7.1 we moved every pointer and keyboard press in our WinUI 3 UI suite (49 suites, PowerShell 7) onto winapp ui. A suite finds the element in-process with UI Automation and hands it to winapp to press. Scoped and typed queries, FontWeight through get-property, --action toggle-on and the DPI in status all work as described against our app. Thank you for those.

Six gaps decide where we still keep input of our own or work around winapp. Each is small, and they came out of the same port, so they are listed together.

  1. The press verbs take no --root / --type, and a text selector is a substring. search, get-property and wait-for take --root, --type and --class-name; click, hover, invoke, drag, scroll and set-value do not, and their selector matches any name that contains it. winapp ui click "Allodia Mail & Calendar" -w <hwnd> pressed a list row named "Welcome to Allodia Mail & Calendar" rather than the title-bar text named exactly that. We now resolve a slug with search … --type, match name and bounds ourselves, and only then press, which costs a second winapp process per press.
  2. --class-name cannot be the whole query. search requires a text selector, so --class-name narrows a text match but cannot stand alone. "The element of class CalendarSurface" cannot be asked without knowing its name, and * matches nothing.
  3. click and hover take no point. touch --at x,y and drag x,y x,y accept coordinates; click and hover accept only a selector. Some presses need a point: inside a WebView2's page, or on part of a control rather than its centre (the header line of an expanded tree row, whose bounds take in its children). We currently right-click a point with drag <p> <p> --right, which works but reads as something else.
  4. No horizontal wheel. scroll --wheel N synthesises vertical notches only. Our calendar pans on a horizontal wheel (a tilt wheel or a touchpad), so we still inject MOUSEEVENTF_HWHEEL ourselves.
  5. touch --gesture pinch has no angle. Pinch and stretch spread along one axis. A diagonal pinch is the only way to exercise a two-axis zoom, which we inject with our own InjectTouchInput helper.
  6. invokableAncestor in get-property carries empty geometry. For a TextBlock inside a list item, the nested object reads "isEnabled": false, "x": 0, "y": 0, "width": 0, "height": 0, while the same ancestor read on its own is enabled with real bounds. A consumer of the nested object gets a disabled element at the origin.

Our code base is open source and can be found here: https://github.com/allodia-eu/mail-calendar/tree/30c8523d2d3865c8a3a6662ff6ef0de165bb401c/clients/windows

Describe the solution you'd like

  1. --root, --type and --class-name on click, hover, invoke, drag, scroll and set-value, with the same meaning they have on search; and an exact-name option (--exact, or name= in the selector) on every verb that takes a selector.
  2. --class-name (and --type) accepted without a text selector.
  3. --at x,y on click and hover, in the physical screen coordinates inspect reports.
  4. A horizontal wheel: --wheel-horizontal N, or --wheel N --axis horizontal.
  5. --angle <degrees> for touch --gesture pinch and stretch.
  6. invokableAncestor populated with the ancestor's real isEnabled and bounds, or those fields left out.

Additional context

Windows 11 Pro ARM64 10.0.26200, winapp 0.7.1, WinUI 3 (Windows App SDK 2.4.0) unpackaged Debug build, PowerShell 7. The cycle through a WebView2 that we hit during the same port is reported separately as a bug.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions