Skip to content

Add ui touch and ui pen commands - #600

Merged
Nikola Metulev (nmetulev) merged 31 commits into
mainfrom
yeelam/ui-touch-pen
Jul 16, 2026
Merged

Nikola Metulev (nmetulev) merged 31 commits into
mainfrom
yeelam/ui-touch-pen

Conversation

@yeelam-gordon

Copy link
Copy Markdown
Contributor

Summary

  • add winapp ui touch for tap, swipe, pinch, stretch, and related touch gestures
  • add winapp ui pen for stylus tap/inking with pressure, tilt, and eraser options
  • regenerate the CLI schema, npm command surface, and UI automation skill/docs

Why

This expands WinApp CLI beyond mouse/keyboard input so touch-first and pen/stylus experiences can be exercised directly in automation.

Adds `winapp ui touch` and `winapp ui pen` so automation can inject touch
gestures (tap, swipe, pinch, stretch) and stylus/inking input (tap, path,
pressure, tilt, eraser) against running Windows apps.

Also updates the generated CLI schema, npm command surface, and UI automation
skill/docs to surface the new commands.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 6, 2026 03:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the winapp ui automation surface beyond mouse/keyboard by adding two new OS-level input-injection commands, ui touch and ui pen, so touch-first and pen/stylus experiences can be driven directly from the CLI. It introduces a testable IPointerInput abstraction (with a native PointerInput P/Invoke implementation using the Windows synthetic-pointer and legacy touch-injection APIs), a pure PointerGesturePlanner for coordinate parsing and gesture expansion, and a new IUiAutomationService.TryGetWindowRect for bounds-checking. It follows the existing gesture-command pattern (resolve session → resolve selector/coords → foreground → stabilize → bounds-check → inject) while adding stricter safety (no_target rejection when no window resolves, and per-point window bounds checks). Schema, npm wrapper, and skill/docs are regenerated to match.

Changes:

  • New ui touch (tap/double-tap/long-press/swipe/pinch/stretch) and ui pen (tap/ink with pressure, tilt, eraser) commands, wired into UiCommand, DI, and the IPointerInput injection abstraction.
  • New helpers: PointerInput (native touch/pen injection with synthetic→legacy fallback), PointerGesturePlanner (parsing + gesture geometry + bounds), plus TryGetWindowRect, new JSON result/error models, and System.Single support in the npm generator/schema context.
  • Unit tests for both commands (happy + several error paths) and regenerated docs/schema/skill artifacts.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/winapp-CLI/WinApp.Cli/Commands/UiTouchCommand.cs New touch command: validation, target/foreground/bounds gating, injection, JSON output.
src/winapp-CLI/WinApp.Cli/Commands/UiPenCommand.cs New pen command: pressure/tilt validation, path/at/selector targeting, injection.
src/winapp-CLI/WinApp.Cli/Commands/UiCommand.cs Registers touch/pen subcommands.
src/winapp-CLI/WinApp.Cli/Helpers/PointerInput.cs Native synthetic-pointer/legacy touch + pen injection with cleanup.
src/winapp-CLI/WinApp.Cli/Helpers/PointerGesturePlanner.cs Coordinate/path parsing, gesture expansion, out-of-bounds detection.
src/winapp-CLI/WinApp.Cli/Helpers/IPointerInput.cs / RealPointerInput.cs Injection abstraction + production delegate.
src/winapp-CLI/WinApp.Cli/Helpers/HostBuilderExtensions.cs DI registration for IPointerInput and the two handlers.
src/winapp-CLI/WinApp.Cli/Helpers/UiJsonContext.cs / UiJsonError.cs New result models (UiTouchResult/UiPenResult/UiPointResult) and no_target/injection_unsupported codes.
src/winapp-CLI/WinApp.Cli/Helpers/CliSchema.cs Adds float to the schema serializer context (for --pressure default).
src/winapp-CLI/WinApp.Cli/Services/IUiAutomationService.cs / UiAutomationService.cs TryGetWindowRect contract + GetWindowRect implementation.
src/winapp-CLI/WinApp.Cli/NativeMethods.txt Adds touch/pen injection P/Invoke entries and structs.
src/winapp-CLI/WinApp.Cli.Tests/* FakePointerInput, TryGetWindowRect fake, and new Touch/Pen command tests.
src/winapp-npm/src/winapp-commands.ts Regenerated uiTouch/uiPen wrappers.
src/winapp-npm/scripts/generate-commands.mjs Maps System.Single → TS number.
scripts/generate-llm-docs.ps1 Adds ui touch/ui pen to the ui-automation skill map.
docs/usage.md, docs/ui-automation.md, docs/fragments/skills/..., docs/cli-schema.json, .github/plugin/.../SKILL.md, .claude/.../SKILL.md Documentation + regenerated schema/skill artifacts for the new commands.

Notes for the author (not anchorable to changed lines): The hand-written "Key subcommands" list in .github/plugin/agents/winapp.agent.md (lines 207-223) still lists ui drag/ui hover/ui send-keys but not the new touch/pen. That file is not auto-generated from the schema, so it should be updated alongside docs/usage.md to keep the shipped Copilot agent aware of the new verbs.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@github-actions

github-actions Bot commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

Build Metrics Report

Binary Sizes

Artifact Baseline Current Delta
CLI (ARM64) 36.56 MB 36.75 MB 📈 +198.0 KB (+0.53%)
CLI (x64) 36.74 MB 36.93 MB 📈 +193.0 KB (+0.51%)
MSIX (ARM64) 15.27 MB 15.35 MB 📈 +78.4 KB (+0.50%)
MSIX (x64) 16.22 MB 16.30 MB 📈 +83.0 KB (+0.50%)
NPM Package 31.89 MB 32.04 MB 📈 +158.5 KB (+0.49%)
NuGet Package 31.92 MB 32.07 MB 📈 +155.6 KB (+0.48%)

Test Results

✅ 3242 passed, 4 skipped out of 3246 tests in 868.2s (+186 tests, +106.1s vs. baseline)

Test Coverage

✅ 83.6% line coverage, 77.5% branch coverage · ✅ no change vs. baseline

CLI Startup Time

34ms median (x64, winapp --version) · ✅ -10ms vs. baseline


Updated 2026-07-16 04:38:14 UTC · commit b21f2a0 · workflow run

@nmetulev Nikola Metulev (nmetulev) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review — produced by GitHub Copilot CLI at a maintainer's request. Every finding comes from an automated, multi-dimensional review plus a hands-on build-and-test of this branch in an isolated worktree. Treat file:line references as pointers to verify, not gospel. Nothing was pushed. Posting as a comment (no approve / request-changes).

Verdict: 🟡 SHIP-WITH-CHANGES — Solid injection safety model and the modern pointer API works without hardware; the gaps are ergonomics and agent discovery.

What I actually ran

Built (~29s, 0 warnings); 70/70 touch+pen unit tests pass. Verified by direct P/Invoke that CreateSyntheticPointerDevice(PT_TOUCH/PT_PEN) succeeds on this no-touchscreen machine (handles returned, err=0) — so the modern path is viable without a digitizer. All validation/error paths return structured JSON + exit 1 (missing_app, invalid gesture, bad coords, pinch requires --distance, fingers 1–10, pressure 0–1, tilt ±90, path parse), and the bounds-check fires before injection.

⚠️ Environmental note: actual on-screen gesture effect could not be visually confirmed — this was a remote/headless desktop and injection hit the same foreground_not_target guard that gates all input verbs (click/drag/hover/scroll/send-keys). Not a PR bug.

Must-fix (High)

  • H1 — agent file not updated. ui touch and ui pen are absent from .github/plugin/agents/winapp.agent.md, so Copilot agents won't autonomously choose them.
  • H2 — swipe has no direction. --gesture swipe --distance N always moves +X (right) (PointerGesturePlanner.cs:125-134); there's no --direction. An agent wanting swipe up/down/left must compute absolute coordinates by hand — inconsistent with ui scroll --direction. Add --direction {right|left|up|down}.

Medium / Low

  • M1 double-checked-lock on _touchInitialized isn't volatile (PointerInput.cs:53-56). Safe on x64 TSO but spec-noncompliant; one-word fix.
  • M2 --gesture long-press with the default --hold-ms 0 silently behaves like a tap. Auto-default (~500ms) or warn.
  • M3 pen inter-point delay is hardcoded to 10ms with no --duration-ms (asymmetric with touch); stroke timing is untested.
  • L1 Interpolate discards intermediate waypoints (latent — only 1–2 pt paths today). L2 pen lacks any timing control.

Fit & recommendation

This is a natural extension of the existing input-verb family and fills real gaps a mouse can't emulate — multi-touch pinch/stretch, pen pressure/tilt/eraser (Ink Workspace, OneNote, Maps zoom). The IPointerInput/Real/Fake abstraction mirrors IMouseInput cleanly and unit-tests run headless. Caveats: these are long-tail scenarios (most agent work is click/type), and — like all input verbs — the injection paths can't be exercised in headless CI. Ship after the swipe-direction (H2) and agent-file (H1) fixes.

@chiaramooney Chiara Mooney (chiaramooney) added this to the 0.5 milestone Jul 14, 2026
Zach Teutsch (zateutsch) and others added 15 commits July 14, 2026 16:33
… default, volatile init, docs & bindings

Addresses AI review of PR #600: swipe --direction {right|left|up|down}; pen --duration-ms; long-press defaults to 500ms; volatile _touchInitialized; add ui touch/pen to agent decision file; regenerate cli-schema, skills, npm bindings; tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
The Debug-built CLI reports assembly version 1.0.0 (version.json's 0.4.1 is
only injected in release builds), and generate-llm-docs derives the doc
version from the schema the CLI emits. Restore the repo version 0.4.1 in the
generated schema, plugin metadata, skill frontmatter, and .claude mirror so
the PR diff contains only the intended touch/pen content changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…en travel-time glide, Contains off-by-one, touch JSON holdMs, docs

H1 — Long-press now emits periodic STATIONARY UPDATE frames (every 40ms) during
the hold instead of a single Thread.Sleep. This prevents Windows touch injection
from cancelling or mis-classifying the contact. Adds HoldFrameIntervalMs=40
constant; the loop clamps the final partial interval so total hold ≈ holdMs.
Multi-contact gestures send a frame per contact. Tap (holdMs<=0) path unchanged.
Pinned with PointerInputFrameTests: long-press emits ≥1 UPDATE between DOWN and
UP; tap emits exactly DOWN+UP. (PointerInput.cs, IPointerInput.cs)

H2 — Pen --duration-ms is now the actual stroke TRAVEL TIME. Added
InjectPenStroke(internal) that interpolates GlideSteps (20) UPDATE frames per
path segment, distributing durationMs via a running-remainder budget so no ms are
systematically lost. durationMs<=0 falls back to the original ~10ms-per-waypoint
cadence. Pen() delegates to InjectPenStroke via a PenFrameSender lambda.
Pinned with PointerInputFrameTests: 2-point stroke with durationMs set emits 20
monotonically-progressing UPDATE frames; positions reach the destination; durationMs=0
emits exactly 1 UPDATE per waypoint. (PointerInput.cs)

M1 — PointerRect.Contains: Right/Bottom are exclusive (GetWindowRect semantics).
Changed p.X<=Right && p.Y<=Bottom to p.X<Right && p.Y<Bottom. Updated XML doc
comment. PointerInputFrameTests covers: (Right-1,Bottom-1) inside; (Right,y) and
(x,Bottom) outside; left/top edges inclusive. (IPointerInput.cs)

M2 — UiTouchResult gains HoldMs property (int) populated with the effective
hold time (500 default for long-press, or explicit --hold-ms). Registered in
UiJsonContext; UiCommandTests.Touch now asserts holdMs appears in JSON output for
long-press (default 500), explicit value, and tap (0). (UiJsonContext.cs,
UiTouchCommand.cs, UiCommandTests.Touch.cs)

L — docs/ui-automation.md touch options table now lists --direction
{right|left|up|down} (default right) and clarifies --hold-ms defaults to 500ms for
long-press. Pen options table now lists --duration-ms and explains it controls
travel time via interpolated UPDATE frames. (docs/ui-automation.md)

Regenerated autogen docs (winapp-commands.ts) with Debug exe; reverted version
stamps to 0.4.1 (no version churn).

NOTE: H1/H2 fixes are unit-test-pinned (PointerInputFrameTests) but NOT
live-validated — shared/remote desktop prevents real touch/pen injection.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…inite pressure

M1 — PointerInput.cs InjectPenStroke: replace budget-based timing (Math.Max(1,
msBudget/stepsLeft) sleep-after-send) with cumulative target-timestamp scheduling.
For frame index k (1..N), targetMs_k = durationMs*k/N; before each frame sleep
max(0, targetMs_k - elapsed). Fixes two bugs:
  - Math.Max(1,...) inflated --duration-ms 1 to ~20ms (20 steps x 1ms floor).
  - Sleep after every frame, including the endpoint, created a trailing dwell beyond
    the scheduled duration.
Add optional Action<int> sleep / Func<long> nowMs seams to InjectPenStroke so unit
tests can inject a fake clock and assert timing without real Thread.Sleep delays.
Replace frame-count-only timing test with injected-clock tests pinning:
  (a) total sleep == durationMs for a 1-segment and 2-segment stroke,
  (b) --duration-ms 1 yields total sleep=1ms, not ~20ms (step-count inflation gone),
  (c) no trailing dwell: last event after endpoint UPDATE is the UP frame, not a sleep.

M2 — UiPenCommand.cs: fold float.IsFinite into the pressure guard so NaN and
Infinity are rejected before injection instead of passing the range check and
causing JSON serialisation to throw later. Updated error message to say
'finite number'. New tests: Pen_PressureNaN_Rejected_NoInjection,
Pen_PressureInfinity_Rejected_NoInjection, Pen_TiltXNonInteger_Rejected_NoInjection
(Option<int> rejects non-integer at parse time; no code change needed for tilt
since int is always finite).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…eduleGlide; ordered-event no-dwell tests; assert invalid_arguments in non-finite pressure tests

M1: Apply the same cumulative-timestamp scheduler to InjectTouchStroke that
round-3 applied to InjectPenStroke. Extracted a shared private ScheduleGlide
helper that both call. Touch and pen timed loops now produce drift-corrected
frames with no Math.Max(1,...) floor and no trailing sleep after the endpoint
frame. Added injectable sleep/nowMs seam to InjectTouchStroke for unit tests.
Files: PointerInput.cs

M2: Rewrote InjectPenStroke_WithDurationMs_NoTrailingDwellAfterEndpoint to
use an ordered event log (List<string> with sleep:N / frame:UPDATE@(x,y) /
frame:UP entries). Assert the tail is exactly endpoint-UPDATE followed by UP
with no sleep between them -- a regression WILL fail this test. Added the
same ordered-event style for the new InjectTouchStroke no-dwell test, plus
three more touch timing tests (total-sleep approx durationMs, small duration
not inflated, monotonic multi-contact).
Files: PointerInputFrameTests.cs

M3: Added optional TextWriter? errorOut to UiJsonError.Emit so callers can
redirect the JSON error to a capturable writer. UiPenCommand passes
parseResult.InvocationConfiguration.Error, which ParseAndInvokeWithCaptureAsync
sets to ConsoleStdErr. Pen_PressureNaN_Rejected_NoInjection now parses the
captured stderr JSON and asserts error.code == 'invalid_arguments'.
Pen_PressureInfinity_Rejected_NoInjection is updated to reflect that
System.CommandLine rejects 'Infinity' at parse time (no handler runs, no JSON
emitted), and asserts exit 1 + PenCalls.Count == 0 + stderr non-empty.
Files: UiJsonError.cs, UiPenCommand.cs, UiCommandTests.Pen.cs

All 103 targeted tests (Pen + PointerInput + Touch) pass. Timing is pinned
via injected fake clock -- no live device injection.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…ath; honor explicit --hold-ms 0

HIGH 1 (PointerInput.cs): Mirror MouseInput.Drag's released-flag pattern in both
InjectTouchStroke and InjectPenStroke. On the normal completion path, the UP/release
frame is sent without any try-catch so a failure propagates to the caller (command
exits non-zero with injection_unsupported, user knows the pointer is stuck). In the
unwind path (!released in finally), the UP is best-effort: exception is swallowed to
avoid masking the original, more-informative hold/glide exception.

LOW 1 (UiTouchCommand.cs): Detect whether --hold-ms was actually supplied via
parseResult.GetResult(HoldOption)?.Tokens.Count > 0. Apply the 500 ms long-press
default only when the option was omitted. An explicit --hold-ms 0 with long-press is
a degenerate combination and is now rejected with a structured invalid_arguments
error instead of being silently rewritten to 500 ms.

Tests added (PointerInputFrameTests.cs):
- InjectTouchStroke_UpFrameFailsOnNormalPath_ExceptionSurfaced
- InjectTouchStroke_GlideFrameThrows_OriginalExceptionPreserved_UpFailureSwallowed
- InjectPenStroke_UpFrameFailsOnNormalPath_ExceptionSurfaced
- InjectPenStroke_GlideFrameThrows_OriginalExceptionPreserved_UpFailureSwallowed

Tests added (UiCommandTests.Touch.cs):
- Touch_LongPress_ExplicitHoldMsZero_RejectedWithInvalidArguments
- Touch_InjectThrowsInvalidOperation_ReturnsNonZeroWithStructuredError

FakePointerInput.ThrowException added so command-level tests can inject failures
without a live injection path. All 109 targeted tests pass (0 failures).
Build: 0 warnings, 0 errors. Zero version churn.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
M1: regenerate docs/npm-usage.md (uiPen/uiTouch were missing); --check now passes.

M3: strengthen injection-failure tests — assert error.code == 'injection_unsupported'
    and empty stdout for both touch (updated) and pen (new test).

M4: change --pressure from Option<float> to Option<string?> so '--pressure nope --json'
    reaches the handler and emits a structured invalid_arguments JSON error instead of
    SCL's plain-text parse-failure message. Updated Pen_PressureInfinity test (now
    produces JSON error from handler rather than raw SCL output). New
    Pen_PressureTypo_Json_EmitsStructuredInvalidArgumentsError test validates the repro.
    Note: --tilt-x/--tilt-y (Option<int>) retain the pre-existing SCL-level rejection
    for non-integer values; only --pressure (the validated repro) is fixed here.

M5: skip --at parse/validation when --path is supplied so 'ui pen --path ... --at nope'
    no longer errors on --at (matching the documented 'ignored when --path is given').

M6 (UiPenCommand) / M8 (UiTouchCommand): narrow InvalidOperationException catch to
    only wrap the actual injection call (pointerInput.Pen/Touch). Pre-injection failures
    from ResolveSessionAsync / element resolution now surface through GenericError
    (internal_error or stale_element) instead of being mis-labeled injection_unsupported.
    Also pass errorOut: parseResult.InvocationConfiguration.Error so test harnesses
    correctly capture the JSON error payload.

M7: remove redundant second SetForegroundWindow+Delay in UiPenCommand. The selector
    branch already calls SetForeground during stable-resolve; the --at and --path
    branches do not, so SetForeground is now conditional on (pathFromOption || at != null).
    Eliminates the superfluous 100ms re-raise that reopened a target-movement race for
    selector-resolved pen taps.

M9: add PointerGesturePlannerTests.cs with planner unit tests covering pinch/stretch
    geometry (two-finger start/end coordinates, convergence/divergence direction),
    finger-count coercion to 2 for pinch/stretch, bounds rejection via FirstOutOfBounds,
    double-tap path structure, and double-tap repetition cycle count (2x InjectTouchStroke
    = 2 DOWN + 2 UP frames).

L1: fix targetLabel to report the effective injection target — atStr when --at was given,
    selectorStr when selector resolved the center (touch); pathStr > atStr > selectorStr
    (pen). Previously selectorStr could win over an explicit --at.

M2 (no code fix): added comment in PointerInput.Touch() noting that the P/Invoke
    device-create → payload-marshal → destroy path requires an interactive-desktop
    test lane; fake-injector tests cover planning/ordering only.

Regen: docs/cli-schema.json, .github/plugin/, .claude/ updated from Debug exe;
    version churn (1.0.0 → 0.4.1) reverted in all generated SKILL.md frontmatter,
    cli-schema.json, and plugin.json.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
H1 — Revert --pressure to Option<float> (numeric type restored)
- UiPenCommand.cs: PressureOption back to Option<float> with default 0.5f
- docs/cli-schema.json: valueType now "System.Single" (was "System.String")
- generate-commands.mjs --check passes; pressure?: number in winapp-commands.ts confirmed

M1 — Reorder semantic validation before missing-app check
- UiPenCommand.cs: pressure/duration/tilt validation runs before the
  missing-app/window guard so `--pressure 5 --json` (no -a) returns
  invalid_arguments, not missing_app

M2 — Shared parse-error → JSON interceptor for ui commands
- Program.cs: check parseResult.Errors before InvokeAsync; if --json
  present emit {"error":{"code":"invalid_arguments",...}} to stdout,
  suppress help banner, exit 1
- BaseCommandTests.cs: same interceptor in ParseAndInvokeWithCaptureAsync
  so tests exercise the same path as production
- New tests: Pen_PressureTypo_NoApp_Json (M1), Pen_TiltXTypo_Json (M2),
  Pen_TiltYTypo_Json (M2); existing tests updated with stdout-empty assertions

M3 — Retry Win32 ERROR_NOT_READY in legacy touch fallback
- PointerInput.cs: SendFrameWithRetry retries up to MaxErrorNotReadyRetries
  (10) on Win32 error 21 with 1ms delay before surfacing the exception
- IsWin32ErrorNotReady helper matches "Win32 error 21:" in message
- New tests: transient-fail (3 retries then success) and always-fail
  (surfaces after 11 calls) in PointerInputFrameTests.cs

M4 — Real double-tap regression test via RunTouchGesture
- PointerInput.cs: RunTouchGesture made internal static with injectable
  sleepInter Action<int> parameter for testable inter-tap delay
- PointerGesturePlannerTests.cs: replaced manual 2x InjectTouchStroke
  test with RunTouchGesture_DoubleTap test asserting 2 DOWN+UP cycles
  and exactly 1 inter-tap sleep of 60ms

M5 — npm regen: both generate-commands and generate-docs checks pass
- docs/cli-schema.json regenerated (System.Single for --pressure, 0.4.1)
- generate-commands.mjs --check: PASS; generate-docs.mjs --check: PASS
- Note: generate-docs drift was pre-existing on origin/main for the
  execWithBuildTools/ExecSyncOptions sections; regenerated here anyway

Runtime validation (debug binary):
  ui pen --pressure nope --json          → invalid_arguments exit 1
  ui pen --tilt-x nope -a __no__ --json  → invalid_arguments exit 1
  ui pen --pressure 0.5 --json           → invalid_arguments (missing target) exit 1
  ui pen --pressure 5 --json             → invalid_arguments exit 1

Tests: 184 filtered CLI tests pass (0 failed); 179 npm tests pass

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
M1: derive JSON mode from selected command's PARSED --json option (not raw
    token scan). New ResolveEffectiveJson() checks that the innermost matched
    command actually owns WinAppRootCommand.JsonOption and reads the parsed
    boolean value, so '--json false' and commands without --json (e.g. complete)
    never trigger the bridge.

M2: unify single-dash typo path with JSON bridge. Previously the typo check
    returned plain-text before the bridge; now when effectiveJson is true the
    typo path also emits a JSON envelope via UiJsonError.Emit.

M3: emit CommandCompletedEvent(exitCode=1) in the bridge early-return so
    telemetry CommandInvoked/CommandCompleted events stay paired.

M4: delete the duplicated parse-error bridge from BaseCommandTests.cs
    ParseAndInvokeWithCaptureAsync. Add InvokeProgramAsync protected static
    helper (same pattern as UpdateNotificationGatingTests). Create new
    ProgramJsonBridgeTests.cs ([DoNotParallelize]) that routes all parse-error
    and bridge-behaviour tests through the real Program.Main. Move the four
    parse-error tests (--pressure nope, --tilt-x nope, --tilt-y nope,
    Infinity pressure) from UiCommandTests.Pen.cs to ProgramJsonBridgeTests.cs.
    Add M1/M2 regression tests: --json false => no envelope; complete wat --json
    => no envelope; -pressure nope --json => JSON envelope; valid parse => no
    false invalid_arguments.

M5: add Pen_PressureOutOfRange_NoApp_EmitsInvalidArguments_NotMissingApp in
    UiCommandTests.Pen.cs: parseable --pressure 5 (> 1.0) with no -a returns
    invalid_arguments (range check) before missing_app (ordering pin).

M6: add RunTouchGesture_NonError21_PropagatesImmediately_CalledExactlyOnce in
    PointerInputFrameTests.cs: non-Win32-error-21 exception propagates on the
    first call with no retry, confirming the when-guard in SendFrameWithRetry.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
M1 (JSON mode split): Add value-aware GetBooleanFlagValue to GlobalOptionPreScan
that understands --json, --json true/false, --json=true/false.  Use it for the
pre-scan so logger config + conflict detection match the real parsed value.
Remove the jsonPreScan gate from ResolveEffectiveJson; always read the parsed
WinAppRootCommand.JsonOption value as the single source of truth.

M3 (UI schema on non-UI commands): Add IsUiDescendant() helper that walks the
Command.Parents chain looking for the 'ui' command.  Gate the parse-error JSON
bridge on IsUiDescendant() so cert info / other non-ui commands fall through to
SCL default error handling rather than getting the nested UI schema.

M4 (UiTouchCommand guard order): Move the missing-app check to AFTER all
app-independent argument validation (gesture allow-list, --at/--to-point
coordinate parsing, --fingers range, --direction, hold/duration/distance
ranges).  Invalid arg values now return invalid_arguments regardless of
whether --app is supplied.

M5 (UiPenCommand guard order): Move --path and --at coordinate parsing to
BEFORE the missing-app check (pressure/duration/tilt were already moved in r8).
Invalid --path / --at values now return invalid_arguments, not missing_app.

Adds regression tests in ProgramJsonBridgeTests for all four fixes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…ash IsUiDescendant gate, M4/M5 test hardening

M1 (pen missing_app parity): Add catch(InvalidOperationException) in both
UiPenCommand and UiTouchCommand outer catch chain. When ResolveSessionAsync
throws because the requested --app is not running, emit missing_app instead
of internal_error. Injection IOE is caught by the inner try/catch and never
reaches the outer catch. Applied to both commands so pen and touch agree.

M2 (single-dash typo bridge gate): Added IsUiDescendant(parsedArgs) gate to
the typo JSON branch in Program.cs so non-UI commands (e.g. cert info) fall
through to plain-text error handling instead of receiving the nested UI schema.

M4 (structural test assertion): Fixed JsonBridge_NonUiCommand_ParseError_
NoNestedUiSchema -- the old assertion matched "error":{ (no space) but the
serializer emits "error": { (indented). Replaced with a helper that
deserialises any JSON found and checks structurally for {error:{code,...}}.

M5 (pen success-path coverage): Added Pen_NonDefaultPressureAndTilt_
PassedThroughToInjector. Uses FakePointerInput (no live injection) to verify
non-default --pressure and --tilt-x/--tilt-y reach the injector and appear
in the success JSON envelope.

New regression tests in ProgramJsonBridgeTests.cs:
  - Pen_AppNotFound_WithAt_ReturnsMissingApp (M1)
  - Pen_AppNotFound_WithPath_ReturnsMissingApp (M1)
  - Touch_AppNotFound_WithAt_ReturnsMissingApp (M1 parity)
  - PenAndTouch_NoTarget_AppNotFound_BothReturnInvalidArguments (M1 ordering)
  - JsonBridge_SingleDashTypo_NonUiCommand_NoNestedUiSchema (M2)
  - JsonBridge_SingleDashTypo_UiCommand_StillGetsNestedSchema (M2 guard)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…pNotFoundException

M2 (boolean pre-scan coerces invalid attached value to true):
- GlobalOptionPreScan.GetBooleanFlagValue: replace the non-'false' shortcut with
  bool.TryParse so that --json=bogus (or any non-bool value) is treated as
  indeterminate (false) rather than true. This prevents the spurious --json/--verbose
  conflict when the user misspells the boolean value; the invalid value now flows
  through to System.CommandLine's normal parse path, which reports it correctly.
  Verified: --json=bogus --verbose no longer produces a conflict; --json=false,
  --json, --json=true, --json true, --json false continue to behave unchanged.
  Added GetBooleanFlagValue regression tests covering all spellings, including the
  new --json=bogus=false case.

M1 (InvalidOperationException catch too broad — mislabels selector ambiguity as missing_app):
- Added AppNotFoundException : InvalidOperationException in WinApp.Cli.Services so
  UiSessionService can throw a typed exception for app-not-found conditions (no process
  found, no window found). The pen/touch outer catch is now ordered:
    COMException  → stale_element
    AppNotFoundException  → missing_app   (specific: session resolution failure)
    InvalidOperationException → invalid_arguments  (generic: selector ambiguity etc.)
    Exception → internal_error
  Before this fix, selector-ambiguity throws from UiAutomationService
  (FindSingleElementAsync: 'Selector matched N elements') were caught as missing_app.
  Now they correctly map to invalid_arguments. Passes errorOut in the new catch blocks
  so ParseAndInvokeWithCaptureAsync-based unit tests can capture the JSON.
  Added fake-driven unit tests: AppNotFoundException -> missing_app (NOT internal_error);
  selector-ambiguity IOE -> invalid_arguments (NOT missing_app, NOT internal_error).
  Updated FakeUiSessionService (ThrowException) and FakeUiAutomationService
  (FindSingleElementThrowException) to support the new test seams.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…gus (#600 M1)

System.CommandLine silently coerces an '='-attached non-boolean value such as
--json=bogus to true (no parse error), while the value-aware pre-scan reads it as
false. That disagreement left logging un-suppressed (emitting the human "no app
found" line) while the parsed --json was true (emitting the JSON envelope), so
BOTH reached stderr and corrupted the machine-readable --json output.

Reject an invalid '='-attached value on --json/--verbose/--quiet up front with a
single clean invalid_arguments error, mirroring how the parser already rejects
other malformed option values (e.g. --pressure nope). The command never runs, so
no logger line can leak alongside the error. --json intent -> JSON envelope;
verbose/quiet -> plain text. Valid --json=true / --json=false and space-separated
forms are unaffected.

- GlobalOptionPreScan.TryFindInvalidBooleanValue detects the invalid attached value
- Program.cs fails fast before command invocation
- +12 tests (9 pre-scan unit + 3 end-to-end bridge)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…e test harness

review-600-r12 found the round-12 --json=bogus fix was under-scoped. This
generalizes and hardens it:

- H1/M2: reject an invalid '='-attached value on ANY boolean option reachable by
  the selected command (its own Option<bool> flags + inherited/global ones), not
  just --json/--verbose/--quiet. Catches --eraser=bogus on `ui pen` (which SCL
  silently coerces to true, switching the pen to its eraser end) and any future
  bool option automatically. TryFindInvalidBooleanValue now scans ALL occurrences
  so a later bad value (--json=true --json=bogus) is still rejected, not accepted.
- M1: move the rejection BEFORE the first-run/update notice so a cold cache no
  longer leaks the first-run banner to stdout ahead of the JSON error.
- M3: make the --verbose/--quiet conflict checks value-aware via
  GetBooleanFlagValue so `--verbose false --json` no longer false-conflicts and
  `--verbose=true --json` correctly conflicts.
- M5: InvokeProgramAsync now isolates Program.Main's internally-built first-run
  and update services from the real ~/.winapp cache (WINAPP_CLI_CACHE_DIRECTORY
  redirect + WINAPP_CLI_UPDATE_CHECK=0), restoring env afterwards. Adds an opt-in
  coldCache parameter for first-run tests.

+13 tests. Build 0/0; 72 bridge/prescan/gating + 151 pen/touch/pointer pass; 8
runtime probes confirmed. Behavior-only — no schema/doc/version churn.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…rrect erase semantics (#600 M6)

Set synthetic pen eraser input to report both inverted and eraser pen flags so downstream ink stacks can recognize the eraser end via either flag.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…ble options (#600 M7, M1)

Cap synthetic touch/pen delay arguments at 60 seconds and reject explicitly supplied options that do not apply to the selected gesture or pen mode before target resolution.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…#600 M3)

Resolve child HWNDs to their GA_ROOT ancestor before reading the bounds rect so touch/pen bounds checks use the documented top-level window rectangle.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
… remote session (#600 id27/id28)

Synthetic touch/pen injection via InjectSyntheticPointerInput can return success
in a remote (RDP / Terminal Services) session without the gesture actually
reaching the target application - pen routing in particular does not survive
Remote Desktop. Previously both commands reported success / exit 0 regardless,
giving agents false confidence that a gesture landed.

Detect remote sessions via GetSystemMetrics(SM_REMOTESESSION) behind the existing
IForegroundGuard seam and, on a successful injection, attach an honest
delivery-uncertainty advisory: a new warnings[] array in the --json envelope
(UiTouchResult/UiPenResult) and a LogWarning line in text mode. The advisory tells
the caller to verify the effect (ui screenshot / ui inspect) and notes delivery is
reliable on a local, physically-attached session. Exit code stays 0 because the
injection call itself did succeed - only delivery is uncertain.

ForegroundGuard.RemoteInjectionWarning is a pure, side-effect-free function so it
is unit-tested without a live remote session, and IsRemoteSession() is on
IForegroundGuard so command behavior is driven by a fake in tests. Adds 5 tests
(touch/pen remote+local + pure composition).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Share the x,y parser and malformed-coordinate heuristic between drag and pointer gesture planning so touch, pen, and drag use the same parsing semantics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Unify the shared touch/pen selector-or-coordinate target resolution, foreground nudge, no-target/window-rect/out-of-bounds/foreground safety checks, injection_unsupported catch, and remote-session warning lookup in PointerCommandSupport. Gesture-specific touch planning and pen argument handling remain in their command handlers to preserve existing validation and result payload behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Make the single-dash long-option typo JSON bridge emit command invoked/completed telemetry for the selected UI command, matching the normal parse-error bridge path. Add an EventSource-backed regression test that verifies ui pen typo telemetry carries the UiPenCommand context and exit code 1.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Add skip-by-default interactive desktop smoke coverage for the real native touch and pen PointerInput paths. The tests require WINAPP_UI_INJECTION_LIVE=1, skip in non-interactive/remote/locked-like sessions, use a throwaway Notepad target, and leave the recorder-based frame tests as the normal CI coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…ing (#600 id29)

Touch/pen inject can report success (exit 0) in a Remote Desktop / VM
session without the input actually reaching the target app; pen routing
over RDP is especially unreliable. The commands already attach a runtime
delivery-uncertainty warning (warnings[] in --json, a warning line in
text) in that case. Document that behavior in the prose UI-automation
docs and the agent skill so a green check is not mistaken for confirmed
delivery, and point users to validate pen flows on a local interactive
desktop.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Resolved conflicts (5 files):
- Program.cs: combined main's logging refactor (ResolveLoggingMode record +
  RunWithTelemetryAsync invoke() delegate seam) with 600's semantic features
  (parse-error->JSON bridge, value-aware boolean flag parsing, GlobalOptionPreScan,
  AppNotFoundException, id17 single-dash typo telemetry). Kept ResolveLoggingMode
  value-aware (GetBooleanFlagValue) so --json false / --verbose false parse right.
- FakeUiServices.cs: union of both throw-hook API sets (via git merge-file).
- UiSessionServiceTests.cs: updated 4 assertions to expect AppNotFoundException
  (600's InvalidOperationException subclass) instead of the base type.
- BaseCommandTests.cs / UiCommandTests.cs: union of DI fakes; took main's
  cancellationToken argument.

All merge-affected suites green (ProgramJsonBridge 41, GlobalOptionPreScan 27,
UiCommandTests 312, UiSessionService 27, PointerInputFrame 28, GesturePlanner 28,
ProgramMain/OptionTypoValidator/CompleteCommand/CustomHelp). Build 0/0, zero
version churn (0.4.1 preserved).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
A synthetic pen stroke jumped straight to the DOWN (contact) frame with no
prior in-range hover. Windows latches the pen transducer type (tip vs.
inverted/eraser end) at the moment the pointer ENTERS RANGE, so a flag set only
on an already-in-contact frame is ignored: PEN_FLAG_INVERTED/ERASER never
reached the target app and --eraser was a silent no-op at delivery (pressure
and tilt came through fine). On-device WM_POINTER validation confirmed
penFlags=0x0 on every frame.

Fix InjectPenStroke to establish in-range state before contact, mirroring the
working reference injectors (draw-stream, EGoTouchRev, zpenflow):

- Emit a hover-arrival frame (INRANGE|UPDATE, pressure 0, no contact) at the
  start point BEFORE DOWN. The send closure carries the pen flags to every
  frame, so the eraser flag is now present on range entry.
- Lift with UP|INRANGE (clean lift to hover) instead of a bare UP; the
  synthetic device is destroyed right after, which retracts the in-range
  pointer.

ComputePenFlags is unchanged (INVERTED|ERASER = 0x6 is correct). No CLI surface
change, so no docs/schema regen.

Update PointerInputFrameTests for the new sequence (hover first, then DOWN;
frame counts +1; glide = in-contact UPDATE frames only) and add
InjectPenStroke_EmitsInRangeHoverArrivalBeforeContact as a regression guard.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
… pen/touch)

PR #599 (ui record) merged to main after this branch. Resolved 4 conflicts:
- UiJsonError.cs: union error codes (no_target/injection_unsupported from #600 +
  ambiguous_selector from #599).
- scripts/generate-llm-docs.ps1: union $SkillCommandMap ui-automation list to
  include ui record AND ui touch/ui pen.
- docs/cli-schema.json, src/winapp-npm/src/winapp-commands.ts: regenerated from
  the merged CLI (both command sets present; version pinned 0.4.1, zero churn).

Regenerated all autogenerated docs/skills/.claude mirror + npm wrapper; both
generated-file parity checks pass. Build 0/0; PointerInputFrame 27 pass/2 skip,
Record 50 pass/1 skip, 0 failures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…er is delivered

The hover-arrival fix added an in-range frame before DOWN, but the send closure
stamped every frame (hover, contact, glide, lift) with ComputePenFlags(eraser),
which returns INVERTED|ERASER (0x6). Windows silently drops any out-of-contact
frame that carries PEN_FLAG_ERASER, so the eraser transducer never latched on
range entry and the stroke was still delivered as an ordinary tip (penFlags 0x0).

Make penFlags a function of the frame's pointer flags: report PEN_FLAG_INVERTED
only on out-of-contact frames (hover-arrival, lift) and the full INVERTED|ERASER
only on in-contact frames (DOWN, glide). This mirrors real hardware, where the
flipped tip reports "inverted" on hover and asserts "eraser" only on contact.

Verified end-to-end against a WM_POINTER receiver: eraser tap and stroke now
deliver penFlags=0x6 on every contact frame (was 0x0). Adds a unit test asserting
ERASER is stripped out of contact and kept on contact.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6919645-3ac6-4180-84ca-176373ee4726
@nmetulev
Nikola Metulev (nmetulev) merged commit 93ed711 into main Jul 16, 2026
22 checks passed
@nmetulev
Nikola Metulev (nmetulev) deleted the yeelam/ui-touch-pen branch July 16, 2026 04:42
Alexandre Zollinger Chohfi (azchohfi) added a commit that referenced this pull request Jul 16, 2026
…iAutomation fake

Main #600 (ui touch/pen) added TryGetWindowRect(long, out PointerRect) to
IUiAutomationService. This branch's GestureTargetingTests QueueUiAutomation fake predates it,
so the head+main merge build failed (CS0535). Implement it as a safe no-op (returns false /
default rect) since the gesture-targeting tests resolve elements via FindSingleElementAsync and
do not depend on a window rect. FakeUiAutomationService already implements it (from main).

Debug + Release(both) 0W/0E; 69 gesture/input/system tests pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants