Repository navigation
Add ui touch and ui pen commands - #600
Conversation
Adds `winapp ui touch` and `winapp ui pen` so automation can inject touch gestures (tap, swipe, pinch, stretch) and stylus/inking input (tap, path, pressure, tilt, eraser) against running Windows apps. Also updates the generated CLI schema, npm command surface, and UI automation skill/docs to surface the new commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR extends the winapp ui automation surface beyond mouse/keyboard by adding two new OS-level input-injection commands, ui touch and ui pen, so touch-first and pen/stylus experiences can be driven directly from the CLI. It introduces a testable IPointerInput abstraction (with a native PointerInput P/Invoke implementation using the Windows synthetic-pointer and legacy touch-injection APIs), a pure PointerGesturePlanner for coordinate parsing and gesture expansion, and a new IUiAutomationService.TryGetWindowRect for bounds-checking. It follows the existing gesture-command pattern (resolve session → resolve selector/coords → foreground → stabilize → bounds-check → inject) while adding stricter safety (no_target rejection when no window resolves, and per-point window bounds checks). Schema, npm wrapper, and skill/docs are regenerated to match.
Changes:
- New
ui touch(tap/double-tap/long-press/swipe/pinch/stretch) andui pen(tap/ink with pressure, tilt, eraser) commands, wired intoUiCommand, DI, and theIPointerInputinjection abstraction. - New helpers:
PointerInput(native touch/pen injection with synthetic→legacy fallback),PointerGesturePlanner(parsing + gesture geometry + bounds), plusTryGetWindowRect, new JSON result/error models, andSystem.Singlesupport in the npm generator/schema context. - Unit tests for both commands (happy + several error paths) and regenerated docs/schema/skill artifacts.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/winapp-CLI/WinApp.Cli/Commands/UiTouchCommand.cs |
New touch command: validation, target/foreground/bounds gating, injection, JSON output. |
src/winapp-CLI/WinApp.Cli/Commands/UiPenCommand.cs |
New pen command: pressure/tilt validation, path/at/selector targeting, injection. |
src/winapp-CLI/WinApp.Cli/Commands/UiCommand.cs |
Registers touch/pen subcommands. |
src/winapp-CLI/WinApp.Cli/Helpers/PointerInput.cs |
Native synthetic-pointer/legacy touch + pen injection with cleanup. |
src/winapp-CLI/WinApp.Cli/Helpers/PointerGesturePlanner.cs |
Coordinate/path parsing, gesture expansion, out-of-bounds detection. |
src/winapp-CLI/WinApp.Cli/Helpers/IPointerInput.cs / RealPointerInput.cs |
Injection abstraction + production delegate. |
src/winapp-CLI/WinApp.Cli/Helpers/HostBuilderExtensions.cs |
DI registration for IPointerInput and the two handlers. |
src/winapp-CLI/WinApp.Cli/Helpers/UiJsonContext.cs / UiJsonError.cs |
New result models (UiTouchResult/UiPenResult/UiPointResult) and no_target/injection_unsupported codes. |
src/winapp-CLI/WinApp.Cli/Helpers/CliSchema.cs |
Adds float to the schema serializer context (for --pressure default). |
src/winapp-CLI/WinApp.Cli/Services/IUiAutomationService.cs / UiAutomationService.cs |
TryGetWindowRect contract + GetWindowRect implementation. |
src/winapp-CLI/WinApp.Cli/NativeMethods.txt |
Adds touch/pen injection P/Invoke entries and structs. |
src/winapp-CLI/WinApp.Cli.Tests/* |
FakePointerInput, TryGetWindowRect fake, and new Touch/Pen command tests. |
src/winapp-npm/src/winapp-commands.ts |
Regenerated uiTouch/uiPen wrappers. |
src/winapp-npm/scripts/generate-commands.mjs |
Maps System.Single → TS number. |
scripts/generate-llm-docs.ps1 |
Adds ui touch/ui pen to the ui-automation skill map. |
docs/usage.md, docs/ui-automation.md, docs/fragments/skills/..., docs/cli-schema.json, .github/plugin/.../SKILL.md, .claude/.../SKILL.md |
Documentation + regenerated schema/skill artifacts for the new commands. |
Notes for the author (not anchorable to changed lines): The hand-written "Key subcommands" list in .github/plugin/agents/winapp.agent.md (lines 207-223) still lists ui drag/ui hover/ui send-keys but not the new touch/pen. That file is not auto-generated from the schema, so it should be updated alongside docs/usage.md to keep the shipped Copilot agent aware of the new verbs.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Build Metrics ReportBinary Sizes
Test Results✅ 3242 passed, 4 skipped out of 3246 tests in 868.2s (+186 tests, +106.1s vs. baseline) Test Coverage✅ 83.6% line coverage, 77.5% branch coverage · ✅ no change vs. baseline CLI Startup Time34ms median (x64, Updated 2026-07-16 04:38:14 UTC · commit |
Nikola Metulev (nmetulev)
left a comment
There was a problem hiding this comment.
🤖 AI-generated review — produced by GitHub Copilot CLI at a maintainer's request. Every finding comes from an automated, multi-dimensional review plus a hands-on build-and-test of this branch in an isolated worktree. Treat
file:linereferences as pointers to verify, not gospel. Nothing was pushed. Posting as a comment (no approve / request-changes).
Verdict: 🟡 SHIP-WITH-CHANGES — Solid injection safety model and the modern pointer API works without hardware; the gaps are ergonomics and agent discovery.
What I actually ran
Built (~29s, 0 warnings); 70/70 touch+pen unit tests pass. Verified by direct P/Invoke that CreateSyntheticPointerDevice(PT_TOUCH/PT_PEN) succeeds on this no-touchscreen machine (handles returned, err=0) — so the modern path is viable without a digitizer. All validation/error paths return structured JSON + exit 1 (missing_app, invalid gesture, bad coords, pinch requires --distance, fingers 1–10, pressure 0–1, tilt ±90, path parse), and the bounds-check fires before injection.
⚠️ Environmental note: actual on-screen gesture effect could not be visually confirmed — this was a remote/headless desktop and injection hit the sameforeground_not_targetguard that gates all input verbs (click/drag/hover/scroll/send-keys). Not a PR bug.
Must-fix (High)
- H1 — agent file not updated.
ui touchandui penare absent from.github/plugin/agents/winapp.agent.md, so Copilot agents won't autonomously choose them. - H2 — swipe has no direction.
--gesture swipe --distance Nalways moves +X (right) (PointerGesturePlanner.cs:125-134); there's no--direction. An agent wanting swipe up/down/left must compute absolute coordinates by hand — inconsistent withui scroll --direction. Add--direction {right|left|up|down}.
Medium / Low
- M1 double-checked-lock on
_touchInitializedisn'tvolatile(PointerInput.cs:53-56). Safe on x64 TSO but spec-noncompliant; one-word fix. - M2
--gesture long-presswith the default--hold-ms 0silently behaves like a tap. Auto-default (~500ms) or warn. - M3 pen inter-point delay is hardcoded to 10ms with no
--duration-ms(asymmetric with touch); stroke timing is untested. - L1
Interpolatediscards intermediate waypoints (latent — only 1–2 pt paths today). L2 pen lacks any timing control.
Fit & recommendation
This is a natural extension of the existing input-verb family and fills real gaps a mouse can't emulate — multi-touch pinch/stretch, pen pressure/tilt/eraser (Ink Workspace, OneNote, Maps zoom). The IPointerInput/Real/Fake abstraction mirrors IMouseInput cleanly and unit-tests run headless. Caveats: these are long-tail scenarios (most agent work is click/type), and — like all input verbs — the injection paths can't be exercised in headless CI. Ship after the swipe-direction (H2) and agent-file (H1) fixes.
… default, volatile init, docs & bindings Addresses AI review of PR #600: swipe --direction {right|left|up|down}; pen --duration-ms; long-press defaults to 500ms; volatile _touchInitialized; add ui touch/pen to agent decision file; regenerate cli-schema, skills, npm bindings; tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
The Debug-built CLI reports assembly version 1.0.0 (version.json's 0.4.1 is only injected in release builds), and generate-llm-docs derives the doc version from the schema the CLI emits. Restore the repo version 0.4.1 in the generated schema, plugin metadata, skill frontmatter, and .claude mirror so the PR diff contains only the intended touch/pen content changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…en travel-time glide, Contains off-by-one, touch JSON holdMs, docs
H1 — Long-press now emits periodic STATIONARY UPDATE frames (every 40ms) during
the hold instead of a single Thread.Sleep. This prevents Windows touch injection
from cancelling or mis-classifying the contact. Adds HoldFrameIntervalMs=40
constant; the loop clamps the final partial interval so total hold ≈ holdMs.
Multi-contact gestures send a frame per contact. Tap (holdMs<=0) path unchanged.
Pinned with PointerInputFrameTests: long-press emits ≥1 UPDATE between DOWN and
UP; tap emits exactly DOWN+UP. (PointerInput.cs, IPointerInput.cs)
H2 — Pen --duration-ms is now the actual stroke TRAVEL TIME. Added
InjectPenStroke(internal) that interpolates GlideSteps (20) UPDATE frames per
path segment, distributing durationMs via a running-remainder budget so no ms are
systematically lost. durationMs<=0 falls back to the original ~10ms-per-waypoint
cadence. Pen() delegates to InjectPenStroke via a PenFrameSender lambda.
Pinned with PointerInputFrameTests: 2-point stroke with durationMs set emits 20
monotonically-progressing UPDATE frames; positions reach the destination; durationMs=0
emits exactly 1 UPDATE per waypoint. (PointerInput.cs)
M1 — PointerRect.Contains: Right/Bottom are exclusive (GetWindowRect semantics).
Changed p.X<=Right && p.Y<=Bottom to p.X<Right && p.Y<Bottom. Updated XML doc
comment. PointerInputFrameTests covers: (Right-1,Bottom-1) inside; (Right,y) and
(x,Bottom) outside; left/top edges inclusive. (IPointerInput.cs)
M2 — UiTouchResult gains HoldMs property (int) populated with the effective
hold time (500 default for long-press, or explicit --hold-ms). Registered in
UiJsonContext; UiCommandTests.Touch now asserts holdMs appears in JSON output for
long-press (default 500), explicit value, and tap (0). (UiJsonContext.cs,
UiTouchCommand.cs, UiCommandTests.Touch.cs)
L — docs/ui-automation.md touch options table now lists --direction
{right|left|up|down} (default right) and clarifies --hold-ms defaults to 500ms for
long-press. Pen options table now lists --duration-ms and explains it controls
travel time via interpolated UPDATE frames. (docs/ui-automation.md)
Regenerated autogen docs (winapp-commands.ts) with Debug exe; reverted version
stamps to 0.4.1 (no version churn).
NOTE: H1/H2 fixes are unit-test-pinned (PointerInputFrameTests) but NOT
live-validated — shared/remote desktop prevents real touch/pen injection.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…inite pressure
M1 — PointerInput.cs InjectPenStroke: replace budget-based timing (Math.Max(1,
msBudget/stepsLeft) sleep-after-send) with cumulative target-timestamp scheduling.
For frame index k (1..N), targetMs_k = durationMs*k/N; before each frame sleep
max(0, targetMs_k - elapsed). Fixes two bugs:
- Math.Max(1,...) inflated --duration-ms 1 to ~20ms (20 steps x 1ms floor).
- Sleep after every frame, including the endpoint, created a trailing dwell beyond
the scheduled duration.
Add optional Action<int> sleep / Func<long> nowMs seams to InjectPenStroke so unit
tests can inject a fake clock and assert timing without real Thread.Sleep delays.
Replace frame-count-only timing test with injected-clock tests pinning:
(a) total sleep == durationMs for a 1-segment and 2-segment stroke,
(b) --duration-ms 1 yields total sleep=1ms, not ~20ms (step-count inflation gone),
(c) no trailing dwell: last event after endpoint UPDATE is the UP frame, not a sleep.
M2 — UiPenCommand.cs: fold float.IsFinite into the pressure guard so NaN and
Infinity are rejected before injection instead of passing the range check and
causing JSON serialisation to throw later. Updated error message to say
'finite number'. New tests: Pen_PressureNaN_Rejected_NoInjection,
Pen_PressureInfinity_Rejected_NoInjection, Pen_TiltXNonInteger_Rejected_NoInjection
(Option<int> rejects non-integer at parse time; no code change needed for tilt
since int is always finite).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…eduleGlide; ordered-event no-dwell tests; assert invalid_arguments in non-finite pressure tests M1: Apply the same cumulative-timestamp scheduler to InjectTouchStroke that round-3 applied to InjectPenStroke. Extracted a shared private ScheduleGlide helper that both call. Touch and pen timed loops now produce drift-corrected frames with no Math.Max(1,...) floor and no trailing sleep after the endpoint frame. Added injectable sleep/nowMs seam to InjectTouchStroke for unit tests. Files: PointerInput.cs M2: Rewrote InjectPenStroke_WithDurationMs_NoTrailingDwellAfterEndpoint to use an ordered event log (List<string> with sleep:N / frame:UPDATE@(x,y) / frame:UP entries). Assert the tail is exactly endpoint-UPDATE followed by UP with no sleep between them -- a regression WILL fail this test. Added the same ordered-event style for the new InjectTouchStroke no-dwell test, plus three more touch timing tests (total-sleep approx durationMs, small duration not inflated, monotonic multi-contact). Files: PointerInputFrameTests.cs M3: Added optional TextWriter? errorOut to UiJsonError.Emit so callers can redirect the JSON error to a capturable writer. UiPenCommand passes parseResult.InvocationConfiguration.Error, which ParseAndInvokeWithCaptureAsync sets to ConsoleStdErr. Pen_PressureNaN_Rejected_NoInjection now parses the captured stderr JSON and asserts error.code == 'invalid_arguments'. Pen_PressureInfinity_Rejected_NoInjection is updated to reflect that System.CommandLine rejects 'Infinity' at parse time (no handler runs, no JSON emitted), and asserts exit 1 + PenCalls.Count == 0 + stderr non-empty. Files: UiJsonError.cs, UiPenCommand.cs, UiCommandTests.Pen.cs All 103 targeted tests (Pen + PointerInput + Touch) pass. Timing is pinned via injected fake clock -- no live device injection. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…ath; honor explicit --hold-ms 0 HIGH 1 (PointerInput.cs): Mirror MouseInput.Drag's released-flag pattern in both InjectTouchStroke and InjectPenStroke. On the normal completion path, the UP/release frame is sent without any try-catch so a failure propagates to the caller (command exits non-zero with injection_unsupported, user knows the pointer is stuck). In the unwind path (!released in finally), the UP is best-effort: exception is swallowed to avoid masking the original, more-informative hold/glide exception. LOW 1 (UiTouchCommand.cs): Detect whether --hold-ms was actually supplied via parseResult.GetResult(HoldOption)?.Tokens.Count > 0. Apply the 500 ms long-press default only when the option was omitted. An explicit --hold-ms 0 with long-press is a degenerate combination and is now rejected with a structured invalid_arguments error instead of being silently rewritten to 500 ms. Tests added (PointerInputFrameTests.cs): - InjectTouchStroke_UpFrameFailsOnNormalPath_ExceptionSurfaced - InjectTouchStroke_GlideFrameThrows_OriginalExceptionPreserved_UpFailureSwallowed - InjectPenStroke_UpFrameFailsOnNormalPath_ExceptionSurfaced - InjectPenStroke_GlideFrameThrows_OriginalExceptionPreserved_UpFailureSwallowed Tests added (UiCommandTests.Touch.cs): - Touch_LongPress_ExplicitHoldMsZero_RejectedWithInvalidArguments - Touch_InjectThrowsInvalidOperation_ReturnsNonZeroWithStructuredError FakePointerInput.ThrowException added so command-level tests can inject failures without a live injection path. All 109 targeted tests pass (0 failures). Build: 0 warnings, 0 errors. Zero version churn. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
M1: regenerate docs/npm-usage.md (uiPen/uiTouch were missing); --check now passes.
M3: strengthen injection-failure tests — assert error.code == 'injection_unsupported'
and empty stdout for both touch (updated) and pen (new test).
M4: change --pressure from Option<float> to Option<string?> so '--pressure nope --json'
reaches the handler and emits a structured invalid_arguments JSON error instead of
SCL's plain-text parse-failure message. Updated Pen_PressureInfinity test (now
produces JSON error from handler rather than raw SCL output). New
Pen_PressureTypo_Json_EmitsStructuredInvalidArgumentsError test validates the repro.
Note: --tilt-x/--tilt-y (Option<int>) retain the pre-existing SCL-level rejection
for non-integer values; only --pressure (the validated repro) is fixed here.
M5: skip --at parse/validation when --path is supplied so 'ui pen --path ... --at nope'
no longer errors on --at (matching the documented 'ignored when --path is given').
M6 (UiPenCommand) / M8 (UiTouchCommand): narrow InvalidOperationException catch to
only wrap the actual injection call (pointerInput.Pen/Touch). Pre-injection failures
from ResolveSessionAsync / element resolution now surface through GenericError
(internal_error or stale_element) instead of being mis-labeled injection_unsupported.
Also pass errorOut: parseResult.InvocationConfiguration.Error so test harnesses
correctly capture the JSON error payload.
M7: remove redundant second SetForegroundWindow+Delay in UiPenCommand. The selector
branch already calls SetForeground during stable-resolve; the --at and --path
branches do not, so SetForeground is now conditional on (pathFromOption || at != null).
Eliminates the superfluous 100ms re-raise that reopened a target-movement race for
selector-resolved pen taps.
M9: add PointerGesturePlannerTests.cs with planner unit tests covering pinch/stretch
geometry (two-finger start/end coordinates, convergence/divergence direction),
finger-count coercion to 2 for pinch/stretch, bounds rejection via FirstOutOfBounds,
double-tap path structure, and double-tap repetition cycle count (2x InjectTouchStroke
= 2 DOWN + 2 UP frames).
L1: fix targetLabel to report the effective injection target — atStr when --at was given,
selectorStr when selector resolved the center (touch); pathStr > atStr > selectorStr
(pen). Previously selectorStr could win over an explicit --at.
M2 (no code fix): added comment in PointerInput.Touch() noting that the P/Invoke
device-create → payload-marshal → destroy path requires an interactive-desktop
test lane; fake-injector tests cover planning/ordering only.
Regen: docs/cli-schema.json, .github/plugin/, .claude/ updated from Debug exe;
version churn (1.0.0 → 0.4.1) reverted in all generated SKILL.md frontmatter,
cli-schema.json, and plugin.json.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
H1 — Revert --pressure to Option<float> (numeric type restored)
- UiPenCommand.cs: PressureOption back to Option<float> with default 0.5f
- docs/cli-schema.json: valueType now "System.Single" (was "System.String")
- generate-commands.mjs --check passes; pressure?: number in winapp-commands.ts confirmed
M1 — Reorder semantic validation before missing-app check
- UiPenCommand.cs: pressure/duration/tilt validation runs before the
missing-app/window guard so `--pressure 5 --json` (no -a) returns
invalid_arguments, not missing_app
M2 — Shared parse-error → JSON interceptor for ui commands
- Program.cs: check parseResult.Errors before InvokeAsync; if --json
present emit {"error":{"code":"invalid_arguments",...}} to stdout,
suppress help banner, exit 1
- BaseCommandTests.cs: same interceptor in ParseAndInvokeWithCaptureAsync
so tests exercise the same path as production
- New tests: Pen_PressureTypo_NoApp_Json (M1), Pen_TiltXTypo_Json (M2),
Pen_TiltYTypo_Json (M2); existing tests updated with stdout-empty assertions
M3 — Retry Win32 ERROR_NOT_READY in legacy touch fallback
- PointerInput.cs: SendFrameWithRetry retries up to MaxErrorNotReadyRetries
(10) on Win32 error 21 with 1ms delay before surfacing the exception
- IsWin32ErrorNotReady helper matches "Win32 error 21:" in message
- New tests: transient-fail (3 retries then success) and always-fail
(surfaces after 11 calls) in PointerInputFrameTests.cs
M4 — Real double-tap regression test via RunTouchGesture
- PointerInput.cs: RunTouchGesture made internal static with injectable
sleepInter Action<int> parameter for testable inter-tap delay
- PointerGesturePlannerTests.cs: replaced manual 2x InjectTouchStroke
test with RunTouchGesture_DoubleTap test asserting 2 DOWN+UP cycles
and exactly 1 inter-tap sleep of 60ms
M5 — npm regen: both generate-commands and generate-docs checks pass
- docs/cli-schema.json regenerated (System.Single for --pressure, 0.4.1)
- generate-commands.mjs --check: PASS; generate-docs.mjs --check: PASS
- Note: generate-docs drift was pre-existing on origin/main for the
execWithBuildTools/ExecSyncOptions sections; regenerated here anyway
Runtime validation (debug binary):
ui pen --pressure nope --json → invalid_arguments exit 1
ui pen --tilt-x nope -a __no__ --json → invalid_arguments exit 1
ui pen --pressure 0.5 --json → invalid_arguments (missing target) exit 1
ui pen --pressure 5 --json → invalid_arguments exit 1
Tests: 184 filtered CLI tests pass (0 failed); 179 npm tests pass
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
M1: derive JSON mode from selected command's PARSED --json option (not raw
token scan). New ResolveEffectiveJson() checks that the innermost matched
command actually owns WinAppRootCommand.JsonOption and reads the parsed
boolean value, so '--json false' and commands without --json (e.g. complete)
never trigger the bridge.
M2: unify single-dash typo path with JSON bridge. Previously the typo check
returned plain-text before the bridge; now when effectiveJson is true the
typo path also emits a JSON envelope via UiJsonError.Emit.
M3: emit CommandCompletedEvent(exitCode=1) in the bridge early-return so
telemetry CommandInvoked/CommandCompleted events stay paired.
M4: delete the duplicated parse-error bridge from BaseCommandTests.cs
ParseAndInvokeWithCaptureAsync. Add InvokeProgramAsync protected static
helper (same pattern as UpdateNotificationGatingTests). Create new
ProgramJsonBridgeTests.cs ([DoNotParallelize]) that routes all parse-error
and bridge-behaviour tests through the real Program.Main. Move the four
parse-error tests (--pressure nope, --tilt-x nope, --tilt-y nope,
Infinity pressure) from UiCommandTests.Pen.cs to ProgramJsonBridgeTests.cs.
Add M1/M2 regression tests: --json false => no envelope; complete wat --json
=> no envelope; -pressure nope --json => JSON envelope; valid parse => no
false invalid_arguments.
M5: add Pen_PressureOutOfRange_NoApp_EmitsInvalidArguments_NotMissingApp in
UiCommandTests.Pen.cs: parseable --pressure 5 (> 1.0) with no -a returns
invalid_arguments (range check) before missing_app (ordering pin).
M6: add RunTouchGesture_NonError21_PropagatesImmediately_CalledExactlyOnce in
PointerInputFrameTests.cs: non-Win32-error-21 exception propagates on the
first call with no retry, confirming the when-guard in SendFrameWithRetry.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
M1 (JSON mode split): Add value-aware GetBooleanFlagValue to GlobalOptionPreScan that understands --json, --json true/false, --json=true/false. Use it for the pre-scan so logger config + conflict detection match the real parsed value. Remove the jsonPreScan gate from ResolveEffectiveJson; always read the parsed WinAppRootCommand.JsonOption value as the single source of truth. M3 (UI schema on non-UI commands): Add IsUiDescendant() helper that walks the Command.Parents chain looking for the 'ui' command. Gate the parse-error JSON bridge on IsUiDescendant() so cert info / other non-ui commands fall through to SCL default error handling rather than getting the nested UI schema. M4 (UiTouchCommand guard order): Move the missing-app check to AFTER all app-independent argument validation (gesture allow-list, --at/--to-point coordinate parsing, --fingers range, --direction, hold/duration/distance ranges). Invalid arg values now return invalid_arguments regardless of whether --app is supplied. M5 (UiPenCommand guard order): Move --path and --at coordinate parsing to BEFORE the missing-app check (pressure/duration/tilt were already moved in r8). Invalid --path / --at values now return invalid_arguments, not missing_app. Adds regression tests in ProgramJsonBridgeTests for all four fixes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…ash IsUiDescendant gate, M4/M5 test hardening
M1 (pen missing_app parity): Add catch(InvalidOperationException) in both
UiPenCommand and UiTouchCommand outer catch chain. When ResolveSessionAsync
throws because the requested --app is not running, emit missing_app instead
of internal_error. Injection IOE is caught by the inner try/catch and never
reaches the outer catch. Applied to both commands so pen and touch agree.
M2 (single-dash typo bridge gate): Added IsUiDescendant(parsedArgs) gate to
the typo JSON branch in Program.cs so non-UI commands (e.g. cert info) fall
through to plain-text error handling instead of receiving the nested UI schema.
M4 (structural test assertion): Fixed JsonBridge_NonUiCommand_ParseError_
NoNestedUiSchema -- the old assertion matched "error":{ (no space) but the
serializer emits "error": { (indented). Replaced with a helper that
deserialises any JSON found and checks structurally for {error:{code,...}}.
M5 (pen success-path coverage): Added Pen_NonDefaultPressureAndTilt_
PassedThroughToInjector. Uses FakePointerInput (no live injection) to verify
non-default --pressure and --tilt-x/--tilt-y reach the injector and appear
in the success JSON envelope.
New regression tests in ProgramJsonBridgeTests.cs:
- Pen_AppNotFound_WithAt_ReturnsMissingApp (M1)
- Pen_AppNotFound_WithPath_ReturnsMissingApp (M1)
- Touch_AppNotFound_WithAt_ReturnsMissingApp (M1 parity)
- PenAndTouch_NoTarget_AppNotFound_BothReturnInvalidArguments (M1 ordering)
- JsonBridge_SingleDashTypo_NonUiCommand_NoNestedUiSchema (M2)
- JsonBridge_SingleDashTypo_UiCommand_StillGetsNestedSchema (M2 guard)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…pNotFoundException
M2 (boolean pre-scan coerces invalid attached value to true):
- GlobalOptionPreScan.GetBooleanFlagValue: replace the non-'false' shortcut with
bool.TryParse so that --json=bogus (or any non-bool value) is treated as
indeterminate (false) rather than true. This prevents the spurious --json/--verbose
conflict when the user misspells the boolean value; the invalid value now flows
through to System.CommandLine's normal parse path, which reports it correctly.
Verified: --json=bogus --verbose no longer produces a conflict; --json=false,
--json, --json=true, --json true, --json false continue to behave unchanged.
Added GetBooleanFlagValue regression tests covering all spellings, including the
new --json=bogus=false case.
M1 (InvalidOperationException catch too broad — mislabels selector ambiguity as missing_app):
- Added AppNotFoundException : InvalidOperationException in WinApp.Cli.Services so
UiSessionService can throw a typed exception for app-not-found conditions (no process
found, no window found). The pen/touch outer catch is now ordered:
COMException → stale_element
AppNotFoundException → missing_app (specific: session resolution failure)
InvalidOperationException → invalid_arguments (generic: selector ambiguity etc.)
Exception → internal_error
Before this fix, selector-ambiguity throws from UiAutomationService
(FindSingleElementAsync: 'Selector matched N elements') were caught as missing_app.
Now they correctly map to invalid_arguments. Passes errorOut in the new catch blocks
so ParseAndInvokeWithCaptureAsync-based unit tests can capture the JSON.
Added fake-driven unit tests: AppNotFoundException -> missing_app (NOT internal_error);
selector-ambiguity IOE -> invalid_arguments (NOT missing_app, NOT internal_error).
Updated FakeUiSessionService (ThrowException) and FakeUiAutomationService
(FindSingleElementThrowException) to support the new test seams.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…gus (#600 M1) System.CommandLine silently coerces an '='-attached non-boolean value such as --json=bogus to true (no parse error), while the value-aware pre-scan reads it as false. That disagreement left logging un-suppressed (emitting the human "no app found" line) while the parsed --json was true (emitting the JSON envelope), so BOTH reached stderr and corrupted the machine-readable --json output. Reject an invalid '='-attached value on --json/--verbose/--quiet up front with a single clean invalid_arguments error, mirroring how the parser already rejects other malformed option values (e.g. --pressure nope). The command never runs, so no logger line can leak alongside the error. --json intent -> JSON envelope; verbose/quiet -> plain text. Valid --json=true / --json=false and space-separated forms are unaffected. - GlobalOptionPreScan.TryFindInvalidBooleanValue detects the invalid attached value - Program.cs fails fast before command invocation - +12 tests (9 pre-scan unit + 3 end-to-end bridge) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…e test harness review-600-r12 found the round-12 --json=bogus fix was under-scoped. This generalizes and hardens it: - H1/M2: reject an invalid '='-attached value on ANY boolean option reachable by the selected command (its own Option<bool> flags + inherited/global ones), not just --json/--verbose/--quiet. Catches --eraser=bogus on `ui pen` (which SCL silently coerces to true, switching the pen to its eraser end) and any future bool option automatically. TryFindInvalidBooleanValue now scans ALL occurrences so a later bad value (--json=true --json=bogus) is still rejected, not accepted. - M1: move the rejection BEFORE the first-run/update notice so a cold cache no longer leaks the first-run banner to stdout ahead of the JSON error. - M3: make the --verbose/--quiet conflict checks value-aware via GetBooleanFlagValue so `--verbose false --json` no longer false-conflicts and `--verbose=true --json` correctly conflicts. - M5: InvokeProgramAsync now isolates Program.Main's internally-built first-run and update services from the real ~/.winapp cache (WINAPP_CLI_CACHE_DIRECTORY redirect + WINAPP_CLI_UPDATE_CHECK=0), restoring env afterwards. Adds an opt-in coldCache parameter for first-run tests. +13 tests. Build 0/0; 72 bridge/prescan/gating + 151 pen/touch/pointer pass; 8 runtime probes confirmed. Behavior-only — no schema/doc/version churn. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…rrect erase semantics (#600 M6) Set synthetic pen eraser input to report both inverted and eraser pen flags so downstream ink stacks can recognize the eraser end via either flag. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…ble options (#600 M7, M1) Cap synthetic touch/pen delay arguments at 60 seconds and reject explicitly supplied options that do not apply to the selected gesture or pen mode before target resolution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…#600 M3) Resolve child HWNDs to their GA_ROOT ancestor before reading the bounds rect so touch/pen bounds checks use the documented top-level window rectangle. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
… remote session (#600 id27/id28) Synthetic touch/pen injection via InjectSyntheticPointerInput can return success in a remote (RDP / Terminal Services) session without the gesture actually reaching the target application - pen routing in particular does not survive Remote Desktop. Previously both commands reported success / exit 0 regardless, giving agents false confidence that a gesture landed. Detect remote sessions via GetSystemMetrics(SM_REMOTESESSION) behind the existing IForegroundGuard seam and, on a successful injection, attach an honest delivery-uncertainty advisory: a new warnings[] array in the --json envelope (UiTouchResult/UiPenResult) and a LogWarning line in text mode. The advisory tells the caller to verify the effect (ui screenshot / ui inspect) and notes delivery is reliable on a local, physically-attached session. Exit code stays 0 because the injection call itself did succeed - only delivery is uncertain. ForegroundGuard.RemoteInjectionWarning is a pure, side-effect-free function so it is unit-tested without a live remote session, and IsRemoteSession() is on IForegroundGuard so command behavior is driven by a fake in tests. Adds 5 tests (touch/pen remote+local + pure composition). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Share the x,y parser and malformed-coordinate heuristic between drag and pointer gesture planning so touch, pen, and drag use the same parsing semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Unify the shared touch/pen selector-or-coordinate target resolution, foreground nudge, no-target/window-rect/out-of-bounds/foreground safety checks, injection_unsupported catch, and remote-session warning lookup in PointerCommandSupport. Gesture-specific touch planning and pen argument handling remain in their command handlers to preserve existing validation and result payload behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Make the single-dash long-option typo JSON bridge emit command invoked/completed telemetry for the selected UI command, matching the normal parse-error bridge path. Add an EventSource-backed regression test that verifies ui pen typo telemetry carries the UiPenCommand context and exit code 1. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Add skip-by-default interactive desktop smoke coverage for the real native touch and pen PointerInput paths. The tests require WINAPP_UI_INJECTION_LIVE=1, skip in non-interactive/remote/locked-like sessions, use a throwaway Notepad target, and leave the recorder-based frame tests as the normal CI coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…ing (#600 id29) Touch/pen inject can report success (exit 0) in a Remote Desktop / VM session without the input actually reaching the target app; pen routing over RDP is especially unreliable. The commands already attach a runtime delivery-uncertainty warning (warnings[] in --json, a warning line in text) in that case. Document that behavior in the prose UI-automation docs and the agent skill so a green check is not mistaken for confirmed delivery, and point users to validate pen flows on a local interactive desktop. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
Resolved conflicts (5 files): - Program.cs: combined main's logging refactor (ResolveLoggingMode record + RunWithTelemetryAsync invoke() delegate seam) with 600's semantic features (parse-error->JSON bridge, value-aware boolean flag parsing, GlobalOptionPreScan, AppNotFoundException, id17 single-dash typo telemetry). Kept ResolveLoggingMode value-aware (GetBooleanFlagValue) so --json false / --verbose false parse right. - FakeUiServices.cs: union of both throw-hook API sets (via git merge-file). - UiSessionServiceTests.cs: updated 4 assertions to expect AppNotFoundException (600's InvalidOperationException subclass) instead of the base type. - BaseCommandTests.cs / UiCommandTests.cs: union of DI fakes; took main's cancellationToken argument. All merge-affected suites green (ProgramJsonBridge 41, GlobalOptionPreScan 27, UiCommandTests 312, UiSessionService 27, PointerInputFrame 28, GesturePlanner 28, ProgramMain/OptionTypoValidator/CompleteCommand/CustomHelp). Build 0/0, zero version churn (0.4.1 preserved). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
A synthetic pen stroke jumped straight to the DOWN (contact) frame with no prior in-range hover. Windows latches the pen transducer type (tip vs. inverted/eraser end) at the moment the pointer ENTERS RANGE, so a flag set only on an already-in-contact frame is ignored: PEN_FLAG_INVERTED/ERASER never reached the target app and --eraser was a silent no-op at delivery (pressure and tilt came through fine). On-device WM_POINTER validation confirmed penFlags=0x0 on every frame. Fix InjectPenStroke to establish in-range state before contact, mirroring the working reference injectors (draw-stream, EGoTouchRev, zpenflow): - Emit a hover-arrival frame (INRANGE|UPDATE, pressure 0, no contact) at the start point BEFORE DOWN. The send closure carries the pen flags to every frame, so the eraser flag is now present on range entry. - Lift with UP|INRANGE (clean lift to hover) instead of a bare UP; the synthetic device is destroyed right after, which retracts the in-range pointer. ComputePenFlags is unchanged (INVERTED|ERASER = 0x6 is correct). No CLI surface change, so no docs/schema regen. Update PointerInputFrameTests for the new sequence (hover first, then DOWN; frame counts +1; glide = in-contact UPDATE frames only) and add InjectPenStroke_EmitsInRangeHoverArrivalBeforeContact as a regression guard. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
… pen/touch) PR #599 (ui record) merged to main after this branch. Resolved 4 conflicts: - UiJsonError.cs: union error codes (no_target/injection_unsupported from #600 + ambiguous_selector from #599). - scripts/generate-llm-docs.ps1: union $SkillCommandMap ui-automation list to include ui record AND ui touch/ui pen. - docs/cli-schema.json, src/winapp-npm/src/winapp-commands.ts: regenerated from the merged CLI (both command sets present; version pinned 0.4.1, zero churn). Regenerated all autogenerated docs/skills/.claude mirror + npm wrapper; both generated-file parity checks pass. Build 0/0; PointerInputFrame 27 pass/2 skip, Record 50 pass/1 skip, 0 failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62b906c-4985-4114-926c-2cf732a814d1
…er is delivered The hover-arrival fix added an in-range frame before DOWN, but the send closure stamped every frame (hover, contact, glide, lift) with ComputePenFlags(eraser), which returns INVERTED|ERASER (0x6). Windows silently drops any out-of-contact frame that carries PEN_FLAG_ERASER, so the eraser transducer never latched on range entry and the stroke was still delivered as an ordinary tip (penFlags 0x0). Make penFlags a function of the frame's pointer flags: report PEN_FLAG_INVERTED only on out-of-contact frames (hover-arrival, lift) and the full INVERTED|ERASER only on in-contact frames (DOWN, glide). This mirrors real hardware, where the flipped tip reports "inverted" on hover and asserts "eraser" only on contact. Verified end-to-end against a WM_POINTER receiver: eraser tap and stroke now deliver penFlags=0x6 on every contact frame (was 0x0). Adds a unit test asserting ERASER is stripped out of contact and kept on contact. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d6919645-3ac6-4180-84ca-176373ee4726
…iAutomation fake Main #600 (ui touch/pen) added TryGetWindowRect(long, out PointerRect) to IUiAutomationService. This branch's GestureTargetingTests QueueUiAutomation fake predates it, so the head+main merge build failed (CS0535). Implement it as a safe no-op (returns false / default rect) since the gesture-targeting tests resolve elements via FindSingleElementAsync and do not depend on a window rect. FakeUiAutomationService already implements it (from main). Debug + Release(both) 0W/0E; 69 gesture/input/system tests pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Summary
winapp ui touchfor tap, swipe, pinch, stretch, and related touch gestureswinapp ui penfor stylus tap/inking with pressure, tilt, and eraser optionsWhy
This expands WinApp CLI beyond mouse/keyboard input so touch-first and pen/stylus experiences can be exercised directly in automation.