Skip to content

Fix screenshot host-path reporting and composite help - #884

Merged
Zach Teutsch (zateutsch) merged 7 commits into
mainfrom
nmetulev-correct-screenshot-host-paths-and-compos
Sep 21, 2026
Merged

Zach Teutsch (zateutsch) merged 7 commits into
mainfrom
nmetulev-correct-screenshot-host-paths-and-compos

Conversation

@nmetulev

@nmetulev Nikola Metulev (nmetulev) commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

Description

Fix plain screenshot output that reported a wrapped guest staging path even after delivering the image to the requested host destination. Screenshot result paths now remain intact for the existing host translation, for both single-window and composite captures. JSON paths and dimensions are unchanged.

Correct screenshot help, reference documentation, and shipped automation guidance to describe existing behavior: default multi-window capture creates one labeled composite PNG. Process-name/PID targeting includes the app's windows and owned windows; a title match or explicit HWND selects one window plus its owned windows. Screen capture of an explicit window includes visible overlays in place.

No capture composition, owned-dialog/tooltip inclusion, focus/foreground safeguards, target isolation, interruption handling, or transfer/publication behavior changes. No new flags or capture modes.

Usage Example

winapp ui screenshot -w <MAIN_HWND> --on sandbox -o C:\captures\shot.png

Before: the image reached C:\captures\shot.png, but plain stdout named a guest C:\WinApp\artifacts\... path split across lines. After: stdout names the delivered host destination. Paths above are illustrative; the original installed-build observation is recorded in #883 and the candidate regression uses actual command rendering and loopback artifact publication.

For overlays in their screen positions, first discover the main window with winapp ui list-windows -a <app>, then use winapp ui screenshot -w <hwnd> --capture-screen. Default multi-window capture still produces labeled side-by-side panels instead.

Related Issue

Fixes #883.

Follow-up (out of scope for this PR): #914 — composite ui screenshot success output ignores --quiet (pre-existing asymmetry surfaced during review of this change).

Type of Change

  • 🐛 Bug fix
  • 📝 Documentation
  • 🧪 Test update

Checklist

  • Regression tests added for rendered plain/JSON paths, publication recovery, quiet behavior, and help
  • Tested locally on Windows with fake UI and real loopback artifact transport (148 focused tests on the merged candidate)
  • Affected usage/reference documentation and generated command descriptions updated
  • Shipped skills and Copilot agent guidance updated

Screenshots / Demo

Nonvisual reporting/help fix; textual before/after above. The screenshot pixels and composition are intentionally unchanged.

Additional Notes

Regression coverage exercises actual rendering at narrow widths, long/spaced and bracket-containing paths, host delivery before success reporting, publication-failure recovery evidence, JSON dimensions, terminal-safe single-window titles, and disabled Information logging. Tests reproduced wrapping failures before the original fix.

Local live-UI tests and candidate commands against the shared Sandbox were not run. Candidate validation uses actual command rendering through loopback guest file transport and host publication; installed .81 evidence is reproduction only, not proof of the merged candidate. Full repository CI and current-head review results are tracked in the lifecycle status comment. Local packaging uses -SkipTests after focused non-UI tests; corporate NuGet configuration is process-scoped, with no global source changes or audit suppression.

Keep screenshot result paths unwrapped until host publication translates them. Cover rendered plain/JSON output, publication recovery, and quiet behavior without changing capture semantics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation is focused and consistent, with the build, targeted regressions, and a real-window screenshot succeeding.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes screenshot host-path reporting and aligns help/documentation with existing composite capture behavior.

Changes:

  • Preserves contiguous paths for host translation and sanitizes window titles.
  • Adds output, publication-failure, quiet-mode, and help regressions.
  • Updates generated and shipped documentation.
File Description
src/​winapp-npm/​src/​winapp-commands.ts Updates generated npm API documentation.
src/​winapp-CLI/​WinApp.Cli/​Commands/​UiScreenshotCommand.cs Fixes plain output and help.
src/​winapp-CLI/​WinApp.Cli.Tests/​UiCommandTests.Screenshot.cs Adds screenshot regressions.
src/​winapp-CLI/​WinApp.Cli.Tests/​TargetUiRoutingTests.cs Exposes the routing harness to related tests.
plugins/​winapp/​skills/​winapp-ui-automation/​SKILL.md Corrects automation guidance.
docs/​ui-automation.md Documents capture scopes and host paths.
docs/​npm-usage.md Updates generated npm reference text.
docs/​cli-schema.json Regenerates screenshot command metadata.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Address automated code-quality feedback; the rendered-output regression matrix still passes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Two documentation surfaces still recommend an app-scoped overlay command that can fail when an owned tooltip or dialog is present.

Review effort: Balanced
Findings: None

Previously missed (2)

In code that hasn't changed since last review

Low severity Fix UI screenshot recipes to resolve and pass an HWND

docs/​ui-automation.md:505

The page still teaches a command that this new paragraph says can fail. After triggering UI, lines 719 and 991 run screenshot -a myapp --capture-screen; if an owned tooltip or dialog is present, the multi-window guard returns invalid_arguments instead of producing a PNG. Update those recipes to resolve an HWND and pass -w <hwnd>.

Low severity Update hover screenshot recipes to pass the tooltip HWND

plugins/​winapp/​skills/​winapp-ui-automation/​SKILL.md:192

The shipped skill still tells agents at lines 225 and 228 to run screenshot -a myapp --capture-screen immediately after hover. When the tooltip is an owned window, that produces multiple matches and fails with invalid_arguments, contradicting the new guidance here. Change those recipes to select and pass -w <hwnd>.

Address Copilot review by resolving the main HWND before hover and popup captures, avoiding ambiguous app-wide screen capture.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev

Copy link
Copy Markdown
Member Author

Addressed both documentation findings in the second Copilot review in 9ef5b60: the canonical hover/popup recipes and shipped skill now discover windows and pass the main window HWND to screenshot --capture-screen. This captures the main screen region with visible overlays in place; selecting the tooltip HWND would instead capture only that small region. No app-scoped screenshot --capture-screen recipes remain under docs or shipped skills. Generated-doc and plugin validation passes. These findings were in the review summary only (no inline threads to resolve).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation matches the requested scope, and all 13 focused regression cases passed.

Review effort: Balanced
Findings: None

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Build Metrics Report

Validation passed. All required build and validation jobs succeeded.

Binary Sizes

Artifact Baseline Current Delta
CLI (ARM64) 57.03 MB 57.03 MB 📈 +1.0 KB (+0.00%)
CLI (x64) 57.08 MB 57.08 MB 📈 +1.0 KB (+0.00%)
MSIX (ARM64) 23.72 MB 23.72 MB 📈 +3.4 KB (+0.01%)
MSIX (x64) 25.16 MB 25.16 MB 📈 +1.0 KB (+0.00%)
NPM Package 49.45 MB 49.45 MB 📈 +0.3 KB (+0.00%)
NuGet Package 49.55 MB 49.56 MB 📈 +1.8 KB (+0.00%)

.NET Test Results (TRX reports)

Other suites are reflected in the overall validation status above.

✅ 7636 passed, 37 skipped out of 7673 tests in 1471.6s (+13 tests, +191.6s vs. baseline)

Test Coverage

✅ 85.6% line coverage, 79.9% branch coverage · ✅ no change vs. baseline

CLI Startup Time

62ms median (x64, winapp --version) · ✅ +5ms vs. baseline

Try This Build

Installs the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing.

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 884
Switching between builds often?

Put the tool on your PATH once:

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPath

Then this build is just:

winapp-pr 884

Run winapp-pr with no arguments to pick from a list of open PRs.


Updated 2026-09-21 21:49:19 UTC · commit 8be071a · workflow run

@nmetulev
Nikola Metulev (nmetulev) marked this pull request as ready for review September 18, 2026 23:38

@zateutsch Zach Teutsch (zateutsch) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review (winappcli pr-review skill) — verify before acting.

Decision: Comment — mergeable as-is

The core fix is correct and well-tested. Switching the saved-path lines from Spectre MarkupLine (which word-wraps mid-path at narrow console widths) to a raw Profile.Out.Writer.WriteLine keeps guest paths contiguous so --on sandbox host-path translation works, and the single-window line correctly sanitizes the untrusted window title. Build is clean and all 58 screenshot tests pass. Nothing introduced by this PR is blocking. The items below are non-blocking, and most are pre-existing on main.

Must fix

None — mergeable as-is.

Non-blocking

1. Sibling doc surfaces still teach the old bare --capture-screen / "separate files" behavior

  • What is wrong: This PR corrects the main screenshot sections but leaves several shipped surfaces contradicting the new contract. Since this is a doc-correction PR, it's worth finishing the job.
  • Show me: docs/usage.md:2135 still says screenshot - Capture window/element as PNG (auto-captures dialogs separately) — this contradicts the new "one labeled composite" behavior. docs/usage.md:2189, later troubleshooting in docs/ui-automation.md, SKILL.md quick-tips/troubleshooting, and the hover shorthand in winapp.agent.md:389 still recommend bare ui screenshot --capture-screen, which now guides -a users into a multi-window invalid_arguments failure. (The main command reference at winapp.agent.md:386 is already accurate — only the hover line is stale.)
  • Why it matters: A user, or the shipped Copilot agent, following the stale tips hits the failed capture the corrected sections warn against.
  • Smallest fix: Update docs/usage.md:2135 to "one labeled composite PNG," and point the leftover bare --capture-screen references at list-windows + -w <hwnd> --capture-screen (or link to the corrected docs/ui-automation.md#screenshot).

2. Composite success line ignores --quiet (pre-existing)

  • What is wrong: The single-window success line is gated on logger.IsEnabled(LogLevel.Information), but the composite line prints on !json with no level gate. This inconsistency exists on main — the PR only changed the body, not the IsComposite && !json condition — so it is not a regression. But this PR touches this exact method and could align it cheaply.
  • Show me: winapp --quiet ui screenshot -a <2-window app> -o shot.png still prints ✓ Saved composite: …; the single-window path stays silent under --quiet.
  • Why it matters: Automation relying on --quiet gets stray output only when capture happens to be composite.
  • Smallest fix: Add && logger.IsEnabled(LogLevel.Information) to the composite non-JSON write (src/winapp-CLI/WinApp.Cli/Commands/UiScreenshotCommand.cs:508).

3. Composite per-window error lines print unsanitized window titles (pre-existing, needs live-sandbox confirmation)

  • What is wrong: This PR adds TerminalText.Sanitize(...) to the single-window title, but the composite error path (✗ HWND … "{Markup.Escape(title)}") still relies on Markup.Escape, which strips Spectre markup, not terminal control sequences. This path is pre-existing and unchanged here (the composite success line prints only the path). Worth noting only because title sanitization is this PR's own theme.
  • Show me: Set a target window title to "\u001b]0;pwned\u0007Evil", force a composite capture where one window fails, run under --on sandbox, and check whether raw escape bytes reach host stdout. This was not reproduced — it needs a live Sandbox slot, so it stays unverified.
  • Why it matters: If the composite error line is replayed raw across the sandbox boundary, an untrusted app title could inject terminal control sequences.
  • Smallest fix: Wrap composite-path titles in TerminalText.Sanitize(...) too (a follow-up; not required to merge this fix).

Minor: the new help says --app "includes the app's windows," but title-based -a resolves to a single window plus its owned windows — a small accuracy gap in otherwise much-improved help.

What was exercised

  • dotnet build WinApp.Cli.Tests.csproj -c Debug (fresh worktree at PR head) — succeeded, 0 warnings/errors.
  • WinApp.Cli.Tests.exe --filter FullyQualifiedName~Screenshot — 58/58 passed, including the new width-40/80/120 × composite × json × publication-fail matrix, the disabled-Information no-output test, and the help-text assertions. The tests exercise the real Profile.Out.Writer path and guest→host path rewriting.
  • Not exercised: the --on sandbox host terminal-escape attempt and live-UI capture — needs a Sandbox slot / real target window, so the security note above stays unverified. This does not affect the merge decision because that path is pre-existing and unchanged by this PR.

@nmetulev Nikola Metulev (nmetulev) added this to the 0.7 milestone Sep 21, 2026
@nmetulev Nikola Metulev (nmetulev) added the agent-preparing Agent is addressing feedback or completing required validation and CI label Sep 21, 2026
@nmetulev

Nikola Metulev (nmetulev) commented Sep 21, 2026 •

Copy link
Copy Markdown
Member Author

State: merged at 2026-09-21T22:13:28Z as bb02c145b7da15f2b0a14982c077f6ed295db918. Active lifecycle labels removed; the renamed ready-for-review label is not retained on merged PRs.
Assessed: head 8be071a47162d20b4d39d3e2cb6c1f2a5a4202cf against actual main@97dea79ad0fc1d712bcc331f2323abec4ab1704a; current integration 0c2691c7f8c6ad42297efb7dcb26df2fb77c2258. Normal nonrewriting merges only. The readiness assessment below is historical evidence for the merged head.
CI: Build and Package35656906463 succeeded for this head/base. Checkout logs confirm the integration SHA above. Required build-and-package check106530432631 completed successfully from GitHub Actions app15368. All four validation lanes, UI E2E, sample matrix/aggregates, docs, CodeQL and metrics pass; samples / build is the expected skipped duplicate build because same-run artifacts are reused. No pending/failed checks.
Local evidence: 148 screenshot/render/publication/safety tests passed on this head. On c74b78bf, scripts\build-cli.ps1 -SkipTests -SkipMsix, scripts\package-msix.ps1 -Tag pr884, and scripts\validate-llm-docs.ps1 -FailOnDrift passed; only subsequent main changes are inspected comments/XML docs in DotNetService/IDotNetService. Earlier full-script attempt failed on a 260-character SignTool staging filename; supported short-tag MSIX rerun signed both architectures. No global setting changes or audit suppression. Local live-UI suite/shared Sandbox candidate run not performed; actual renderer plus loopback guest/host publication regressions cover the reporting fix. Installed .81 evidence remains reproduction only.
Review: Current-head independent Copilot5272055682 has no findings. Static C# precheck found nothing obvious (not a CodeQL guarantee). Full review bodies, top-level comments, PR body, every inline thread and nested reply page reassessed; no unaddressed in-scope findings or substantive human reservation. Human documentation suggestions implemented; explicitly nonblocking pre-existing quiet and unverified error-title follow-ups deferred by scope decision, not fixed.
Pending: None; PR is merged.
Blocker / owner / next step: None. This agent performed lifecycle cleanup only; it did not execute the merge or enable auto-merge.
Resume: Monitoring stopped. Closed/merged lifecycle cleanup complete; do not reopen. pr-lifecycle followed from merged SKILL.md because cached registry could not invoke it.

@nmetulev

Copy link
Copy Markdown
Member Author

Disposition of the nonblocking follow-ups in review5257953369: composite --quiet behavior is pre-existing and intentionally preserved in this reporting/help-only fix; it is deferred, not claimed fixed. The composite per-window error/title renderer is also unchanged against main, and this PR does not add a title to the raw composite saved-path line or alter the host relay. The terminal-escape concern was explicitly unverified in the review; no candidate live Sandbox experiment was run, so I am not claiming a confirmed bug or a fix. Per the agreed scope, no unrelated quiet/security expansion is included. The remaining directly affected documentation and title-target wording are being corrected in this PR.

Correct sibling overlay tips and process-name/PID versus title matching descriptions after main integration. Preserve existing capture, quiet, and safety behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev

Copy link
Copy Markdown
Member Author

Addressed the in-scope documentation points from review5257953369 in37fa426f: usage now describes one labeled composite PNG; remaining hover/troubleshooting/recording-workaround tips link to the canonical screenshot workflow or explicitly select a discovered main HWND. Help and canonical docs now distinguish process-name/PID app-wide capture from title-based selection of one matching window plus its owned windows. Generated schema/npm descriptions refreshed; help assertions and 148 focused rendering/publication/safety tests pass. The previously stated nonblocking pre-existing quiet and unverified composite-error-title follow-ups remain deferred, not implemented.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The solution builds cleanly, 76 focused tests pass, plugin validation succeeds, and the built CLI exposes the corrected help.

Review effort: Balanced
Findings: None

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation is focused, consistent with artifact publication behavior, and the build plus focused regression suites passed.

Review effort: Balanced
Findings: None

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The solution builds cleanly, all 13 focused tests pass, and no unresolved correctness issues were found.

Review effort: Balanced
Findings: None

@nmetulev Nikola Metulev (nmetulev) added ready-for-review Agent work and technical checks complete; awaiting review or re-review, not approval or merge and removed agent-preparing Agent is addressing feedback or completing required validation and CI labels Sep 21, 2026
@zateutsch
Zach Teutsch (zateutsch) merged commit bb02c14 into main Sep 21, 2026
43 checks passed
@zateutsch
Zach Teutsch (zateutsch) deleted the nmetulev-correct-screenshot-host-paths-and-compos branch September 21, 2026 22:13
@nmetulev Nikola Metulev (nmetulev) removed the ready-for-review Agent work and technical checks complete; awaiting review or re-review, not approval or merge label Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Sandbox screenshot text reports a guest path and help misdescribes composite output

3 participants