Skip to content

Snapshots: App Capture turns on Chrome/Electron accessibility itself - #1803

Merged
backnotprop merged 2 commits into
feat/shots-macosfrom
snapshots-chrome-ax
Oct 8, 2026
Merged

backnotprop merged 2 commits into
feat/shots-macosfrom
snapshots-chrome-ax

Conversation

@backnotprop

@backnotprop backnotprop commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

App Capture (⌥⇧⌘5, toolbar App Capture) no longer needs Chrome's accessibility mode turned on by hand.

What it does (AXEnablement.swift, after Codex's Appshots enableAccessibilityIfNeeded):

  • Chromium browsers (Chrome, Chromium, Edge, Brave, Arc, Dia, Opera, Vivaldi, … by bundle id or a Chromium * Framework.framework) and Electron apps (Electron Framework.framework: Slack, VS Code, Discord, Notion, Cursor, …): AXManualAccessibility = true on the application element BEFORE the walk, then poll up to 600 ms for content.
  • Any other app: read first; ask with AXManualAccessibility only when the tree comes back empty (an unknown attribute is refused harmlessly).
  • AXEnhancedUserInterface is not written by default: AppKit animates every window frame change while it is on (window managers feel it) and Chromium treats it as a screen reader. Only a known Chromium that refuses AXManualAccessibility gets it, for that capture only: it is reset to false afterwards whether it read as off or could not be read, and left alone only when it already read true (a screen reader set it).
  • One 2 s budget for asking, waiting, the walk and the reset (the old path could take 4.5 s). Every AX message gets its own messaging timeout clamped to the remaining budget (children do not inherit the app element's), and the work stops 100 ms early so the reset fits. Exclusion list apps are never asked; secure fields still skipped. No text → the picture alone, textUnavailable: "this app gave no text, so the picture is sent on its own".

Restore policy: AXManualAccessibility is written once per process (pid + launch date) and never toggled per capture.

For users: after the first App Capture of Chrome (or Slack, VS Code, …), that app keeps its accessibility tree until it quits. That costs it a little CPU and memory while it runs, the same as when VoiceOver, a password manager or any other assistive app is running. Nothing is saved; quitting the app ends it.

Why not restore: Turning it off after each capture would make Chrome drop and rebuild the tree (seconds on a big page) every time; nothing persists to disk, and it is the state any assistive app leaves Chrome in. If the tree is empty again later, writing true again is not a toggle.

Tests: swift test in apps/snapshots-macos (new test target, 16 tests: bundle/framework detection, step, fallback, the reset rule, messaging timeout clamp, settle time inside budget, pid reuse, exclusion never asked; the walk is generic over a tree source, so a fake tree checks the secure-field skip (only role and subrole are queried, nothing of its children) and that a slow tree stops at the deadline). --selftest checks the decisions, prints tree: …, exits 1 on a failed check or a budget overrun, and gains --ax-app <bundle id>. The first revision's --selftest run here wrote AXManualAccessibility to the frontmost app (Cursor), so it is not run against the owner's apps again; the review fixes are covered by swift test. The full-capture budget with the reset is bounded by construction (per-message clamp + 100 ms reserve); a live timing against an unresponsive app was not measured.

Owner live check: quit and reopen Chrome (so a11y is off), open a page with text (e.g. example.com), press ⌥⇧⌘5; the snapshot's text should hold the page text (page … / heading Example Domain). The line in ~/.plannotator/snapshots/app.log reads app text com.google.Chrome: chromium: AXManualAccessibility set, tree ready in N ms. A second capture shows no write. Terminal alternative: PlannotatorSnapshots --selftest /tmp/x --ax-app com.google.Chrome from a terminal with Accessibility.

… itself

AXManualAccessibility is written on the application element before the walk
for Chromium browsers and Electron apps (after Codex's Appshots), once per
process, with a bounded 600 ms settle inside the 2 s budget. Other apps are
asked only when their tree comes back empty. AXEnhancedUserInterface only as a
per-capture fallback for a Chromium that refuses AXManualAccessibility.
No text: the picture goes on its own. Unit tests via swift test; --selftest
checks the decisions and gains --ax-app.
Every AX message gets its own messaging timeout clamped to the remaining
budget (children did not inherit the app element's 0.25 s), and the work
stops 100 ms early so the AXEnhancedUserInterface reset fits in the 2 s.
That reset now always runs unless the attribute read true beforehand.
The walk is generic over a tree source; a fake tree tests the secure-field
skip and the deadline. Documents the persistent AXManualAccessibility cost.
@backnotprop
backnotprop marked this pull request as ready for review October 8, 2026 19:44
@backnotprop
backnotprop merged commit 7ed636a into feat/shots-macos Oct 8, 2026
@backnotprop
backnotprop deleted the snapshots-chrome-ax branch October 8, 2026 19:44
backnotprop added a commit that referenced this pull request Oct 8, 2026
…o snapshots-review-fixes

Conflicts resolved as a union:
- mod snapshots.ts: the mod's own lease and claim kept, with #1804's older-CLI
  answer, the quiet loop end, and nothing spawned (no git, claim folder or lease)
  until a hub exists.
- mod snapshots.test.ts: both sets of tests; the off-switch test follows the
  new on-by-default rule.
- SelfTest.swift: the security checks and #1803's enablement checks both count
  toward the exit status. Package.swift: both test targets.
- packages/shared/package.json: both exports. AGENTS.md: #1804's text with the
  review fixes applied (install rules, URL scheme, hub resend, message format,
  one lease and claim per host).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant