Skip to content

fix(context-drop): use ax-read subprocess for read_selection parity with voice - #891

Closed
sonichi wants to merge 1 commit into
mainfrom
fix/context-drop-use-ax-read-subprocess
Closed

fix(context-drop): use ax-read subprocess for read_selection parity with voice#891
sonichi wants to merge 1 commit into
mainfrom
fix/context-drop-use-ax-read-subprocess

Conversation

@sonichi

@sonichi sonichi commented May 19, 2026

Copy link
Copy Markdown
Owner

Summary

Replaces in-process AX+Cmd+C duplicate logic in dropContext() with a subprocess call to ax-read — the same primitive voice-agent's read_selection tool uses (lives in skills/personal-deictic, private memory-sync). Falls back to legacy in-process logic (with #890's changeCount fix) when binary is unavailable.

Closes #890 (superseded). Per Chi 2026-05-19: pivoted from minimal changeCount-only port to full ax-read switchover.

Three upgrades over legacy

  1. No clipboard corruption on failed copy. ax-read restores prior pasteboard after Cmd+C probe — Sutando.app left it mutated.
  2. changeCount check. Distinguishes "Cmd+C fired" vs "clipboard already populated" — fixes the Discord-input bug where stale Arize email kept getting redropped.
  3. Rich metadata. Task file now includes app/window/url/path lines above the text. LLM core gets situational context that was previously discarded.

Binary path resolution

  1. $SUTANDO_MEMORY_DIR/skills/personal-deictic/ax-read
  2. $SUTANDO_PRIVATE_DIR/skills/personal-deictic/ax-read (legacy alias)
  3. ~/.sutando/memory-sync/skills/personal-deictic/ax-read

If none exist, subprocess fails, or JSON malformed → falls through to legacy AX in-process + simulateCopy (also patched with changeCount, so fallback is safe).

Test plan

  • Swift parse-clean (swiftc -parse)
  • ax-read direct invocation returns expected JSON shape {app, window_title, url, selected, cursor, screenshot_path, path}
  • Smoke: focused on Discord (contenteditable input), ax-read returns {path: "none", selected: ""} — confirms it correctly reports "nothing selected" instead of dropping stale clipboard
  • Rebuild Sutando.app + install + grant Accessibility
  • Positive path: select text in TextEdit → ⌃C → verify task file has app/window metadata + correct text
  • Negative path: cursor in Discord input (no selection) → ⌃C → verify notification says "Nothing selected" + no task written + clipboard unchanged
  • Fallback path: rename ax-read binary, retry ⌃C → verify legacy in-process path fires + still works with changeCount fix

Context

Chi hit this twice today (2026-05-19):

  1. ~6:40am PT: dropped Arize email via ⌃C from Pages → switched to Discord input, hit ⌃C → got Arize email re-dropped (stale clipboard, AX-fail-then-Cmd+C-noop)
  2. ~7:45am PT: tested voice agent on same Discord input → worked. Asked "does the app use the same read_selection?" — answer was no, they were two separate impls. This PR makes them one.

🤖 Generated with Claude Code

@sonichi @claude
fix(context-drop): use ax-read subprocess for read_selection parity w…
4e39fc3 
…ith voice

Replaces in-process AX+Cmd+C duplicate logic with a subprocess call to
`ax-read` — the same primitive voice-agent's `read_selection` tool uses
via `skills/personal-deictic`. Falls back to legacy in-process logic
(now with PR #890's changeCount check) when the binary is unavailable.

Closes #890 (superseded). Per Chi 2026-05-19: "replace" — pivoted from
the minimal changeCount-only port to the full ax-read switchover.

Three behavioral upgrades over legacy:

1. **No clipboard corruption on failed copy.** ax-read restores the
   prior pasteboard string after its Cmd+C probe. Sutando.app's old
   path left the clipboard mutated, so successive ⌃C attempts in
   apps that can't expose selection (Discord input box, Notion,
   Slack) compounded the problem.

2. **NSPasteboard.changeCount check.** Distinguishes "Cmd+C actually
   fired" from "clipboard was already populated" — the bug Chi hit
   today where ⌃C in Discord input silently dropped the previous
   Arize-email content because no new copy ever happened.

3. **Rich metadata in task file.** ax-read returns
   `{app, window_title, url, selected, screenshot_path, path}`. The
   drop now includes app + window + url lines above the selected
   text, giving downstream consumers (LLM core, voice agent) the
   situational context that was previously discarded.

Binary path resolution:
1. `$SUTANDO_MEMORY_DIR/skills/personal-deictic/ax-read`
2. `$SUTANDO_PRIVATE_DIR/skills/personal-deictic/ax-read` (legacy alias)
3. `~/.sutando/memory-sync/skills/personal-deictic/ax-read`

If none exist or the binary fails, falls through to the legacy AX
in-process + simulateCopy path (also patched with the changeCount
fix from #890, so the fallback is still safe).

Smoke-tested locally: `ax-read` invoked while focused on Discord
returns `{app:Discord, path:none, selected:""}` — matches expected
"no selection" output. Swift parse-clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vidhuUC
vidhuUC approved these changes May 19, 2026
View reviewed changes

@vidhuUC vidhuUC left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE. Clean ax-read switchover for context-drop. Three concrete wins over the in-process path (changeCount check, pasteboard restore, richer metadata) are all grounded — #890's changeCount fix is the right reference. Fallback chain (SUTANDO_MEMORY_DIR → SUTANDO_PRIVATE_DIR → default path) correctly matches PR #876's env rename + legacy alias convention. The early return after ax-read success is correct — it won't fall into the legacy changeCount path unnecessarily. CI green. No blocking issues.

@sonichi

sonichi commented May 19, 2026

Copy link
Copy Markdown
Owner Author

Closing per design re-think — the subprocess approach (Sutando.app spawning ax-read CLI) hits a TCC scope quirk specific to LSUIElement menu-bar agents that voice's Node-spawn doesn't trigger. Will pursue Swift Package extraction in #892 so ax-read.swift's logic lives in a shared SPM module imported by both Sutando.app and ax-read CLI — single source of truth, no subprocess, no LSUIElement issue. Testing locally first before re-pushing.

@sonichi sonichi closed this May 19, 2026
sonichi added a commit that referenced this pull request May 19, 2026
@sonichi @claude
fix(context-drop): TCC-stable AX trust + flipped priority + region-ca…
ef62646 
…pture fallback

Folds three changes onto the ax-read subprocess approach (PR #891
groundwork) to make ⌃C reliable across rebuilds:

1. **AXIsProcessTrustedWithOptions(prompt:true) at startup.**
   When Sutando.app's codesign identity changes (ad-hoc → cert, rename,
   rotate), the prior TCC Accessibility grant orphans silently — macOS
   returns AX errors without re-prompting. Empirical case 2026-05-19:
   Sutando.app was re-signed; AXIsProcessTrusted=false on first
   launch despite the prior grant still appearing toggled-on in
   System Settings. Adding the prompt call at startup forces macOS
   to re-bind the entry to the current binary on first AX use, or
   pop the standard "Sutando wants Accessibility" dialog if no
   matching entry exists. Idempotent when already trusted. Result
   logged so future drift is grep-able from session 0.

2. **Flip dropContext priority: ax-read FIRST, clipboard image second.**
   Original order checked clipboard PNG/TIFF BEFORE attempting AX read.
   Result: any lingering screenshot in the clipboard pre-empted text-
   selection drops, so ⌃C in Discord with a selection would silently
   drop yesterday's screenshot instead. The dominant ⌃C use case is
   "drop the highlighted text" — so ax-read runs first; only when it
   returns empty (`path=none`) do we fall through to the clipboard
   image branches.

3. **screencapture -i -c last-resort fallback.**
   When ax-read returns empty AND clipboard has no image AND legacy
   AX returns empty AND simulated Cmd+C doesn't advance pasteboard
   changeCount, instead of giving up with "Nothing selected" we now
   launch `screencapture -i -c`. macOS draws the crosshair, user
   drag-selects a region (or Esc to cancel). On completion the
   resulting PNG/TIFF clipboard content is captured as an image
   drop. Per Chi 2026-05-19: every ⌃C now has SOME path to
   producing a drop instead of dead-ending at "Nothing selected."

Also requires (out-of-band, not in this PR) codesigning Sutando.app
with an embedded designated requirement so TCC keys on bundle
identifier alone, surviving cdhash churn across rebuilds:

    codesign --requirements '=designated => identifier "com.sutando.menubar"'

Verified: rebuilt binary with different cdhash + same identifier passes
`AXIsProcessTrusted=true` on first run after one initial grant.

Tested locally:
- ⌃C with text selected in Discord → drops via ax-read (path=ax)
- ⌃C with image in clipboard, text selected → text wins (was: image won)
- ⌃C with nothing selected anywhere → screencapture crosshair fires
- Esc during screencapture → "Cancelled — nothing dropped"

Supersedes previous proxy-via-web-client approach on this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sonichi sonichi mentioned this pull request May 19, 2026
Merged
6 tasks done
sonichi added a commit that referenced this pull request May 19, 2026
@sonichi @claude
fix(context-drop): TCC-stable AX trust + flipped priority + region-ca…
5cd2ad4 
…pture fallback

Folds three changes onto the ax-read subprocess approach (PR #891
groundwork) to make ⌃C reliable across rebuilds:

1. **AXIsProcessTrustedWithOptions(prompt:true) at startup.**
   When Sutando.app's codesign identity changes (ad-hoc → cert, rename,
   rotate), the prior TCC Accessibility grant orphans silently — macOS
   returns AX errors without re-prompting. Empirical case 2026-05-19:
   Sutando.app was re-signed; AXIsProcessTrusted=false on first
   launch despite the prior grant still appearing toggled-on in
   System Settings. Adding the prompt call at startup forces macOS
   to re-bind the entry to the current binary on first AX use, or
   pop the standard "Sutando wants Accessibility" dialog if no
   matching entry exists. Idempotent when already trusted. Result
   logged so future drift is grep-able from session 0.

2. **Flip dropContext priority: ax-read FIRST, clipboard image second.**
   Original order checked clipboard PNG/TIFF BEFORE attempting AX read.
   Result: any lingering screenshot in the clipboard pre-empted text-
   selection drops, so ⌃C in Discord with a selection would silently
   drop yesterday's screenshot instead. The dominant ⌃C use case is
   "drop the highlighted text" — so ax-read runs first; only when it
   returns empty (`path=none`) do we fall through to the clipboard
   image branches.

3. **screencapture -i -c last-resort fallback.**
   When ax-read returns empty AND clipboard has no image AND legacy
   AX returns empty AND simulated Cmd+C doesn't advance pasteboard
   changeCount, instead of giving up with "Nothing selected" we now
   launch `screencapture -i -c`. macOS draws the crosshair, user
   drag-selects a region (or Esc to cancel). On completion the
   resulting PNG/TIFF clipboard content is captured as an image
   drop. Per Chi 2026-05-19: every ⌃C now has SOME path to
   producing a drop instead of dead-ending at "Nothing selected."

Also requires (out-of-band, not in this PR) codesigning Sutando.app
with an embedded designated requirement so TCC keys on bundle
identifier alone, surviving cdhash churn across rebuilds:

    codesign --requirements '=designated => identifier "com.sutando.menubar"'

Verified: rebuilt binary with different cdhash + same identifier passes
`AXIsProcessTrusted=true` on first run after one initial grant.

Tested locally:
- ⌃C with text selected in Discord → drops via ax-read (path=ax)
- ⌃C with image in clipboard, text selected → text wins (was: image won)
- ⌃C with nothing selected anywhere → screencapture crosshair fires
- Esc during screencapture → "Cancelled — nothing dropped"

Supersedes previous proxy-via-web-client approach on this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sonichi added a commit that referenced this pull request May 19, 2026
@sonichi @claude
fix(context-drop): TCC-stable AX trust + flipped priority + region-ca…
58d9e02 
…pture fallback

Folds three changes onto the ax-read subprocess approach (PR #891
groundwork) to make ⌃C reliable across rebuilds:

1. **AXIsProcessTrustedWithOptions(prompt:true) at startup.**
   When Sutando.app's codesign identity changes (ad-hoc → cert, rename,
   rotate), the prior TCC Accessibility grant orphans silently — macOS
   returns AX errors without re-prompting. Empirical case 2026-05-19:
   Sutando.app was re-signed; AXIsProcessTrusted=false on first
   launch despite the prior grant still appearing toggled-on in
   System Settings. Adding the prompt call at startup forces macOS
   to re-bind the entry to the current binary on first AX use, or
   pop the standard "Sutando wants Accessibility" dialog if no
   matching entry exists. Idempotent when already trusted. Result
   logged so future drift is grep-able from session 0.

2. **Flip dropContext priority: ax-read FIRST, clipboard image second.**
   Original order checked clipboard PNG/TIFF BEFORE attempting AX read.
   Result: any lingering screenshot in the clipboard pre-empted text-
   selection drops, so ⌃C in Discord with a selection would silently
   drop yesterday's screenshot instead. The dominant ⌃C use case is
   "drop the highlighted text" — so ax-read runs first; only when it
   returns empty (`path=none`) do we fall through to the clipboard
   image branches.

3. **screencapture -i -c last-resort fallback.**
   When ax-read returns empty AND clipboard has no image AND legacy
   AX returns empty AND simulated Cmd+C doesn't advance pasteboard
   changeCount, instead of giving up with "Nothing selected" we now
   launch `screencapture -i -c`. macOS draws the crosshair, user
   drag-selects a region (or Esc to cancel). On completion the
   resulting PNG/TIFF clipboard content is captured as an image
   drop. Per Chi 2026-05-19: every ⌃C now has SOME path to
   producing a drop instead of dead-ending at "Nothing selected."

Also requires (out-of-band, not in this PR) codesigning Sutando.app
with an embedded designated requirement so TCC keys on bundle
identifier alone, surviving cdhash churn across rebuilds:

    codesign --requirements '=designated => identifier "com.sutando.menubar"'

Verified: rebuilt binary with different cdhash + same identifier passes
`AXIsProcessTrusted=true` on first run after one initial grant.

Tested locally:
- ⌃C with text selected in Discord → drops via ax-read (path=ax)
- ⌃C with image in clipboard, text selected → text wins (was: image won)
- ⌃C with nothing selected anywhere → screencapture crosshair fires
- Esc during screencapture → "Cancelled — nothing dropped"

Supersedes previous proxy-via-web-client approach on this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sonichi added a commit that referenced this pull request May 19, 2026
@sonichi @claude
fix(context-drop): TCC-stable AX trust + flipped priority + region-ca…
a677dfb 
…pture fallback (#902)

Folds three changes onto the ax-read subprocess approach (PR #891
groundwork) to make ⌃C reliable across rebuilds:

1. **AXIsProcessTrustedWithOptions(prompt:true) at startup.**
   When Sutando.app's codesign identity changes (ad-hoc → cert, rename,
   rotate), the prior TCC Accessibility grant orphans silently — macOS
   returns AX errors without re-prompting. Empirical case 2026-05-19:
   Sutando.app was re-signed; AXIsProcessTrusted=false on first
   launch despite the prior grant still appearing toggled-on in
   System Settings. Adding the prompt call at startup forces macOS
   to re-bind the entry to the current binary on first AX use, or
   pop the standard "Sutando wants Accessibility" dialog if no
   matching entry exists. Idempotent when already trusted. Result
   logged so future drift is grep-able from session 0.

2. **Flip dropContext priority: ax-read FIRST, clipboard image second.**
   Original order checked clipboard PNG/TIFF BEFORE attempting AX read.
   Result: any lingering screenshot in the clipboard pre-empted text-
   selection drops, so ⌃C in Discord with a selection would silently
   drop yesterday's screenshot instead. The dominant ⌃C use case is
   "drop the highlighted text" — so ax-read runs first; only when it
   returns empty (`path=none`) do we fall through to the clipboard
   image branches.

3. **screencapture -i -c last-resort fallback.**
   When ax-read returns empty AND clipboard has no image AND legacy
   AX returns empty AND simulated Cmd+C doesn't advance pasteboard
   changeCount, instead of giving up with "Nothing selected" we now
   launch `screencapture -i -c`. macOS draws the crosshair, user
   drag-selects a region (or Esc to cancel). On completion the
   resulting PNG/TIFF clipboard content is captured as an image
   drop. Per Chi 2026-05-19: every ⌃C now has SOME path to
   producing a drop instead of dead-ending at "Nothing selected."

Also requires (out-of-band, not in this PR) codesigning Sutando.app
with an embedded designated requirement so TCC keys on bundle
identifier alone, surviving cdhash churn across rebuilds:

    codesign --requirements '=designated => identifier "com.sutando.menubar"'

Verified: rebuilt binary with different cdhash + same identifier passes
`AXIsProcessTrusted=true` on first run after one initial grant.

Tested locally:
- ⌃C with text selected in Discord → drops via ax-read (path=ax)
- ⌃C with image in clipboard, text selected → text wins (was: image won)
- ⌃C with nothing selected anywhere → screencapture crosshair fires
- Esc during screencapture → "Cancelled — nothing dropped"

Supersedes previous proxy-via-web-client approach on this branch.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
liususan091219 pushed a commit to liususan091219/sutando-fork that referenced this pull request May 28, 2026
@sonichi @claude
fix(context-drop): TCC-stable AX trust + flipped priority + region-ca…
d2ee6c4 
…pture fallback (sonichi#902)

Folds three changes onto the ax-read subprocess approach (PR sonichi#891
groundwork) to make ⌃C reliable across rebuilds:

1. **AXIsProcessTrustedWithOptions(prompt:true) at startup.**
   When Sutando.app's codesign identity changes (ad-hoc → cert, rename,
   rotate), the prior TCC Accessibility grant orphans silently — macOS
   returns AX errors without re-prompting. Empirical case 2026-05-19:
   Sutando.app was re-signed; AXIsProcessTrusted=false on first
   launch despite the prior grant still appearing toggled-on in
   System Settings. Adding the prompt call at startup forces macOS
   to re-bind the entry to the current binary on first AX use, or
   pop the standard "Sutando wants Accessibility" dialog if no
   matching entry exists. Idempotent when already trusted. Result
   logged so future drift is grep-able from session 0.

2. **Flip dropContext priority: ax-read FIRST, clipboard image second.**
   Original order checked clipboard PNG/TIFF BEFORE attempting AX read.
   Result: any lingering screenshot in the clipboard pre-empted text-
   selection drops, so ⌃C in Discord with a selection would silently
   drop yesterday's screenshot instead. The dominant ⌃C use case is
   "drop the highlighted text" — so ax-read runs first; only when it
   returns empty (`path=none`) do we fall through to the clipboard
   image branches.

3. **screencapture -i -c last-resort fallback.**
   When ax-read returns empty AND clipboard has no image AND legacy
   AX returns empty AND simulated Cmd+C doesn't advance pasteboard
   changeCount, instead of giving up with "Nothing selected" we now
   launch `screencapture -i -c`. macOS draws the crosshair, user
   drag-selects a region (or Esc to cancel). On completion the
   resulting PNG/TIFF clipboard content is captured as an image
   drop. Per Chi 2026-05-19: every ⌃C now has SOME path to
   producing a drop instead of dead-ending at "Nothing selected."

Also requires (out-of-band, not in this PR) codesigning Sutando.app
with an embedded designated requirement so TCC keys on bundle
identifier alone, surviving cdhash churn across rebuilds:

    codesign --requirements '=designated => identifier "com.sutando.menubar"'

Verified: rebuilt binary with different cdhash + same identifier passes
`AXIsProcessTrusted=true` on first run after one initial grant.

Tested locally:
- ⌃C with text selected in Discord → drops via ax-read (path=ax)
- ⌃C with image in clipboard, text selected → text wins (was: image won)
- ⌃C with nothing selected anywhere → screencapture crosshair fires
- Esc during screencapture → "Cancelled — nothing dropped"

Supersedes previous proxy-via-web-client approach on this branch.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@vidhuUC vidhuUC vidhuUC approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sonichi @vidhuUC