chandlervdw · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026
diff --git a/.changeset/ios-sim-loop.md b/.changeset/ios-sim-loop.md
@@ -0,0 +1,5 @@
+---
+"subtext": minor
+---
+
+Add the `subtext-ios-sim` skill (preview): drive and inspect an iOS app in the local Simulator — annotated accessibility snapshots, taps and typing by identifier or sightmap component, live console, and a run log for producer/consumer sightmap authoring.
diff --git a/.changeset/ios-sim-react-native.md b/.changeset/ios-sim-react-native.md
@@ -0,0 +1,5 @@
+---
+"subtext": minor
+---
+
+`subtext-ios-sim` now drives React Native apps: role-less `Pressable`s with a `testID` count as interactive, React Navigation's tabs read as `tab` elements and its JS tab bar blocks taps like a native one, typing keeps up with controlled `TextInput`s, and system sheets such as "Save Password?" show up in the snapshot as an `[Overlay]` you can dismiss. Adds a React Native sample app and an e2e run against it (`test/e2e.sh --fixture RNSampleApp`).
diff --git a/docs/ios-sim-loop/FIELD-NOTES.md b/docs/ios-sim-loop/FIELD-NOTES.md
@@ -0,0 +1,72 @@
+# subtext-ios-sim field notes
+
+A running record of what came up while building and testing the iOS Simulator loop: on the development Mac ("dev Mac") and in field tests on a second Mac against a real React Native app ("field Mac"). Newest entries at the top of each section. Add to it from either machine; keep it free of org IDs, credentials and internal-only details.
+
+Status values: **fixed** (commit noted), **in progress**, **open**, **upstream** (outside this skill), **won't fix**.
+
+## Skill, CLI and helper
+
+| Found | Where | Issue | Status |
+|---|---|---|---|
+| 2026-10-28 | field Mac (retest, 7fa7aef) | The Fullstory session ended mid-run (SDK: `Received HTTP Status Code: 206, terminating FullStory session`), and nothing in subtext-sim noticed: `session-url` and `status` kept returning the dead URL. | fixed 701aa25: `session-ended` run-log entry; `session-url`/`status` report it; the skill has the producer relaunch. Matched on the SDK's log text, which hasn't been checked against a real 206 run yet |
+| 2026-10-28 | field Mac (retest) | The selector skip rule merged three components onto the same `[testid*="-option-"]` match, and the wrong one won. | fixed 701aa25: skipping needs an exact leaf; full-chain matches win; leftover conflicts labelled `(ancestor skipped)`. Cost: `[testid="list"] [testid^="row-"]` no longer matches when the list isn't on screen |
+| 2026-10-28 | field Mac (retest) | `click MobileTopBarAvatar` reported success but opened the Expo dev menu: the dev-menu window's touch area is larger than its accessibility frame. | fixed 701aa25: 24 pt cover zone around a dev-menu-like floating element (heuristic; the helper doesn't report windows; untested on a real Expo tree) |
+| 2026-10-28 | field Mac (retest) | The empty-tree diagnosis was only fixed in `snapshot`; `click`, `fill` and `scroll` still give the "long lists" advice. | fixed 701aa25: one shared diagnosis |
+| 2026-10-28 | field Mac (retest) | Coverage misses labelled, tappable `other` elements (e.g. cards) and counts keyboard keys while the keyboard is up. | fixed 701aa25 |
+| 2026-10-28 | field Mac (retest) | `console list --grep -i 206` took `-i` as the pattern; `fill "" --clear` reported the placeholder as the value; clicks right after a sheet closes or a back navigation fail until retried. | fixed 701aa25: `-i` flag; placeholder vs value; click/fill retry for up to 1.5 s |
+| 2026-10-28 | field Mac (retest) | `doctor` used sightmap 0.36.0 from PATH rather than the pinned 0.36.1. | fixed 701aa25: PATH still wins; doctor says so and names the pinned fallback |
+| 2026-10-28 | field Mac (retest) | Metro in a nested git worktree ignores edits: Watchman roots at the parent checkout, and `--clear` doesn't help; restarting Metro does. | fixed 701aa25: skill note |
+| 2026-10-28 | field Mac | Empty accessibility tree while app accessibility was already on. The cause was an always-mounted view with `accessibilityViewIsModal = YES` hiding its siblings, but the error said to enable accessibility. | fixed 61a37a8: errors list modal/hidden-root views first and skip the accessibility advice when it's on; a `doctor --app` probe is still open |
+| 2026-10-28 | field Mac | `snapshot` crashes with `extract.startsWith is not a function`: sightmap 0.36 emits `extract` as an object, and the bridge only handled strings. | fixed 61a37a8: both forms accepted; npx pin is now sightmap 0.36.1; `doctor` reports the resolved version |
+| 2026-10-28 | field Mac | `session-url` missed the URL when React Native logged it as a multi-argument line. | fixed 61a37a8: any Fullstory session URL in any console entry, newest wins |
+| 2026-10-28 | field Mac | An existing corpus matched 0%: session corpora use `[testid=…]` while the simulator exposes `identifier`, and wrapper views aren't accessibility elements, so parent chains never match. A draft simulator corpus reached 56–100% per screen. | fixed 61a37a8: `testid` aliases `identifier`; descendant chains skip ancestors whose identifier nothing carries. sightmap's own `explain`/`coverage` don't apply that rule, so they can report fewer matches |
+| 2026-10-28 | field Mac | Skill YAML examples fail `sightmap validate` on 0.36 (`extract: text` is deprecated; native views have no route). | fixed 61a37a8: examples and fixtures pass 0.36 `validate`, which now runs in tests and the e2e. Views need a `route`, so native views use placeholder routes like `/login` |
+| 2026-10-28 | field Mac | `start --bundle-id` on a running session relaunched the previous app while printing "relaunched". | fixed 61a37a8: errors on a different app; reports what it installed and relaunched |
+| 2026-10-28 | field Mac | `tap-xy` accepted points far off the screen; units (points vs pixels) weren't stated. | fixed 61a37a8: off-screen points refused; units are points |
+| 2026-10-28 | field Mac | `--sightmap-dir` ties the corpus to the session, so a draft corpus can't be tried against a running session. | open: `--corpus DIR` idea |
+| 2026-10-28 | field Mac | The dev-menu floating button is its own window, and it's the only visible element when the tree is empty. | fixed 61a37a8: snapshot adds a `[Mostly empty]` note (size heuristic, untested against a real dev-menu tree) |
+| 2026-10-28 | field Mac | Producer and consumer ran one after the other: the test prompt told the consumer to wait for the producer, and the skill allowed either. | fixed e9fb6b3: the skill requires concurrent work and explains SDK delivery |
+| 2026-10-28 | both | macOS keychain prompt on the first helper build: SwiftPM offered a saved `github.com` credential while downloading idb's public frameworks. | fixed 4410044: `--disable-keychain --disable-netrc` |
+| 2026-10-28 | dev Mac | The SwiftUI fixture was assembled by hand with `swiftc`. It lacked the simulated entitlements Xcode embeds, so the Fullstory SDK's background uploads all failed and no session reached Fullstory. Looked like a server or org problem until device logs showed every `/rec/bundle` upload failing inside the simulator. | fixed c38b875: committed `SimSample.xcodeproj`, built with `xcodebuild` |
+| 2026-10-28 | dev Mac | iOS shows a "Save Password?" sheet after sign-in once the app has a real identity (Xcode build). It briefly swallowed taps. | documented; a fresh `snapshot` clears it |
+| 2026-10-28 | dev Mac | Mobile network capture is off by default and can't be set by API or the Subtext MCP; it's the main onboarding roadblock for network data. | fixed 0227741: `doctor` prints it as a manual check |
+| 2026-10-28 | dev Mac (clean-machine run) | The `owner/repo#ref` marketplace shorthand cloned over SSH, which fails without GitHub SSH keys. | fixed 19e7fb5: README leads with the HTTPS URL |
+| 2026-10-28 | dev Mac (clean-machine run) | The fixture build ignored `SUBTEXT_SIM_CACHE`; the helper build used the global SwiftPM cache. | fixed 19e7fb5 |
+| 2026-10-28 | dev Mac (clean-machine run) | `console list` was swamped by Apple framework errors. | fixed 19e7fb5: hidden by default, `--all` shows them |
+| 2026-10-28 | dev Mac | Enabling app accessibility only takes effect at boot, so the first `start` on a fresh simulator reboots it once. | documented |
+| 2026-10-28 | dev Mac | iOS 26 SwiftUI tab bars appear as one "Tab Bar" element with no tabs; switching tabs needs `tap-xy`. | open |
+| 2026-10-28 | dev Mac | Scrolling a long list to a far row takes ~3–4 s per screenful. | open |
+
+## Fullstory data and the Subtext MCP
+
+| Found | Where | Issue | Status |
+|---|---|---|---|
+| 2026-10-28 | field Mac (retest) | Sessions ended with HTTP 206, which the ingest server returns for a bad request it won't accept, mainly a bundle that's too large; the SDK then terminates the session. The SDK also logged "Maximum number of views reached". Only the first 26 s of one session arrived, and none of the run's own requests were ingested. Sessions lived 51 s to 5.5 min. | upstream / app: bundle size on large React Native trees |
+| 2026-10-28 | field Mac (retest) | Duplicate network events are unchanged (65 unique requests appeared as 130 events), and `detail` still shows response headers only. | upstream |
+| 2026-10-28 | field Mac | Every network request appears twice at the same millisecond in the session; the server's own logs show one. | upstream: capture or ingest |
+| 2026-10-28 | field Mac | `review-zoom` at `detail` returned response headers only: no request headers, bodies or query strings. Check against the org's header allowlists and body rules before calling it a bug. | upstream / open |
+| 2026-10-28 | field Mac | `review-zoom` at `standard` collapses network traffic into counts; endpoint names only show at `machine`. | fixed 61a37a8: skill recommends `machine` for naming endpoints |
+| 2026-10-28 | field Mac | Network data lagged 35 s to 2 min behind the app, longer than the ~20 s upload interval. | upstream / open |
+| 2026-10-28 | dev Mac | `review-open` reports "invalid session_url" when a session's IDs don't exist server-side, which reads like a bad link rather than "this session never arrived". | upstream: clearer error; skill explains it (701aa25) |
+
+## Setup friction
+
+| Found | Where | Issue | Status |
+|---|---|---|---|
+| 2026-10-28 | field Mac | `pod install` crashed without a UTF-8 locale. | fixed 61a37a8: README prerequisite (`export LANG=en_US.UTF-8`) |
+| 2026-10-28 | field Mac | `subtext-verify` installs disabled from the marketplace. | open |
+| 2026-10-28 | field Mac | Stale Metro modules in git worktrees made edits appear not to apply; `--clear` fixed it. | fixed 61a37a8: skill note |
+| 2026-10-28 | field Mac | Upgrading an existing install from the branch: `plugin update` says "already at latest", because the version doesn't change. | documented: `marketplace update`, then uninstall and install |
+
+## What worked (field Mac, React Native)
+
+- Every `testID` surfaced as `identifier`; `snapshot`, `click`, `fill` and `scroll --until` each took under a second.
+- A dev client with Metro worked as well as Release for the live loop, and was cheaper.
+- Recorded taps lined up with producer steps to within about 30 ms once the clock offset was known.
+- Network capture had zero errors across about 900 events, and every `/api/*` route got a `requests:` name.
+
+## What worked (field Mac retest, 7fa7aef)
+
+- 8 of 9 re-checks pass: properties no longer crash, `session-url` works, the existing corpus matches every screen (it matched 0% before), `validate` passes, bundle-id switching, off-screen taps and `machine` grain all work. The empty-tree fix is partial.
+- The producer and consumer ran concurrently. The consumer started 25 s after the producer and sent four coverage requests; the producer acted on two (45 s and 2m42s later). One wasn't acted on because the producer's brief limited relaunches.
+- Timings: `doctor` 9.2 s, `start` 23.8 s, `snapshot` 0.26 s average, `click` 0.32 s average, `relaunch` to a new session URL 10 s.
diff --git a/docs/ios-sim-loop/PLAN.md b/docs/ios-sim-loop/PLAN.md
@@ -0,0 +1,104 @@
+# iOS Simulator live local loop: build plan
+
+**Goal:** an agent drives and inspects an iOS app in the local Simulator the same way `sightmap browser` does for web. It snapshots the UI tree, acts on components by id or component query, reads console output, and authors `.sightmap/` from what it sees. Everything runs on the developer's Mac: no Appium, no cloud, Simulator only.
+
+**Status:** prototype on branch `chandler/ios-live-loop`. It ships as a first-party skill in the Subtext plugin, so it can be tested on any Mac by installing the plugin from this branch. The intent is to move the command surface upstream as `sightmap device …` once it's proven.
+
+## Shape
+
+```
+agent ──► subtext-sim CLI ──► daemon ──► sim helper (FBSimulatorControl) ──► iOS Simulator
+                               │  └──► simctl (lifecycle, screenshot, log stream)
+                               └──► .sightmap/ matcher (sightmap CLI, offline)
+network (v1): app's Fullstory SDK session ──► Subtext MCP review-open / review-zoom
+```
+
+| Piece | Path | What it is |
+|---|---|---|
+| Skill | `skills/subtext-ios-sim/SKILL.md` | Teaches agents the workflow and commands |
+| CLI | `skills/subtext-ios-sim/bin/subtext-sim` + `lib/*.mjs` | Node ≥18, **no npm dependencies**. It mirrors `sightmap browser` commands. |
+| Daemon | `lib/daemon.mjs` | Long-lived; owns the helper process and the device log stream. The CLI talks to it over a Unix socket recorded in `.sightmap/.sim-session`. |
+| Helper | `skills/subtext-ios-sim/helper/` | Swift package built on FBSimulatorControl through idb's released binary xcframeworks (pinned). It builds on first use into `~/Library/Caches/subtext-sim/<version>/`. |
+| Fixture app | `skills/subtext-ios-sim/fixtures/SampleApp/` | A small SwiftUI app with `accessibilityIdentifier`s, a long list, a form and network calls, plus `build.sh`. It's used for the demo and for testing. |
+
+## Helper protocol (stable contract between CLI and helper)
+
+The helper is a long-lived process speaking JSON lines over stdin/stdout, one request per line:
+
+```json
+{"id": 1, "cmd": "describe", "args": {"udid": "…"}}
+{"id": 1, "ok": true, "result": {…}}
+{"id": 2, "ok": false, "error": {"code": "not_booted", "message": "…"}}
+```
+
+| cmd | args | result |
+|---|---|---|
+| `hello` | — | `{version, xcode, fbsimctl}` |
+| `describe` | `udid` | `{screen: {w, h}, root: Node}` where `Node = {role, type, label, identifier, value, frame: {x, y, w, h}, enabled, children: [Node]}`. `identifier` is `AXUniqueId`; points are top-left origin. |
+| `tap` | `udid, x, y` | `{}` |
+| `type` | `udid, text` | `{}` |
+| `swipe` | `udid, x1, y1, x2, y2, duration?` | `{}` |
+| `button` | `udid, name` (`home`, `lock`, `siri`) | `{}` |
+| `key` | `udid, key` (`return`, `delete`, `tab`…) | `{}` |
+
+Lifecycle (boot, install, launch, terminate, openurl), screenshots and logs go through `xcrun simctl` from Node, not the helper.
+
+## CLI surface (v1)
+
+| Command | Behavior |
+|---|---|
+| `subtext-sim doctor` | Checks Xcode (not just the CLT), simctl, iOS runtimes, the Swift toolchain, Node, the sightmap CLI and the helper build; prints exact fixes. Builds the helper if needed. |
+| `subtext-sim start [--device NAME\|UDID] [--app PATH] [--bundle-id ID] [--detach]` | Boots the simulator, enables accessibility, installs and launches the app, starts the daemon, writes `.sightmap/.sim-session`. |
+| `subtext-sim status` / `stop` | Reports session health / shuts everything down. |
+| `subtext-sim snapshot [--out F] [--tree-out F] [--json]` | describe → normalized tree → sightmap matching → annotated tree with stable per-snapshot ids. |
+| `subtext-sim click <id\|component-query>` | Resolves to an element, **refuses off-screen targets** (suggests scrolling), taps its center. |
+| `subtext-sim fill <id\|query> <text>` | Taps, then types. |
+| `subtext-sim scroll <up\|down> [--in id\|query]` / `swipe` / `tap-xy` / `button` / `key` | Lower-level input. |
+| `subtext-sim openurl <url>` / `screenshot --out F` | Deep links / PNG. |
+| `subtext-sim console list [--limit N] [--grep S]` / `console get <i>` | App process logs from the device log stream, live. |
+| `subtext-sim session-url` | The Fullstory session URL from the SDK's log output, when the SDK is present. |
+
+**Network (v1):** read through the Subtext MCP. The agent passes `session-url` to `review-open`, then calls `review-zoom` with `{"network": "standard"}` (or `"detail"` for headers and bodies). That data arrives on the SDK's upload interval, roughly 20 s, so the skill tells agents to batch: run the flow, then read once. Faster or live delivery depends on SDK work outside this branch.
+
+**Producer / consumer roles:** the skill splits the loop into two subagent roles, so the upload delay never blocks anyone.
+- **Producer:** drives the app with `subtext-sim` (start, snapshot, click/fill/scroll, openurl). It works through flows quickly and writes a run log to `.sightmap/.sim-runs/<run>.jsonl`. Each entry records a step: timestamp, action, target component or id, screen and view. The log also records the session URL.
+- **Consumer:** follows the producer's run log. It reads that session's captured data through the Subtext MCP (`review-open`, then `review-zoom` on network, console and interaction), lining the data up against the producer's steps. With it, the consumer authors or cleans up `.sightmap/`:
+  - names `requests:` and `properties:`;
+  - adds `messages:`;
+  - fixes components and views the producer found unnamed or ambiguous.
+
+  It re-reads as new data lands, roughly every upload interval.
+
+The orchestrating agent runs both, or the producer runs alone when only UI work is needed. Data the consumer reads depends on the org's mobile capture settings: network capture must be on for network data; resource uploading isn't needed.
+
+**Element model:** `role ← type`, `identifier ← AXUniqueId`, `text ← label`, `value`, `bounds`. These map onto sightmap selectors as `<role>[identifier="…"]`, `[text="…"]`, `[value="…"]`, so `.sightmap/` YAML stays platform-neutral.
+
+**Deviations found while building:**
+- Enabling app accessibility only takes effect at boot, so the first `start` on a fresh simulator reboots it once.
+- `stop` keeps the simulator running unless `--shutdown` is passed.
+- Rows in long lists that haven't been rendered yet report "not found"; `scroll --until` handles it.
+
+## Build phases
+
+1. **In parallel:**
+   - **A, helper:** the Swift helper and `build.sh`, tested against a booted simulator.
+   - **B, sightmap bridge:** the normalizer and sightmap bridge (the offline tree format, matching, annotated output), developed against recorded `describe` fixtures.
+   - **C, app, skill and install path:** the fixture app, the skill draft, `doctor`, and installing the plugin from a branch.
+2. **D, CLI and daemon:** wire up the helper, simctl, the log stream and the bridge into the commands above.
+3. **E, verification:**
+   - local end to end on the fixture app;
+   - a clean-machine dry run: fresh clone of the branch outside any dev checkout, isolated plugin config and cache, plugin installed from the clone, `doctor` → `start` → `snapshot` → `click` → `console`;
+   - after the branch is pushed, a test on a second Mac from the branch.
+
+## Testing on another Mac
+
+1. Prerequisites: macOS 15+, Xcode 26+ with an iOS Simulator runtime, Node 18+.
+2. Install the plugin from the branch (exact command is in the skill's README once verified), then ask the agent to run `subtext-sim doctor`.
+3. Try the fixture app: `fixtures/SampleApp/build.sh`, then `subtext-sim start --app <built .app>`.
+
+## Known limits (v1)
+
+- Simulator only; Apple Silicon only (idb's prebuilt frameworks are arm64).
+- No hover, file upload or arbitrary viewport sizes.
+- Network data isn't live, and depends on the app's Fullstory SDK and org network-capture settings.
+- The helper relies on Apple private frameworks through FBSimulatorControl, so new Xcode releases may need a helper update.