Repository navigation
Add subtext-ios-sim: local iOS Simulator loop (preview) - #39
Closed
chandlervdw wants to merge 13 commits into
Closed
chandlervdw wants to merge 13 commits into
chandlervdw wants to merge 13 commits into
Conversation
Drive and inspect an iOS app in the local Simulator the way sightmap browser does for web: an FBSimulatorControl-based helper for the accessibility tree and HID input, a dependency-free Node CLI and daemon (subtext-sim), sightmap component matching on iOS trees, live console from the device log, and a run log for producer/consumer sightmap authoring. Includes a SwiftUI fixture app and an e2e script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
- README: lead with the HTTPS marketplace URL (shorthand may clone over SSH) - SampleApp build.sh honors SUBTEXT_SIM_CACHE - helper build.sh keeps SwiftPM's artifact cache under SUBTEXT_SIM_CACHE, so a fresh cache dir is a truly cold, self-contained build - console list hides Apple framework errors by default; --all shows them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
…anual check Network capture is off by default for mobile and isn't settable via API, so it's the main onboarding roadblock for network data; doctor now says where to turn it on until it can read the setting itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
Add a React Native sample app (fixtures/RNSampleApp: login, React Navigation
tabs and stack, a 200-row FlatList, the SwiftUI fixture's testIDs) whose
build.sh generates the native project from the pinned RN template and builds
a Release .app with the JS bundle embedded. test/e2e.sh takes --fixture to
run the same flow against it.
Fixes found running it:
- RN accessible Views arrive as GenericElement: map them to `other`, and
count ones with a testID as interactive so readiness and coverage work.
- React Navigation tabs ("Products, tab, 1 of 3" buttons) normalize to
`tab "Products"`; their row stands in for the missing tab bar container
when checking whether a target is covered.
- Controlled TextInputs dropped keys from a burst of HID events; pause 50 ms
between keys, and let --clear's deletions settle before typing.
- iOS's "Save Password?" sheet lives in another process and emptied the
frontmost tree; describe the process under the screen's center instead and
render it as an [Overlay] whose buttons can be clicked.
- LogBox toasts (Debug builds) are reported with the tap-xy for their ✕ and
refuse taps underneath.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
The swiftc hand-assembled bundle lacked the simulated entitlements Xcode embeds in simulator apps, so the Fullstory SDK's background uploads all failed and no session reached Fullstory. Build a committed SimSample.xcodeproj instead; FS_ORG_ID injects the SDK, OrgId and VerboseLogging at build time, so no org ID lives in the repo. Verified: e2e passes, and with FS_ORG_ID the app's POST is readable via review-open / review-zoom (status, timing, response headers). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
On a cold build SwiftPM downloads idb's public xcframeworks from github.com and, when the login keychain holds a github.com credential, shows a "swift-package wants to use your confidential information" keychain prompt. The downloads are public, so pass --disable-keychain and --disable-netrc and never ask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
…delivery Spell out how Fullstory SDK data reaches agents (in-app capture, ~20 s bundle uploads, point-in-time review-open reads at 10 credits, network capture and Xcode-build requirements) and make the producer and consumer work at the same time: the orchestrator starts the consumer as soon as a session URL exists, the consumer re-reads every ~30 s within a credit budget, and asks the producer for coverage through a shared notes file. Ownership of .sightmap/ sections is split to avoid conflicting edits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
- Bridge accepts sightmap's object `extract` form (0.34+); npx pin moves to 0.36.1 and doctor reports the resolved sightmap version and range - session-url finds any Fullstory session URL in any console entry - Selectors: `testid` aliases `identifier`; descendant chains skip ancestors whose identifier no element carries (wrapper views aren't accessibility elements) - Skill and fixture YAML pass `sightmap validate`; validate runs in tests and the e2e - start errors when asked for a different app than the running session and reports what it installed/relaunched - tap-xy/swipe reject off-screen points (units are points) - Empty-tree errors list modal/hidden-root views first and drop the accessibility advice when it's already enabled - Skill: dev client + Metro is fine, inside-vs-outside capture, empty-tree causes, testIDs on accessible elements, Metro --clear, `machine` grain for naming endpoints; README: UTF-8 locale for pod install Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
- Detect Fullstory session end (SDK 206/202 termination lines): log a session-ended run-log entry, make session-url/status report it, and have the producer relaunch for a new session - Selector skip rule needs an exact leaf and prefers full-chain matches; remaining skip-induced conflicts are labelled - click/fill refuse taps inside a floating dev-menu button's cover zone - snapshot, click, fill and scroll share one empty-tree diagnosis - Coverage counts labelled tappable `other` elements and ignores keyboard keys - CLI: -i/--ignore-case for console grep, no flag-looking values, placeholder vs value after fill --clear, bounded settle-and-retry in click/fill, clearer doctor sightmap wording - Docs: Metro/Watchman in nested worktrees, what "invalid session_url" usually means, --corpus not yet available Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chandler Van De Water <chandler.vdw@gmail.com>
Collaborator
Author
|
Superseded. The simulator tool moved upstream as |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new first-party skill,
subtext-ios-sim, that lets an agent drive and inspect an iOS app in the local Simulator the waysightmap browserdoes for web:.sightmap/components.click/fillby accessibility identifier, snapshot id, or component query. Off-screen targets are refused with a scroll hint, andscroll --untilscrolls to a target..sightmap/.sim-runs/*.jsonl). A producer subagent drives the app while a consumer subagent reads the app's Fullstory session through the Subtext MCP and authorsrequests:,properties:andmessages:. The split absorbs the SDK's ~20 s upload interval.Everything runs on the developer's Mac: no Appium, no cloud, Simulator only. Design notes are in
docs/ios-sim-loop/PLAN.md.How it's built
helper/: a Swift helper on Meta's FBSimulatorControl, using idb v1.7.7's binary xcframeworks (pinned by checksum), with a JSON-lines stdio protocol. It builds on first use with only Xcode installed.bin/subtext-sim+lib/: a dependency-free Node CLI and daemon. The daemon owns the helper, the log stream, the last snapshot and the run log.lib/normalize.mjs,lib/sightmap-bridge.mjs: match sightmap selectors over the normalized iOS tree. The corpus is loaded viasightmap export, and matching agrees withsightmap explainon every node of four recorded trees.fixtures/SampleApp/: a SwiftUI fixture app with a.sightmap/, plustest/e2e.sh.Testing
node --test test/*.test.mjs: 28 pass, including parity tests against the real sightmap binary.test/e2e.shon an iPhone 17 Pro Max simulator (Xcode 26.6, iOS 26.2) passes in ~81 s. It signs in by component name, scrolls to an off-screen product row, opens it, adds it to the cart, reads the console line and screenshots the result.Draft because
Setup prompts testers may see
starton a simulator that has never had app accessibility enabled reboots it once.snapshotclears it.Known limits (v1)
tap-xy.🤖 Generated with Claude Code