Support iOS and macOS applications (Mach-O) - #216
Draft
greeng3 wants to merge 13 commits into
Draft
Conversation
The ios branch defined only DART_TARGET_OS_MACOS_IOS. A real Dart iOS build defines DART_TARGET_OS_MACOS as well (runtime/platform/globals.h), and code such as Dart::VersionString() tests the latter first, so the VM does not compile with the sub-flag alone. Defining it exposes an unfinished Mach-O stub in ElfHelper::MapLibAppSo that cannot compile: it returns a bool from a LibAppInfo function and references platform/mach_o.h, which only exists in recent SDKs. Remove it; Mach-O loading lands separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Blutter read Android libapp.so only. Add MachOHelper, which locates the four snapshot symbols in LC_SYMTAB and lays the segments out at their virtual addresses rather than mapping the file flat: App's __DATA is zero-fill and its vmaddr range overlaps __LINKEDIT under a flat mapping, and the VM writes into that BSS while loading the snapshot. Fat binaries are parsed and the arm64 slice preferred, falling back to x86_64; that path is untested for want of a universal sample (#5). Mach-O structures are declared locally instead of coming from platform/mach_o.h, which is absent from older SDKs. DartApp dispatches on the file magic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extract_dart_info.py parsed ELF only, so an iOS app needed --dart-version and still guessed the target wrong. Add Mach-O parsing (thin and fat) that reads the embedded engine version string out of Flutter.framework/Flutter and derives os/arch from the load commands. find_lib_files() now also accepts a bundle's Frameworks directory, where each binary sits inside its own .framework. The Mach-O path has no engine-id fallback for beta/dev channels the way the ELF path does (#4). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arm64 analyzer assumed compressed pointers throughout. iOS does not use them, and every crash and every one of the 903 analysis errors on the test app traced back to that one assumption. It is now 0 errors over 424 disassembly files. Disassembler_arm64.h: CSREG_DART_HEAP was gated on DART_COMPRESSED_POINTERS, but HEAP_BITS carries the write-barrier mask either way and CodeAnalyzer_arm64.cpp refers to the register unconditionally, so the guard could never have been off. DartClass.cpp: segfault on Object, which has no super class. The class id was read out of the null pointer and then guarded with "superCid > 0"; uncompressed, that garbage reads back negative and the guard never fired. Test the pointer instead. CodeAnalyzer_arm64.cpp, five matchers, by error count: - static field offset divided by kWordSize/kCompressedWordSize rather than shifted right by one (586) - isSmiScaledOperand() for the sxtw-vs-lsl scaling of a Smi index, 5 sites (194) - args-descriptor element at Array::data_offset() rather than a hardcoded two pointers (77) - isSmiUntagToNative() for the sbfx-vs-asr untag, 3 sites (43) - String payloads accepted in the array-element assertion (3) The decompression handlers are now compiled out when pointers are uncompressed, since no such instruction is emitted. Only the String-payload row is meant to change compressed-build behaviour, and only by accepting a match that was previously logged as an error; the rest should be byte-identical. That has not been shown against an Android build — see #1, which blocks merging. getArrayOp() still hardcodes an uncompressed Array data offset, left alone deliberately (#2). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issue #5: both readers parse a fat header and prefer the arm64 slice, and neither had ever run against a universal binary. The ordering cases carry the weight. A real universal build — checked against AppFlowy 0.13.1 macos-universal — lists x86_64 **first** and arm64 second, so an implementation that quietly took the first slice would pick the wrong architecture and fail later somewhere unrelated. Fixtures are synthesized rather than committed, since a real universal binary is hundreds of megabytes of third-party software. tests/fat_macho.py builds a fat container around arbitrary thin images. The loader test wraps a real thin App supplied via BLUTTER_TEST_MACHO, with a stub x86_64 slice placed first, and requires the analysis to match the thin run file for file and line for line after masking host addresses — so the synthetic container is validated against genuine Mach-O content, not just against itself. Eleven cases: thin, arm64-first, x86_64-first, x86_64-only fallback, no supported slice, fat recognition, 64-bit fat header, plus the four loader comparisons. Refs: #5 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FAT_MAGIC_64 lists its slices as fat_arch_64, 32 bytes each rather than 20. Both readers only tested for FAT_MAGIC/FAT_CIGAM, so such a file fell through: the Python side reported "Unsupported Mach-O image" naming the *slice* magic it had landed on, and the C++ side did not recognise the file as Mach-O at all and let it be treated as an ELF, failing later with something unrelated. Neither reader supports the format, which is fine — Flutter does not ship binaries large enough to need it. Saying so plainly is the difference between a one-line answer and an afternoon. Refs: #5 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three gaps between an iOS app and a macOS one, all of which #10 is about. find_lib_files: a macOS app ships FlutterMacOS.framework, not Flutter.framework, and a versioned bundle (Versions/A/<name> plus a top-level symlink) where iOS is flat. Layouts are built from empty files — this is pathfinding, so contents are irrelevant and no sample binary is needed. Also pins the existing Android and iOS shapes, and that a *directory* sharing the binary's name is skipped rather than returned. Engine version from a universal binary: every slice carries its own version string, so searching the whole file finds whichever slice comes first. A real universal engine lists x64 first, so a macOS app reported arch x64 while its snapshot said arm64 — the analysis would then have been attempted with the wrong Dart VM entirely. DartLibInfo defaults: Android is compressed, iOS is not, and neither is macOS, which the default missed by testing only for iOS. Verified against AppFlowy 0.13.1, which ships "arm64 macos no-compressed-pointers". Refs: #10 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
find_lib_files() knew the Android and iOS shapes only, so a macOS app stopped at "Cannot find libflutter file" before anything else could be tried. Framework lookup is now generated per name and covers the flat iOS layout, the versioned macOS one, and Versions/Current as well as Versions/A. isfile() follows symlinks, so a bundle's top-level link resolves, while a directory that happens to share the name is skipped. Refs: #10 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every slice of a universal binary carries its own copy of the Dart version string, and the search ran over the whole file, so it returned whichever slice came first. A real universal engine lists x64 first: AppFlowy's FlutterMacOS reported "macos_x64" while the snapshot beside it said arm64. Nothing downstream would have caught that. Blutter would have fetched and built an x64 Dart VM, spent the time doing it, and only then failed against an arm64 snapshot. _find_slice now records the slice extent alongside its offset, and slice_data() exposes just that slice for content searches. Refs: #10 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things stopped it. The platform guard accepted only android or ios, and the target flags gave everything non-Android both DART_TARGET_OS_MACOS and DART_TARGET_OS_MACOS_IOS. The _IOS sub-flag selects iOS-specific behaviour, so defining it for a desktop target misreports the target and takes iOS code paths. Each target now gets what a real Dart build of it defines: android ANDROID, ios MACOS plus MACOS_IOS, macos MACOS alone — verified by configuring all three and reading back the generated flags, and that an unknown target still fails with a clear message. Also defaults macOS to uncompressed pointers. Both Apple targets ship that way; only iOS was covered before macOS could be analyzed at all. This is the --dart-version path only — the normal path reads the snapshot flags. Refs: #10 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The suite skips tests whose module will not import, which is right for a developer without requests or pyelftools installed and wrong for a runner: with the dependencies missing it skips 19 tests and still exits green, putting a tick on a run that tested almost nothing. BLUTTER_TESTS_STRICT=1 re-raises the ImportError instead, so a failed dependency install fails the run. Sample-driven skips are untouched — a runner legitimately has no built binary and no 180 MB sample. Refs: #14 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issue #4. Two facts from real binaries shape these, both checked rather than assumed: - A macOS engine embeds engine ids exactly as an ELF one does — AppFlowy 0.13.1 carries two. An iOS engine embeds none; the only 40-hex runs in one are byte tables, not strings. So the lookup can rescue a macOS app and cannot rescue an iOS one, and the iOS case has to fail with something a reader can act on. - Both carry LC_BUILD_VERSION naming the platform, and their header names the architecture, so the target need not depend on the version string that may be missing in the first place. Covers the string present, engine ids collected, a missing string not being fatal, the target surviving a missing string, the header outranking a disagreeing string, the platform falling back to the string on images with no LC_BUILD_VERSION, and nothing-to-go-on. Two opt-in classes assert the real iOS and macOS engines behave as recorded, so the asymmetry above is pinned rather than remembered. Refs: #4 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ELF path recovers the Dart version of a beta or dev channel engine — which does not always name one — from the engine ids beside it, via the Flutter SDK archive. The Mach-O path asserted instead, so such an app failed outright. Both formats now share one fallback in extract_dart_info(), and the Mach-O reader returns the engine ids it finds instead of an empty list. That means the target can no longer come from the version string, since the whole point is that the string may be absent. It now comes from the image: the header names the architecture and LC_BUILD_VERSION names the platform, both present in real iOS and macOS engines. The string is consulted only as a fallback for the platform on older images that carry no LC_BUILD_VERSION. This is also more truthful for a universal binary, where each slice carries its own string. Verified against the real macOS engine: its two engine ids resolve through the archive to Dart 3.11.5, the same answer its version string gives. Blanking the string out of a copy — same length, so every offset is preserved — leaves the target still reported as macos/arm64 and the version recovered as 3.11.5 end to end. An iOS engine carries no engine ids, so nothing can rescue it. Rather than the previous bare assertion, that now reads: Cannot determine the Dart version: the engine names no version and carries no engine ids. Pass --dart-version to say which one to build. Fixes #4 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blutter reads Android
libapp.so. This teaches it to read the Mach-O images aniOS or macOS Flutter application ships —
App.framework/AppalongsideFlutter.framework/FlutterorFlutterMacOS.framework/…/FlutterMacOS.Validated on two real applications:
The two macOS errors are pre-existing arm64 matcher gaps (a
cbnzfallthroughwhere a branch is expected, and two thread-offset loads before a
blr) withnothing platform-specific about them.
The loader
MachOHelperlocates the four snapshot symbols inLC_SYMTABand lays thesegments out at their virtual addresses rather than mapping the file flat.
That detail is load-bearing:
App's__DATAis zero-fill and its vmaddr rangeoverlaps
__LINKEDITunder a flat mapping, and the VM writes into that BSSwhile loading the snapshot.
Mach-O structures are declared locally rather than taken from
<platform/mach_o.h>, which only exists in recent SDKs.DartAppdispatches onthe file magic. Universal binaries are supported, preferring the arm64 slice; a
64-bit fat header is rejected with a clear message rather than misparsed, since
its slice entries are 32 bytes rather than 20.
The analyzer
iOS and macOS do not use compressed pointers, and the arm64 analyzer assumed
throughout that they were in use. Every crash and all 903 initial analysis
errors on the iOS sample traced back to that one assumption.
The matchers now handle both representations:
isSmiScaledOperand()for thesxtw-vs-lsl scaling of a Smi index,
isSmiUntagToNative()for the sbfx-vs-asruntag, the static field offset divided by
kWordSize / kCompressedWordSizerather than shifted by one, and the args-descriptor element read at
Array::data_offset()rather than a hardcoded two pointers.CSREG_DART_HEAPloses a
DART_COMPRESSED_POINTERSguard it could never have honoured, sinceCodeAnalyzer_arm64.cpprefers to the register unconditionally.DartClassstops reading a class id out of the null pointer for
Object, which has nosuper class — uncompressed, that garbage reads back negative and the old
superCid > 0guard never fired.These changes touch code shared with the Android path, so they were
validated against a compressed build: the same Android
libapp.sothrough thepre-change and post-change binaries produces byte-identical output — all
1800 asm files,
objs.txt,pp.txtand the IDA scripts — with 0 analysiserrors on both sides.
One caution for anyone repeating that: a raw diff shows 257 of 1800 files
differing, and it is noise. Blutter prints host addresses of its own process
into its output, in hex and in decimal, so running the same binary twice
reproduces the same 257-file diff. Run that control first.
Build and detection
scripts/CMakeLists.txtdefines what a real Dart build of each target defines —android
DART_TARGET_OS_ANDROID, iosDART_TARGET_OS_MACOSandDART_TARGET_OS_MACOS_IOS, macosDART_TARGET_OS_MACOSalone. Defining onlythe
_IOSsub-flag does not compile, becauseDart::VersionString()tests theparent flag first; defining it for a desktop target takes iOS code paths.
extract_dart_info.pyparses Mach-O, so the Dart version and target aredetected without
--dart-version. The target is read from the image — theheader names the architecture,
LC_BUILD_VERSIONnames the platform — ratherthan from the embedded version string, because a universal binary carries one
string per slice and beta/dev engines may carry none at all. When the version
string is absent, the engine ids are looked up against the Flutter SDK archive,
matching what the ELF path already did. A macOS engine embeds those ids; an iOS
engine does not, so that case now fails with an actionable message naming
--dart-versioninstead of an assertion.Tests
A
tests/directory, standard library only, no test framework introduced: 48tests run with
python -m unittest discover -s tests. Roughly half need nothingbut
requestsandpyelftools; the rest opt in to a built binary or a sampleapplication via environment variables and skip cleanly without them.
BLUTTER_TESTS_STRICT=1turns a missing dependency into a failure rather than askip, so a runner cannot report success over a suite that did not execute.
Universal-binary fixtures are synthesized rather than committed — a real one is
hundreds of megabytes of third-party software — but the loader test wraps a
real thin
Appin a fat container with a stub slice placed first andrequires the analysis to match the thin run line for line, so the synthetic
container is checked against genuine Mach-O content.
Known follow-ups, not addressed here
payload base, giving a wrong (negative) index on a small number of accesses.
A String byte read and a TypedData byte read are indistinguishable from the
displacement alone, so this needs a String entry in
ArrayOp::ArrayTyperather than another constant.
Opened as a draft to keep it tracked, not to ask for review yet.
Independent of #214 and #215 — this branch sits directly on
mainand applieswithout either of them, so all three can be taken in any order, or separately.