swift-semantic-string

Semantic markup and post-processing for rendered output.

The package ships two products:

Product	What it does
Semantic	Builds semantically typed strings with a SwiftUI-like declarative syntax. Each piece of text carries a semantic type (keyword, type name, variable, …), so downstream consumers can apply syntax highlighting, accessibility annotations, or any type-aware rendering. No dependencies.
OutputTransformer	The shared Transformer namespace and its Module protocol — the contract a post-processing step implements (declare parameters, take an input, return an output, be enabled or not). No dependencies, not even on Semantic.

OutputTransformer holds no concrete transformer. A transformer's tokens carry the vocabulary of whatever it describes (bitsNeededForTag, ivar offset), which belongs to the library that owns that domain rather than to a general-purpose string package. The concrete modules therefore ship elsewhere and all extend this one namespace, so a consumer importing several of them still writes Transformer.CType unqualified:

Modules	Home
SwiftFieldOffset, SwiftVTableOffset, SwiftMemberAddress, SwiftTypeLayout, SwiftEnumLayout	MachOSwiftSection
CType, ObjCIvarOffset	MachOObjCSection

Requirements

• Swift 6.2+
• No external dependencies
• Apple platforms: macOS 10.15+, iOS 13+, macCatalyst 13+, tvOS 13+, watchOS 6+, visionOS 1+
• Also builds on Linux and Windows. The only system dependency is a platform locking primitive (os_unfair_lock, pthread_mutex_t, or SRWLOCK), selected behind #if canImport.

Installation

Swift Package Manager

dependencies: [
    .package(url: "https://github.com/MxIris-Reverse-Engineering/swift-semantic-string.git", from: "0.3.0")
]

Then add "Semantic" — and "OutputTransformer" if you are writing a transformer module of your own — as dependencies of your target.

Quick Start

import Semantic

// Use the result builder to compose semantic strings declaratively
@SemanticStringBuilder
var declaration: SemanticString {
    Keyword("public")
    Space()
    Keyword("struct")
    Space()
    TypeName(kind: .struct, "MyType")
}

// Access the plain text
print(declaration.string) // "public struct MyType"

// Enumerate with semantic info for rendering
declaration.enumerate { text, type in
    switch type {
    case .keyword:
        renderAsKeyword(text)
    case .type:
        renderAsTypeName(text)
    default:
        renderAsPlainText(text)
    }
}

Semantic Types

SemanticType categorizes each text fragment:

Type	Description
.keyword	Language keywords (public, func, class, ...)
.variable	Variable names
.numeric	Numeric literals
.argument	Function argument labels
.comment	Comments and annotations
.error	Error indicators
.type(TypeKind, Context)	Type references — TypeKind: .enum, .struct, .class, .protocol, .other; Context: .declaration or .name
.member(Context)	Member references (property/method)
.function(Context)	Function references
.standard	Plain text with no special meaning
.other	Uncategorized

Components

Atomic Components

Atomic components are leaf nodes — each holds a single string and its semantic type.

Keyword("func")                        // .keyword
Variable("count")                      // .variable
Numeric("42")                          // .numeric
Argument("name")                       // .argument
Comment("// TODO")                     // .comment
TypeName(kind: .struct, "Int")         // .type(.struct, .name)
TypeDeclaration(kind: .class, "Foo")   // .type(.class, .declaration)
FunctionName("viewDidLoad")            // .function(.name)
FunctionDeclaration("init")            // .function(.declaration)
MemberName("count")                    // .member(.name)
MemberDeclaration("title")             // .member(.declaration)
Standard("(")                          // .standard
Space()                                // " "
BreakLine()                            // "\n"
Indent(level: 2)                       // "        " (4 spaces per level)

Composite Components

Composite components combine multiple children.

Group

Groups components together, with an optional separator:

let modifiers = Group {
    Keyword("public")
    Keyword("static")
    Keyword("func")
}.separator(Space())
// "public static func"

Joined

Joins items with a separator, automatically filtering out empty items:

Joined(separator: ", ", prefix: "(", suffix: ")") {
    if hasLabel {
        Standard("label")
    }
    Standard("value")
}
// With hasLabel=true:  "(label, value)"
// With hasLabel=false: "(value)"

ForEach / ForEachIndexed

Iterate over collections:

ForEach(parameters, separator: ", ") { param in
    Standard(param.name)
    Standard(": ")
    TypeName(kind: .other, param.typeName)
}

ForEachIndexed(items) { item, info in
    if !info.isFirst {
        Standard(", ")
    }
    Standard(item.name)
}

IfLet

Conditional content based on optionals:

IfLet(superclass) { name in
    Standard(": ")
    TypeName(kind: .class, name)
}

DeclarationBlock

Structured declaration with header, braces, and indented body:

DeclarationBlock(level: 0) {
    Keyword("struct")
    Space()
    TypeName(kind: .struct, "Point")
} body: {
    MemberList(level: 1) {
        SemanticString {
            Keyword("var")
            Space()
            Variable("x")
            Standard(": ")
            TypeName(kind: .struct, "Int")
        }
        SemanticString {
            Keyword("var")
            Space()
            Variable("y")
            Standard(": ")
            TypeName(kind: .struct, "Int")
        }
    }
}
// struct Point {
//     var x: Int
//     var y: Int
// }

Result Builder

@SemanticStringBuilder supports the full range of Swift result builder features:

@SemanticStringBuilder
func describe(_ decl: SomeDecl) -> SemanticString {
    // Conditionals
    if decl.isPublic {
        Keyword("public")
        Space()
    }

    // Switch / if-else
    if decl.isStatic {
        Keyword("static")
        Space()
    }

    Keyword("func")
    Space()
    FunctionName(decl.name)

    // For loops
    Joined(separator: ", ", prefix: "(", suffix: ")") {
        for param in decl.parameters {
            Standard(param)
        }
    }

    // Optionals via IfLet
    IfLet(decl.returnType) { ret in
        Standard(" -> ")
        TypeName(kind: .other, ret)
    }
}

Manipulating Semantic Strings

SemanticString provides a rich API for transformation and inspection:

let ss = SemanticString { ... }

// Concatenation
let combined = ss + otherString
ss += moreContent

// Transformation
let uppercased = ss.map { AtomicComponent(string: $0.string.uppercased(), type: $0.type) }
let allStandard = ss.replacing { _ in .standard }

// Wrapping
let wrapped = ss.parenthesized()   // "(content)"
let generic = ss.angleBracketed()  // "<content>"

// Trimming
let trimmed = ss.trimmingWhitespace()

// Conditional operations
let result = ss
    .prefixed(with: "public ", if: isPublic)
    .suffixed(with: "?", if: isOptional)

// Querying
ss.contains(type: .keyword)
ss.hasPrefix("func")
ss.starts(with: .keyword)

// Filtering
let keywords = ss.filter(byType: .keyword)

// Slicing
let first3 = ss.prefix(3)
let rest = ss.dropFirst()

Codable Support

SemanticString conforms to Codable, serializing as an array of AtomicComponent values:

let data = try JSONEncoder().encode(semanticString)
let decoded = try JSONDecoder().decode(SemanticString.self, from: data)

Zero-length components are not stored

A component whose string is empty contributes nothing to any rendering, and building one through a component (Keyword("")) has always discarded it. Constructing a string from atomic components directly does the same, so both ways of assembling the same content agree:

let components = [
    AtomicComponent(string: "a", type: .standard),
    AtomicComponent(string: "", type: .standard),   // dropped
    AtomicComponent(string: "b", type: .standard),
]
SemanticString(components: components).count   // 2, not 3

This holds on every construction path — hand-built arrays, streamed appends, decoding, and the flattening of custom components whose buildComponents() emits an empty atom. A zero-length entry does not survive into count, components, isEmpty, subscripting, prefix / suffix, equality, hashing, or a Codable round trip (which is therefore always idempotent: encoded output never contains an empty component). The emptiness measures all coincide: isEmpty ⟺ count == 0 ⟺ first == nil ⟺ string.isEmpty, and equal values agree on all of them. Callers that pair a map result positionally against the source array should filter empties themselves first, and callers migrating from the element-tree storage should note that map-ing a component to "" now removes it (shifting indices) instead of keeping an empty placeholder. The same applies across versions: a payload encoded by the element-tree version that contains zero-length components decodes here with those entries dropped — component indices recorded alongside such a payload (a span map, a selection anchor, a decoration table) shift past every dropped entry and must be re-derived from the decoded value.

Empty atoms contribute no text directly, and container emptiness decisions see the filtered form: an item or element consisting only of empty atoms produces no row in MemberList/Rows/BlockList, no separator in Group/Joined/ForEach, and no braces content in DeclarationBlock. This unifies two shapes that historically behaved differently — Keyword("") (whose default buildComponents() returns nothing) was always skipped, while a custom component emitting one empty atom was rendered as an empty row. Both now flatten to the same nothing and get the same treatment.

Components are flattened when they are appended

A component's buildComponents() runs at the moment the component is appended or placed in a container — not when the result is rendered. Its output is copied into storage and the component itself is released. buildComponents() takes no context and is expected to be deterministic, so the flattening can only ever be done once, and doing it eagerly keeps every read path free of side effects.

This matters if a custom component reads mutable state that is resolved after assembly — a deferred-resolution pattern where a printer builds the structure first and fills in names, offsets, or addresses afterwards:

final class LateBoundName { var value = "placeholder" }

struct NameReference: SemanticStringComponent {
    let holder: LateBoundName
    func buildComponents() -> [AtomicComponent] {
        [AtomicComponent(string: holder.value, type: .standard)]
    }
}

let holder = LateBoundName()
let group = Group([SemanticString(NameReference(holder: holder))])
holder.value = "resolved"
SemanticString(group).string   // "placeholder" — captured at append time

The element-tree storage retained the component and called buildComponents() at render time, so the same code produced "resolved". Resolve such values before appending them, or store a SemanticString placeholder and rebuild the container once the value is known.

Read-Only Content: FrozenSemanticString

SemanticString is the builder — composable, streamable, mutable. When a string is fully built and will only ever be read (rendered, encoded, exported, searched), freeze it:

let frozen = declaration.frozen()

print(frozen.string)   // free — the full text is the storage

frozen.enumerateSpans { spanText, type, identifier in
    render(spanText, as: type, linkingTo: identifier)
}

Freezing collapses the per-token component representation into three flat allocations:

Storage	Contents
text: String	The complete UTF-8 text, stored exactly once
spans: [Span]	8 bytes per token — UTF-8 length, semantic typeCode, identifierIndex
identifierTable: [String]	Interned span identifiers, referenced 1-based (0 means none)

Every stored property is let, so FrozenSemanticString is unconditionally Sendable with no locks and no copy-on-write. Freezing is one-way by design: there is no mutation API and no conversion back, which pins the "build, then read-only" lifecycle at the type level.

On a large corpus (3.46M tokens, 27.3 MB of text) this cut resident memory by roughly 78% against the per-token representation, with printing time unchanged.

Notes:

• Use enumerateSpans(_:) on hot paths. frozen.components materializes [AtomicComponent] and recreates exactly the per-token allocation cost freezing removed — it exists for compatibility and tests.
• Every span boundary falls on a Unicode scalar boundary — that is the guarantee to build on. Grapheme cluster (Character) alignment holds only within a token: Span.length is a UInt16, so a token longer than 65535 UTF-8 bytes is split into consecutive spans carrying the same type and identifier, at cluster boundaries (a single cluster that itself exceeds 65535 bytes degrades to scalar boundaries). Boundaries between tokens sit wherever the builder put the tokens, so adjacent tokens can split one Character — e.g. a keyword followed by a combining mark typed as a separate token. Consumers that concatenate text see no difference; styling or selection code that needs cluster-aligned runs must merge adjacent spans, and a token-by-token count comparison against SemanticString.components will see long-token splits. Note that the split also breaks value round trips: rebuilding a SemanticString from frozen.components after a long-token split produces a value that is not == to the source and hashes differently, even though its string matches — across a freeze/thaw boundary, compare rendered text (or re-freeze and compare snapshots), not SemanticString values.
• FrozenSemanticString conforms to Codable with a columnar encoding (one string plus four homogeneous arrays) an order of magnitude smaller on the wire than SemanticString's array-of-objects form. The two formats are intentionally not interchangeable — both ends of any persistence or cross-process channel must agree on which one they use.
• SemanticType values encode to stable UInt8 codes. Codes are assigned append-only, and a code from a newer version decodes as .other rather than failing. Note that resolving to .other is lossy in one direction: a value decoded from a newer encoder keeps the original code in spans, but round-tripping it through components and re-freezing rewrites it to .other's code. Forward and backward compatible readers should walk enumerateSpans or spans rather than components.
• Freeze at a storage boundary — once the string is final. There is deliberately no way to compact a SemanticString in place and keep building with it: the element boundaries a builder exposes are semantic (a container like MemberList treats one element as one row), and collapsing them silently changes layout. frozen() gives you the compact representation with that lifecycle enforced by the type.
• frozen() never inflates the value it is called on, or any value sharing its storage. The components are the storage (there is no component cache to fill), and the text reuses the string cache only when a previous render already paid for it — otherwise it is computed locally and not published.
• Decoding validates the structural invariants a reader would otherwise trap on: column counts must agree, span lengths must cover the text exactly, no span may be zero-length, and every span boundary must fall on a Unicode scalar boundary. Two fields are deliberately kept verbatim rather than rejected — an unknown typeCode resolves to .other and an out-of-range identifier index resolves to nil in every reader — so forward-compatible payloads (and this type's own encoder output) stay decodable. The unchecked init(text:spans:identifierTable:) trusts the caller instead — use it only for parts you produced yourself. Decoding is the path for anything that arrived from another process; note that validation cannot bound what JSONDecoder itself materializes while parsing, so cap the byte size of untrusted payloads before handing them to a decoder.
• Equality and hashing compare rendered content, not storage layout: identifierTable shape (duplicate or unreferenced entries, which a decoded payload keeps as they arrived) does not affect them. Two different SemanticStrings can still freeze to equal snapshots, because a token over 65535 bytes and the same text split across adjacent same-type tokens produce identical spans — compare the source strings when token identity matters.
• frozen() interns identifiers by String equality, which is Unicode-canonical: two identifiers whose bytes differ but compare canonically equal (an NFC and an NFD spelling of the same name) collapse into one table entry, and every span referencing either resolves to whichever spelling was interned first. SemanticString's own Codable path preserves identifier bytes exactly; freezing is the only path that can rewrite them. Mangled names are ASCII and unaffected — if identifiers can carry non-ASCII, keep them in a single normalization form before building.

License

MIT