[Design] OpenAB Mac Agent — cloud brain (k8s) + thin macOS executor over Tailscale

[chaodu-obk]

OpenAB Mac Agent — Design Proposal

Goal

Run an AI agent that can fully operate a fresh Mac mini at home (browser automation, GUI control, shell) from anywhere. The agent "brain" runs in a sandbox — home, cloud, k8s, anywhere. The Mac mini runs a thin executor daemon. All traffic goes over Tailscale. OpenAB Connect/Remote (Mac/iOS app) lets the human control the agent and view the remote desktop from anywhere.

Mental model — three cleanly separated roles

Role	Component	Runs
🧠 Brain	Agent (via openab broker) + openab-pty for human shell access	Sandbox — k8s / ECS / anywhere
🎮 Remote control	OpenAB Connect / Remote (Mac / iOS app)	Wherever the human is
🦾 Hands & feet	OpenAB Mac App (oab-mc-agent)	Mac mini fleet on isolated home VLAN

Perfect isolation between all three layers — every cross-boundary edge goes through the tailnet.

Ecosystem overview

flowchart TB
    subgraph human["👤 Human — anywhere"]
        CHAT["Chat client<br/>Discord / Slack / Telegram"]
        CONNECT["🎮 OpenAB Connect / Remote<br/>(Mac / iOS app)"]
    end

    subgraph tailnet["🔐 Tailscale tailnet — WireGuard encrypted overlay + ACLs"]
        NET(("tailnet"))
    end

    subgraph sandbox["🧠 Agent Sandbox — k8s / ECS / anywhere"]
        OAB["openab<br/>Open Agent Broker<br/>chat ⇄ ACP JSON-RPC"]
        AGENT["ACP Agent CLI<br/>(Kiro / Claude / …)<br/>the brain: LLM loop, planning"]
        PTY["openab-pty runtime<br/>sandboxed PTY over WSS<br/>per-session tokens"]
        WORK["shared workspace"]
        OAB -->|"stdio"| AGENT
        AGENT --- WORK
        PTY --- WORK
    end

    subgraph home["🏠 Home — isolated VLAN / guest Wi-Fi (no LAN access)"]
        subgraph mini["Mac mini (×N fleet)"]
            MCA["🦾 OpenAB Mac App<br/>(oab-mc-agent)<br/>thin executor daemon"]
            SCK["ScreenCaptureKit<br/>screenshot / stream"]
            INJ["CGEvent + AX<br/>input injection"]
            CDP["Chrome<br/>remote CDP"]
            SH["shell /<br/>AppleScript"]
            MCA --- SCK
            MCA --- INJ
            MCA --- SH
            MCA -.manages.- CDP
        end
    end

    CHAT -->|"natural-language tasks"| OAB
    CONNECT -->|"attach terminal<br/>HTTPS + WSS"| NET
    CONNECT -->|"live desktop view<br/>video stream"| NET
    AGENT -->|"operate Mac:<br/>daemon API + Playwright over CDP"| NET
    NET --> PTY
    NET --> MCA
    NET --> CDP

    classDef brain fill:#e8f4fd,stroke:#3a87c8,color:#000
    classDef hands fill:#e9f7ef,stroke:#2e9e5b,color:#000
    classDef control fill:#fff4e6,stroke:#d9822b,color:#000
    classDef net fill:#f3e8fd,stroke:#8e5bc8,color:#000
    class AGENT,OAB,PTY,WORK brain
    class MCA,SCK,INJ,CDP,SH hands
    class CHAT,CONNECT control
    class NET net

Loading

Design principle: brain/body separation

• Mac mini side stays thin: a pure executor daemon that rarely changes. All intelligence (LLM loop, task orchestration, retry strategy) lives in the sandbox and can be rolled out independently.
• Browser control is native to this split: Chrome runs on the Mac mini with --remote-debugging-port; the sandboxed agent drives it directly via playwright.connectOverCDP("http://<tailnet-ip>:9222"). No custom protocol needed for browser tasks.

Components

1. Network — Tailscale

• Mac mini: system Tailscale app.
• Sandbox: preferred options in order — (a) tsnet embedded in the agent binary (userspace, no privileged pod), (b) Tailscale Kubernetes Operator, (c) tailscaled sidecar.
• Ephemeral + tagged auth keys so pods auto-join the tailnet on recreation.
• ACLs are mandatory: CDP and the executor daemon have no built-in auth. Lock ports to the agent pod's tag. Add a shared-token check in the daemon as a second layer.

2. Mac mini — OpenAB Mac App / oab-mc-agent (Swift)

• LaunchAgent (must run in the Aqua user session for GUI access — not a LaunchDaemon).
• WebSocket (or gRPC) server bound to the tailnet IP.
• Capabilities:
  • Screen capture: ScreenCaptureKit (SCStream) — on-demand screenshots + optional live stream.
  • Input injection: CGEvent + Accessibility (AXUIElement) for non-browser GUI apps.
  • Shell / AppleScript execution.
  • Managed Chrome instance with persistent profile and remote CDP.
• Video pipeline (phase 2): SCStream → VideoToolbox H.264/HEVC hardware encode → WebSocket/UDP → client-side VideoToolbox decode → AVSampleBufferDisplayLayer render.

3. Sandbox — Agent Brain

• Computer-use style LLM loop: screenshot → reason → act.
• Playwright over CDP for browser tasks; daemon API for GUI/shell tasks.
• Handles Mac-offline detection, task pause/resume, retries.
• openab brokers chat-driven instructions; openab-pty gives the human a sandboxed shell beside the agent workspace.

4. OpenAB Connect / Remote (Swift native, macOS/iOS)

• Connects to the sandbox for task submission, status, and terminal attach (via openab-pty).
• Connects directly to the Mac mini over the tailnet only when live desktop view is needed.
• Swift chosen over Tauri: native VideoToolbox decode for streaming, shared Codable protocol models with the daemon, and a free path to an iOS client.

Isolation & hardening (defense in depth)

1. VLAN / guest Wi-Fi isolation — Mac minis live on a dedicated home network segment that cannot reach any other hosts on the home LAN. A compromised Mac mini can't pivot into the home network.
2. Tailscale ACLs — only tagged agent pods can reach the executor daemon / CDP ports.
3. Daemon shared token — last line of defense on the API itself.

This makes the Mac minis behave like disposable "compute appliances" — safe to let an autonomous agent drive them.

macOS setup (one-time, manual by design)

TCC permissions cannot be granted programmatically; a setup wizard should guide:

• Accessibility, Screen Recording, Automation, (optionally) Full Disk Access.
• Auto-login enabled, screen lock/saver disabled so the GUI session stays alive.

Known constraints / risks

• Latency: screenshot → sandbox → LLM → action round trip; acceptable when the tailnet gets a direct (hole-punched) connection, worse via DERP relay.
• CDP is unauthenticated: mitigated by Tailscale ACLs + daemon token.
• Session lock: locked/logged-out GUI session breaks input injection; requires auto-login config.

Phased plan

1. PoC: minimal daemon — WebSocket server + shell exec + on-demand JPEG screenshots; use built-in macOS Screen Sharing over Tailscale for live view.
2. Browser automation: managed Chrome + CDP, agent drives via Playwright from the sandbox.
3. GUI automation: CGEvent/AX injection driven by the LLM loop.
4. Productize: VideoToolbox streaming pipeline integrated into OpenAB Connect/Remote; multi-Mac fleet support.

---

Requested by Pahud (Discord thread, 2026-09-22).

[github-actions]

Thanks for the report! A maintainer will triage this shortly and apply the appropriate label. Once a type label is added, the issue will be re-validated automatically.

[chaodu-obk]

Refined mental model (from Pahud)

Three cleanly separated roles:

Role	Component	Location
🧠 Brain	Agent (sandbox)	Anywhere — home, cloud, k8s
🎮 Remote control	OpenAB connect/remote	Wherever the human is
🦾 Hands & feet	oab-mc-agent on Mac mini(s)	Home, on the tailnet

• The Agent runs in a sandbox anywhere; you control it via OpenAB connect/remote from anywhere.
• One or more Mac minis run oab-mc-agent, joined to the tailnet — the Agent can operate them fully, including seeing the screen.
• Perfect isolation between all three layers.

Additional hardening: isolated home network segment

Put the Mac minis on a dedicated, isolated Wi-Fi segment (guest network / VLAN) that cannot reach any other hosts on the home LAN. Combined with Tailscale ACLs, this gives defense in depth:

1. VLAN/guest isolation → a compromised Mac mini can't pivot into the home network.
2. Tailscale ACLs → only tagged Agent pods can reach the executor daemon / CDP ports.
3. Daemon shared token → last line of defense on the API itself.

This also makes the Mac minis behave like disposable "compute appliances" — safe to let an autonomous agent drive them.

[chaodu-obk]

Ecosystem overview (mermaid)

How the Mac Agent fits into the existing OpenAB ecosystem — openab (broker), openab-pty (sandboxed terminal), OpenAB Connect/Remote (Mac/iOS client), and the new OpenAB Mac App (oab-mc-agent):

flowchart TB
    subgraph human["👤 Human — anywhere"]
        CHAT["Chat client<br/>Discord / Slack / Telegram"]
        CONNECT["🎮 OpenAB Connect / Remote<br/>(Mac / iOS app)"]
    end

    subgraph tailnet["🔐 Tailscale tailnet — WireGuard encrypted overlay + ACLs"]
        NET(("tailnet"))
    end

    subgraph sandbox["🧠 Agent Sandbox — k8s / ECS / anywhere"]
        OAB["openab<br/>Open Agent Broker<br/>chat ⇄ ACP JSON-RPC"]
        AGENT["ACP Agent CLI<br/>(Kiro / Claude / …)<br/>the brain: LLM loop, planning"]
        PTY["openab-pty runtime<br/>sandboxed PTY over WSS<br/>per-session tokens"]
        WORK["shared workspace"]
        OAB -->|"stdio"| AGENT
        AGENT --- WORK
        PTY --- WORK
    end

    subgraph home["🏠 Home — isolated VLAN / guest Wi-Fi (no LAN access)"]
        subgraph mini["Mac mini (×N fleet)"]
            MCA["🦾 OpenAB Mac App<br/>(oab-mc-agent)<br/>thin executor daemon"]
            SCK["ScreenCaptureKit<br/>screenshot / stream"]
            INJ["CGEvent + AX<br/>input injection"]
            CDP["Chrome<br/>remote CDP"]
            SH["shell /<br/>AppleScript"]
            MCA --- SCK
            MCA --- INJ
            MCA --- SH
            MCA -.manages.- CDP
        end
    end

    CHAT -->|"natural-language tasks"| OAB
    CONNECT -->|"attach terminal<br/>HTTPS + WSS"| NET
    CONNECT -->|"live desktop view<br/>video stream"| NET
    AGENT -->|"operate Mac:<br/>daemon API + Playwright over CDP"| NET
    NET --> PTY
    NET --> MCA
    NET --> CDP

    classDef brain fill:#e8f4fd,stroke:#3a87c8,color:#000
    classDef hands fill:#e9f7ef,stroke:#2e9e5b,color:#000
    classDef control fill:#fff4e6,stroke:#d9822b,color:#000
    classDef net fill:#f3e8fd,stroke:#8e5bc8,color:#000
    class AGENT,OAB,PTY,WORK brain
    class MCA,SCK,INJ,CDP,SH hands
    class CHAT,CONNECT control
    class NET net

Loading

Role mapping

Role	Component	Runs
🧠 Brain	Agent (via openab broker) + openab-pty for human shell access	Sandbox — k8s/ECS/anywhere
🎮 Remote control	OpenAB Connect / Remote (Mac/iOS app)	Wherever the human is
🦾 Hands & feet	OpenAB Mac App (oab-mc-agent)	Mac mini fleet on isolated home VLAN

Every cross-boundary edge goes through the tailnet: the sandbox has no routable ingress, the Mac minis are unreachable from the home LAN, and Connect/Remote reaches both only as a tailnet peer allowed by ACLs.