[qingyun-wu]

Summary

Promotes the owner's untracked WIP src/mentra-bridge.ts (322 lines) into a tracked file and wires it to the DeviceSession registry added in #739.

When a Mentra glasses session connects, the bridge now POSTs to /vision/register with the full DeviceProfile (mentra-live: 400×240 green HUD center, HD 119° camera, 3-mic array, tap gestures), and unregisters on teardown. Best-effort — silently falls back to legacy push-mode-only behavior if /vision/register 404s, so the bridge keeps working against pre-#739 voice-agent.

Depends on #739 for the runtime endpoint (graceful degradation when missing).

Diff scope

• 320 of 322 lines are unchanged from the owner's working-tree WIP.
• Added: registerDevice(), unregisterDevice(), call sites in onSession() and teardown().

Test plan

• npx tsc --noEmit clean.
• Manual: with feat(device-session): minimum-viable scaffold for §5 Now item 3 #739 merged + @mentra/sdk installed + MENTRA_API_KEY set, tsx src/mentra-bridge.ts → pair glasses → curl :7847/vision/devices shows mentra-live-<sessionId> as active with the correct profile.
• Manual against pre-feat(device-session): minimum-viable scaffold for §5 Now item 3 #739 voice-agent: bridge logs register endpoint unavailable; continuing in legacy mode and the existing push-mode path still works.

Roadmap impact

• §5 Now item 2 (Mentra Live wire-up): scaffolding is in repo. Profile + capabilities flow through the registry; the output router (next PR) can consume them.
• TODO 1 in the file header (DeviceSession + profile catalog): partially resolved — full output-router consumption is the follow-up.

🤖 Generated with Claude Code

[sonichi]

LGTM — clean DeviceSession wiring on top of the WIP promotion. The graceful degradation against pre-#739 voice-agent is exactly the right shape for a bridge that has to ship before its dependency is universally deployed.

Things I like:

• 404 graceful degradation in registerDevice() — when voice-agent predates #739 (no /vision/register endpoint), logs "register endpoint unavailable; continuing in legacy mode" and falls through to the existing push-mode-only path. Bridge keeps working across the rollout window without needing a coordinated deploy.
• deviceId = 'mentra-live-${sessionId}' — uniquely scopes concurrent glasses pairings. Two Mentra units paired at once won't collide in the registry, and /vision/devices can show both with distinct IDs.
• Profile capabilities sourced from the Mentra SDK's caps object — mic, camera.enabled, hud.enabled are derived rather than hardcoded. Defensive against a future Mentra device that ships without one of these (e.g., glasses with no display).
• pushSourceLabel: SOURCE_LABEL passed in — preserves the existing push-mode source identification, so downstream vision-tools.ts keeps distinguishing mentra frames from browser/laptop.
• Teardown ordering correct: stop frame timer → stop managed camera stream → stop vision push → unregisterDevice. Producers stopped before the registry slot is released. No torn-down session left active in the registry.
• unregisterDevice fire-and-forget with empty catch — fine; if the bridge is teardown-racing the voice-agent dying, we don't want to error out the teardown chain on a network failure.

Two non-blocking observations:

1. caps?: any defeats type safety. The caps parameter in registerDevice(deviceId, caps?: any) and call sites use caps?.hasMicrophone, caps?.hasCamera, etc. — these are real Mentra SDK fields. Casting via any means a future SDK rename (hasMicrophone → hasMic) silently breaks the profile inference (booleans become undefined, then !! → false, so the profile gets mic: false for devices that actually have one). Either:

   • Import the Mentra capabilities type from @mentra/sdk and use it.
   • Declare a local interface MentraCaps { hasMicrophone?: boolean; hasCamera?: boolean; hasDisplay?: boolean; modelName?: string } and use that.

   Either makes the SDK-rename failure mode a compile error instead of silent capability misreporting.

2. HUD profile fields hardcoded for mentra-live. resolution: '400x240', color: 'green', position: 'center' matches the §7 fleet matrix entry. When the second Mentra variant lands (Even G2 = 576×288 lower-right), you'll need a lookup table — profileId === 'mentra-live' is the right branch point but it's an implicit assumption. A MENTRA_PROFILES: Record<string, Partial<DeviceProfile>> map keyed by SDK device model name would make the second-variant addition a one-row append rather than an if/else fork.

The "320 of 322 lines unchanged from owner's WIP" framing is the right way to lower this PR's review-risk — the dependency wiring is the only new logic, and it's defensive in the right places.

Ready to merge after #739 lands (which I already LGTM'd).

[liususan091219]

Reviewed: new mentra-bridge.ts streams glasses camera → /vision/frame at 3s thumbnails + tap-to-capture full frames + task emit. HUD render via session.layouts.showTextWall. Registers via /vision/register (PR #739 DeviceSession). WIP-tagged. Note: BEHIND main, rebase before merge. — Lucy (Mac Studio bot)

[liususan091219]

LGTM — first non-test consumer of the DeviceSession + vision-pipeline contract, with the right WIP discipline (lazy SDK load so tsc --noEmit stays green on every box; legacy-mode fallback when register endpoint is 404). The four TODO comments at the top are unusually disciplined — each names the §4/§5/§8 roadmap anchor it punts on, so future-me knows the deferrals are intentional, not oversights.

Things I like:

1. Lazy SDK load via createRequire + loadSdk() (mentra-bridge.ts:64-79) — import of @mentra/sdk happens only inside start(). Means the file imports cleanly on any box without the SDK installed (CI, fresh checkout, peer node), and tsc --noEmit always passes. The alternative (top-level import) would have made this bridge a CI dependency rather than an optional component.

2. Legacy-mode fallback on register 404 (mentra-bridge.ts:120-122) — if voice-agent predates the /vision/register endpoint, the bridge logs the version mismatch and continues in legacy push mode. So a partial deploy (newer mentra-bridge on a node with older voice-agent) doesn't brick the bridge; it degrades to the prior behavior. This is the kind of forward/backward compat care that makes rolling deploys safe.

3. Per-session deviceId = mentra-live-${sessionId} (mentra-bridge.ts:226) — two simultaneous glasses pairings get distinct ids in the registry. Matches #739's Map<deviceId, DeviceSession> shape; without per-session ids, the second pair would silently displace the first via last-activate-wins.

4. Frame-size sanity check (mentra-bridge.ts:283) — if (buf.byteLength < 2048) return skips blank/tiny frames before paying the POST cost. 2KB is a reasonable JPEG floor — even a blank 720p JPEG is ~5KB. Saves the round-trip on the obvious junk.

5. Lazy push-mode re-entry on tick (mentra-bridge.ts:285-288) — if voice-agent comes up after the bridge starts, the next interval tries startVisionPush() again. So you can tsx src/mentra-bridge.ts first, then start voice-agent, and frames flow without restarting the bridge.

6. Photo tap fires task even without live voice session (mentra-bridge.ts:332-334) — writeTask('I just took a photo with my glasses. What do you see?') runs unconditionally. So even if voice-agent is down, the task queue captures the user intent and the proactive loop picks it up later. This is the "drop in, react later" pattern that makes the bridge survive transient outages.

7. Comprehensive teardown (mentra-bridge.ts:236-244) — clearInterval → stopManagedStream → stopVisionPush → unregisterDevice. Order matters (stop the producer before the consumer) and the try-wrapped unregisterDevice is best-effort. Same pattern as the discord-bridge's session cleanup.

8. Four roadmap-anchored TODOs (mentra-bridge.ts:42-60) — §4 OC substrate adoption, §4 3-tier access control (mentra session implicitly owner-trusted), §5 Next device-aware output routing (showTextWall is static strings), §5 Next omnia-agent.yaml manifest. Each TODO names the bet/pitfall it's deferring, so a future reader can decide priority without re-reading the roadmap.

Non-blocking observations:

1. MENTRA_PACKAGE defaults to com.sutando.glasses (mentra-bridge.ts:80) — that package id needs to be registered at portal.mentra.glass/ per the setup notes. If a user runs this without registering, the Mentra portal rejects the session at handshake. Worth surfacing in the connect-failure log path.

2. writeTask(text) writes to TASKS_DIR without parents=true (mentra-bridge.ts:185) — if a fresh install starts the bridge before voice-agent has created tasks/, the writeFileSync throws ENOENT. Cheap fix: mkdirSync(TASKS_DIR, { recursive: true }) once at startup, or pass parents: true semantics. (See #762's mkdir fixes for the same class of bug.)

3. onTranscription forwards all final text to writeTask (mentra-bridge.ts:343-348) — this is the §4 access-control TODO playing out: any voice text from the glasses session becomes a task with no access-tier metadata. When the access-control milestone lands, this needs access_tier: owner (or whatever the gate decides) injected into the task header.

4. Push-mode failures silently drop frames (mentra-bridge.ts:289-290) — if (!ok) pushActive = false; // poll will retry next tick. If voice-agent goes down mid-session, the bridge keeps polling thumbnails but every POST 503s for the duration. A backoff (or "stop polling after N consecutive failures, resume on register-success") would reduce log spam.

5. Hard-coded HUD shape in registerDevice (mentra-bridge.ts:108-112) — resolution: '400x240', color: 'green', position: 'center' baked into the bridge instead of read from caps. Once #759/#760 land, this PR could shrink to { deviceId, profileId: 'mentra-live' } and the server enriches from the OC catalog. Already noted in TODO #3 about output routing; just confirming the shrink path.

6. onPhotoTaken writes a hard-coded prompt (mentra-bridge.ts:332) — "I just took a photo with my glasses. What do you see?" Fine for V1, but should become a configurable string (or template) once the bridge gets more use cases. Cheap follow-up.

7. No package.json entry for @mentra/sdk — intentional (lazy load), but means setup requires the manual npm i @mentra/sdk step in the docstring. Once the bridge stabilizes, dropping it into optionalDependencies would let npm install discover it without breaking installs on non-glasses nodes.

Cross-PR linkages:

• First real consumer of #739 DeviceSession — registerDevice() POSTs the full DeviceProfile shape today; will shrink to {deviceId, profileId} once #760 lands.
• First real consumer of #747 access-policy gate — when this PR registers, #747's attachPolicy(session) runs against category: 'glasses', granting the glasses default-allowed capability set.
• Pairs with #745 voice-agent wire — both PRs need to be live for the full glasses→HUD round trip. This bridge sends frames; #745 routes assistant replies back to HUD via routeReply.
• §5 Now item 2 milestone — "Mentra wire-up." This PR is the milestone. The four TODOs at the top map the §5/§4 follow-ups.
• OC adoption tension (TODO #1 / §4 bet) — this bridge talks directly to vision-tools, not through OC's device_register protocol. Deliberate shortcut named in the TODO so the next person knows the OC-adapter rewrite is the intended end state.
• Workspace fallback (mentra-bridge.ts:84) — process.env.SUTANDO_WORKSPACE || process.cwd() is the TS-side pattern flagged in my #762 review obs 2 as the place where a workspace.ts shim should land.

Tests: None in this PR. WIP marker in the title is honest about the test gap. Worth adding before this goes from WIP→merged: (a) registerDevice POST shape against a mock vision-tools server, (b) frame-size threshold (1KB drops, 3KB passes), (c) teardown order on session disconnect. The lazy-SDK design means tests can mock loadSdk() cleanly without needing @mentra/sdk installed in CI.

WIP → not blocking on merge if Susan/qy want the bridge in-tree to dogfood. Otherwise the test gap + access-control TODO are the two items worth resolving before unmarking WIP.

— Lucy (Mac Studio bot)

[vidhuUC]

APPROVE

Clean promotion of owner WIP into tracked code. The DeviceSession wiring is minimal and correct — POST to /vision/register on connect, unregister on teardown, graceful 404 fallback for pre-#739 voice-agent. 320/322 lines unchanged from owner's working tree is reassuring.

The TODOs in the header are honest and well-scoped — access control, OC protocol migration, HUD routing via device profile, skill manifest. These are the right follow-ups in the right order.

One note: the implicit owner-trust for any paired Mentra session is called out as a TODO. Until that's addressed, this bridge should only be used in a single-owner setup (which is the current reality anyway). Acceptable for now.

Depends on #739. Merge after #739.