Repository navigation
docs(mentra-channel): design — MentraOS app server as full-duplex client-side adapter - #2707
qingyun-wu wants to merge 3 commits into
Conversation
…ent-side adapter Verified SDK surface (cloud-docs.mentra.glass): @mentraos/sdk AppSession, webhook-initiated sessions, onTranscription in, layouts/audio out. Adapter lives client-side (MentraOS only talks to a server we host); inbound via broker /v1/ingest (Bee pattern), outbound via GET /v1/result long-poll + glasses render, INTEGRATION_FALLBACK_ROOM_MENTRA for glasses-off delivery. v1 is wake-phrase-gated — no ambient firehose. Owner opens: dev-account identity, wake phrase, first device. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Coverage Gate✅ No Python changes — nothing to gate (bar: 95% on changed lines). |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at exact head 309155dc84f475552a2546397c3242e0016be0d0. Formal REQUEST_CHANGES is unavailable on the author's own PR.
[P1] skills/mentra-channel/DESIGN.md:44-52,72-77: the proposed fallback is unconditional, not glasses-off-only. Backend #444's Hub.add_result checks deliverer_for(source); when there is no native broker-side deliverer, any configured INTEGRATION_FALLBACK_ROOM_<SOURCE> immediately replaces the result room and writes to the Matrix AppService sink. A client-side Mentra adapter is not registered as a broker deliverer, so source=mentra plus INTEGRATION_FALLBACK_ROOM_MENTRA will send every result to the Mentra DM room even while the app server also retrieves it through GET /v1/result/<task-id> and renders it on live glasses. The broker has no session-liveness signal to make the distinction claimed here.
Revise the design so exactly one component owns delivery. For example, let the app server poll the result and choose live glasses versus a fallback-room post based on its actual AppSession state, or add an explicit broker/app delivery contract that reports live-session availability. The unit and live test plan must assert no Matrix duplicate on the live-session path and exactly one fallback delivery after session loss.
I verified the Mentra SDK surface against the current official SDK docs (AppSession, transcription events, layouts, and audio APIs), and verified the broker result endpoint plus backend #444 dispatch code. git diff --check and the hardcoded-path scan pass; hosted clean-install is still running.
Worst case is every private glasses answer being duplicated into a Matrix room, expanding exposure and notification noise. The design is not merge-ready until delivery ownership/fallback timing is corrected.
|
Approval signal: I do not see any blocking issues in this docs-only design PR. The architecture keeps the MentraOS API key and native session handling in the client-side app server, uses the broker only through the existing Verification:
Reviewed by Qingyun's Personal Codex. |
… 2026-08-06) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@cla-assistant check |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at exact head bc2548bec76f19196110cad6c5e9158e9e26f873. Formal REQUEST_CHANGES is unavailable on the author's own PR.
The EKS deployment correction is accurate, but it does not address the existing delivery-ownership blocker at skills/mentra-channel/DESIGN.md:44-52,75-80.
[P1] The fallback is still described as glasses-off-only even though backend #444 applies it unconditionally. A client-side Mentra app is not a broker deliverer, so INTEGRATION_FALLBACK_ROOM_MENTRA redirects every source=mentra result to Matrix as soon as the result is added. The app server simultaneously polls the same result and renders it on live glasses. The broker has no live-session signal that could make the fallback wait until the glasses disconnect.
Define one delivery owner: either have the session-aware app choose glasses versus Matrix after polling, or add an explicit availability/claim contract. Tests must prove exactly one live-session delivery and exactly one post-session fallback. Static diff and hardcoded-path checks pass; clean-install was still running at refresh.
Worst case remains duplication of every private glasses response into a Matrix room. The design is not merge-ready until fallback timing and ownership are corrected.
|
Review update for head The new delta is the deployment wording change from an EC2/Caddy note to the EKS ingress/deployment prerequisite. I do not see a new blocking issue in that docs-only change, and Reviewed by Qingyun's Personal Codex. |
|
Folded into #2708 at the owner's request — the design doc now ships alongside the scaffold it specifies (single review+merge). No content lost: DESIGN.md carried over at its EKS-corrected head. |
|
Approval signal for head The deployment wording update is docs-only, the local diff hygiene check passed, and the hosted required checks are now green, including Reviewed by Qingyun's Personal Codex. |
…es lane (scaffold) Implements the #2707 design with every registration input the owner completed today (packageName 'sutando', server https://chat.ag2.space/mentra, webview /mentra/webview, MENTRAOS_API_KEY in the vault): - scripts/core.ts — PURE logic, zero imports, tested by the root tsx suite without the SDK installed: wake-phrase gate (case/punctuation tolerant; phrase-alone yields no task; ambient firehose stays out BY CONTRACT), sparrow-safe deterministic task ids (session+seq, hostile-session-id sanitization), lane task shape, env>manifest config resolution. - scripts/server.ts — THIN wiring: /webhook/session-start spawns an SDK AppSession (dynamic import — --check and tests never need the dep); final transcriptions → gate → POST /v1/ingest (source=mentra); reply leg long-polls the broker's existing GET /v1/result/<id> and renders via showTextWall + audio.speak; no-result-before-timeout stays quiet because the broker's INTEGRATION_FALLBACK_ROOM_MENTRA owns glasses-off delivery. /webview serves the minimal status page the console registration points at. - Self-contained package.json (@mentraos/sdk) — repo root deps untouched. - manifest.json config-only, strict-lint clean from the start. 5/5 core tests; lint-skill clean; tsc scope unaffected (skills/ excluded). Live wiring lands with the wear test: EC2 deploy (git-tracked) + Caddy /mentra route + lane registry entry are the deploy-time pieces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Design-only PR (no code) — the Mentra slice of the wearables order the owner pinned today (Bee → Mentra → Even Realities), sibling in spirit to the Teams design doc (ag2-space/ag2space-backend#443).
Verified SDK facts (cloud-docs.mentra.glass, checked today):
@mentraos/sdk,AppSession({packageName, apiKey, cloudUrl: wss://cloud.mentraos.com/app-ws}), webhook-initiated sessions (POST /webhook/session-startwith sessionId/userId),session.events.onTranscription()inbound,session.layouts.showTextWall()/session.audio.speak()outbound. Full-duplex, event-push.Key architecture calls (all in the doc):
skills/mentra-channel/, TypeScript) — MentraOS only talks to a server we host, so unlike Teams this cannot be a broker module. Inbound reuses the Bee pattern: normalized tasks → broker/v1/ingest,source: mentra. Zero broker changes.GET /v1/result/<id>and renders on the glasses while the session lives;INTEGRATION_FALLBACK_ROOM_MENTRA(backend#444's config mechanism) covers glasses-off delivery. No new broker→app push channel invented.Owner decisions the doc surfaces: dev-account identity, wake phrase, first device to pair (Mentra Live vs G1-under-MentraOS — interacts with the Even Hub track), and the public-endpoint deploy target (EC2, git-tracked per the hard rule).
Evidence: docs-only diff (
git diff --stat origin/main...HEAD→ 1 file, +99); SDK claims each carry the source they were verified against.🤖 Generated with Claude Code