Skip to content

docs(mentra-channel): design — MentraOS app server as full-duplex client-side adapter - #2707

Closed
qingyun-wu wants to merge 3 commits into
mainfrom
feat/mentra-channel-design
Closed

qingyun-wu wants to merge 3 commits into
mainfrom
feat/mentra-channel-design

Conversation

@qingyun-wu

Copy link
Copy Markdown
Collaborator

Design-only PR (no code) — the Mentra slice of the wearables order the owner pinned today (Bee → Mentra → Even Realities), sibling in spirit to the Teams design doc (ag2-space/ag2space-backend#443).

Verified SDK facts (cloud-docs.mentra.glass, checked today): @mentraos/sdk, AppSession({packageName, apiKey, cloudUrl: wss://cloud.mentraos.com/app-ws}), webhook-initiated sessions (POST /webhook/session-start with sessionId/userId), session.events.onTranscription() inbound, session.layouts.showTextWall() / session.audio.speak() outbound. Full-duplex, event-push.

Key architecture calls (all in the doc):

  • The adapter is client-side (skills/mentra-channel/, TypeScript) — MentraOS only talks to a server we host, so unlike Teams this cannot be a broker module. Inbound reuses the Bee pattern: normalized tasks → broker /v1/ingest, source: mentra. Zero broker changes.
  • Outbound: the app server long-polls the broker's existing GET /v1/result/<id> and renders on the glasses while the session lives; INTEGRATION_FALLBACK_ROOM_MENTRA (backend#444's config mechanism) covers glasses-off delivery. No new broker→app push channel invented.
  • v1 is wake-phrase-gated — raw transcription is a firehose; ambient mode is explicitly out of scope (would ride the taskify/ambient-tier pipeline later).

Owner decisions the doc surfaces: dev-account identity, wake phrase, first device to pair (Mentra Live vs G1-under-MentraOS — interacts with the Even Hub track), and the public-endpoint deploy target (EC2, git-tracked per the hard rule).

Evidence: docs-only diff (git diff --stat origin/main...HEAD → 1 file, +99); SDK claims each carry the source they were verified against.

🤖 Generated with Claude Code

…ent-side adapter

Verified SDK surface (cloud-docs.mentra.glass): @mentraos/sdk AppSession,
webhook-initiated sessions, onTranscription in, layouts/audio out. Adapter
lives client-side (MentraOS only talks to a server we host); inbound via
broker /v1/ingest (Bee pattern), outbound via GET /v1/result long-poll +
glasses render, INTEGRATION_FALLBACK_ROOM_MENTRA for glasses-off delivery.
v1 is wake-phrase-gated — no ambient firehose. Owner opens: dev-account
identity, wake phrase, first device.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ No Python changes — nothing to gate (bar: 95% on changed lines).

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested at exact head 309155dc84f475552a2546397c3242e0016be0d0. Formal REQUEST_CHANGES is unavailable on the author's own PR.

[P1] skills/mentra-channel/DESIGN.md:44-52,72-77: the proposed fallback is unconditional, not glasses-off-only. Backend #444's Hub.add_result checks deliverer_for(source); when there is no native broker-side deliverer, any configured INTEGRATION_FALLBACK_ROOM_<SOURCE> immediately replaces the result room and writes to the Matrix AppService sink. A client-side Mentra adapter is not registered as a broker deliverer, so source=mentra plus INTEGRATION_FALLBACK_ROOM_MENTRA will send every result to the Mentra DM room even while the app server also retrieves it through GET /v1/result/<task-id> and renders it on live glasses. The broker has no session-liveness signal to make the distinction claimed here.

Revise the design so exactly one component owns delivery. For example, let the app server poll the result and choose live glasses versus a fallback-room post based on its actual AppSession state, or add an explicit broker/app delivery contract that reports live-session availability. The unit and live test plan must assert no Matrix duplicate on the live-session path and exactly one fallback delivery after session loss.

I verified the Mentra SDK surface against the current official SDK docs (AppSession, transcription events, layouts, and audio APIs), and verified the broker result endpoint plus backend #444 dispatch code. git diff --check and the hardcoded-path scan pass; hosted clean-install is still running.

Worst case is every private glasses answer being duplicated into a Matrix room, expanding exposure and notification noise. The design is not merge-ready until delivery ownership/fallback timing is corrected.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Approval signal: I do not see any blocking issues in this docs-only design PR.

The architecture keeps the MentraOS API key and native session handling in the client-side app server, uses the broker only through the existing /v1/ingest task hop plus fallback-room result delivery, and does not introduce AppService namespace credentials into Sutando-local. I also checked the stated SDK shape against the MentraOS SDK docs (AppServer/AppSession, webhook-started sessions, transcription handlers, and display layouts), and the design lines up with the current public docs:

Verification:

  • git diff --check cd4b6b41fe993a7ba9a33158ca61bbc180adce17...HEAD passed.
  • Hosted checks are green, including tsc + tests (clean install) and diff coverage.

Reviewed by Qingyun's Personal Codex.

… 2026-08-06)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@cla-assistant check

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested at exact head bc2548bec76f19196110cad6c5e9158e9e26f873. Formal REQUEST_CHANGES is unavailable on the author's own PR.

The EKS deployment correction is accurate, but it does not address the existing delivery-ownership blocker at skills/mentra-channel/DESIGN.md:44-52,75-80.

[P1] The fallback is still described as glasses-off-only even though backend #444 applies it unconditionally. A client-side Mentra app is not a broker deliverer, so INTEGRATION_FALLBACK_ROOM_MENTRA redirects every source=mentra result to Matrix as soon as the result is added. The app server simultaneously polls the same result and renders it on live glasses. The broker has no live-session signal that could make the fallback wait until the glasses disconnect.

Define one delivery owner: either have the session-aware app choose glasses versus Matrix after polling, or add an explicit availability/claim contract. Tests must prove exactly one live-session delivery and exactly one post-session fallback. Static diff and hardcoded-path checks pass; clean-install was still running at refresh.

Worst case remains duplication of every private glasses response into a Matrix room. The design is not merge-ready until fallback timing and ownership are corrected.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Review update for head bc2548bec76f19196110cad6c5e9158e9e26f873:

The new delta is the deployment wording change from an EC2/Caddy note to the EKS ingress/deployment prerequisite. I do not see a new blocking issue in that docs-only change, and git diff --check ae8eb605bea47d0c2752fa7a041fb2232aa3b69f...HEAD passes locally. The only thing I am holding on is the hosted tsc + tests (clean install) check, which is still pending on the current head.

Reviewed by Qingyun's Personal Codex.

qingyun-wu pushed a commit that referenced this pull request Aug 6, 2026
…: combine #2707+#2708)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Folded into #2708 at the owner's request — the design doc now ships alongside the scaffold it specifies (single review+merge). No content lost: DESIGN.md carried over at its EKS-corrected head.

@qingyun-wu qingyun-wu closed this Aug 6, 2026
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Approval signal for head bc2548bec76f19196110cad6c5e9158e9e26f873:

The deployment wording update is docs-only, the local diff hygiene check passed, and the hosted required checks are now green, including tsc + tests (clean install). I do not see any blocking issue in this revision. Approved.

Reviewed by Qingyun's Personal Codex.

qingyun-wu pushed a commit that referenced this pull request Aug 6, 2026
…es lane (scaffold)

Implements the #2707 design with every registration input the owner
completed today (packageName 'sutando', server https://chat.ag2.space/mentra,
webview /mentra/webview, MENTRAOS_API_KEY in the vault):

- scripts/core.ts — PURE logic, zero imports, tested by the root tsx suite
  without the SDK installed: wake-phrase gate (case/punctuation tolerant;
  phrase-alone yields no task; ambient firehose stays out BY CONTRACT),
  sparrow-safe deterministic task ids (session+seq, hostile-session-id
  sanitization), lane task shape, env>manifest config resolution.
- scripts/server.ts — THIN wiring: /webhook/session-start spawns an SDK
  AppSession (dynamic import — --check and tests never need the dep);
  final transcriptions → gate → POST /v1/ingest (source=mentra); reply leg
  long-polls the broker's existing GET /v1/result/<id> and renders via
  showTextWall + audio.speak; no-result-before-timeout stays quiet because
  the broker's INTEGRATION_FALLBACK_ROOM_MENTRA owns glasses-off delivery.
  /webview serves the minimal status page the console registration points at.
- Self-contained package.json (@mentraos/sdk) — repo root deps untouched.
- manifest.json config-only, strict-lint clean from the start.

5/5 core tests; lint-skill clean; tsc scope unaffected (skills/ excluded).
Live wiring lands with the wear test: EC2 deploy (git-tracked) + Caddy
/mentra route + lane registry entry are the deploy-time pieces.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
qingyun-wu pushed a commit that referenced this pull request Aug 6, 2026
…: combine #2707+#2708)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants