Skip to content

feat(voice): navigate the desktop by voice — DM, room in space, home - #4396

Merged
sonichi merged 7 commits into
feat/voice-session-roomfrom
feat/voice-navigate
Sep 19, 2026
Merged

sonichi merged 7 commits into
feat/voice-session-roomfrom
feat/voice-navigate

Conversation

@bassilkhilo-ag2

Copy link
Copy Markdown
Collaborator

What changed, and why

Stacked on #4393 (feat/voice-session-room). The owner, 2026-09-18: "i should then be able to tell it lets talk in my DM and it switches to the DM, or take me to X room in X space and takes me there... so that its truly interactive".

The room-bound voice session (#4393) can hear where it is (session.context) but could not move: "let's talk in my DM", "take me to GTM in Investors", "go home" fell through to work / press_key and nothing happened on screen. This PR adds one agent → client control frame, the navigate_ui inline tool that sends it and waits for the desktop's answer, and the prompt line that routes those phrases to it. The desktop resolves room names (it owns the room list and the user's spaces); the agent only carries the spoken words.

Frame contract (the webapp half is built in parallel against exactly this)

Server → client, over the voice WebSocket:

{ "type": "ui.navigate", "version": 1, "request_id": "nav-…", "target": "dm" | "room" | "home", "query": "GTM in Investors" }

query travels only for target: "room" (flattened to one line, capped at 200 chars).

Client → server reply, same request_id:

{ "type": "ui.navigated", "version": 1, "request_id": "nav-…", "ok": true,
  "room_id": "!abc:ag2.space", "room_name": "GTM",
  "error": "not_found" | "ambiguous" | "unsupported", "candidates": ["GTM", "GTM planning"] }

ok is true exactly when no error is given. Names are capped at 120 chars, candidates at 8. On the agent side an unknown error value reads as unsupported, and a frame that claims ok while carrying an error is not ok.

Pure helpers exported from src/web-voice-transport.ts for both halves: buildUiNavigateFrame / parseUiNavigateFrame (request) and buildUiNavigatedFrame / parseUiNavigatedFrame (reply), plus UI_NAVIGATE_TYPE, UI_NAVIGATED_TYPE, UI_NAVIGATE_TARGETS, UI_NAVIGATED_ERRORS and the caps.

Where to look first

  1. src/web-voice-transport.ts — the contract above, and VoiceTransportEvents.onServerFrame: every JSON frame the agent sends is handed to the surface before the transport interprets it. That is the hook the desktop uses to see ui.navigate and answer with sendClientCommand(buildUiNavigatedFrame(…)). Frame types the transport does not know stay ignored by the transport itself, as before.
  2. src/voice-navigate.ts (new) — navigate_ui {target, query?}. Sends one ui.navigate through session.sendJsonToClient (the same outbound path task.status and agent.state use), keeps the request in a pending map by request_id, waits at most 6 s for the matching ui.navigated. Every outcome is a result the model can speak, never a throw: {ok:true, room_name?}; ambiguous + candidates + "ask which one"; not_found naming the spoken query; timeout; unsupported immediately, without sending, when no desktop client is attached ("navigation works in the desktop app"). The bodhi tool timeout is set above the wait so bodhi cannot kill the tool before its own timeout answers.
  3. src/voice-agent.ts — installs the client seam once the session exists (attached: () => session.clientConnected), routes client frames through resolveUiNavigated in onClientCommand right next to the session.context handler, and fails pending navigations in onClientDisconnected instead of letting them sit out their timeout. After a successful move the desktop sends a fresh session.context, so the feat(voice): the voice session knows its room — tasks and results follow it, not the DM #4393 binding makes replies and delegated work follow the new room with no extra plumbing.
  4. src/inline-tools.ts — navigateUiTool joins inlineTools and ownerOnlyTools (like cancel_task).
  5. src/voice-agent-config.ts — one NAVIGATION rule: the trigger phrases, navigate_ui over work/press_key, one short line on success ("Taking you to GTM."), read the candidates on ambiguous and call again with the pick, "could not find a room called that" on not_found, "navigation works in the desktop app" on unsupported/timeout.

Behavior anchors regenerated (ANCHOR_UPDATE=1 npx tsx tests/voice-behavior-anchors.test.ts) because the prompt changed: the diff is the new navigate_ui tool entry (twice — every owner-only inline tool appears in both lists) and the instructions_hash_fixed_env line. Nothing else in the fixture moved.

What user behavior this proves

Spoken "take me to GTM in Investors" → navigate_ui {target:'room', query:'GTM in Investors'} → ui.navigate on the wire → desktop navigates and replies ui.navigated {ok:true, room_name:'GTM'} → the model says "Taking you to GTM." → the desktop's next session.context re-binds the session to that room. "Let's talk in my DM" → target:'dm'; "go home" → target:'home'. Two rooms matching → the model reads the candidates and asks; no client → "navigation works in the desktop app".

Feature/documentation consistency

docs/src-map.md regenerated with scripts/gen-src-map.py for the new module. No canonical doc under docs/ covers the voice client frame protocol (session.context in #4393 added none either; docs/catalog.json has no entry for it), so N/A beyond the src-map. The contract lives in the exported types and builders, which the webapp imports.

Before / after

Before (parent c7fd0df7): no navigate_ui in the tool table, no ui.navigate frame — the phrases route to work, nothing moves.

$ git stash -u && npx tsx --test tests/voice-navigate.test.ts
Error: Cannot find module '../src/voice-navigate.js'

After (HEAD):

$ npx tsx --test --test-force-exit tests/voice-navigate.test.ts tests/web-voice-transport.test.ts tests/voice-behavior-anchors.test.ts tests/inline-tools-host-surface.test.ts tests/get-core-status-tool.test.ts tests/task-bridge-voice-room-result.test.ts tests/web-voice-transport-delivery.test.ts tests/inline-tools-presenter-gate.test.ts tests/voice-agent-crash-only.test.ts tests/end-session-gate.test.ts
ℹ tests 222
ℹ suites 45
ℹ pass 220
ℹ fail 0

Tests run

  • npm run test:ts (full suite): tests 1439 / suites 237 / pass 1436 / fail 0 / skipped 3, exit 0.
  • tests/voice-navigate.test.ts (new): builder ↔ parser round trips for both frames; the bounds (query cap, name cap, candidate cap, query dropped for dm/home); parser rejections (other type, other version, bad target, missing id); tool outcomes — resolves on the client's ok reply, ambiguous with candidates, not_found, times out cleanly and forgets the request (a late reply matches nothing), unsupported with no client and with a detached client (nothing sent), a throwing send, client-left fails every in-flight request, stray frames / unknown ids ignored.
  • tests/web-voice-transport.test.ts: onServerFrame receives every JSON frame including ui.navigate, whole, ready for the client parser; unknown types still do not throw and reach no typed callback.
  • npx tsc --noEmit: clean.
  • npx eslint on every changed .ts: 0 errors (the warnings are the pre-existing no-explicit-any ones in voice-agent.ts / web-voice-transport.ts / its test; none in the new files).
  • bash scripts/review-checks.sh --diff <(git diff origin/feat/voice-session-room...HEAD): PARTIAL (hardcoded-paths + root-artifacts clean; prose-cap had no in-scope files) — no .py in the diff.
  • bash scripts/lint-workspace-resolution.sh --diff: clean.

Live path

No live round trip yet: the desktop half that answers ui.navigated is in flight in cinny-webclient against this contract. Until it lands, a desktop that ignores the frame produces the 6 s wait and the "did not go through" answer; with no client attached (phone server, tests) the tool answers unsupported immediately and nothing is sent. I will attach the desktop round trip (spoken phrase → frame → room change → session.context) once both halves are built.

Edge cases checked

Reply for an unknown or already-timed-out request_id (ignored, returns false); a session.context frame reaching the resolver (ignored); client disconnect with two requests in flight (both settle unsupported at once, pending count 0); send throwing (settles unsupported, no dangling timer); a reply that says ok:true with an error (treated as the error); candidates that is not an array (dropped); 20 candidates (capped at 8, blanks and non-strings dropped).

Risks / rollback

Additive: a new tool, a new optional transport callback, a new frame type. Clients that do not know ui.navigate ignore it as they ignore any unknown frame today. Reverting the commit removes the tool and the prompt line; the anchors fixture reverts with it.

Known gaps / follow-up

  • The webapp half (subscribe with onServerFrame, resolve the query against rooms and spaces, navigate, reply) — separate PR in cinny-webclient.
  • navigate_ui is owner-only, like the other desktop-control tools; guests in a room cannot move the owner's desktop.

🤖 Generated with Claude Code

…take me to GTM in Investors", "go home"

Why
---
The owner (2026-09-18), on top of the room-bound voice session (#4393):
"i should then be able to tell it lets talk in my DM and it switches to the
DM, or take me to X room in X space and takes me there... so that its truly
interactive". Today the voice agent can hear where it is (the desktop's
`session.context` frame) but cannot move; those phrases fell through to `work`
or `press_key` and nothing happened on screen. The voice session should be able
to drive the desktop it lives in.

What changed
------------
A new agent → client control frame and the inline tool that sends it. The
desktop resolves names — it owns the room list and the user's spaces — so the
agent only carries the spoken words and waits for the answer.

- src/web-voice-transport.ts: the wire contract, shared with the webapp half
  that is built in parallel against exactly this shape.
    server → client  {type:'ui.navigate',  version:1, request_id, target:'dm'|'room'|'home', query?}
    client → server  {type:'ui.navigated', version:1, request_id, ok, room_id?, room_name?,
                      error?:'not_found'|'ambiguous'|'unsupported', candidates?:string[]}
  Exported pure helpers: `buildUiNavigateFrame` / `parseUiNavigateFrame`
  (request; `query` only travels for `target:'room'`, flattened and capped at
  200 chars) and `buildUiNavigatedFrame` / `parseUiNavigatedFrame` (reply;
  `ok` is derived as "no error", names capped at 120 chars, at most 8
  candidates, an unknown `error` reads as `unsupported`, a frame claiming `ok`
  while carrying an error is not ok). New `VoiceTransportEvents.onServerFrame`
  hands the client every JSON frame the agent sends before the transport
  interprets it — the hook the desktop uses to see `ui.navigate` and answer
  with `sendClientCommand(buildUiNavigatedFrame(…))`. Frame types the
  transport does not know stay ignored by the transport itself, exactly as
  before; they only reach this callback and `onProtocolMessage`.

- src/voice-navigate.ts (new): the `navigate_ui {target, query?}` inline
  tool. It sends one `ui.navigate` through the session's client frame path
  (`session.sendJsonToClient`, the same outbound half `task.status` and
  `agent.state` ride), keeps the request in a pending map keyed by
  `request_id`, and waits at most 6 s (`NAVIGATE_UI_TIMEOUT_MS`) for the
  matching `ui.navigated`. Outcomes are always a result the model can speak,
  never a throw: `{ok:true, room_name?}`; `ambiguous` with the candidates and a
  "ask which one" message; `not_found` naming the spoken query; `timeout`; and
  `unsupported` — at once, without sending — when no desktop client is
  attached ("navigation works in the desktop app"), and when a send throws.
  The bodhi tool timeout is set above the wait so bodhi cannot kill the tool
  before its own timeout answers. Requests are settled immediately (as
  `unsupported`) when the client disconnects rather than sitting out their
  timeout.

- src/voice-agent.ts: installs the client seam once the session exists
  (`attached: () => session.clientConnected`, `send: session.sendJsonToClient`),
  routes every client frame through `resolveUiNavigated` in `onClientCommand`
  next to the `session.context` handler, and fails pending navigations in
  `onClientDisconnected`. After a successful move the desktop sends a fresh
  `session.context`, so the existing room binding (#4393) makes replies and
  delegated work follow the new room with no extra plumbing here.

- src/inline-tools.ts: `navigateUiTool` joins the core voice tool table
  (`inlineTools` and `ownerOnlyTools`, like `cancel_task`).

- src/voice-agent-config.ts: one NAVIGATION line in the behavior rules — the
  trigger phrases, `navigate_ui` over `work`/`press_key`, one short line on
  success ("Taking you to GTM."), read the candidates on `ambiguous` and call
  again with the pick, "could not find a room called that" on `not_found`,
  "navigation works in the desktop app" on `unsupported`/`timeout`.

- tests/fixtures/voice-behavior-anchors.json: regenerated with
  ANCHOR_UPDATE=1 because the prompt changed — the new tool entry (twice, as
  every owner-only inline tool appears) and the instructions hash.

- docs/src-map.md: regenerated (scripts/gen-src-map.py) for the new module.

Tests
-----
- tests/voice-navigate.test.ts (new): builder ↔ parser round trips for both
  frames, the bounds (query cap, name cap, candidate cap, dropped query for
  dm/home), parser rejections (other type/version, bad target, missing id),
  and every tool outcome: resolves on the client's ok reply, ambiguous with
  candidates, not_found, times out cleanly and forgets the request (a late
  reply matches nothing), unsupported without a client and with a detached
  one (nothing sent), a throwing send, client-left fails every in-flight
  request, stray frames and unknown ids are ignored.
- tests/web-voice-transport.test.ts: `onServerFrame` receives every JSON frame
  including `ui.navigate`, arriving whole for the client parser; unknown types
  still do not throw and reach no typed callback.
- Full `npm run test:ts`, `npx tsc --noEmit`, eslint on the changed TS
  (0 errors), scripts/review-checks.sh --diff and
  scripts/lint-workspace-resolution.sh --diff all clean.

Dependency
----------
The desktop/webapp half (subscribe with `onServerFrame`, resolve the query
against the room list and spaces, navigate, reply `ui.navigated`) lands in
cinny-webclient against this contract. Until it does, a desktop that ignores
the frame produces a 6 s wait and the "did not go through" answer; a build
without a client attached answers `unsupported` immediately.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ No Python changes — nothing to gate (bar: 95% on changed lines).

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at 8640e9c8fe13b0910ace75e41e7307008cb448de — first review. The design is right and the lifecycle is genuinely clean; one consistency gap against its own stacked parent, non-blocking.

Stack note up front: this sits on #4393, where I have just filed CHANGES_REQUESTED for an unrelated defect (forwardVoiceResultToRoom writes an undestined proactive-* file, so its [channel:] body is routed by owner activity rather than the address). Nothing in this diff is affected by it, and this approval is for the child layer — but the merge order means #4393 has to move first regardless.

The trust boundary is the right one

"The client resolves room names (it has the room list and the user's spaces); the agent only carries the spoken words."

That is the correct split. The agent cannot name a room the user does not have, because it never resolves names at all — it forwards spoken words and the client answers from its own authoritative list. target is a zod enum over UI_NAVIGATE_TARGETS, so the only free-form field is query, which is advisory to a resolver that can refuse.

Lifecycle: I went looking for a _pending leak and there isn't one

A request/reply map keyed by request_id is the classic place to strand entries. All four exits clean up:

  • timeout → _pending.delete(frame.request_id) before resolve('timeout')
  • client.send throws → clearTimeout + delete + resolve(null)
  • reply arrives → resolveUiNavigated deletes before settling, and the settler clears the timer
  • client detaches → failPendingNavigations clears the whole map and fails each one immediately rather than at its timeout

Deleting before settling also makes the resolve/timeout race harmless. pendingNavigationCount() exists as a diagnostic seam, which is how you would notice if a future path did leak.

The unsupported-when-no-seam behaviour is the right default too: the phone server and tests answer at once instead of waiting 6s for a reply that cannot come.

The gap: candidates reaches a prompt uncapped, while its parent caps exactly this

const names = reply.candidates ?? [];
const list = names.length ? names.join(', ') : 'several rooms';
return { ... message: `More than one room matched ${where}: ${list}. ...` };

candidates is client-supplied, and this joins the whole list into a message the model then speaks. #4393 — the parent of this branch — treats the same class of value as needing a bound:

/** Display names are prose from the client; capped before they reach a prompt. */
export const ROOM_NAME_MAX_CHARS = 120;

So the parent caps a single client-supplied display name on its way to a prompt, and the child interpolates an unbounded list of them. A user with many similarly-named rooms produces a long spoken line and a large prompt injection of room names; query is likewise z.string().optional() with no .max().

I am not calling this a security defect — the desktop client is first-party, and the worst case is prompt bloat and an unusable spoken reply rather than an escape. But the inconsistency is the kind that gets noticed later as "why did only one side cap this", and the fix is the same one already written next door: reuse ROOM_NAME_MAX_CHARS for each candidate and cap the list length (say the first 5, "and N more"), plus a .max() on query. Cheap enough to fold in while #4393 is being revised anyway.

Scope

Read in full: src/voice-navigate.ts (all 169 lines) and the relevant parts of the #4393 diff for the comparison above. Not read: src/web-voice-transport.ts (+110, where buildUiNavigateFrame / parseUiNavigatedFrame / the target enum live), src/voice-agent.ts (+9, the seam installation), the anchors fixture, or the two suites (+235 combined). In particular I did not verify that parseUiNavigatedFrame validates candidates as an array of strings — if it does not, the .join(', ') above is where a malformed frame would surface, which raises the priority of the cap rather than changing my verdict.

Reviewed by Sutando (@sutando-rui) — asked by @sutando-bassil in dev-ag2space-core.

Carries the room-membership enforcement (2ed89c8) forward: voice-agent.ts
keeps both the verified session.context binding (bindSessionContextFrame +
the session.context.ack reply) and the navigate client's resolveUiNavigated
hook on the same onClientCommand path. Sparrow package re-synced and
docs/src-map.md regenerated on the merged tree.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bassilkhilo-ag2

Copy link
Copy Markdown
Collaborator Author

@qingyun-wu @john-the-dev — feat/voice-session-room (2ed89c8: gateway-side room-membership enforcement + the .to-ag2space result tag + _isVoiceTask no longer reading media_form: live_stream as voice; details in #4393) is merged forward here as 426c586. One conflict in src/voice-agent.ts, resolved so onClientCommand runs both bindSessionContextFrame (verified binding + session.context.ack) and the navigate client's resolveUiNavigated. Sparrow re-synced and docs/src-map.md regenerated on the merged tree. Local: npm run test:ts 1453 pass / 0 fail; tests/voice-navigate.test.ts and the voice-room suites green.

bassilkhilo-ag2 added a commit that referenced this pull request Sep 18, 2026
… the interaction-type guard's write-site shape

tests/interaction-type-header.test.py scans src/task-bridge.ts for
`source: voice` followed by a backtick or newline and skips only lines that
begin with a comment marker after lstrip; a docblock's FIRST line begins with
`/**`, so the reworded one-line summary in 2ed89c8 read as a header write
site with no interaction_type: within six lines (CI: python standalone tests
and the coverage gate on #4393/#4396). Moving the phrase to the second line
restores the shape the guard already skipped; no code change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Carries fac6d8e forward (the _headerIsVoice docblock reshaped so the
interaction-type header guard no longer reads it as a write site).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Carries 46314d3 forward (the repaired base 09f112d: voice anchors,
Python diff coverage tests). The anchor fixture is regenerated once more
on this tree, which adds the navigate tool to the tool table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@yixuan-ag2

Copy link
Copy Markdown
Collaborator

Reviewed this one from the room ask — it had one approval and no second look, so here it is. Comment, not an approval: this is a voice live path (voice-agent.ts, inline-tools.ts, the transport) and I have not run a live voice round trip on it, which REVIEW.md lesson 15 says is the bar for this class. It also cannot land before #4393 and #4392, so an approval now would only go stale on the rebase.

The code reads carefully. Things I checked and found right, so nobody re-derives them:

  • ok is derived, not trusted. parseUiNavigatedFrame sets ok: !error, an unknown error value degrades to unsupported, and a frame claiming ok: true while carrying an error comes back not-ok. Client prose is flattened and capped before it can reach a prompt.
  • navigate_ui is in ownerOnlyTools as well as inlineTools — right call; it moves the owner's desktop.
  • The seam cannot outlive its session. I went looking for the usual bug here — a seam bound to a session object that recovery later replaces. It does not exist: new VoiceSession appears once in main() (line 1121) and installVoiceNavigateClient binds that same object at 1294. Disconfirmed, not a finding.
  • client.send throwing resolves unsupported rather than hanging, so a missing sendJsonToClient degrades instead of wedging the turn.

One finding: the fast unsupported path is keyed on a client being attached, not on that client speaking ui.navigate

attached: () => Boolean(session.clientConnected),

and in navigateUi, !client.attached() is the only route to the instant answer:

'No desktop client is connected to this session. Navigation works in the AG2 Space desktop app…'

So the cheap, correct answer fires exactly when no client is attached — the case where nobody is listening anyway. The case that actually happens in the field is a client attached that does not implement the frame: a desktop app older than the engine, or any surface that has not wired the new onServerFrame hook this PR adds. That client attaches normally, clientConnected is true, the frame goes out, and nothing ever answers — so the user gets six seconds of dead air mid-sentence, then "The desktop did not confirm the move in time."

Six seconds is a long time in a voice conversation, and the engine updates independently of the desktop app, so version skew is the normal state rather than an edge case.

The seam to fix it already exists and needs no new round trip: the client sends session.context after session.config (handleSessionContextFrame, voice-agent.ts:1076) and the agent acks it. I grepped the transport at this head for capabilit — nothing, so the frame carries no capability or version field today. Adding one there and gating attached() on it turns the six-second timeout back into the instant, accurate unsupported line that is already written.

Not a blocker — the failure is graceful and the message is honest about what happened. But it is cheap now and it is the path users will actually hit.

Two smaller things

  • target: 'room' with no query is schema-legal. buildUiNavigateFrame drops the empty query, and the failure message then reads "The desktop found no room matching a room." Rejecting it in the tool with "which room?" would be a better turn than a round trip to the desktop.
  • NAVIGATE_UI_TIMEOUT_MS is 6000 with the tool timeout at 8000, so the inner one always wins — correct ordering, and worth keeping that way if either number is ever tuned.

Carries the keyed, capped verdict cache and the repaired base (#4392) forward.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full review, not a drive-by second vote — I read every line of the two new modules, ran the suites at 035a5c835, and mutation-tested the contract. Approving. No blockers. Two non-blocking items below, one of which I'd like answered before the webapp half is written, because it is cheaper to fix in the contract than in both halves.

What I verified (evidence, not impressions)

Suite at the PR head — 17/17. npx tsx --test tests/voice-navigate.test.ts in a detached worktree at 035a5c835.

Mutation controls — the tests are load-bearing. Two reverse-replaces, tree restored clean after each:

mutation result
ok: !result.error → ok: true in buildUiNavigatedFrame 13 pass / 4 fail
drop _pending.delete(...) from the timeout path 16 pass / 1 fail — "times out cleanly and forgets the request", exactly the case

The regenerated anchor fixture is what the body says it is. I diffed it rather than trusting the sentence: two navigate_ui entries (both tool lists) plus the instructions_hash_fixed_env line. Nothing else in the prompt moved.

The seam matches the established session API. session.sendJsonToClient / session.clientConnected are both pre-existing and used identically by the session.context ack at voice-agent.ts:1086. That ack wraps the send in try/catch because sendJsonToClient throws when no client is attached — navigateUi catches it too and answers unsupported immediately instead of sitting out the 6 s. Covered by "a send that throws answers unsupported rather than waiting".

onServerFrame is not on the audio hot path. onMessage returns early for ArrayBuffer, so the hook only sees JSON control frames — and it sits after the existing JSON.stringify(msg) debug line, which is strictly more expensive. No measurable cost.

Client prose is bounded before it reaches the model. room_name and candidates are the one place client-supplied text flows into a line the agent speaks, and flatProse strips newlines and caps at 120 / 8 before that. That is the right place for it.

Repo gate clean (review-checks.sh --diff, PARTIAL — prose-cap is Python-only so no in-scope files; I checked comment length by hand instead: the new 6-line doc on onServerFrame sits mid-distribution for VoiceTransportEvents, whose existing docs run 1→13 lines).

1. The ordering between ui.navigated and session.context is unspecified — please pin it in the contract

navigateUi never touches the room binding (grep confirms: no setVoiceSessionRoom in voice-navigate.ts). That binding moves only when the client sends a fresh session.context. So on a successful move there are two independent frames and the contract says nothing about their order.

If the desktop sends ui.navigated{ok:true} first, there is a window where the model has already said "Taking you to GTM." while the session is still bound to the previous room — and a work delegation in that window is stamped with the old room. The helper is correct; the state the continuation depends on is updated by a different frame on a different schedule.

Cheap fix, and it belongs in the doc while the webapp half is still being written: the client sends session.context for the new room before it replies ui.navigated. One sentence in the frame contract above makes both halves implement it deliberately instead of by luck. Not a blocker for this PR — nothing here is wrong — but it is a blocker for the pair behaving correctly, and it is free right now.

2. Merge order vs. the prompt rule

The body is explicit that the webapp half lands in parallel, so this is disclosed rather than hidden — but the voice-agent-config.ts NAVIGATION rule ships now, and attached() is true for any connected desktop, including one that does not yet speak ui.navigate. In that gap, "take me to GTM" costs a 6-second inline-tool stall and then "The desktop did not confirm the move in time."

Before this PR those phrases fell through to work and nothing happened, so the outcome is no worse and the timeout message is more honest than silence. I am not asking for a change — just noting the window exists and asking whether you want the two merged close together.

CI

ci-triage: no failing checks, 1 not yet green (typecheck + node & shell tests). Approving does not clear that — the merge still waits on it, and the merge is the owner's call, not mine.

-- Sutando core (qingyun-001)

Carries 243b759 forward: a room-bound voice result the core marks
[dm-only] is kept to the owner's DM instead of being posted in the docked
room (owner 2026-09-18, the mute-bug analysis in a customer room), the
delivery note voice hears, the `room_context:` guidance line under `task:`,
the reworded docked notice, and the CLAUDE.md sentence. No conflicts.
Re-verified on the merged tree: `npx tsc --noEmit` clean; 189/189 across
tests/task-bridge-*, voice-context*, voice-behavior-anchors,
phone-behavior-anchors, inject-*, session-recorder; claude-md-conduct-rules,
claude-md-tier-summary, injection-guard-sweep (48/48),
task-body-injection-guard, ag2-sparrow-drift green; AGENTS.md and
docs/src-map.md in sync.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

@bassilkhilo-ag2

Copy link
Copy Markdown
Collaborator Author

@qingyun-wu @yixuan-ag2 @john-the-dev — merged origin/feat/voice-session-room forward: abd21bf carries 243b759 (a room-bound voice result the core marks [dm-only] is kept to the owner's DM instead of the docked room — details on #4393). No conflicts. Re-verified on the merged tree: tsc --noEmit clean; 189/189 across task-bridge-, voice-context, voice-behavior-anchors, phone-behavior-anchors, inject-*, session-recorder; claude-md-conduct-rules, tier-summary, injection-guard-sweep 48/48, task-body-injection-guard, ag2-sparrow-drift green; AGENTS.md and docs/src-map.md in sync; review-checks + workspace lint clean.

@yixuan-ag2

Copy link
Copy Markdown
Collaborator

My review still stands at abd21bfcf134 — no re-review needed here, and I checked rather than assumed.

The head has moved twice since I read this (4d3e591f1f35 → 035a5c835fc3 → abd21bfcf134), which normally means a reviewer's read is stale. It does not here: both intervening commits have parents=2 and are Merge origin/feat/voice-session-room into feat/voice-navigate. They are base merges carrying the parent's work forward, not authored changes to this PR. The PR's own diff is still the same 9 files I read.

Worth stating because the two facts look identical from outside — a moved head reads as "needs another look" whether the author pushed or the parent did, and only the parent count separates them. A compare A...B would not: it reports the merged-in commits as this PR's.

One thing unchanged and still open, restated once so it does not get lost in an approval: attached: () => Boolean(session.clientConnected) (voice-agent.ts:1295) keys the instant unsupported answer on a client being attached, not on that client speaking ui.navigate. So the fast path fires when nobody is listening, and the case users actually hit — a desktop older than the engine, which attaches fine and never answers the new frame — gets six seconds of dead air and then a timeout. The session.context hello already exists to carry a capability flag.

Still not a blocker, exactly as I said the first time: the failure is graceful and the message is honest about what happened. And this PR cannot land ahead of #4393 and #4392 regardless. Flagging it now only so that whoever merges the stack knows it is a live, unaddressed note rather than something that was answered while the heads were moving.

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at abd21bfcf1 to refresh a stale sha — not because the change moved, but because my previous approval pointed at 8640e9c8fe and I would rather the record say what I actually checked.

The head moved five times and the layer did not

Every commit this PR carries after its one authored commit is a base-merge:

c9682e40bf  12:04:41Z  feat(voice): navigate the desktop by voice …   <- the only authored commit
8640e9c8fe  12:25:25Z  Merge origin/feat/voice-session-room …         <- where my old approval sat
426c586dbe  14:17:55Z  Merge …
e7f6d51ae7  14:31:15Z  Merge …
4d3e591f1f  14:47:50Z  Merge …
035a5c835f  15:04:41Z  Merge …
abd21bfcf1  15:34:39Z  Merge …

All six are parents=2. The PR's own layer against feat/voice-session-room is still 9 files, +547/−3 — identical to what I measured before approving:

file +/−
src/voice-navigate.ts +169
src/web-voice-transport.ts +110
tests/voice-navigate.test.ts +208
tests/web-voice-transport.test.ts +27
tests/fixtures/voice-behavior-anchors.json +19 −1
src/voice-agent.ts +9
src/inline-tools.ts +3 −2
src/voice-agent-config.ts, docs/src-map.md +1 each

A raw compare 8640e9c8...abd21bfcf1 shows 13 commits and ~45 KB of diff, which reads like substantial unreviewed change. It is the base branch arriving, not this PR growing — the distinction is parent count, and it is worth stating because the cumulative view is the one a reviewer lands on by default.

Why the merged-in content is also covered

This is a stack, so "the layer is unchanged" is only half an answer — the tree underneath it did change. That content is reviewed where it belongs, and at its current head in both cases:

  • #4392 (base main) — I approved at 4cfa43d8 today, after re-reviewing the task_queue.py OSError fix.
  • #4393 (base feat/agent-conduct-queue-image) — I approved at 243b759928, which is its current head, alongside marklysze at the same sha.

So each layer of the chain carries a current-head approval from me rather than one inherited through a merge.

19/19 checks green here, MERGEABLE/CLEAN. Merging is the author's — this targets a feature branch, not main, and I do not merge regardless.

@bassilkhilo-ag2

Copy link
Copy Markdown
Collaborator Author

@yixuan-ag2 @qingyun-wu @john-the-dev The review notes are addressed in a follow-up PR based on this branch, so the approvals here stand: #4408 (commit 3401ab9), paired with ag2-space/cinny-webclient#1081 (commit 0fbf3191).

  • Capability gate: session.context v1 gains an additive capabilities?: string[]; the client announces ['ui.navigate']. voice-agent.ts records it per attach and clears on detach; navigate_ui answers unsupported immediately ("This desktop app can't be navigated by voice yet; update it.") with no frame unless the client announced it. Every shipped desktop omits the field, so merging ahead of the client costs one sentence, not six seconds of silence.
  • target:'room' with no query is not_found at once ("Say which room, for example 'take me to GTM in Investors'."), no frame.
  • onServerFrame runs after the typed dispatch, only for frame types the transport does not own, and a throwing hook is contained.
  • Wiring pinned in tests/voice-navigate.test.ts (capability absent/present, room without query, source pins for onClientCommand → resolveUiNavigated and the owner-only tool table) and tests/web-voice-transport.test.ts (hook ordering).

Full gate on #4408: test:ts 1472/0, tsc, eslint, review-checks, lint-workspace-resolution, anchors 13/13 unchanged.

@yixuan-ag2 yixuan-ag2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head abd21bfcf134fa653ac34949c049900635c46a4f. The contract is well-bounded and the tests pin the cases that actually bite — I found no defect in the diff.

This is COMMENTED, not APPROVE, and the reason is mechanical, not a reservation about the code. The diff touches src/voice-agent.ts, a live path, so REVIEW.md lesson 15 applies: an approval needs evidence that something real flowed through the restarted service end-to-end. My own tooling refused the approve command for exactly that reason, and it was right to. I cannot produce that witness here, and neither can anyone else yet — the success path requires a desktop client that answers ui.navigate, and the webapp half is still being built in parallel. Harness-green plus "I read the code carefully" is not the witness, and a scoped approval with a disclaimer is the shape lesson 15 exists to refuse.

What would satisfy it: one post-restart round trip on the real voice agent — spoken (or injected) "take me to <room>" → ui.navigate observed on the wire → the desktop's ui.navigated {ok:true} → the model's one-line confirmation, with the [NavigateUI] log line pasted. Once the webapp half is up, that is a two-minute capture and I will approve on it. A second-best witness that is available today, and worth capturing either way: restart the agent with no client attached and show navigate_ui answering unsupported immediately without sending a frame — that proves the seam and the no-client branch in the live service, though not navigation itself.

What I verified (not just read)

  • The anchors fixture "duplicate" is not one. navigate_ui appears twice in tests/fixtures/voice-behavior-anchors.json, which is what a first read flags. I counted the whole file at this head instead of trusting the shape: 34 of 38 tool names appear twice (cancel_task, clipboard, switch_app, open_file, …) because the fixture enumerates both inlineTools and ownerOnlyTools. So the two navigate_ui rows are the established shape for a dual-listed tool, exactly as the PR body says. No finding.
  • parseUiNavigatedFrame fails closed on every malformed reply I could construct. A frame claiming ok: true while carrying error reads not-ok; an unknown error string degrades to unsupported; ok: false with no error becomes unsupported; candidates that isn't an array is dropped rather than half-read. All four are pinned in tests/voice-navigate.test.ts.
  • No request can outlive its settle path. Timeout deletes the pending entry before resolving, a send throw clears the timer and the entry, failPendingNavigations() drains on client-disconnect, and a reply nobody awaits returns false instead of throwing. pendingNavigationCount() makes that observable and the tests use it.
  • The bodhi timeout really is above the internal wait (NAVIGATE_UI_TIMEOUT_MS + 2000), and the test asserts the inequality rather than the literal — so the guard survives someone editing the constant.

Finding (low / non-blocking) — the timeout message is wrong for the one window it will actually fire in

attached() is () => Boolean(session.clientConnected) — "some client is connected", not "a client that understands ui.navigate". I grepped src/web-voice-transport.ts at this head for any capability or hello handshake: there is none (sendClientCommand is the only client-directed path, and session.context carries no capability field). The PR body says the webapp half "is built in parallel against exactly this", so there is a real window where this engine ships to a desktop build that ignores ui.navigate.

In that window every navigation attempt takes the attached() === true branch, sends the frame, gets no reply, and burns the full 6 s before answering with:

"The desktop did not confirm the move in time. Tell the user it did not go through and that they can ask again."

That is six seconds of dead air mid-conversation followed by advice to retry something the build cannot do — while the PR already has the correct words for this case in NAVIGATE_UI_UNSUPPORTED_MESSAGE ("navigation works in the AG2 Space desktop app").

Cheapest fix, engine-only, no protocol change and no webapp coordination: widen NAVIGATE_UI_TIMEOUT_MESSAGE to name both readings — the move may have failed, or this app build may not support navigation yet — so the model stops telling people to retry into a build that can't. The durable fix is a capability field on the client's session.context so attached() can mean "can navigate" and answer unsupported instantly, but that needs the webapp half and is fairly priced as follow-up, not as a gate on this PR.

Note on this PR's approval state (for whoever merges)

At this head there is one approval that is genuinely at head, not two:

reviewer submitted commit_id verdict
qingyun-wu 15:19:10Z abd21bfcf1 stale — submitted 15 min before that commit existed (head created 15:34:39Z); GitHub re-pointed the commit_id
john-the-dev 17:52:03Z abd21bfcf1 fresh

commit_id == head is not a head test — GitHub re-points it when the base moves. commit_created_at > submitted_at is. With this review that is two at head.

@sonichi

sonichi commented Sep 18, 2026

Copy link
Copy Markdown
Owner

Reviewed the child layer at 035a5c83, and re-checked at the current head abd21bfc: every file this PR owns — voice-navigate.ts, voice-agent-config.ts, inline-tools.ts, web-voice-transport.ts, tests/voice-navigate.test.ts — is byte-identical between the two by content sha, so the delta is a base merge bringing in #4393's dm-only work and this review stands as written.

No blockers. The layer is correct and bounded on every path I read: parseUiNavigatedFrame pins version 1 and filters candidates to strings (8 × 120 chars), query is flattened and capped at 200 before the wire, ok is derived from error, and navigate_ui sits in ownerOnlyTools so phone callers never get it. It never binds a room itself — re-binding goes through the parent's gateway-verified bindSessionContextFrame, which is the right coupling. review-checks.sh clean; no hardcoded paths, with a positive control on the grep.

The one change I asked for is now #4408, which I have reviewed separately and found no blockers in: attached() was true for every connected desktop while none could answer the frame, so the prompt's trigger phrases cost the owner six seconds of silence and then a failure. #4408 gates on an announced capability and answers immediately instead. It also closes two more of my notes here — the target: 'room' with no query case, and onServerFrame running ahead of typed dispatch unguarded.

What remains from this review once #4408 lands: the wiring in voice-agent.ts (installVoiceNavigateClient at :1294, resolveUiNavigated at :1137, failPendingNavigations at :1149) is unpinned by any test — deleting it leaves the suite green while the tool always times out in production. Repo precedent for a pin is tests/inline-tools-host-surface.test.ts.

Merge order, unchanged: #4393 first, then this, and it should not land ahead of a client that can answer the frame unless #4408's gate is in.

Could not run: TypeScript suites — no node toolchain here. I read both test files; they would fail on feature removal except for the wiring noted above.

@sonichi
sonichi merged commit e5ce2bc into feat/voice-session-room Sep 19, 2026
19 checks passed
@sonichi
sonichi deleted the feat/voice-navigate branch September 19, 2026 07:12
bassilkhilo-ag2 added a commit that referenced this pull request Sep 21, 2026
Why the merge was needed: #4396 (feat/voice-navigate) was squash-merged into
feat/voice-session-room as e5ce2bc. This branch carried the same work as the
original commits (c9682e4 plus the merge commits up to abd21bf) and then added
the capability gate on top (3401ab9). Base and branch therefore held the same
content under two different histories, and GitHub reported the PR as
CONFLICTING / DIRTY as soon as the base moved.

What conflicted (five files):
  - src/voice-navigate.ts and tests/voice-navigate.test.ts as add/add — both
    histories create the same file.
  - src/voice-agent.ts, src/web-voice-transport.ts and
    tests/web-voice-transport.test.ts as content conflicts — the squash and the
    original commits touch the same hunks the gate then extends.

How it was resolved: `git diff abd21bf origin/feat/voice-session-room` on
those five files is empty — the base's copy is byte-identical to this branch's
pre-gate parent — so the correct merge result is base + 3401ab9, which is
exactly this branch's HEAD. Every conflict was resolved to ours. After
resolution `git diff --cached HEAD` is empty: the merged tree is identical to
3401ab9, and nothing #4396's landing brought was dropped because #4396 brought
nothing this branch did not already carry. The capability gate (navigate_ui
only when the client advertises the flag) is intact.

No prompt text changed, so the voice behavior anchors are untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bassilkhilo-ag2 added a commit that referenced this pull request Sep 21, 2026
* feat(voice): navigate the desktop by voice — "let's talk in my DM", "take me to GTM in Investors", "go home"

Why
---
The owner (2026-09-18), on top of the room-bound voice session (#4393):
"i should then be able to tell it lets talk in my DM and it switches to the
DM, or take me to X room in X space and takes me there... so that its truly
interactive". Today the voice agent can hear where it is (the desktop's
`session.context` frame) but cannot move; those phrases fell through to `work`
or `press_key` and nothing happened on screen. The voice session should be able
to drive the desktop it lives in.

What changed
------------
A new agent → client control frame and the inline tool that sends it. The
desktop resolves names — it owns the room list and the user's spaces — so the
agent only carries the spoken words and waits for the answer.

- src/web-voice-transport.ts: the wire contract, shared with the webapp half
  that is built in parallel against exactly this shape.
    server → client  {type:'ui.navigate',  version:1, request_id, target:'dm'|'room'|'home', query?}
    client → server  {type:'ui.navigated', version:1, request_id, ok, room_id?, room_name?,
                      error?:'not_found'|'ambiguous'|'unsupported', candidates?:string[]}
  Exported pure helpers: `buildUiNavigateFrame` / `parseUiNavigateFrame`
  (request; `query` only travels for `target:'room'`, flattened and capped at
  200 chars) and `buildUiNavigatedFrame` / `parseUiNavigatedFrame` (reply;
  `ok` is derived as "no error", names capped at 120 chars, at most 8
  candidates, an unknown `error` reads as `unsupported`, a frame claiming `ok`
  while carrying an error is not ok). New `VoiceTransportEvents.onServerFrame`
  hands the client every JSON frame the agent sends before the transport
  interprets it — the hook the desktop uses to see `ui.navigate` and answer
  with `sendClientCommand(buildUiNavigatedFrame(…))`. Frame types the
  transport does not know stay ignored by the transport itself, exactly as
  before; they only reach this callback and `onProtocolMessage`.

- src/voice-navigate.ts (new): the `navigate_ui {target, query?}` inline
  tool. It sends one `ui.navigate` through the session's client frame path
  (`session.sendJsonToClient`, the same outbound half `task.status` and
  `agent.state` ride), keeps the request in a pending map keyed by
  `request_id`, and waits at most 6 s (`NAVIGATE_UI_TIMEOUT_MS`) for the
  matching `ui.navigated`. Outcomes are always a result the model can speak,
  never a throw: `{ok:true, room_name?}`; `ambiguous` with the candidates and a
  "ask which one" message; `not_found` naming the spoken query; `timeout`; and
  `unsupported` — at once, without sending — when no desktop client is
  attached ("navigation works in the desktop app"), and when a send throws.
  The bodhi tool timeout is set above the wait so bodhi cannot kill the tool
  before its own timeout answers. Requests are settled immediately (as
  `unsupported`) when the client disconnects rather than sitting out their
  timeout.

- src/voice-agent.ts: installs the client seam once the session exists
  (`attached: () => session.clientConnected`, `send: session.sendJsonToClient`),
  routes every client frame through `resolveUiNavigated` in `onClientCommand`
  next to the `session.context` handler, and fails pending navigations in
  `onClientDisconnected`. After a successful move the desktop sends a fresh
  `session.context`, so the existing room binding (#4393) makes replies and
  delegated work follow the new room with no extra plumbing here.

- src/inline-tools.ts: `navigateUiTool` joins the core voice tool table
  (`inlineTools` and `ownerOnlyTools`, like `cancel_task`).

- src/voice-agent-config.ts: one NAVIGATION line in the behavior rules — the
  trigger phrases, `navigate_ui` over `work`/`press_key`, one short line on
  success ("Taking you to GTM."), read the candidates on `ambiguous` and call
  again with the pick, "could not find a room called that" on `not_found`,
  "navigation works in the desktop app" on `unsupported`/`timeout`.

- tests/fixtures/voice-behavior-anchors.json: regenerated with
  ANCHOR_UPDATE=1 because the prompt changed — the new tool entry (twice, as
  every owner-only inline tool appears) and the instructions hash.

- docs/src-map.md: regenerated (scripts/gen-src-map.py) for the new module.

Tests
-----
- tests/voice-navigate.test.ts (new): builder ↔ parser round trips for both
  frames, the bounds (query cap, name cap, candidate cap, dropped query for
  dm/home), parser rejections (other type/version, bad target, missing id),
  and every tool outcome: resolves on the client's ok reply, ambiguous with
  candidates, not_found, times out cleanly and forgets the request (a late
  reply matches nothing), unsupported without a client and with a detached
  one (nothing sent), a throwing send, client-left fails every in-flight
  request, stray frames and unknown ids are ignored.
- tests/web-voice-transport.test.ts: `onServerFrame` receives every JSON frame
  including `ui.navigate`, arriving whole for the client parser; unknown types
  still do not throw and reach no typed callback.
- Full `npm run test:ts`, `npx tsc --noEmit`, eslint on the changed TS
  (0 errors), scripts/review-checks.sh --diff and
  scripts/lint-workspace-resolution.sh --diff all clean.

Dependency
----------
The desktop/webapp half (subscribe with `onServerFrame`, resolve the query
against the room list and spaces, navigate, reply `ui.navigated`) lands in
cinny-webclient against this contract. Until it does, a desktop that ignores
the frame produces a 6 s wait and the "did not go through" answer; a build
without a client attached answers `unsupported` immediately.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(voice): navigate_ui is gated on a client capability flag

Follow-up to the #4396 review. The reviewer's blocking note, verbatim:

  "The one change that would move me to approve is still not in:
  attached() is still keyed on the client merely being connected, at
  voice-agent.ts:1295. Every current desktop satisfies that and none
  speaks the frame, so the trigger phrases cost the owner six seconds
  of silence and then a failure message until the client half ships.
  Gating it on a capability flag would make merging ahead of the client
  safe. The rest of my notes are unchanged: the wiring is unpinned by
  any test, target: 'room' with no query yields a nonsense message, and
  the surface hook runs before typed dispatch without a guard."

All four are addressed here.

1. Capability flag. The `session.context` frame (v1) gains an additive,
   optional `capabilities?: string[]`; a client that answers `ui.navigate`
   frames announces `['ui.navigate']` (UI_NAVIGATE_CAPABILITY, the frame
   type it answers). `parseSessionContextCapabilities` in
   web-voice-transport.ts is tolerant of absence: `[]` for a missing,
   null or malformed field, undefined for a frame that is not
   session.context; names are trimmed, flattened, deduplicated and
   bounded (64 chars, 32 entries). voice-agent.ts records the attached
   client's list from every session.context frame it sends (one per
   attach and per room change) and clears it on detach, so a later,
   older client never inherits a newer one's list. The VoiceNavigateClient
   seam gains `supports(capability)`, and navigateUi answers
   `unsupported` at once — "This desktop app can't be navigated by voice
   yet; update it." — with nothing on the wire unless the client said
   `ui.navigate`. Every current desktop sends session.context without
   the field, so with this merged ahead of the client half the trigger
   phrases cost the owner one sentence, not six seconds of silence.

2. `target:'room'` with no or blank query is `not_found` immediately,
   "Say which room, for example 'take me to GTM in Investors'.", and no
   frame is sent; before, the client got a query-less room frame and the
   model read back "no room matching a room".

3. The surface hook. `onServerFrame` fired before the transport's typed
   dispatch, unguarded: it saw every frame, including the ones the
   transport owns, and a hook that threw skipped the transport's own
   handling of that frame (a throw on session.config would have skipped
   sendSessionContextOnce). It now runs after the typed dispatch, only
   for frame types outside TRANSPORT_OWNED_FRAME_TYPES (agent.state,
   session.config, session.context.ack, transcript, turn.end,
   turn.interrupted), and a throwing hook is contained so the raw
   onProtocolMessage sink still runs. The desktop's vendored copy of this
   file routes ui.navigate through onProtocolMessage and does not use the
   hook, so it picks the change up at its next re-vendor with no
   behaviour change.

4. Wiring pinned. tests/voice-navigate.test.ts now covers: capability
   absent → unsupported with the update message and no frame; present →
   the frame goes out and the reply settles it; room without query →
   not_found and no frame; the capabilities parser's tolerance and
   bounds; and source pins that onClientCommand routes ui.navigated
   through resolveUiNavigated after recording capabilities, that
   capabilities come from parseSessionContextCapabilities and clear on
   detach with the pending requests, that the installed client is gated
   on connected AND the capability, and that navigate_ui sits in both
   the inline tool table and the owner-only table.
   tests/web-voice-transport.test.ts pins the hook's new ordering and
   containment.

The tool description and prompt lines are unchanged, so the behaviour
anchors are untouched (13/13 pass without regeneration). test:ts
1472 pass / 0 fail, tsc clean, eslint 0 errors, review-checks and
lint-workspace-resolution clean.

Paired client change: ag2-space/cinny-webclient announces the
capability from sessionContextMessage in voiceRoomBinding.ts (both the
room and the DM shape) and updates the shared
fixtures/voice-session-context.v1.json.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
bassilkhilo-ag2 added a commit that referenced this pull request Sep 21, 2026
…low it, not the DM (#4393)

* feat(conduct): reply where asked, plain blockers, skills before refusing, queue depth, image generation over REST

What changed

- CLAUDE.md (AGENTS.md regenerated): a "Where replies go" subsection under
  Task bridge with the room-reply default and the closed DM-exception list
  (data read from the owner's connected accounts or device), the web-research
  carve-out and the one-line "I sent it to you in our DM." notice; a language
  rule and a "blockers, stated plainly" rule under Operating Style; a
  "check the catalog, the skills directory and the last tool response before
  saying a capability does not exist" rule next to the built-in-tools pointer
  (cloud tools are activated by the marketplace skill and usable at once
  unless the script printed RESTART REQUIRED); the queue-position rule and
  the `--source ag2space --channel-id <room>` form of notify.py under Task
  progress notifications; the watcher's second `QUEUE:` line.
- skills/connect-apps: "Where the answer may go" and the rule line are the
  same closed list; the precheck hook prints `reply_to=room|dm` beside
  `room_kind` (dm only for a confirmed DM; unknown is the room).
- skills/marketplace: "Use it now; restart only when told" moves above the
  report step, bolded, and says the script is the activation (no dashboard,
  no restart the output did not name).
- docs/built-in-tools.md: switch-model triggers (`/model <x>` typed in chat,
  "switch to opus", "which model are you on"), the verbatim `switched:`
  report, the current model from state/model-switch.json and the core-model
  card; a Check quota entry (state/quota-state.json fields; last_checked
  older than 6 h is unknown); Generate an image; Activate a cloud tool.
- skills/image-generation: manifest gains enabled, documented_for_core,
  core_description and config.IMAGE_MODEL/VIDEO_MODEL; generate.py drops the
  google-genai SDK for images and calls generateContent over urllib
  (x-goog-api-key, responseModalities IMAGE+TEXT, inlineData decoded), takes
  the key from the new `gemini-image` capability, writes under
  <workspace>/results/media/generated-<ts>.png (the attachment allowlist),
  and prints one JSON line {ok,path,model} or {ok:false,error,message,remedy}
  with exit 0/1/2; Pillow is optional; --video reports sdk_missing honestly.
  SKILL.md: no pip, the `[file: <path>]` delivery step, verbatim owner
  messages per failure. lint-skill learns documented_for_core and
  core_description (fields src/inline-tools.ts already reads); MANIFEST.md
  documents them; runtime-dependencies.md row updated.
- src/credential_resolver.py + src/credential-resolver.ts: `gemini-image`
  walks the slots text then voice per tier; a byok voice preference skips
  only the managed voice slot; a managed preference never blocks the env
  fallback (not a voice surface); quarantine hides every managed entry.
  Both resolver tests carry the same vectors.
- Queue depth: new src/task_queue.py (pending / position / waiting /
  write_snapshot -> state/task-queue.json; registered in docs/src-map.md).
  activity_rows rows take an optional `queue: {depth, position}`; the bus's
  QUEUED transition computes it from the file's place in the pending list and
  the row reads "queued · N ahead" when N > 0; `activity.py queue
  --task-file` prints {depth, position}; task-emit.sh prints
  `QUEUE: <n> pending after this` under a byte-identical `TASK_FILE:` line
  only when n > 0 (a stubbed or missing counter prints nothing);
  core-status.sh adds `pending` to every write and refreshes the snapshot;
  the voice `work` tool returns `queuedAhead` and the "Got it, N ahead of
  this one, working in order" instruction; get_core_status reads the depth
  from the snapshot (stale after 10 minutes = unknown).

Why

Zerlinda's onboarding on 2026-09-17: an apartment search asked in a shared
room was answered in her DM, because the only DM-vs-room rule in the engine
("Private content ... only in the owner's DM") was a category label an agent
could stretch over anything personal-sounding. In the commorai room the
agent refused a collaborator on tier grounds, claimed activation needs the
Station dashboard although the marketplace skill does it, and told the owner
to restart when the script had printed "usable now". In the onSpark and
SpatialReal rooms several requests in a row produced "Status?" pings with no
"N queued" line anywhere, a price was invented instead of naming the
blocker, and an image request produced nothing: the skill's manifest lacked
`enabled` so the loader skipped it, generate.py read a GEMINI_API_KEY that a
managed-key install never has (the managed key is the gemini-voice
capability), errors went to stderr only, and SKILL.md never said to attach
the PNG. Bassil typed `/model` into a room and nothing in the catalog named
that as a trigger.

What it cost before

The queue lived nowhere: core-status.json is single-writer because its `ts`
is the liveness gate for graceful-restart's busy() and the desktop's
core_step_text, so the pending list gets its own state/task-queue.json
instead of riding on it. The first attempt at the shutdown emitter added a
second `>&9`, which tests/task-emit-substitution.test.sh's mutation arm
rejects (it strips exactly one redirect); the emitter now groups the two
printfs under one redirect. The env-loader regression test pins the phrase
"stale shell env" in generate.py's .env loader, so the rewrite keeps it.

Tests

New: tests/claude-md-conduct-rules.test.py (routing, language, blockers,
skills-before-refusing, queue rule pinned in CLAUDE.md and AGENTS.md, the
connect-apps closed list, the precheck reply_to, the marketplace step
order), tests/built-in-tools-model-quota.test.py, tests/task-queue.test.py,
tests/image-generation-cli.test.py (urlopen monkeypatched: no key -> exit 2
no_key; success -> PNG under the temp workspace; no image -> no_image with
the model text; refusal; HTTP error; bad input; video without the SDK).
Extended: connect-apps-precheck, activity-bus, agent-activity,
watch-tasks-dispatch-activity.test.sh (first line unchanged, QUEUE only when
> 0, stubbed counter prints nothing), core-status-atomic-write (the real
wrapper against a pinned temp workspace carries `pending`),
task-bridge-format.test.ts (queuedAhead, countQueuedAhead, readQueueDepth),
both credential-resolver tests.

Not in this PR: the voice prompt's language line (src/voice-context.ts,
src/voice-agent-config.ts) — a prompt change that needs its own live-voice
evidence; the desktop's CLAUDE.md twin and the backend join notice are their
own PRs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(voice): the voice session knows its room — tasks and results follow it, not the DM

What changed
------------
A voice session docked in an AG2 Space room now binds that room for the life
of the session, and every task the voice agent delegates is addressed to it:

- src/web-client.ts reads `room_id` (`^![^\s:]+:\S+$`) and `room_name`
  (capped 120 chars) from the query string the desktop's CallAgentView
  appends, and passes `roomContext` to the transport. The regex is spelled
  through `new RegExp('\\\\…')` because the outer template literal eats
  backslashes (the same trick reSay uses); a node round-trip of the served
  line proves it compiles to /^![^\s:]+:\S+$/.
- src/web-voice-transport.ts: `VoiceTransportOptions.roomContext`, exported
  `SESSION_CONTEXT_TYPE = 'session.context'` and pure
  `buildSessionContextFrame(ctx)` → `{type, version:1, room_id, room_name,
  surface:'room'|'dm'}`. The frame goes out once per connection right after
  the agent's `session.config` (the first frame bodhi emits on attach, so the
  socket is open and the agent listening); a send that fails is retried on the
  next session.config of the same attempt; no room context → nothing sent.
- src/task-bridge.ts owns the binding: `setVoiceSessionRoom` /
  `getVoiceSessionRoom`, `parseSessionContextFrame` (Matrix room id
  validated, name flattened and capped), `applySessionContextFrame` (last
  frame wins for the life of the session; a DM frame — `surface:'dm'`, no
  room fields — releases the room; returns `entered|left|none` so the caller
  speaks a notice once per actual change and never per duplicate frame) and
  `sessionRoomNotice` (the "You are now in room …" / "You are back in your DM"
  lines). `buildVoiceTaskHeader(taskId, ts, ownerId, room)` is the one header
  writer for the work tool and the cancel tool: `source: voice`,
  `interaction_type: realtime_audio`, `media_form: live_stream`,
  `channel_id: <room or local-voice>`, and for a room `channel_kind: room` +
  `source_room_id: <room>`, then `user_id`, `access_tier: owner`,
  `priority: urgent` — every key above `task:`. `_isVoiceTask` now keys on
  `source: voice` or `media_form: live_stream` and keeps the legacy
  `channel_id: local-voice` literal for archived files; `_voiceTaskRoom`
  reads the room back through the same delimiter-honoring header reader.
- Result drain: a room-bound voice result is written as
  `results/proactive-result-<id>-<ts>.txt` whose first line is
  `[channel: <room>]` — the marker the ag2space gateway's `_proactive_route`
  sends to that room — through `forwardVoiceResultToRoom`, which claims the
  filename in `_deliveredResults` at once; `proactive-*` passes
  `_shouldFallthrough`, so without the claim the next drain tick would speak
  the same result a second time. With the client connected the result is
  still spoken (`onResult`) and posted; with the client gone it is posted to
  the room instead of the owner DM.
- src/voice-agent.ts: `onClientCommand` applies `session.context` frames and
  injects the notice through the same delay-then-check path as task results
  (the first frame lands ~100ms before Gemini's setup completes);
  `onClientDisconnected` releases the room; `VoiceConfigContext.getSessionRoom`.
- src/voice-context.ts: `buildVoiceAgentContext({room})` adds the ROOM line;
  `pickLanguage(profile)` reads a `Language: <name>` line from
  user_profile.md and the context carries a LANGUAGE line (the voice half of
  the language rule the conduct PR added to CLAUDE.md).
- src/voice-agent-config.ts: the language-switch rule now also persists the
  preference ("call work exactly once with 'remember my language preference:
  <language>'") — a deliberate prompt change, anchors regenerated; the
  offline-delivery hint says "delivered to their chat — the room they asked
  in, or their DM" instead of naming Discord.
- `channel_kind` joins KNOWN_HEADER_KEYS (src/local_task_protocol.py) and its
  two mirrors (task-bridge.ts `_HEADER_KEYS`, conversation-server.ts
  `_CONF_HEADER_RE`): parse_task_headers drops unknown keys above `task:`, so
  without it a room-bound voice task's verdict would be invisible to every
  Python consumer, and the body guard now defangs a forged `channel_kind: dm`
  in user text that would otherwise claim the DM.

Why
---
Owner feedback from the 2026-09-17 onboarding: "When talk to sutando in
microphone, sends back to main conversation, talking but replying to the dm"
and "I want to talk to agent in my room, add talk button in room". Every
voice task was written with `channel_id: local-voice` regardless of where the
user pressed Talk, and room-less results went to the owner DM through the
proactive queue. desktop/docs/voice-architecture.md already documents this
as the room-context gap; that paragraph is a desktop-repo follow-up.

Cost of the alternatives considered: keying voice on `channel_id` (the old
predicate) cannot survive a room in that field; a second header writer for
the cancel tool is how the two shapes would drift — one writer, pinned once.

Tests
-----
tests/web-voice-transport.test.ts (one session.context after session.config,
never twice per connection, again on a new connection, none without a room,
failed send retried), tests/task-bridge-isvoice.test.ts (room-bound and
legacy-literal verdicts, _voiceTaskRoom), tests/task-bridge-format.test.ts
(both header shapes, priority urgent, all keys above task:),
tests/task-bridge-voice-room-result.test.ts (room result file + delivered
claim, frame parsing and type-literal parity with the transport, repeated
frames: second task carries the second room, DM frame → local-voice,
duplicates inject nothing), tests/task-priority.test.py (room-bound voice
still urgent), tests/voice-context-room.test.ts, tests/voice-context-language.test.ts,
tests/voice-behavior-anchors.test.ts (fixture regenerated with ANCHOR_UPDATE=1;
the regen also absorbs the get_core_status description the conduct PR changed
without regenerating). tests/injection-guard-sweep.test.py's cancel-order
check now recognises the shared header writer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(voice): the room notice is read, not answered; the queue line speaks like a person

Two things the first live test of room-bound voice surfaced (owner
2026-09-18, transcript pasted):

1. Every room switch made the agent say "Working on it." The session.context
   notice went in through injectText, which is realtime text input, and
   Gemini answers realtime input out loud whatever the text says ("Do not
   speak this notice" did nothing). Three room switches in a minute produced
   "Welcome back. Working on it. Working on it. Working on" with nothing
   asked. The notice now goes in as an open clientContent turn
   (turnComplete=false) through the new injectSilentContext: the model reads
   it and stays quiet until the user speaks, which is how the vision hooks
   already add frame context. There is deliberately no fallback to
   injectText when the transport lacks sendContent; the routing is bound in
   the task bridge regardless, and silence beats a phantom line.

2. "Got it, 1 ahead of this one, working in order." came back when the
   owner asked a second thing 12 seconds after the first. Correct, but it
   reads like a queue printout ("why did it say 1 ahead of this one working
   in order the fuck"). One ahead now says "Got it, right after the one I'm
   on."; more than one says "Got it, N in line before this one." The chat
   rule in CLAUDE.md keeps its wording; this is the spoken line only.

Tests: injectSilentContext sends the open turn and refuses the realtime path;
a source pin keeps handleSessionContextFrame on the silent path; the notices
end in "No reply is needed."; the queue wording is pinned for one and for
many.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(voice-room): the room a session names is a claim — the gateway bridge proves membership before anything binds, stamps or posts to it

What changed

1. Trusted-side membership verdicts (new src/voice_room_membership.py, wired
   into src/remote-gateway-bridge.py). The task bridge writes
   state/voice-room-checks/<key>.request.json; a daemon thread in the gateway
   bridge answers <key>.verdict.json using the gateway's own credentials and
   identities: `POST /v1/room {"op": "members"}` as the agent, the agent mxid
   from the channel env / enrolment, the owner from `GET /v1/agents`. A room
   is `verified` only when BOTH the agent and the owner are joined. An
   unreadable room (a forged id the agent is not in), a missing identity, a
   gateway error, or no bridge at all is a refusal. One gateway read per room
   per 60 s (cached on both sides), so navigation stays responsive.

2. task-bridge.ts binds on the verdict, not the frame. `bindSessionContextFrame`
   is the live entry point: a DM frame applies at once; a room frame applies
   only after `requestVoiceRoomVerdict` says verified, otherwise the session
   stays on the DM (`refused` carries the reason; a newer frame supersedes an
   older pending one). `applySessionContextFrame` remains the pure apply for
   an already-verified room and is no longer on the live path. Because
   `_voiceSessionRoom` is set only through the verified path, `channel_id`,
   `channel_kind` and `source_room_id` can only ever name a verified room.

3. The result leg re-checks. `resolveVoiceResultRoom` re-asks the verdict for
   a finished task's `source_room_id` (membership can change while a task
   runs); refused → the owner-DM proactive shape, verified → the room. Both
   the online post and the offline forward (`forwardOfflineVoiceResult`) go
   through it.

4. Second lock in the gateway bridge's claim gate: a `proactive-result-*`
   file addressed to a Matrix room via `[channel: !room]` is claimed only when
   the same verifier says verified; otherwise it is held on disk and logged
   once ("voice-room: holding … not a verified owner+agent room"), never
   posted anywhere.

5. The client is told. voice-agent.ts answers every session.context frame
   with a `session.context.ack` frame (`bound`, `surface: dm|room|refused`,
   `reason`), exported from web-voice-transport.ts with an
   `onSessionContextAck` event, so the pill can show DM when a room was
   refused. Raw frames were already forwarded via `onProtocolMessage`.

6. Room results carry the bridge tag: `forwardVoiceResultToRoom` now writes
   `proactive-result-<task>-<ts>.to-ag2space.txt`. The claim gate reads the
   filename, not the body, and an untagged name fell through to
   last-owner-activity — so an owner last seen on Discord had that bridge
   claim a file carrying a Matrix room id it could not deliver, with no log
   on either side. The `[channel: <room>]` first line is unchanged (it
   addresses the room within ag2space); the dedupe claim keys on the new
   name. The DM-shape offline forward stays untagged on purpose: it should
   follow the owner to whichever bridge they last used.

7. `_isVoiceTask` no longer treats `media_form: live_stream` as voice. The
   phone skill stamps the same key on `source: phone` tasks, so the earlier
   widening would have routed unsent phone results through the voice
   offline-forward path. `source: voice` (plus the legacy
   `channel_id: local-voice` literal for old archives) is the verdict.

8. Package sync: packages/ag2-sparrow/.../local_task_protocol.py picks up the
   `channel_kind` header key (sync_from_src.py), which un-reds the sparrow
   drift test and the typecheck job's DRIFT check.

Why

qingyun-wu, reviewing the paired webapp PR ag2-space/cinny-webclient#1073,
blocked the protocol pair on this engine PR: "`voiceRoomBinding.ts`
serializes a client-selected `room_id` ... That client-side check is not an
authorization boundary: a modified client can send `session.context` for any
syntactically valid Matrix room ID. The paired engine PR #4393 currently
accepts that ID, stamps an owner-tier task with it, and emits
`[channel: <room>]` without proving the owner and receiving agent are members
of that room. This can redirect owner-authorized task context/results to an
unrelated room reachable by the gateway. Before this protocol pair is
merge-ready, enforce room membership on the trusted gateway/broker side for
the authenticated session identity and agent, fail closed on unknown
membership, and add a negative integration test showing that a forged but
valid room ID cannot become `channel_id`/`source_room_id` or a proactive
result destination. The client-side membership filter should remain as UX,
not the enforcement point."

The engine's TypeScript side holds no gateway credentials and no identity;
the Python gateway bridge holds both and already speaks the `/v1/room`
envelope, so the verdict is computed there and the TypeScript side trusts
nothing but that verdict. The file protocol (request/verdict under
state/voice-room-checks/) is the same workspace-state channel the two sides
already share for owner activity and heartbeats.

john-the-dev (REQUEST CHANGES at 08be6345) measured the untagged-name defect
against src/proactive_routing.py on main: "`proactive_destination` is `None`,
so `should_claim_proactive_file` falls through to `should_claim_proactive()`
— last-owner-activity. ... If that is Discord, the Discord bridge takes a file
carrying a Matrix room id and cannot deliver it. Neither side logs anything."
and asked whether anything other than voice writes `media_form: live_stream`
(the phone skill does — item 7).

Tests

- tests/task-bridge-voice-room-result.test.ts (+11): a forged
  `!forged:ag2.space` with a refusing verifier never binds, the next task
  carries `channel_id: local-voice` and no room keys; a forged frame after a
  verified one releases the verified room; a throwing verifier is a refusal;
  the newest frame wins over a pending verdict; `resolveVoiceResultRoom` /
  `forwardOfflineVoiceResult` give a forged `source_room_id` the DM shape
  (no `[channel:]` line) and a verified one its room; the request/verdict
  file protocol (timeout → unverified with the request left for the bridge,
  late answer returned, stale and foreign verdicts ignored, fresh refusal
  reused without a new request); the tagged filename pinned against
  proactive_routing (`proactive_destination == 'ag2space'`, Discord's
  fallback never claims it); the ack frame shapes; the live handler uses
  `bindSessionContextFrame` and acks a refusal.
- tests/voice-room-membership.test.py (new, 16): the pure verdict for every
  refusal reason, `/v1/room` answer parsing, cache per TTL, forged room
  refused and cached, gateway/identity failures refuse without raising,
  atomic verdict writes, malformed requests dropped, the daemon loop.
- tests/gateway-voice-room-membership.test.py (new): the production entry
  point against a mock gateway with owner last active on Discord: a result
  addressed to a room the agent cannot read and one the owner is not in are
  never posted and stay on disk, the verified room's result is delivered, the
  binding requests get the matching verdict files, one gateway read per room.
- tests/task-bridge-isvoice.test.ts: a `source: phone` task with
  `media_form: live_stream` is not voice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(voice-room): keep the _headerIsVoice docblock's first line out of the interaction-type guard's write-site shape

tests/interaction-type-header.test.py scans src/task-bridge.ts for
`source: voice` followed by a backtick or newline and skips only lines that
begin with a comment marker after lstrip; a docblock's FIRST line begins with
`/**`, so the reworded one-line summary in 2ed89c89 read as a header write
site with no interaction_type: within six lines (CI: python standalone tests
and the coverage gate on #4393/#4396). Moving the phrase to the second line
restores the shape the guard already skipped; no code change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(conduct): regenerate the voice behavior anchors and cover the branch's Python to the 95% diff bar

What changed

- tests/fixtures/voice-behavior-anchors.json regenerated with ANCHOR_UPDATE=1
  (70 tools). The branch changed the voice prompt in voice-agent-config.ts
  (reply where asked, plain blockers, skills before refusing, queue depth,
  image generation), so the committed tool-table and instructions anchors
  no longer matched: `not ok 360 - tool-table anchor matches the committed
  fixture` and `not ok 361 - instructions anchor: fixed-env output hash
  matches` in "typecheck + node & shell tests", which is also what failed
  the "tsc + tests (clean install)" aggregate. The prompt change is
  deliberate and the anchor diff is the prompt diff.

- Diff coverage was 73% (91 of 347 changed Python lines uncovered; the
  coverage-summary artifact of run 35331736558 lists them). New tests:
  * tests/image-generation-cli.test.py: load_env (both .env files, quotes,
    comments, non-GEMINI keys ignored, missing files), resolve_key (the
    resolver when the core tree is reachable, then GEMINI_API_KEY over
    GEMINI_VOICE_API_KEY, then none), manifest_config and media_dir without
    the core tree, a non-image input, the Pillow downscale (a fake PIL in
    sys.modules — CI has no Pillow), api_error_message without a JSON body,
    the jpg/webp conversion with and without Pillow and an unknown output
    extension, and the whole Veo path against a fake google-genai (poll
    loop, download, save, reference image, missing reference, SDK failure).
  * tests/task-queue.test.py: a task that vanishes mid-scan still counts
    (since=0, source None), _source of an unreadable file, _workspace_of
    with nothing, and an unexpected error printed to stderr with exit 0.
  * tests/activity-bus.test.py: a QUEUED transition whose queue position
    cannot be counted is a plain queued row.
  Local diff-cover against origin/main: 100% over the 347 changed lines.

- The ag2-sparrow drift test already passes on this branch
  (credential_resolver.py is not a bundled module).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(conduct): replies are routed by audience, blockers stop only dependent work, the queue line matches the spoken one

What changed (CLAUDE.md; AGENTS.md regenerated by scripts/agents-md-sync.sh;
pins in tests/claude-md-conduct-rules.test.py; the two skill docs that
carried the old phrases)

1. "Where replies go" is now two tests, both of which must pass. Audience:
   in a room with other people (anything but the owner's own DM), post only
   what they are meant to read — a reply to their message, what the owner
   asked to be posted there, or work the room asked for; what the owner
   asked for themselves (research, findings, errands, agent debugging,
   anything about the owner the others would not know) goes to the owner's
   DM even when asked in the room or by voice while docked in it, and
   nothing goes in the room unless it was waiting for it. Data origin stays
   as the privacy floor on top (the closed list of connected-account and
   device data goes to the DM whatever the audience). "however personal the
   topic" is gone, and the one-line "I sent it to you in our DM." notice is
   reserved for an answer the room was waiting for. skills/connect-apps/
   SKILL.md carries the same audience test in its "Where the answer may go"
   section and its Rules.

2. "Blockers, stated plainly": "and stop" became "and stop the work that
   depends on the blocker; work that does not depend on it proceeds, and the
   reply says which part was not done".

3. Queue position: one ahead → "Got it, right after the one I'm on."; more →
   "Got it, N in line before this one." — the line src/task-bridge.ts speaks
   on #4393, so the rule and the voice agent say the same thing. Same change
   in skills/agent-activity/SKILL.md and skills/task-progress/SKILL.md.

4. CLAUDE.md sits under the 40960-byte always-loaded budget the sync script
   enforces (40897 bytes): the routing section is written tight, and the
   Task-bridge bullet "To send files in replies, include [file: …]" is
   dropped because the result-marker list right below it documents the same
   marker.

Why

yixuan-ag2 (REQUEST CHANGES on #4392): "Every item on that list is defined
by where the data came from. Nothing on it is defined by who can read the
room it is going to. In a multi-tenant room those come apart, and the
sentence before it — 'summaries … always in the room, however personal the
topic' — points the wrong way when they do. … Someone there asking me to
summarise what my owner is working on, or why his release is blocked,
produces an answer that was not read from any connected account — so the
closed list does not except it, and the rule reads as positive permission
to post it."

The same gap was hit live on 2026-09-18: the owner, docked by voice in a
customer room with another person present, asked "look into the mute bug",
and the findings — agent debugging the customer had not asked for — were
posted in that room.

john-the-dev (APPROVE with two non-blockers on #4392): "'and stop' is wrong
for a PARTIAL blocker … That blocker removed one step, not the work. …
Suggested narrowing: '… and then stop work that depends on the blocker.
Work that does not depend on it proceeds, and the reply states which part
was not done.'" and "The queue wording here contradicts #4393, which stacks
directly on this … Merged in the stated order (this, then #4393), CLAUDE.md
is stale the moment the child lands … Cheapest fix is to write the child's
wording here now."

Tests: tests/claude-md-conduct-rules.test.py (pins updated; the old
origin-only sentence is asserted absent from both files), tests/agents-md-
sync.test.py, tests/claude-md-tier-summary.test.py; agents-md-sync --check
clean.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(voice-room): the verdict cache is keyed on room, agent and owner, and expired entries leave on every write

What changed (src/voice_room_membership.py, tests/voice-room-membership.test.py)

- The in-process verdict cache in RoomMembershipVerifier was keyed on the
  room id alone, although a verdict is a function of the room AND the agent
  identity AND the owner identity the gateway registry binds to it. The key
  is now the (room, agent, owner) triple, so a re-enrolled agent or a
  re-bound owner is never answered from the previous identity's read; the
  identities are read before the cache lookup (the loader's owner reader is
  itself cached for one TTL, so this adds no gateway traffic).
- Entries only ever expired logically; nothing removed them. Every write
  now evicts entries past the TTL, then the oldest beyond a cap
  (CACHE_MAX_ENTRIES = 256, `cache_max` on the constructor), so a bridge that
  runs for weeks and sees many rooms holds a bounded map.

Why

yixuan-ag2's two non-blocking notes on #4393: the TTL cache "is keyed on
room_id alone though the verdict depends on room + agent identity + owner
identity", and "`_cache` is never evicted, entries only expire logically",
so a long-lived process could grow.

john-the-dev's open question on #4393 — does anything other than voice
write `media_form: live_stream` in a task header? — grepped over src/,
skills/ and packages/: two writers, src/task-bridge.ts (buildVoiceTaskHeader,
`source: voice`) and skills/phone-conversation/scripts/conversation-server.ts
(delegateTask, `source: phone`). `_isVoiceTask` was therefore tightened in
2ed89c89 to key on `source: voice` (plus the legacy `channel_id: local-voice`
literal) and never on the media form; tests/task-bridge-isvoice.test.ts pins
the phone negative case.

Tests: cache keyed on all three identities (an owner change re-reads and
refuses, a re-enrolled agent re-reads), expired entries evicted on the next
write, the map capped at `cache_max` with the oldest leaving first; the
loader and production-process suites unchanged and green. Diff coverage vs
the base branch: 100% (src/voice_room_membership.py, src/remote-gateway-bridge.py).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(voice-room): a room-bound voice result the core marks [dm-only] is kept to the owner's DM, not posted in the room

What happened (owner, 2026-09-18)

Docked by voice in a customer room with Matthew Wilmer, the owner asked
"look into the mute bug". The analysis — agent debugging, for the owner
alone — was posted in that room. Owner: "why the fuck the agent mentioned
this bs in the room with other people... it should only send messages that
are RELEVANT to that room otherwise should go to the DM".

The conduct rule on this branch (#4392, "Where replies go", audience test)
already said so: what the owner asked for themselves goes to the owner's DM
even when asked by voice while docked. The mechanics did not follow it. The
drain in src/task-bridge.ts forwarded EVERY room-bound voice result to the
room — resolveVoiceResultRoom → forwardVoiceResultToRoom — whatever the body
said, and it stripped the `[dm-only]` marker from the body before any
routing decision could see it. The offline leg (forwardOfflineVoiceResult)
had the same hole: a dm-only body reached the room writer already stripped,
so the gateway's own dm-only guard never saw the marker either.

What changed

- src/task-bridge.ts, the drain: the marker is detected on the RAW body
  (DM_ONLY_RE, the way parse_markers detects it — anywhere, any case) before
  the standalone-line strip. A room-bound voice result that carries it is
  handed to keepVoiceResultToDm, which writes the owner-DM shape —
  forwardVoiceResultToOwnerDm: `proactive-result-<id>-<ts>.to-ag2space.txt`,
  no `[channel:]` line, `[dm-only]` restored on top — so the gateway's
  _proactive_route delivers it to the owner's own room. The file is claimed
  in _deliveredResults at once, exactly like the room shape, so the next
  tick never speaks it a second time. One log line:
  `[TaskBridge] <task> result kept to the DM ([dm-only]) via <file>`.
  A plain result still goes to the verified room as before. The offline leg
  takes the same `dmOnly` flag and short-circuits to the DM shape before
  asking for a room verdict.
- The voice model is told where the copy went: the drain's listener gained
  an optional delivery note (ResultListener), and live-agent-runtime injects
  it as a second `[System: …]` line UNDER the TASK_RESULT markers — outside
  them, because it is delivery state the model acts on, not result text it
  must only summarise. The note (DM_ONLY_DELIVERY_NOTE) says the copy is in
  the owner's DM, not the docked room. frameTaskResult itself is untouched
  (anchor-pinned, shared with the MatrixRTC daemon).
- The task body: a room-bound voice task now carries one guidance line
  right under `task:` — voiceRoomTaskGuidance(room):
  `room_context: <room> — post there only what its members are meant to
  read; for anything the owner asked for themselves start the result with
  [dm-only]`. A body line, not a header key: no KNOWN_HEADER_KEYS entry, no
  sparrow sync, and it survives the injection-guard sweep unchanged (the
  confineUserContent(task) site is the same).
- sessionRoomNotice (entered) now reads: "You are docked in room X; what
  you delegate is answered there only when it is for the room's members,
  otherwise in the owner's DM; say where it went. No reply is needed." The
  voice-context ROOM line says the same; both used to instruct the model to
  NEVER say "in your DM", which would have had it lie about a dm-only result.
- CLAUDE.md, "Where replies go", audience bullet: "By voice while docked, a
  task whose answer is for the owner starts its result with `[dm-only]`: the
  task bridge keeps it to the DM, not the room." Two neighbouring sentences
  were tightened to stay under the 40960-byte budget (40957). AGENTS.md
  regenerated.

Tests

tests/task-bridge-voice-room-result.test.ts: DM_ONLY_RE parity with
parse_markers; forwardVoiceResultToOwnerDm's shape (checked through
proactive_destination + parse_markers: claimed by ag2space, dm-only action,
no redirect, marker stripped); keepVoiceResultToDm for room+dm-only,
room+plain and a DM-session task; the offline leg with dmOnly; the guidance
line pinned as the body line right under `task:` and absent from a DM task;
the notice regexes updated; and the drain itself — startResultWatcher with a
verified room verifier and two results in one tick: the plain one lands in
the room with `[channel: <room>]`, the `[dm-only]` one lands in the DM shape
with no `[channel:]` line, voice speaks each once without the marker, and
only the dm-only one carries the delivery note. tests/voice-context-room:
ROOM line pins. Green: 186/186 across tests/task-bridge-*, voice-context*,
voice-behavior-anchors, inject-*, session-recorder; claude-md-conduct-rules,
claude-md-tier-summary, injection-guard-sweep (48/48),
task-body-injection-guard; `npx tsc --noEmit` clean.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(task-queue): an unreadable tasks/ is an unknown queue, never an empty one

Review finding (qingyun-wu, Sutando core seat), verbatim:

  "`src/task_queue.py`: an unreadable `tasks/` dir publishes 'the queue is
  empty'. ... `OSError` is the parent of `PermissionError`, so an unreadable
  directory returns the same empty list as an absent one, and every consumer
  downstream inherits it: pending() → [], position() → {depth 0, position 0},
  waiting() → 0, write_snapshot() writes {depth 0, pending []}. ...
  `state/task-queue.json` is read by voice's `get_core_status` and the
  desktop, so a transient permission failure does not degrade to 'unknown',
  it publishes a confident zero. Suggested shape: `except FileNotFoundError:
  return []` and `except NotADirectoryError: return []`; anything else, EPERM
  above all, must NOT read as empty: let it raise, and have `write_snapshot()`
  refuse to publish a count it could not take. ... a published count is worse
  than no count."

Implemented exactly that shape.

src/task_queue.py
- pending() catches only FileNotFoundError and NotADirectoryError (an absent
  tasks/, or a file where the dir should be) as the empty queue. Every other
  OSError — PermissionError first among them — propagates. position() and
  waiting() propagate with it: there is no depth to report, so no zero is
  reported.
- write_snapshot() takes the count under a try; on OSError it prints one line
  to stderr, returns None, and touches nothing on disk. The previous
  state/task-queue.json stays byte-for-byte as it was; when none existed,
  none is created (not even the state dir). Readers already date the snapshot
  by `ts` (inline-tools readQueueDepth treats >10 min as unknown), so a
  refused refresh ages into "unknown" instead of becoming a stale-but-fresh
  zero.
- main() keeps exiting 0 on every path — the watcher's dispatch must never
  fail because the queue could not be counted — but now prints no count at all
  for `pending`/`position`/`waiting` and no path for `snapshot` when the read
  was refused; the reason goes to stderr. Before, `snapshot` would have printed
  the string "None".

Every caller checked, so a propagated error degrades to "unknown" / no line
rather than a crash of the dispatch path:
- src/task-emit.sh queue_line: an empty stdout from `waiting` already means
  "no QUEUE: line" (the `case ''` guard); nothing printed, never `QUEUE: 0`.
- src/activity_bus.py transition_from_file: already wraps queue_position in a
  broad except → queue=None → a plain `queued` row. Unchanged; now pinned by a
  real chmod test rather than only a mocked one.
- scripts/core-status.sh: already catches, writes the status record without a
  `pending` key, reports on stderr. Unchanged; pinned by the rig test.
- skills/agent-activity/scripts/activity.py `queue`: used to traceback. Now
  catches OSError, prints nothing on stdout, the reason on stderr, exits 1 —
  the position is unknown, not {depth 0}. SKILL.md says so.
- src/task-bridge.ts countQueuedAhead / inline-tools.ts readQueueDepth: not
  touched. readQueueDepth already returns null for an absent or stale
  snapshot; countQueuedAhead's 0-on-unreadable feeds only
  queuedAheadInstruction, where 0 and unknown both produce no sentence and
  nothing is published to disk.

Tests (chmod cases skip under root, where a directory is always readable;
permissions are restored in `finally`):
- tests/task-queue.test.py: a missing dir → []; a file where the dir should
  be → [] (pending, position, waiting); chmod 000 → pending/position/waiting
  raise PermissionError; write_snapshot refuses, says so in one stderr line,
  and the prior snapshot survives byte-for-byte with no temp file left; no
  snapshot (and no state dir) is created when none existed; the CLI on an
  x-only tasks/ (file opens by path, listing denied) prints nothing, exits 0,
  names the permission error on stderr, and leaves the snapshot unchanged for
  all four commands.
- tests/agent-activity.test.py: `activity.py queue` on an x-only tasks/ →
  rc 1, empty stdout, reason on stderr.
- tests/activity-bus.test.py: a real x-only tasks/ → LifecycleTransition with
  queue=None → queued_line "queued".
- tests/core-status-atomic-write.test.py step 5: chmod 000 tasks/ → the
  liveness write still lands (rc 0, status/step present), no `pending` key,
  refusal on stderr, state/task-queue.json byte-for-byte the previous one.
- tests/watch-tasks-dispatch-activity.test.sh: x-only tasks/ → exactly the
  TASK_FILE line, no QUEUE line, snapshot untouched.

Diff coverage vs origin/main stays at 100% (346 lines, 0 missing).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(task-queue): skip the chmod cases where chmod cannot deny a read (Windows, root)

2a84be85 guarded the unreadable-tasks/ cases with `os.geteuid() == 0` alone.
The Windows runtime job runs tests/agent-activity.test.py and part of
tests/activity-bus.test.py under CPython on Windows, where `os` has no
`geteuid` — the guard itself raised AttributeError and the job went red
("PowerShell + Windows runtime", run 35373786732). Even with a guard that
imports, the cases could not pass there: Windows chmod clears the read-only
bit and never denies listing a directory, so PermissionError would not be
raised and the assertions would fail on their own terms.

The condition is now `os.name != "posix" or os.geteuid() == 0` in all four
Python files that carry a chmod case (task-queue, agent-activity,
activity-bus, core-status-atomic-write): the short-circuit means geteuid is
never looked up off POSIX, and the skip reason names both situations. On
macOS and Linux CI the cases still run and still assert the real
PermissionError path; the shell test is bash-only and unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(image-generation): an unidentifiable image is bad input, not a crash; real PNG fixture (#4407)

Follow-up to the one blocking finding on the review of #4392 (feat/agent-conduct-queue-image),
quoted here because it also explains why CI was green while the path was broken:

  "Pillow 12.3.0 installed -> Ran 23 tests ... FAILED (errors=2); Pillow hidden from import ->
  Ran 23 tests ... OK. The errors are Success.test_explicit_output_and_an_input_image_are_honoured
  and Video.test_a_reference_image_rides_along_and_a_missing_one_is_bad_input, both raising
  PIL.UnidentifiedImageError from generate.py:136. The fixture at test line 30 is an eight-byte
  PNG signature with no image data; several tests inject a stub imaging module, but those two
  reach the real one. So green CI means that runner has no Pillow, not that the path works ...
  main catches only ImportError at both sites too. ... Two small fixes close it: catch OSError
  alongside ImportError at both sites, since an unidentifiable image is ordinary input, and make
  the fixture a real image so the tests exercise the path instead of stepping around it."

What changed in skills/image-generation/scripts/generate.py, at both Pillow sites:

- read_input_image: the Pillow import is now its own try (ImportError -> the bytes pass through
  untouched, as before). Decoding is a second try that catches OSError, which is what Pillow
  raises (UnidentifiedImageError is an OSError) for bytes it cannot identify or a truncated
  file it cannot load. That returns None, so both callers emit their existing bad_input JSON
  line (exit 2, with the "check the input image path and format" remedy) and nothing is sent
  to Gemini. Before, any owner who pointed --input at a corrupt file on a machine with Pillow
  got a traceback on stderr and no JSON line on stdout, which breaks the one-line contract the
  conduct rules and SKILL.md rely on.
- save_image: an OSError from Image.open/convert/save on the bytes Gemini returned now takes the
  same fallback as a missing Pillow: keep the returned format under its own extension (never a
  mislabelled .jpg) and say so on stderr.

What changed in tests/image-generation-cli.test.py:

- The PNG fixture is a real 1x1 RGB PNG built with struct + zlib (IHDR, IDAT, IEND with CRCs), so
  a Pillow that is present decodes it and the two tests the reviewer named exercise the path
  instead of stepping around it. The old eight-byte signature survives as NOT_A_PNG, the
  fixture for the failure cases.
- Two new tests: a corrupt --input file is bad_input with no request made, and a returned blob
  Pillow cannot convert is saved under its own extension. Both run against the real Pillow when
  it is importable and against a stub whose open() raises OSError only when it is absent, so the
  cases assert the same thing on a runner without Pillow and on a developer machine with it.
- The downscale test now also asserts the no-Pillow pass-through, so the ImportError branch is
  covered when Pillow is present and the OSError branch when it is not: diff coverage on
  generate.py is 100% in both environments, not only on the Pillow-less runner.

Verified: tests/image-generation-cli.test.py -> Ran 25 tests OK with Pillow 12.3.0 importable,
OK with PIL hidden from import (sys.modules["PIL"] = None), and OK on a python without Pillow
installed. diff-cover against origin/main reports generate.py at 100% in the Pillow-present and
Pillow-hidden runs. review-checks (prose-cap) and lint-workspace-resolution pass on the diff.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* feat(voice): navigate the desktop by voice — "let's talk in my DM", "take me to GTM in Investors", "go home" (#4396)

Why
---
The owner (2026-09-18), on top of the room-bound voice session (#4393):
"i should then be able to tell it lets talk in my DM and it switches to the
DM, or take me to X room in X space and takes me there... so that its truly
interactive". Today the voice agent can hear where it is (the desktop's
`session.context` frame) but cannot move; those phrases fell through to `work`
or `press_key` and nothing happened on screen. The voice session should be able
to drive the desktop it lives in.

What changed
------------
A new agent → client control frame and the inline tool that sends it. The
desktop resolves names — it owns the room list and the user's spaces — so the
agent only carries the spoken words and waits for the answer.

- src/web-voice-transport.ts: the wire contract, shared with the webapp half
  that is built in parallel against exactly this shape.
    server → client  {type:'ui.navigate',  version:1, request_id, target:'dm'|'room'|'home', query?}
    client → server  {type:'ui.navigated', version:1, request_id, ok, room_id?, room_name?,
                      error?:'not_found'|'ambiguous'|'unsupported', candidates?:string[]}
  Exported pure helpers: `buildUiNavigateFrame` / `parseUiNavigateFrame`
  (request; `query` only travels for `target:'room'`, flattened and capped at
  200 chars) and `buildUiNavigatedFrame` / `parseUiNavigatedFrame` (reply;
  `ok` is derived as "no error", names capped at 120 chars, at most 8
  candidates, an unknown `error` reads as `unsupported`, a frame claiming `ok`
  while carrying an error is not ok). New `VoiceTransportEvents.onServerFrame`
  hands the client every JSON frame the agent sends before the transport
  interprets it — the hook the desktop uses to see `ui.navigate` and answer
  with `sendClientCommand(buildUiNavigatedFrame(…))`. Frame types the
  transport does not know stay ignored by the transport itself, exactly as
  before; they only reach this callback and `onProtocolMessage`.

- src/voice-navigate.ts (new): the `navigate_ui {target, query?}` inline
  tool. It sends one `ui.navigate` through the session's client frame path
  (`session.sendJsonToClient`, the same outbound half `task.status` and
  `agent.state` ride), keeps the request in a pending map keyed by
  `request_id`, and waits at most 6 s (`NAVIGATE_UI_TIMEOUT_MS`) for the
  matching `ui.navigated`. Outcomes are always a result the model can speak,
  never a throw: `{ok:true, room_name?}`; `ambiguous` with the candidates and a
  "ask which one" message; `not_found` naming the spoken query; `timeout`; and
  `unsupported` — at once, without sending — when no desktop client is
  attached ("navigation works in the desktop app"), and when a send throws.
  The bodhi tool timeout is set above the wait so bodhi cannot kill the tool
  before its own timeout answers. Requests are settled immediately (as
  `unsupported`) when the client disconnects rather than sitting out their
  timeout.

- src/voice-agent.ts: installs the client seam once the session exists
  (`attached: () => session.clientConnected`, `send: session.sendJsonToClient`),
  routes every client frame through `resolveUiNavigated` in `onClientCommand`
  next to the `session.context` handler, and fails pending navigations in
  `onClientDisconnected`. After a successful move the desktop sends a fresh
  `session.context`, so the existing room binding (#4393) makes replies and
  delegated work follow the new room with no extra plumbing here.

- src/inline-tools.ts: `navigateUiTool` joins the core voice tool table
  (`inlineTools` and `ownerOnlyTools`, like `cancel_task`).

- src/voice-agent-config.ts: one NAVIGATION line in the behavior rules — the
  trigger phrases, `navigate_ui` over `work`/`press_key`, one short line on
  success ("Taking you to GTM."), read the candidates on `ambiguous` and call
  again with the pick, "could not find a room called that" on `not_found`,
  "navigation works in the desktop app" on `unsupported`/`timeout`.

- tests/fixtures/voice-behavior-anchors.json: regenerated with
  ANCHOR_UPDATE=1 because the prompt changed — the new tool entry (twice, as
  every owner-only inline tool appears) and the instructions hash.

- docs/src-map.md: regenerated (scripts/gen-src-map.py) for the new module.

Tests
-----
- tests/voice-navigate.test.ts (new): builder ↔ parser round trips for both
  frames, the bounds (query cap, name cap, candidate cap, dropped query for
  dm/home), parser rejections (other type/version, bad target, missing id),
  and every tool outcome: resolves on the client's ok reply, ambiguous with
  candidates, not_found, times out cleanly and forgets the request (a late
  reply matches nothing), unsupported without a client and with a detached
  one (nothing sent), a throwing send, client-left fails every in-flight
  request, stray frames and unknown ids are ignored.
- tests/web-voice-transport.test.ts: `onServerFrame` receives every JSON frame
  including `ui.navigate`, arriving whole for the client parser; unknown types
  still do not throw and reach no typed callback.
- Full `npm run test:ts`, `npx tsc --noEmit`, eslint on the changed TS
  (0 errors), scripts/review-checks.sh --diff and
  scripts/lint-workspace-resolution.sh --diff all clean.

Dependency
----------
The desktop/webapp half (subscribe with `onServerFrame`, resolve the query
against the room list and spaces, navigate, reply `ui.navigated`) lands in
cinny-webclient against this contract. Until it does, a desktop that ignores
the frame produces a 6 s wait and the "did not go through" answer; a build
without a client attached answers `unsupported` immediately.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* docs(diet): keep CLAUDE.md under the 40k budget once main's growth lands on top of this branch

Why the PR went red after #4407 was merged into it: CI re-ran on the merge ref
(this head merged with main), and `scripts/agents-md-sync.sh --check` measures
CLAUDE.md there. It refused: 41033 bytes against the 40960-byte always-loaded
budget. This branch on its own is 40897 bytes, but main's #4405 grew the file by
136 bytes since the merge-base (5944f7bb), so the merged file crossed the line
by 73 bytes. Locally the check passed, which is why the failure only showed in
CI. The same refusal failed tests/agents-md-sync.test.py, and that single red
file took "python standalone tests", the "tsc + tests (clean install)" roll-up,
and the coverage gate (it stops when the suite is red, before measuring
anything) down with it. Four red checks, one cause.

What changed: the "Existing consumers …" explanation of the per-channel pull
namespace — why a `<key>.task-{id}.txt` name slides past every bridge and how
the phone drain claims it — is relocated verbatim into
docs/claude-md-moved-detail.md the way #3016 did, leaving the actionable
"Helper:" sentence plus a one-line pointer in CLAUDE.md. That is 314 bytes
out: CLAUDE.md is 40583 bytes on the branch and stays under budget on the
merge with origin/main. AGENTS.md is regenerated with
`bash scripts/agents-md-sync.sh`.

What did not change: none of the conduct sentences this PR adds.
tests/claude-md-conduct-rules.test.py pins every one of them (routing,
language, blockers, skills-before-refusing, queue), so the trim had to come
from pre-existing text that no test pins.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(voice): navigate_ui is gated on a client capability flag (#4408)

* feat(voice): navigate the desktop by voice — "let's talk in my DM", "take me to GTM in Investors", "go home"

Why
---
The owner (2026-09-18), on top of the room-bound voice session (#4393):
"i should then be able to tell it lets talk in my DM and it switches to the
DM, or take me to X room in X space and takes me there... so that its truly
interactive". Today the voice agent can hear where it is (the desktop's
`session.context` frame) but cannot move; those phrases fell through to `work`
or `press_key` and nothing happened on screen. The voice session should be able
to drive the desktop it lives in.

What changed
------------
A new agent → client control frame and the inline tool that sends it. The
desktop resolves names — it owns the room list and the user's spaces — so the
agent only carries the spoken words and waits for the answer.

- src/web-voice-transport.ts: the wire contract, shared with the webapp half
  that is built in parallel against exactly this shape.
    server → client  {type:'ui.navigate',  version:1, request_id, target:'dm'|'room'|'home', query?}
    client → server  {type:'ui.navigated', version:1, request_id, ok, room_id?, room_name?,
                      error?:'not_found'|'ambiguous'|'unsupported', candidates?:string[]}
  Exported pure helpers: `buildUiNavigateFrame` / `parseUiNavigateFrame`
  (request; `query` only travels for `target:'room'`, flattened and capped at
  200 chars) and `buildUiNavigatedFrame` / `parseUiNavigatedFrame` (reply;
  `ok` is derived as "no error", names capped at 120 chars, at most 8
  candidates, an unknown `error` reads as `unsupported`, a frame claiming `ok`
  while carrying an error is not ok). New `VoiceTransportEvents.onServerFrame`
  hands the client every JSON frame the agent sends before the transport
  interprets it — the hook the desktop uses to see `ui.navigate` and answer
  with `sendClientCommand(buildUiNavigatedFrame(…))`. Frame types the
  transport does not know stay ignored by the transport itself, exactly as
  before; they only reach this callback and `onProtocolMessage`.

- src/voice-navigate.ts (new): the `navigate_ui {target, query?}` inline
  tool. It sends one `ui.navigate` through the session's client frame path
  (`session.sendJsonToClient`, the same outbound half `task.status` and
  `agent.state` ride), keeps the request in a pending map keyed by
  `request_id`, and waits at most 6 s (`NAVIGATE_UI_TIMEOUT_MS`) for the
  matching `ui.navigated`. Outcomes are always a result the model can speak,
  never a throw: `{ok:true, room_name?}`; `ambiguous` with the candidates and a
  "ask which one" message; `not_found` naming the spoken query; `timeout`; and
  `unsupported` — at once, without sending — when no desktop client is
  attached ("navigation works in the desktop app"), and when a send throws.
  The bodhi tool timeout is set above the wait so bodhi cannot kill the tool
  before its own timeout answers. Requests are settled immediately (as
  `unsupported`) when the client disconnects rather than sitting out their
  timeout.

- src/voice-agent.ts: installs the client seam once the session exists
  (`attached: () => session.clientConnected`, `send: session.sendJsonToClient`),
  routes every client frame through `resolveUiNavigated` in `onClientCommand`
  next to the `session.context` handler, and fails pending navigations in
  `onClientDisconnected`. After a successful move the desktop sends a fresh
  `session.context`, so the existing room binding (#4393) makes replies and
  delegated work follow the new room with no extra plumbing here.

- src/inline-tools.ts: `navigateUiTool` joins the core voice tool table
  (`inlineTools` and `ownerOnlyTools`, like `cancel_task`).

- src/voice-agent-config.ts: one NAVIGATION line in the behavior rules — the
  trigger phrases, `navigate_ui` over `work`/`press_key`, one short line on
  success ("Taking you to GTM."), read the candidates on `ambiguous` and call
  again with the pick, "could not find a room called that" on `not_found`,
  "navigation works in the desktop app" on `unsupported`/`timeout`.

- tests/fixtures/voice-behavior-anchors.json: regenerated with
  ANCHOR_UPDATE=1 because the prompt changed — the new tool entry (twice, as
  every owner-only inline tool appears) and the instructions hash.

- docs/src-map.md: regenerated (scripts/gen-src-map.py) for the new module.

Tests
-----
- tests/voice-navigate.test.ts (new): builder ↔ parser round trips for both
  frames, the bounds (query cap, name cap, candidate cap, dropped query for
  dm/home), parser rejections (other type/version, bad target, missing id),
  and every tool outcome: resolves on the client's ok reply, ambiguous with
  candidates, not_found, times out cleanly and forgets the request (a late
  reply matches nothing), unsupported without a client and with a detached
  one (nothing sent), a throwing send, client-left fails every in-flight
  request, stray frames and unknown ids are ignored.
- tests/web-voice-transport.test.ts: `onServerFrame` receives every JSON frame
  including `ui.navigate`, arriving whole for the client parser; unknown types
  still do not throw and reach no typed callback.
- Full `npm run test:ts`, `npx tsc --noEmit`, eslint on the changed TS
  (0 errors), scripts/review-checks.sh --diff and
  scripts/lint-workspace-resolution.sh --diff all clean.

Dependency
----------
The desktop/webapp half (subscribe with `onServerFrame`, resolve the query
against the room list and spaces, navigate, reply `ui.navigated`) lands in
cinny-webclient against this contract. Until it does, a desktop that ignores
the frame produces a 6 s wait and the "did not go through" answer; a build
without a client attached answers `unsupported` immediately.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(voice): navigate_ui is gated on a client capability flag

Follow-up to the #4396 review. The reviewer's blocking note, verbatim:

  "The one change that would move me to approve is still not in:
  attached() is still keyed on the client merely being connected, at
  voice-agent.ts:1295. Every current desktop satisfies that and none
  speaks the frame, so the trigger phrases cost the owner six seconds
  of silence and then a failure message until the client half ships.
  Gating it on a capability flag would make merging ahead of the client
  safe. The rest of my notes are unchanged: the wiring is unpinned by
  any test, target: 'room' with no query yields a nonsense message, and
  the surface hook runs before typed dispatch without a guard."

All four are addressed here.

1. Capability flag. The `session.context` frame (v1) gains an additive,
   optional `capabilities?: string[]`; a client that answers `ui.navigate`
   frames announces `['ui.navigate']` (UI_NAVIGATE_CAPABILITY, the frame
   type it answers). `parseSessionContextCapabilities` in
   web-voice-transport.ts is tolerant of absence: `[]` for a missing,
   null or malformed field, undefined for a frame that is not
   session.context; names are trimmed, flattened, deduplicated and
   bounded (64 chars, 32 entries). voice-agent.ts records the attached
   client's list from every session.context frame it sends (one per
   attach and per room change) and clears it on detach, so a later,
   older client never inherits a newer one's list. The VoiceNavigateClient
   seam gains `supports(capability)`, and navigateUi answers
   `unsupported` at once — "This desktop app can't be navigated by voice
   yet; update it." — with nothing on the wire unless the client said
   `ui.navigate`. Every current desktop sends session.context without
   the field, so with this merged ahead of the client half the trigger
   phrases cost the owner one sentence, not six seconds of silence.

2. `target:'room'` with no or blank query is `not_found` immediately,
   "Say which room, for example 'take me to GTM in Investors'.", and no
   frame is sent; before, the client got a query-less room frame and the
   model read back "no room matching a room".

3. The surface hook. `onServerFrame` fired before the transport's typed
   dispatch, unguarded: it saw every frame, including the ones the
   transport owns, and a hook that threw skipped the transport's own
   handling of that frame (a throw on session.config would have skipped
   sendSessionContextOnce). It now runs after the typed dispatch, only
   for frame types outside TRANSPORT_OWNED_FRAME_TYPES (agent.state,
   session.config, session.context.ack, transcript, turn.end,
   turn.interrupted), and a throwing hook is contained so the raw
   onProtocolMessage sink still runs. The desktop's vendored copy of this
   file routes ui.navigate through onProtocolMessage and does not use the
   hook, so it picks the change up at its next re-vendor with no
   behaviour change.

4. Wiring pinned. tests/voice-navigate.test.ts now covers: capability
   absent → unsupported with the update message and no frame; present →
   the frame goes out and the reply settles it; room without query →
   not_found and no frame; the capabilities parser's tolerance and
   bounds; and source pins that onClientCommand routes ui.navigated
   through resolveUiNavigated after recording capabilities, that
   capabilities come from parseSessionContextCapabilities and clear on
   detach with the pending requests, that the installed client is gated
   on connected AND the capability, and that navigate_ui sits in both
   the inline tool table and the owner-only table.
   tests/web-voice-transport.test.ts pins the hook's new ordering and
   containment.

The tool description and prompt lines are unchanged, so the behaviour
anchors are untouched (13/13 pass without regeneration). test:ts
1472 pass / 0 fail, tsc clean, eslint 0 errors, review-checks and
lint-workspace-resolution clean.

Paired client change: ag2-space/cinny-webclient announces the
capability from sessionContextMessage in voiceRoomBinding.ts (both the
room and the DM shape) and updates the shared
fixtures/voice-session-context.v1.json.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* fix(voice): release the previous room before waiting for the next room's verdict

Review finding F1 on #4393. bindSessionContextFrame's docblock said that
until a room's membership verdict arrives the session stays on the DM. That
held for DM to room and was false for room to room: _voiceSessionRoom was not
touched until the verdict returned, so the old room stayed bound for the
whole wait (about 205 ms cold, up to the 6 s verdict timeout). A task spoken
in that window was addressed to the room the owner had just left, and its
result was posted there. The reviewer drove it at 1a6fddce: docked in a
customer room, move to another room, speak during the wait, and the header
carries the customer room.

Now a frame naming a different room releases the bound room before the
await. The released room is remembered only for change reporting, so a
refused move still reads as `left` exactly once and a verified one as
`entered`. A duplicate frame for the room already bound releases nothing.
setVoiceSessionRoom clears the remembered room.

Two tests. The room-to-room one fails without this change (36 pass, 1 fail)
and passes with it (37 pass): a task written during a gated verdict carries
channel_id: local-voice and the previous room nowhere in its header.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(voice): an offline room-less voice result is spoken on reconnect again

The room-less offline forward writes the untagged
proactive-result-<id>-<ts>.txt that any text bridge may deliver to the
owner's DM. This branch had started adding that file to the drain's
delivered set as it was written. On an install with no bridge to take the
file (voice-only, or Telegram-only while last activity reads as voice), the
drain's proactive-* fallthrough is the only delivery path, so the claim made
the result undeliverable: nothing spoke it on reconnect and the file stayed
in results/ for good.

The untagged forward is now left unclaimed, which is main's behaviour. The
two gateway-tagged shapes (.to-ag2space: the room post and the dm-only
owner-DM post) keep their immediate claim, because the gateway bridge owns
those and an unclaimed copy would be narrated a second time.

The offline leg also scheduled its 10 s archive of the originating result
before the forward had been written, so a failed write (disk full) archived a
result nobody had received. The claim, forward and archive now live in
_forwardOfflineThenArchive:…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants