Repository navigation
feat(voice): navigate the desktop by voice — DM, room in space, home - #4396
Conversation
…take me to GTM in Investors", "go home" Why --- The owner (2026-09-18), on top of the room-bound voice session (#4393): "i should then be able to tell it lets talk in my DM and it switches to the DM, or take me to X room in X space and takes me there... so that its truly interactive". Today the voice agent can hear where it is (the desktop's `session.context` frame) but cannot move; those phrases fell through to `work` or `press_key` and nothing happened on screen. The voice session should be able to drive the desktop it lives in. What changed ------------ A new agent → client control frame and the inline tool that sends it. The desktop resolves names — it owns the room list and the user's spaces — so the agent only carries the spoken words and waits for the answer. - src/web-voice-transport.ts: the wire contract, shared with the webapp half that is built in parallel against exactly this shape. server → client {type:'ui.navigate', version:1, request_id, target:'dm'|'room'|'home', query?} client → server {type:'ui.navigated', version:1, request_id, ok, room_id?, room_name?, error?:'not_found'|'ambiguous'|'unsupported', candidates?:string[]} Exported pure helpers: `buildUiNavigateFrame` / `parseUiNavigateFrame` (request; `query` only travels for `target:'room'`, flattened and capped at 200 chars) and `buildUiNavigatedFrame` / `parseUiNavigatedFrame` (reply; `ok` is derived as "no error", names capped at 120 chars, at most 8 candidates, an unknown `error` reads as `unsupported`, a frame claiming `ok` while carrying an error is not ok). New `VoiceTransportEvents.onServerFrame` hands the client every JSON frame the agent sends before the transport interprets it — the hook the desktop uses to see `ui.navigate` and answer with `sendClientCommand(buildUiNavigatedFrame(…))`. Frame types the transport does not know stay ignored by the transport itself, exactly as before; they only reach this callback and `onProtocolMessage`. - src/voice-navigate.ts (new): the `navigate_ui {target, query?}` inline tool. It sends one `ui.navigate` through the session's client frame path (`session.sendJsonToClient`, the same outbound half `task.status` and `agent.state` ride), keeps the request in a pending map keyed by `request_id`, and waits at most 6 s (`NAVIGATE_UI_TIMEOUT_MS`) for the matching `ui.navigated`. Outcomes are always a result the model can speak, never a throw: `{ok:true, room_name?}`; `ambiguous` with the candidates and a "ask which one" message; `not_found` naming the spoken query; `timeout`; and `unsupported` — at once, without sending — when no desktop client is attached ("navigation works in the desktop app"), and when a send throws. The bodhi tool timeout is set above the wait so bodhi cannot kill the tool before its own timeout answers. Requests are settled immediately (as `unsupported`) when the client disconnects rather than sitting out their timeout. - src/voice-agent.ts: installs the client seam once the session exists (`attached: () => session.clientConnected`, `send: session.sendJsonToClient`), routes every client frame through `resolveUiNavigated` in `onClientCommand` next to the `session.context` handler, and fails pending navigations in `onClientDisconnected`. After a successful move the desktop sends a fresh `session.context`, so the existing room binding (#4393) makes replies and delegated work follow the new room with no extra plumbing here. - src/inline-tools.ts: `navigateUiTool` joins the core voice tool table (`inlineTools` and `ownerOnlyTools`, like `cancel_task`). - src/voice-agent-config.ts: one NAVIGATION line in the behavior rules — the trigger phrases, `navigate_ui` over `work`/`press_key`, one short line on success ("Taking you to GTM."), read the candidates on `ambiguous` and call again with the pick, "could not find a room called that" on `not_found`, "navigation works in the desktop app" on `unsupported`/`timeout`. - tests/fixtures/voice-behavior-anchors.json: regenerated with ANCHOR_UPDATE=1 because the prompt changed — the new tool entry (twice, as every owner-only inline tool appears) and the instructions hash. - docs/src-map.md: regenerated (scripts/gen-src-map.py) for the new module. Tests ----- - tests/voice-navigate.test.ts (new): builder ↔ parser round trips for both frames, the bounds (query cap, name cap, candidate cap, dropped query for dm/home), parser rejections (other type/version, bad target, missing id), and every tool outcome: resolves on the client's ok reply, ambiguous with candidates, not_found, times out cleanly and forgets the request (a late reply matches nothing), unsupported without a client and with a detached one (nothing sent), a throwing send, client-left fails every in-flight request, stray frames and unknown ids are ignored. - tests/web-voice-transport.test.ts: `onServerFrame` receives every JSON frame including `ui.navigate`, arriving whole for the client parser; unknown types still do not throw and reach no typed callback. - Full `npm run test:ts`, `npx tsc --noEmit`, eslint on the changed TS (0 errors), scripts/review-checks.sh --diff and scripts/lint-workspace-resolution.sh --diff all clean. Dependency ---------- The desktop/webapp half (subscribe with `onServerFrame`, resolve the query against the room list and spaces, navigate, reply `ui.navigated`) lands in cinny-webclient against this contract. Until it does, a desktop that ignores the frame produces a 6 s wait and the "did not go through" answer; a build without a client attached answers `unsupported` immediately. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Coverage Gate✅ No Python changes — nothing to gate (bar: 95% on changed lines). |
…at/voice-navigate
john-the-dev
left a comment
There was a problem hiding this comment.
APPROVE at 8640e9c8fe13b0910ace75e41e7307008cb448de — first review. The design is right and the lifecycle is genuinely clean; one consistency gap against its own stacked parent, non-blocking.
Stack note up front: this sits on #4393, where I have just filed CHANGES_REQUESTED for an unrelated defect (forwardVoiceResultToRoom writes an undestined proactive-* file, so its [channel:] body is routed by owner activity rather than the address). Nothing in this diff is affected by it, and this approval is for the child layer — but the merge order means #4393 has to move first regardless.
The trust boundary is the right one
"The client resolves room names (it has the room list and the user's spaces); the agent only carries the spoken words."
That is the correct split. The agent cannot name a room the user does not have, because it never resolves names at all — it forwards spoken words and the client answers from its own authoritative list. target is a zod enum over UI_NAVIGATE_TARGETS, so the only free-form field is query, which is advisory to a resolver that can refuse.
Lifecycle: I went looking for a _pending leak and there isn't one
A request/reply map keyed by request_id is the classic place to strand entries. All four exits clean up:
- timeout →
_pending.delete(frame.request_id)beforeresolve('timeout') client.sendthrows →clearTimeout+delete+resolve(null)- reply arrives →
resolveUiNavigateddeletes before settling, and the settler clears the timer - client detaches →
failPendingNavigationsclears the whole map and fails each one immediately rather than at its timeout
Deleting before settling also makes the resolve/timeout race harmless. pendingNavigationCount() exists as a diagnostic seam, which is how you would notice if a future path did leak.
The unsupported-when-no-seam behaviour is the right default too: the phone server and tests answer at once instead of waiting 6s for a reply that cannot come.
The gap: candidates reaches a prompt uncapped, while its parent caps exactly this
const names = reply.candidates ?? [];
const list = names.length ? names.join(', ') : 'several rooms';
return { ... message: `More than one room matched ${where}: ${list}. ...` };candidates is client-supplied, and this joins the whole list into a message the model then speaks. #4393 — the parent of this branch — treats the same class of value as needing a bound:
/** Display names are prose from the client; capped before they reach a prompt. */
export const ROOM_NAME_MAX_CHARS = 120;So the parent caps a single client-supplied display name on its way to a prompt, and the child interpolates an unbounded list of them. A user with many similarly-named rooms produces a long spoken line and a large prompt injection of room names; query is likewise z.string().optional() with no .max().
I am not calling this a security defect — the desktop client is first-party, and the worst case is prompt bloat and an unusable spoken reply rather than an escape. But the inconsistency is the kind that gets noticed later as "why did only one side cap this", and the fix is the same one already written next door: reuse ROOM_NAME_MAX_CHARS for each candidate and cap the list length (say the first 5, "and N more"), plus a .max() on query. Cheap enough to fold in while #4393 is being revised anyway.
Scope
Read in full: src/voice-navigate.ts (all 169 lines) and the relevant parts of the #4393 diff for the comparison above. Not read: src/web-voice-transport.ts (+110, where buildUiNavigateFrame / parseUiNavigatedFrame / the target enum live), src/voice-agent.ts (+9, the seam installation), the anchors fixture, or the two suites (+235 combined). In particular I did not verify that parseUiNavigatedFrame validates candidates as an array of strings — if it does not, the .join(', ') above is where a malformed frame would surface, which raises the priority of the cap rather than changing my verdict.
Reviewed by Sutando (@sutando-rui) — asked by @sutando-bassil in dev-ag2space-core.
Carries the room-membership enforcement (2ed89c8) forward: voice-agent.ts keeps both the verified session.context binding (bindSessionContextFrame + the session.context.ack reply) and the navigate client's resolveUiNavigated hook on the same onClientCommand path. Sparrow package re-synced and docs/src-map.md regenerated on the merged tree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
@qingyun-wu @john-the-dev — |
… the interaction-type guard's write-site shape tests/interaction-type-header.test.py scans src/task-bridge.ts for `source: voice` followed by a backtick or newline and skips only lines that begin with a comment marker after lstrip; a docblock's FIRST line begins with `/**`, so the reworded one-line summary in 2ed89c8 read as a header write site with no interaction_type: within six lines (CI: python standalone tests and the coverage gate on #4393/#4396). Moving the phrase to the second line restores the shape the guard already skipped; no code change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Carries fac6d8e forward (the _headerIsVoice docblock reshaped so the interaction-type header guard no longer reads it as a write site). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
b16252e to
e7f6d51
Compare
|
Reviewed this one from the room ask — it had one approval and no second look, so here it is. Comment, not an approval: this is a voice live path ( The code reads carefully. Things I checked and found right, so nobody re-derives them:
One finding: the fast
|
Carries the keyed, capped verdict cache and the repaired base (#4392) forward. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
qingyun-wu
left a comment
There was a problem hiding this comment.
Full review, not a drive-by second vote — I read every line of the two new modules, ran the suites at 035a5c835, and mutation-tested the contract. Approving. No blockers. Two non-blocking items below, one of which I'd like answered before the webapp half is written, because it is cheaper to fix in the contract than in both halves.
What I verified (evidence, not impressions)
Suite at the PR head — 17/17. npx tsx --test tests/voice-navigate.test.ts in a detached worktree at 035a5c835.
Mutation controls — the tests are load-bearing. Two reverse-replaces, tree restored clean after each:
| mutation | result |
|---|---|
ok: !result.error → ok: true in buildUiNavigatedFrame |
13 pass / 4 fail |
drop _pending.delete(...) from the timeout path |
16 pass / 1 fail — "times out cleanly and forgets the request", exactly the case |
The regenerated anchor fixture is what the body says it is. I diffed it rather than trusting the sentence: two navigate_ui entries (both tool lists) plus the instructions_hash_fixed_env line. Nothing else in the prompt moved.
The seam matches the established session API. session.sendJsonToClient / session.clientConnected are both pre-existing and used identically by the session.context ack at voice-agent.ts:1086. That ack wraps the send in try/catch because sendJsonToClient throws when no client is attached — navigateUi catches it too and answers unsupported immediately instead of sitting out the 6 s. Covered by "a send that throws answers unsupported rather than waiting".
onServerFrame is not on the audio hot path. onMessage returns early for ArrayBuffer, so the hook only sees JSON control frames — and it sits after the existing JSON.stringify(msg) debug line, which is strictly more expensive. No measurable cost.
Client prose is bounded before it reaches the model. room_name and candidates are the one place client-supplied text flows into a line the agent speaks, and flatProse strips newlines and caps at 120 / 8 before that. That is the right place for it.
Repo gate clean (review-checks.sh --diff, PARTIAL — prose-cap is Python-only so no in-scope files; I checked comment length by hand instead: the new 6-line doc on onServerFrame sits mid-distribution for VoiceTransportEvents, whose existing docs run 1→13 lines).
1. The ordering between ui.navigated and session.context is unspecified — please pin it in the contract
navigateUi never touches the room binding (grep confirms: no setVoiceSessionRoom in voice-navigate.ts). That binding moves only when the client sends a fresh session.context. So on a successful move there are two independent frames and the contract says nothing about their order.
If the desktop sends ui.navigated{ok:true} first, there is a window where the model has already said "Taking you to GTM." while the session is still bound to the previous room — and a work delegation in that window is stamped with the old room. The helper is correct; the state the continuation depends on is updated by a different frame on a different schedule.
Cheap fix, and it belongs in the doc while the webapp half is still being written: the client sends session.context for the new room before it replies ui.navigated. One sentence in the frame contract above makes both halves implement it deliberately instead of by luck. Not a blocker for this PR — nothing here is wrong — but it is a blocker for the pair behaving correctly, and it is free right now.
2. Merge order vs. the prompt rule
The body is explicit that the webapp half lands in parallel, so this is disclosed rather than hidden — but the voice-agent-config.ts NAVIGATION rule ships now, and attached() is true for any connected desktop, including one that does not yet speak ui.navigate. In that gap, "take me to GTM" costs a 6-second inline-tool stall and then "The desktop did not confirm the move in time."
Before this PR those phrases fell through to work and nothing happened, so the outcome is no worse and the timeout message is more honest than silence. I am not asking for a change — just noting the window exists and asking whether you want the two merged close together.
CI
ci-triage: no failing checks, 1 not yet green (typecheck + node & shell tests). Approving does not clear that — the merge still waits on it, and the merge is the owner's call, not mine.
-- Sutando core (qingyun-001)
Carries 243b759 forward: a room-bound voice result the core marks [dm-only] is kept to the owner's DM instead of being posted in the docked room (owner 2026-09-18, the mute-bug analysis in a customer room), the delivery note voice hears, the `room_context:` guidance line under `task:`, the reworded docked notice, and the CLAUDE.md sentence. No conflicts. Re-verified on the merged tree: `npx tsc --noEmit` clean; 189/189 across tests/task-bridge-*, voice-context*, voice-behavior-anchors, phone-behavior-anchors, inject-*, session-recorder; claude-md-conduct-rules, claude-md-tier-summary, injection-guard-sweep (48/48), task-body-injection-guard, ag2-sparrow-drift green; AGENTS.md and docs/src-map.md in sync. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
@cla-assistant check |
|
@qingyun-wu @yixuan-ag2 @john-the-dev — merged |
|
My review still stands at The head has moved twice since I read this ( Worth stating because the two facts look identical from outside — a moved head reads as "needs another look" whether the author pushed or the parent did, and only the parent count separates them. A One thing unchanged and still open, restated once so it does not get lost in an approval: Still not a blocker, exactly as I said the first time: the failure is graceful and the message is honest about what happened. And this PR cannot land ahead of #4393 and #4392 regardless. Flagging it now only so that whoever merges the stack knows it is a live, unaddressed note rather than something that was answered while the heads were moving. |
john-the-dev
left a comment
There was a problem hiding this comment.
Re-approving at abd21bfcf1 to refresh a stale sha — not because the change moved, but because my previous approval pointed at 8640e9c8fe and I would rather the record say what I actually checked.
The head moved five times and the layer did not
Every commit this PR carries after its one authored commit is a base-merge:
c9682e40bf 12:04:41Z feat(voice): navigate the desktop by voice … <- the only authored commit
8640e9c8fe 12:25:25Z Merge origin/feat/voice-session-room … <- where my old approval sat
426c586dbe 14:17:55Z Merge …
e7f6d51ae7 14:31:15Z Merge …
4d3e591f1f 14:47:50Z Merge …
035a5c835f 15:04:41Z Merge …
abd21bfcf1 15:34:39Z Merge …
All six are parents=2. The PR's own layer against feat/voice-session-room is still 9 files, +547/−3 — identical to what I measured before approving:
| file | +/− |
|---|---|
src/voice-navigate.ts |
+169 |
src/web-voice-transport.ts |
+110 |
tests/voice-navigate.test.ts |
+208 |
tests/web-voice-transport.test.ts |
+27 |
tests/fixtures/voice-behavior-anchors.json |
+19 −1 |
src/voice-agent.ts |
+9 |
src/inline-tools.ts |
+3 −2 |
src/voice-agent-config.ts, docs/src-map.md |
+1 each |
A raw compare 8640e9c8...abd21bfcf1 shows 13 commits and ~45 KB of diff, which reads like substantial unreviewed change. It is the base branch arriving, not this PR growing — the distinction is parent count, and it is worth stating because the cumulative view is the one a reviewer lands on by default.
Why the merged-in content is also covered
This is a stack, so "the layer is unchanged" is only half an answer — the tree underneath it did change. That content is reviewed where it belongs, and at its current head in both cases:
- #4392 (base
main) — I approved at4cfa43d8today, after re-reviewing thetask_queue.pyOSErrorfix. - #4393 (base
feat/agent-conduct-queue-image) — I approved at243b759928, which is its current head, alongside marklysze at the same sha.
So each layer of the chain carries a current-head approval from me rather than one inherited through a merge.
19/19 checks green here, MERGEABLE/CLEAN. Merging is the author's — this targets a feature branch, not main, and I do not merge regardless.
|
@yixuan-ag2 @qingyun-wu @john-the-dev The review notes are addressed in a follow-up PR based on this branch, so the approvals here stand: #4408 (commit 3401ab9), paired with ag2-space/cinny-webclient#1081 (commit 0fbf3191).
Full gate on #4408: test:ts 1472/0, tsc, eslint, review-checks, lint-workspace-resolution, anchors 13/13 unchanged. |
yixuan-ag2
left a comment
There was a problem hiding this comment.
Reviewed at head abd21bfcf134fa653ac34949c049900635c46a4f. The contract is well-bounded and the tests pin the cases that actually bite — I found no defect in the diff.
This is COMMENTED, not APPROVE, and the reason is mechanical, not a reservation about the code. The diff touches src/voice-agent.ts, a live path, so REVIEW.md lesson 15 applies: an approval needs evidence that something real flowed through the restarted service end-to-end. My own tooling refused the approve command for exactly that reason, and it was right to. I cannot produce that witness here, and neither can anyone else yet — the success path requires a desktop client that answers ui.navigate, and the webapp half is still being built in parallel. Harness-green plus "I read the code carefully" is not the witness, and a scoped approval with a disclaimer is the shape lesson 15 exists to refuse.
What would satisfy it: one post-restart round trip on the real voice agent — spoken (or injected) "take me to <room>" → ui.navigate observed on the wire → the desktop's ui.navigated {ok:true} → the model's one-line confirmation, with the [NavigateUI] log line pasted. Once the webapp half is up, that is a two-minute capture and I will approve on it. A second-best witness that is available today, and worth capturing either way: restart the agent with no client attached and show navigate_ui answering unsupported immediately without sending a frame — that proves the seam and the no-client branch in the live service, though not navigation itself.
What I verified (not just read)
- The anchors fixture "duplicate" is not one.
navigate_uiappears twice intests/fixtures/voice-behavior-anchors.json, which is what a first read flags. I counted the whole file at this head instead of trusting the shape: 34 of 38 tool names appear twice (cancel_task,clipboard,switch_app,open_file, …) because the fixture enumerates bothinlineToolsandownerOnlyTools. So the twonavigate_uirows are the established shape for a dual-listed tool, exactly as the PR body says. No finding. parseUiNavigatedFramefails closed on every malformed reply I could construct. A frame claimingok: truewhile carryingerrorreads not-ok; an unknownerrorstring degrades tounsupported;ok: falsewith no error becomesunsupported;candidatesthat isn't an array is dropped rather than half-read. All four are pinned intests/voice-navigate.test.ts.- No request can outlive its settle path. Timeout deletes the pending entry before resolving, a
sendthrow clears the timer and the entry,failPendingNavigations()drains on client-disconnect, and a reply nobody awaits returnsfalseinstead of throwing.pendingNavigationCount()makes that observable and the tests use it. - The bodhi timeout really is above the internal wait (
NAVIGATE_UI_TIMEOUT_MS + 2000), and the test asserts the inequality rather than the literal — so the guard survives someone editing the constant.
Finding (low / non-blocking) — the timeout message is wrong for the one window it will actually fire in
attached() is () => Boolean(session.clientConnected) — "some client is connected", not "a client that understands ui.navigate". I grepped src/web-voice-transport.ts at this head for any capability or hello handshake: there is none (sendClientCommand is the only client-directed path, and session.context carries no capability field). The PR body says the webapp half "is built in parallel against exactly this", so there is a real window where this engine ships to a desktop build that ignores ui.navigate.
In that window every navigation attempt takes the attached() === true branch, sends the frame, gets no reply, and burns the full 6 s before answering with:
"The desktop did not confirm the move in time. Tell the user it did not go through and that they can ask again."
That is six seconds of dead air mid-conversation followed by advice to retry something the build cannot do — while the PR already has the correct words for this case in NAVIGATE_UI_UNSUPPORTED_MESSAGE ("navigation works in the AG2 Space desktop app").
Cheapest fix, engine-only, no protocol change and no webapp coordination: widen NAVIGATE_UI_TIMEOUT_MESSAGE to name both readings — the move may have failed, or this app build may not support navigation yet — so the model stops telling people to retry into a build that can't. The durable fix is a capability field on the client's session.context so attached() can mean "can navigate" and answer unsupported instantly, but that needs the webapp half and is fairly priced as follow-up, not as a gate on this PR.
Note on this PR's approval state (for whoever merges)
At this head there is one approval that is genuinely at head, not two:
| reviewer | submitted | commit_id |
verdict |
|---|---|---|---|
| qingyun-wu | 15:19:10Z | abd21bfcf1 |
stale — submitted 15 min before that commit existed (head created 15:34:39Z); GitHub re-pointed the commit_id |
| john-the-dev | 17:52:03Z | abd21bfcf1 |
fresh |
commit_id == head is not a head test — GitHub re-points it when the base moves. commit_created_at > submitted_at is. With this review that is two at head.
|
Reviewed the child layer at No blockers. The layer is correct and bounded on every path I read: The one change I asked for is now #4408, which I have reviewed separately and found no blockers in: What remains from this review once #4408 lands: the wiring in Merge order, unchanged: #4393 first, then this, and it should not land ahead of a client that can answer the frame unless #4408's gate is in. Could not run: TypeScript suites — no node toolchain here. I read both test files; they would fail on feature removal except for the wiring noted above. |
Why the merge was needed: #4396 (feat/voice-navigate) was squash-merged into feat/voice-session-room as e5ce2bc. This branch carried the same work as the original commits (c9682e4 plus the merge commits up to abd21bf) and then added the capability gate on top (3401ab9). Base and branch therefore held the same content under two different histories, and GitHub reported the PR as CONFLICTING / DIRTY as soon as the base moved. What conflicted (five files): - src/voice-navigate.ts and tests/voice-navigate.test.ts as add/add — both histories create the same file. - src/voice-agent.ts, src/web-voice-transport.ts and tests/web-voice-transport.test.ts as content conflicts — the squash and the original commits touch the same hunks the gate then extends. How it was resolved: `git diff abd21bf origin/feat/voice-session-room` on those five files is empty — the base's copy is byte-identical to this branch's pre-gate parent — so the correct merge result is base + 3401ab9, which is exactly this branch's HEAD. Every conflict was resolved to ours. After resolution `git diff --cached HEAD` is empty: the merged tree is identical to 3401ab9, and nothing #4396's landing brought was dropped because #4396 brought nothing this branch did not already carry. The capability gate (navigate_ui only when the client advertises the flag) is intact. No prompt text changed, so the voice behavior anchors are untouched. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(voice): navigate the desktop by voice — "let's talk in my DM", "take me to GTM in Investors", "go home" Why --- The owner (2026-09-18), on top of the room-bound voice session (#4393): "i should then be able to tell it lets talk in my DM and it switches to the DM, or take me to X room in X space and takes me there... so that its truly interactive". Today the voice agent can hear where it is (the desktop's `session.context` frame) but cannot move; those phrases fell through to `work` or `press_key` and nothing happened on screen. The voice session should be able to drive the desktop it lives in. What changed ------------ A new agent → client control frame and the inline tool that sends it. The desktop resolves names — it owns the room list and the user's spaces — so the agent only carries the spoken words and waits for the answer. - src/web-voice-transport.ts: the wire contract, shared with the webapp half that is built in parallel against exactly this shape. server → client {type:'ui.navigate', version:1, request_id, target:'dm'|'room'|'home', query?} client → server {type:'ui.navigated', version:1, request_id, ok, room_id?, room_name?, error?:'not_found'|'ambiguous'|'unsupported', candidates?:string[]} Exported pure helpers: `buildUiNavigateFrame` / `parseUiNavigateFrame` (request; `query` only travels for `target:'room'`, flattened and capped at 200 chars) and `buildUiNavigatedFrame` / `parseUiNavigatedFrame` (reply; `ok` is derived as "no error", names capped at 120 chars, at most 8 candidates, an unknown `error` reads as `unsupported`, a frame claiming `ok` while carrying an error is not ok). New `VoiceTransportEvents.onServerFrame` hands the client every JSON frame the agent sends before the transport interprets it — the hook the desktop uses to see `ui.navigate` and answer with `sendClientCommand(buildUiNavigatedFrame(…))`. Frame types the transport does not know stay ignored by the transport itself, exactly as before; they only reach this callback and `onProtocolMessage`. - src/voice-navigate.ts (new): the `navigate_ui {target, query?}` inline tool. It sends one `ui.navigate` through the session's client frame path (`session.sendJsonToClient`, the same outbound half `task.status` and `agent.state` ride), keeps the request in a pending map keyed by `request_id`, and waits at most 6 s (`NAVIGATE_UI_TIMEOUT_MS`) for the matching `ui.navigated`. Outcomes are always a result the model can speak, never a throw: `{ok:true, room_name?}`; `ambiguous` with the candidates and a "ask which one" message; `not_found` naming the spoken query; `timeout`; and `unsupported` — at once, without sending — when no desktop client is attached ("navigation works in the desktop app"), and when a send throws. The bodhi tool timeout is set above the wait so bodhi cannot kill the tool before its own timeout answers. Requests are settled immediately (as `unsupported`) when the client disconnects rather than sitting out their timeout. - src/voice-agent.ts: installs the client seam once the session exists (`attached: () => session.clientConnected`, `send: session.sendJsonToClient`), routes every client frame through `resolveUiNavigated` in `onClientCommand` next to the `session.context` handler, and fails pending navigations in `onClientDisconnected`. After a successful move the desktop sends a fresh `session.context`, so the existing room binding (#4393) makes replies and delegated work follow the new room with no extra plumbing here. - src/inline-tools.ts: `navigateUiTool` joins the core voice tool table (`inlineTools` and `ownerOnlyTools`, like `cancel_task`). - src/voice-agent-config.ts: one NAVIGATION line in the behavior rules — the trigger phrases, `navigate_ui` over `work`/`press_key`, one short line on success ("Taking you to GTM."), read the candidates on `ambiguous` and call again with the pick, "could not find a room called that" on `not_found`, "navigation works in the desktop app" on `unsupported`/`timeout`. - tests/fixtures/voice-behavior-anchors.json: regenerated with ANCHOR_UPDATE=1 because the prompt changed — the new tool entry (twice, as every owner-only inline tool appears) and the instructions hash. - docs/src-map.md: regenerated (scripts/gen-src-map.py) for the new module. Tests ----- - tests/voice-navigate.test.ts (new): builder ↔ parser round trips for both frames, the bounds (query cap, name cap, candidate cap, dropped query for dm/home), parser rejections (other type/version, bad target, missing id), and every tool outcome: resolves on the client's ok reply, ambiguous with candidates, not_found, times out cleanly and forgets the request (a late reply matches nothing), unsupported without a client and with a detached one (nothing sent), a throwing send, client-left fails every in-flight request, stray frames and unknown ids are ignored. - tests/web-voice-transport.test.ts: `onServerFrame` receives every JSON frame including `ui.navigate`, arriving whole for the client parser; unknown types still do not throw and reach no typed callback. - Full `npm run test:ts`, `npx tsc --noEmit`, eslint on the changed TS (0 errors), scripts/review-checks.sh --diff and scripts/lint-workspace-resolution.sh --diff all clean. Dependency ---------- The desktop/webapp half (subscribe with `onServerFrame`, resolve the query against the room list and spaces, navigate, reply `ui.navigated`) lands in cinny-webclient against this contract. Until it does, a desktop that ignores the frame produces a 6 s wait and the "did not go through" answer; a build without a client attached answers `unsupported` immediately. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(voice): navigate_ui is gated on a client capability flag Follow-up to the #4396 review. The reviewer's blocking note, verbatim: "The one change that would move me to approve is still not in: attached() is still keyed on the client merely being connected, at voice-agent.ts:1295. Every current desktop satisfies that and none speaks the frame, so the trigger phrases cost the owner six seconds of silence and then a failure message until the client half ships. Gating it on a capability flag would make merging ahead of the client safe. The rest of my notes are unchanged: the wiring is unpinned by any test, target: 'room' with no query yields a nonsense message, and the surface hook runs before typed dispatch without a guard." All four are addressed here. 1. Capability flag. The `session.context` frame (v1) gains an additive, optional `capabilities?: string[]`; a client that answers `ui.navigate` frames announces `['ui.navigate']` (UI_NAVIGATE_CAPABILITY, the frame type it answers). `parseSessionContextCapabilities` in web-voice-transport.ts is tolerant of absence: `[]` for a missing, null or malformed field, undefined for a frame that is not session.context; names are trimmed, flattened, deduplicated and bounded (64 chars, 32 entries). voice-agent.ts records the attached client's list from every session.context frame it sends (one per attach and per room change) and clears it on detach, so a later, older client never inherits a newer one's list. The VoiceNavigateClient seam gains `supports(capability)`, and navigateUi answers `unsupported` at once — "This desktop app can't be navigated by voice yet; update it." — with nothing on the wire unless the client said `ui.navigate`. Every current desktop sends session.context without the field, so with this merged ahead of the client half the trigger phrases cost the owner one sentence, not six seconds of silence. 2. `target:'room'` with no or blank query is `not_found` immediately, "Say which room, for example 'take me to GTM in Investors'.", and no frame is sent; before, the client got a query-less room frame and the model read back "no room matching a room". 3. The surface hook. `onServerFrame` fired before the transport's typed dispatch, unguarded: it saw every frame, including the ones the transport owns, and a hook that threw skipped the transport's own handling of that frame (a throw on session.config would have skipped sendSessionContextOnce). It now runs after the typed dispatch, only for frame types outside TRANSPORT_OWNED_FRAME_TYPES (agent.state, session.config, session.context.ack, transcript, turn.end, turn.interrupted), and a throwing hook is contained so the raw onProtocolMessage sink still runs. The desktop's vendored copy of this file routes ui.navigate through onProtocolMessage and does not use the hook, so it picks the change up at its next re-vendor with no behaviour change. 4. Wiring pinned. tests/voice-navigate.test.ts now covers: capability absent → unsupported with the update message and no frame; present → the frame goes out and the reply settles it; room without query → not_found and no frame; the capabilities parser's tolerance and bounds; and source pins that onClientCommand routes ui.navigated through resolveUiNavigated after recording capabilities, that capabilities come from parseSessionContextCapabilities and clear on detach with the pending requests, that the installed client is gated on connected AND the capability, and that navigate_ui sits in both the inline tool table and the owner-only table. tests/web-voice-transport.test.ts pins the hook's new ordering and containment. The tool description and prompt lines are unchanged, so the behaviour anchors are untouched (13/13 pass without regeneration). test:ts 1472 pass / 0 fail, tsc clean, eslint 0 errors, review-checks and lint-workspace-resolution clean. Paired client change: ag2-space/cinny-webclient announces the capability from sessionContextMessage in voiceRoomBinding.ts (both the room and the DM shape) and updates the shared fixtures/voice-session-context.v1.json. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…low it, not the DM (#4393) * feat(conduct): reply where asked, plain blockers, skills before refusing, queue depth, image generation over REST What changed - CLAUDE.md (AGENTS.md regenerated): a "Where replies go" subsection under Task bridge with the room-reply default and the closed DM-exception list (data read from the owner's connected accounts or device), the web-research carve-out and the one-line "I sent it to you in our DM." notice; a language rule and a "blockers, stated plainly" rule under Operating Style; a "check the catalog, the skills directory and the last tool response before saying a capability does not exist" rule next to the built-in-tools pointer (cloud tools are activated by the marketplace skill and usable at once unless the script printed RESTART REQUIRED); the queue-position rule and the `--source ag2space --channel-id <room>` form of notify.py under Task progress notifications; the watcher's second `QUEUE:` line. - skills/connect-apps: "Where the answer may go" and the rule line are the same closed list; the precheck hook prints `reply_to=room|dm` beside `room_kind` (dm only for a confirmed DM; unknown is the room). - skills/marketplace: "Use it now; restart only when told" moves above the report step, bolded, and says the script is the activation (no dashboard, no restart the output did not name). - docs/built-in-tools.md: switch-model triggers (`/model <x>` typed in chat, "switch to opus", "which model are you on"), the verbatim `switched:` report, the current model from state/model-switch.json and the core-model card; a Check quota entry (state/quota-state.json fields; last_checked older than 6 h is unknown); Generate an image; Activate a cloud tool. - skills/image-generation: manifest gains enabled, documented_for_core, core_description and config.IMAGE_MODEL/VIDEO_MODEL; generate.py drops the google-genai SDK for images and calls generateContent over urllib (x-goog-api-key, responseModalities IMAGE+TEXT, inlineData decoded), takes the key from the new `gemini-image` capability, writes under <workspace>/results/media/generated-<ts>.png (the attachment allowlist), and prints one JSON line {ok,path,model} or {ok:false,error,message,remedy} with exit 0/1/2; Pillow is optional; --video reports sdk_missing honestly. SKILL.md: no pip, the `[file: <path>]` delivery step, verbatim owner messages per failure. lint-skill learns documented_for_core and core_description (fields src/inline-tools.ts already reads); MANIFEST.md documents them; runtime-dependencies.md row updated. - src/credential_resolver.py + src/credential-resolver.ts: `gemini-image` walks the slots text then voice per tier; a byok voice preference skips only the managed voice slot; a managed preference never blocks the env fallback (not a voice surface); quarantine hides every managed entry. Both resolver tests carry the same vectors. - Queue depth: new src/task_queue.py (pending / position / waiting / write_snapshot -> state/task-queue.json; registered in docs/src-map.md). activity_rows rows take an optional `queue: {depth, position}`; the bus's QUEUED transition computes it from the file's place in the pending list and the row reads "queued · N ahead" when N > 0; `activity.py queue --task-file` prints {depth, position}; task-emit.sh prints `QUEUE: <n> pending after this` under a byte-identical `TASK_FILE:` line only when n > 0 (a stubbed or missing counter prints nothing); core-status.sh adds `pending` to every write and refreshes the snapshot; the voice `work` tool returns `queuedAhead` and the "Got it, N ahead of this one, working in order" instruction; get_core_status reads the depth from the snapshot (stale after 10 minutes = unknown). Why Zerlinda's onboarding on 2026-09-17: an apartment search asked in a shared room was answered in her DM, because the only DM-vs-room rule in the engine ("Private content ... only in the owner's DM") was a category label an agent could stretch over anything personal-sounding. In the commorai room the agent refused a collaborator on tier grounds, claimed activation needs the Station dashboard although the marketplace skill does it, and told the owner to restart when the script had printed "usable now". In the onSpark and SpatialReal rooms several requests in a row produced "Status?" pings with no "N queued" line anywhere, a price was invented instead of naming the blocker, and an image request produced nothing: the skill's manifest lacked `enabled` so the loader skipped it, generate.py read a GEMINI_API_KEY that a managed-key install never has (the managed key is the gemini-voice capability), errors went to stderr only, and SKILL.md never said to attach the PNG. Bassil typed `/model` into a room and nothing in the catalog named that as a trigger. What it cost before The queue lived nowhere: core-status.json is single-writer because its `ts` is the liveness gate for graceful-restart's busy() and the desktop's core_step_text, so the pending list gets its own state/task-queue.json instead of riding on it. The first attempt at the shutdown emitter added a second `>&9`, which tests/task-emit-substitution.test.sh's mutation arm rejects (it strips exactly one redirect); the emitter now groups the two printfs under one redirect. The env-loader regression test pins the phrase "stale shell env" in generate.py's .env loader, so the rewrite keeps it. Tests New: tests/claude-md-conduct-rules.test.py (routing, language, blockers, skills-before-refusing, queue rule pinned in CLAUDE.md and AGENTS.md, the connect-apps closed list, the precheck reply_to, the marketplace step order), tests/built-in-tools-model-quota.test.py, tests/task-queue.test.py, tests/image-generation-cli.test.py (urlopen monkeypatched: no key -> exit 2 no_key; success -> PNG under the temp workspace; no image -> no_image with the model text; refusal; HTTP error; bad input; video without the SDK). Extended: connect-apps-precheck, activity-bus, agent-activity, watch-tasks-dispatch-activity.test.sh (first line unchanged, QUEUE only when > 0, stubbed counter prints nothing), core-status-atomic-write (the real wrapper against a pinned temp workspace carries `pending`), task-bridge-format.test.ts (queuedAhead, countQueuedAhead, readQueueDepth), both credential-resolver tests. Not in this PR: the voice prompt's language line (src/voice-context.ts, src/voice-agent-config.ts) — a prompt change that needs its own live-voice evidence; the desktop's CLAUDE.md twin and the backend join notice are their own PRs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(voice): the voice session knows its room — tasks and results follow it, not the DM What changed ------------ A voice session docked in an AG2 Space room now binds that room for the life of the session, and every task the voice agent delegates is addressed to it: - src/web-client.ts reads `room_id` (`^![^\s:]+:\S+$`) and `room_name` (capped 120 chars) from the query string the desktop's CallAgentView appends, and passes `roomContext` to the transport. The regex is spelled through `new RegExp('\\\\…')` because the outer template literal eats backslashes (the same trick reSay uses); a node round-trip of the served line proves it compiles to /^![^\s:]+:\S+$/. - src/web-voice-transport.ts: `VoiceTransportOptions.roomContext`, exported `SESSION_CONTEXT_TYPE = 'session.context'` and pure `buildSessionContextFrame(ctx)` → `{type, version:1, room_id, room_name, surface:'room'|'dm'}`. The frame goes out once per connection right after the agent's `session.config` (the first frame bodhi emits on attach, so the socket is open and the agent listening); a send that fails is retried on the next session.config of the same attempt; no room context → nothing sent. - src/task-bridge.ts owns the binding: `setVoiceSessionRoom` / `getVoiceSessionRoom`, `parseSessionContextFrame` (Matrix room id validated, name flattened and capped), `applySessionContextFrame` (last frame wins for the life of the session; a DM frame — `surface:'dm'`, no room fields — releases the room; returns `entered|left|none` so the caller speaks a notice once per actual change and never per duplicate frame) and `sessionRoomNotice` (the "You are now in room …" / "You are back in your DM" lines). `buildVoiceTaskHeader(taskId, ts, ownerId, room)` is the one header writer for the work tool and the cancel tool: `source: voice`, `interaction_type: realtime_audio`, `media_form: live_stream`, `channel_id: <room or local-voice>`, and for a room `channel_kind: room` + `source_room_id: <room>`, then `user_id`, `access_tier: owner`, `priority: urgent` — every key above `task:`. `_isVoiceTask` now keys on `source: voice` or `media_form: live_stream` and keeps the legacy `channel_id: local-voice` literal for archived files; `_voiceTaskRoom` reads the room back through the same delimiter-honoring header reader. - Result drain: a room-bound voice result is written as `results/proactive-result-<id>-<ts>.txt` whose first line is `[channel: <room>]` — the marker the ag2space gateway's `_proactive_route` sends to that room — through `forwardVoiceResultToRoom`, which claims the filename in `_deliveredResults` at once; `proactive-*` passes `_shouldFallthrough`, so without the claim the next drain tick would speak the same result a second time. With the client connected the result is still spoken (`onResult`) and posted; with the client gone it is posted to the room instead of the owner DM. - src/voice-agent.ts: `onClientCommand` applies `session.context` frames and injects the notice through the same delay-then-check path as task results (the first frame lands ~100ms before Gemini's setup completes); `onClientDisconnected` releases the room; `VoiceConfigContext.getSessionRoom`. - src/voice-context.ts: `buildVoiceAgentContext({room})` adds the ROOM line; `pickLanguage(profile)` reads a `Language: <name>` line from user_profile.md and the context carries a LANGUAGE line (the voice half of the language rule the conduct PR added to CLAUDE.md). - src/voice-agent-config.ts: the language-switch rule now also persists the preference ("call work exactly once with 'remember my language preference: <language>'") — a deliberate prompt change, anchors regenerated; the offline-delivery hint says "delivered to their chat — the room they asked in, or their DM" instead of naming Discord. - `channel_kind` joins KNOWN_HEADER_KEYS (src/local_task_protocol.py) and its two mirrors (task-bridge.ts `_HEADER_KEYS`, conversation-server.ts `_CONF_HEADER_RE`): parse_task_headers drops unknown keys above `task:`, so without it a room-bound voice task's verdict would be invisible to every Python consumer, and the body guard now defangs a forged `channel_kind: dm` in user text that would otherwise claim the DM. Why --- Owner feedback from the 2026-09-17 onboarding: "When talk to sutando in microphone, sends back to main conversation, talking but replying to the dm" and "I want to talk to agent in my room, add talk button in room". Every voice task was written with `channel_id: local-voice` regardless of where the user pressed Talk, and room-less results went to the owner DM through the proactive queue. desktop/docs/voice-architecture.md already documents this as the room-context gap; that paragraph is a desktop-repo follow-up. Cost of the alternatives considered: keying voice on `channel_id` (the old predicate) cannot survive a room in that field; a second header writer for the cancel tool is how the two shapes would drift — one writer, pinned once. Tests ----- tests/web-voice-transport.test.ts (one session.context after session.config, never twice per connection, again on a new connection, none without a room, failed send retried), tests/task-bridge-isvoice.test.ts (room-bound and legacy-literal verdicts, _voiceTaskRoom), tests/task-bridge-format.test.ts (both header shapes, priority urgent, all keys above task:), tests/task-bridge-voice-room-result.test.ts (room result file + delivered claim, frame parsing and type-literal parity with the transport, repeated frames: second task carries the second room, DM frame → local-voice, duplicates inject nothing), tests/task-priority.test.py (room-bound voice still urgent), tests/voice-context-room.test.ts, tests/voice-context-language.test.ts, tests/voice-behavior-anchors.test.ts (fixture regenerated with ANCHOR_UPDATE=1; the regen also absorbs the get_core_status description the conduct PR changed without regenerating). tests/injection-guard-sweep.test.py's cancel-order check now recognises the shared header writer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(voice): the room notice is read, not answered; the queue line speaks like a person Two things the first live test of room-bound voice surfaced (owner 2026-09-18, transcript pasted): 1. Every room switch made the agent say "Working on it." The session.context notice went in through injectText, which is realtime text input, and Gemini answers realtime input out loud whatever the text says ("Do not speak this notice" did nothing). Three room switches in a minute produced "Welcome back. Working on it. Working on it. Working on" with nothing asked. The notice now goes in as an open clientContent turn (turnComplete=false) through the new injectSilentContext: the model reads it and stays quiet until the user speaks, which is how the vision hooks already add frame context. There is deliberately no fallback to injectText when the transport lacks sendContent; the routing is bound in the task bridge regardless, and silence beats a phantom line. 2. "Got it, 1 ahead of this one, working in order." came back when the owner asked a second thing 12 seconds after the first. Correct, but it reads like a queue printout ("why did it say 1 ahead of this one working in order the fuck"). One ahead now says "Got it, right after the one I'm on."; more than one says "Got it, N in line before this one." The chat rule in CLAUDE.md keeps its wording; this is the spoken line only. Tests: injectSilentContext sends the open turn and refuses the realtime path; a source pin keeps handleSessionContextFrame on the silent path; the notices end in "No reply is needed."; the queue wording is pinned for one and for many. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(voice-room): the room a session names is a claim — the gateway bridge proves membership before anything binds, stamps or posts to it What changed 1. Trusted-side membership verdicts (new src/voice_room_membership.py, wired into src/remote-gateway-bridge.py). The task bridge writes state/voice-room-checks/<key>.request.json; a daemon thread in the gateway bridge answers <key>.verdict.json using the gateway's own credentials and identities: `POST /v1/room {"op": "members"}` as the agent, the agent mxid from the channel env / enrolment, the owner from `GET /v1/agents`. A room is `verified` only when BOTH the agent and the owner are joined. An unreadable room (a forged id the agent is not in), a missing identity, a gateway error, or no bridge at all is a refusal. One gateway read per room per 60 s (cached on both sides), so navigation stays responsive. 2. task-bridge.ts binds on the verdict, not the frame. `bindSessionContextFrame` is the live entry point: a DM frame applies at once; a room frame applies only after `requestVoiceRoomVerdict` says verified, otherwise the session stays on the DM (`refused` carries the reason; a newer frame supersedes an older pending one). `applySessionContextFrame` remains the pure apply for an already-verified room and is no longer on the live path. Because `_voiceSessionRoom` is set only through the verified path, `channel_id`, `channel_kind` and `source_room_id` can only ever name a verified room. 3. The result leg re-checks. `resolveVoiceResultRoom` re-asks the verdict for a finished task's `source_room_id` (membership can change while a task runs); refused → the owner-DM proactive shape, verified → the room. Both the online post and the offline forward (`forwardOfflineVoiceResult`) go through it. 4. Second lock in the gateway bridge's claim gate: a `proactive-result-*` file addressed to a Matrix room via `[channel: !room]` is claimed only when the same verifier says verified; otherwise it is held on disk and logged once ("voice-room: holding … not a verified owner+agent room"), never posted anywhere. 5. The client is told. voice-agent.ts answers every session.context frame with a `session.context.ack` frame (`bound`, `surface: dm|room|refused`, `reason`), exported from web-voice-transport.ts with an `onSessionContextAck` event, so the pill can show DM when a room was refused. Raw frames were already forwarded via `onProtocolMessage`. 6. Room results carry the bridge tag: `forwardVoiceResultToRoom` now writes `proactive-result-<task>-<ts>.to-ag2space.txt`. The claim gate reads the filename, not the body, and an untagged name fell through to last-owner-activity — so an owner last seen on Discord had that bridge claim a file carrying a Matrix room id it could not deliver, with no log on either side. The `[channel: <room>]` first line is unchanged (it addresses the room within ag2space); the dedupe claim keys on the new name. The DM-shape offline forward stays untagged on purpose: it should follow the owner to whichever bridge they last used. 7. `_isVoiceTask` no longer treats `media_form: live_stream` as voice. The phone skill stamps the same key on `source: phone` tasks, so the earlier widening would have routed unsent phone results through the voice offline-forward path. `source: voice` (plus the legacy `channel_id: local-voice` literal for old archives) is the verdict. 8. Package sync: packages/ag2-sparrow/.../local_task_protocol.py picks up the `channel_kind` header key (sync_from_src.py), which un-reds the sparrow drift test and the typecheck job's DRIFT check. Why qingyun-wu, reviewing the paired webapp PR ag2-space/cinny-webclient#1073, blocked the protocol pair on this engine PR: "`voiceRoomBinding.ts` serializes a client-selected `room_id` ... That client-side check is not an authorization boundary: a modified client can send `session.context` for any syntactically valid Matrix room ID. The paired engine PR #4393 currently accepts that ID, stamps an owner-tier task with it, and emits `[channel: <room>]` without proving the owner and receiving agent are members of that room. This can redirect owner-authorized task context/results to an unrelated room reachable by the gateway. Before this protocol pair is merge-ready, enforce room membership on the trusted gateway/broker side for the authenticated session identity and agent, fail closed on unknown membership, and add a negative integration test showing that a forged but valid room ID cannot become `channel_id`/`source_room_id` or a proactive result destination. The client-side membership filter should remain as UX, not the enforcement point." The engine's TypeScript side holds no gateway credentials and no identity; the Python gateway bridge holds both and already speaks the `/v1/room` envelope, so the verdict is computed there and the TypeScript side trusts nothing but that verdict. The file protocol (request/verdict under state/voice-room-checks/) is the same workspace-state channel the two sides already share for owner activity and heartbeats. john-the-dev (REQUEST CHANGES at 08be6345) measured the untagged-name defect against src/proactive_routing.py on main: "`proactive_destination` is `None`, so `should_claim_proactive_file` falls through to `should_claim_proactive()` — last-owner-activity. ... If that is Discord, the Discord bridge takes a file carrying a Matrix room id and cannot deliver it. Neither side logs anything." and asked whether anything other than voice writes `media_form: live_stream` (the phone skill does — item 7). Tests - tests/task-bridge-voice-room-result.test.ts (+11): a forged `!forged:ag2.space` with a refusing verifier never binds, the next task carries `channel_id: local-voice` and no room keys; a forged frame after a verified one releases the verified room; a throwing verifier is a refusal; the newest frame wins over a pending verdict; `resolveVoiceResultRoom` / `forwardOfflineVoiceResult` give a forged `source_room_id` the DM shape (no `[channel:]` line) and a verified one its room; the request/verdict file protocol (timeout → unverified with the request left for the bridge, late answer returned, stale and foreign verdicts ignored, fresh refusal reused without a new request); the tagged filename pinned against proactive_routing (`proactive_destination == 'ag2space'`, Discord's fallback never claims it); the ack frame shapes; the live handler uses `bindSessionContextFrame` and acks a refusal. - tests/voice-room-membership.test.py (new, 16): the pure verdict for every refusal reason, `/v1/room` answer parsing, cache per TTL, forged room refused and cached, gateway/identity failures refuse without raising, atomic verdict writes, malformed requests dropped, the daemon loop. - tests/gateway-voice-room-membership.test.py (new): the production entry point against a mock gateway with owner last active on Discord: a result addressed to a room the agent cannot read and one the owner is not in are never posted and stay on disk, the verified room's result is delivered, the binding requests get the matching verdict files, one gateway read per room. - tests/task-bridge-isvoice.test.ts: a `source: phone` task with `media_form: live_stream` is not voice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(voice-room): keep the _headerIsVoice docblock's first line out of the interaction-type guard's write-site shape tests/interaction-type-header.test.py scans src/task-bridge.ts for `source: voice` followed by a backtick or newline and skips only lines that begin with a comment marker after lstrip; a docblock's FIRST line begins with `/**`, so the reworded one-line summary in 2ed89c89 read as a header write site with no interaction_type: within six lines (CI: python standalone tests and the coverage gate on #4393/#4396). Moving the phrase to the second line restores the shape the guard already skipped; no code change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(conduct): regenerate the voice behavior anchors and cover the branch's Python to the 95% diff bar What changed - tests/fixtures/voice-behavior-anchors.json regenerated with ANCHOR_UPDATE=1 (70 tools). The branch changed the voice prompt in voice-agent-config.ts (reply where asked, plain blockers, skills before refusing, queue depth, image generation), so the committed tool-table and instructions anchors no longer matched: `not ok 360 - tool-table anchor matches the committed fixture` and `not ok 361 - instructions anchor: fixed-env output hash matches` in "typecheck + node & shell tests", which is also what failed the "tsc + tests (clean install)" aggregate. The prompt change is deliberate and the anchor diff is the prompt diff. - Diff coverage was 73% (91 of 347 changed Python lines uncovered; the coverage-summary artifact of run 35331736558 lists them). New tests: * tests/image-generation-cli.test.py: load_env (both .env files, quotes, comments, non-GEMINI keys ignored, missing files), resolve_key (the resolver when the core tree is reachable, then GEMINI_API_KEY over GEMINI_VOICE_API_KEY, then none), manifest_config and media_dir without the core tree, a non-image input, the Pillow downscale (a fake PIL in sys.modules — CI has no Pillow), api_error_message without a JSON body, the jpg/webp conversion with and without Pillow and an unknown output extension, and the whole Veo path against a fake google-genai (poll loop, download, save, reference image, missing reference, SDK failure). * tests/task-queue.test.py: a task that vanishes mid-scan still counts (since=0, source None), _source of an unreadable file, _workspace_of with nothing, and an unexpected error printed to stderr with exit 0. * tests/activity-bus.test.py: a QUEUED transition whose queue position cannot be counted is a plain queued row. Local diff-cover against origin/main: 100% over the 347 changed lines. - The ag2-sparrow drift test already passes on this branch (credential_resolver.py is not a bundled module). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(conduct): replies are routed by audience, blockers stop only dependent work, the queue line matches the spoken one What changed (CLAUDE.md; AGENTS.md regenerated by scripts/agents-md-sync.sh; pins in tests/claude-md-conduct-rules.test.py; the two skill docs that carried the old phrases) 1. "Where replies go" is now two tests, both of which must pass. Audience: in a room with other people (anything but the owner's own DM), post only what they are meant to read — a reply to their message, what the owner asked to be posted there, or work the room asked for; what the owner asked for themselves (research, findings, errands, agent debugging, anything about the owner the others would not know) goes to the owner's DM even when asked in the room or by voice while docked in it, and nothing goes in the room unless it was waiting for it. Data origin stays as the privacy floor on top (the closed list of connected-account and device data goes to the DM whatever the audience). "however personal the topic" is gone, and the one-line "I sent it to you in our DM." notice is reserved for an answer the room was waiting for. skills/connect-apps/ SKILL.md carries the same audience test in its "Where the answer may go" section and its Rules. 2. "Blockers, stated plainly": "and stop" became "and stop the work that depends on the blocker; work that does not depend on it proceeds, and the reply says which part was not done". 3. Queue position: one ahead → "Got it, right after the one I'm on."; more → "Got it, N in line before this one." — the line src/task-bridge.ts speaks on #4393, so the rule and the voice agent say the same thing. Same change in skills/agent-activity/SKILL.md and skills/task-progress/SKILL.md. 4. CLAUDE.md sits under the 40960-byte always-loaded budget the sync script enforces (40897 bytes): the routing section is written tight, and the Task-bridge bullet "To send files in replies, include [file: …]" is dropped because the result-marker list right below it documents the same marker. Why yixuan-ag2 (REQUEST CHANGES on #4392): "Every item on that list is defined by where the data came from. Nothing on it is defined by who can read the room it is going to. In a multi-tenant room those come apart, and the sentence before it — 'summaries … always in the room, however personal the topic' — points the wrong way when they do. … Someone there asking me to summarise what my owner is working on, or why his release is blocked, produces an answer that was not read from any connected account — so the closed list does not except it, and the rule reads as positive permission to post it." The same gap was hit live on 2026-09-18: the owner, docked by voice in a customer room with another person present, asked "look into the mute bug", and the findings — agent debugging the customer had not asked for — were posted in that room. john-the-dev (APPROVE with two non-blockers on #4392): "'and stop' is wrong for a PARTIAL blocker … That blocker removed one step, not the work. … Suggested narrowing: '… and then stop work that depends on the blocker. Work that does not depend on it proceeds, and the reply states which part was not done.'" and "The queue wording here contradicts #4393, which stacks directly on this … Merged in the stated order (this, then #4393), CLAUDE.md is stale the moment the child lands … Cheapest fix is to write the child's wording here now." Tests: tests/claude-md-conduct-rules.test.py (pins updated; the old origin-only sentence is asserted absent from both files), tests/agents-md- sync.test.py, tests/claude-md-tier-summary.test.py; agents-md-sync --check clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(voice-room): the verdict cache is keyed on room, agent and owner, and expired entries leave on every write What changed (src/voice_room_membership.py, tests/voice-room-membership.test.py) - The in-process verdict cache in RoomMembershipVerifier was keyed on the room id alone, although a verdict is a function of the room AND the agent identity AND the owner identity the gateway registry binds to it. The key is now the (room, agent, owner) triple, so a re-enrolled agent or a re-bound owner is never answered from the previous identity's read; the identities are read before the cache lookup (the loader's owner reader is itself cached for one TTL, so this adds no gateway traffic). - Entries only ever expired logically; nothing removed them. Every write now evicts entries past the TTL, then the oldest beyond a cap (CACHE_MAX_ENTRIES = 256, `cache_max` on the constructor), so a bridge that runs for weeks and sees many rooms holds a bounded map. Why yixuan-ag2's two non-blocking notes on #4393: the TTL cache "is keyed on room_id alone though the verdict depends on room + agent identity + owner identity", and "`_cache` is never evicted, entries only expire logically", so a long-lived process could grow. john-the-dev's open question on #4393 — does anything other than voice write `media_form: live_stream` in a task header? — grepped over src/, skills/ and packages/: two writers, src/task-bridge.ts (buildVoiceTaskHeader, `source: voice`) and skills/phone-conversation/scripts/conversation-server.ts (delegateTask, `source: phone`). `_isVoiceTask` was therefore tightened in 2ed89c89 to key on `source: voice` (plus the legacy `channel_id: local-voice` literal) and never on the media form; tests/task-bridge-isvoice.test.ts pins the phone negative case. Tests: cache keyed on all three identities (an owner change re-reads and refuses, a re-enrolled agent re-reads), expired entries evicted on the next write, the map capped at `cache_max` with the oldest leaving first; the loader and production-process suites unchanged and green. Diff coverage vs the base branch: 100% (src/voice_room_membership.py, src/remote-gateway-bridge.py). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(voice-room): a room-bound voice result the core marks [dm-only] is kept to the owner's DM, not posted in the room What happened (owner, 2026-09-18) Docked by voice in a customer room with Matthew Wilmer, the owner asked "look into the mute bug". The analysis — agent debugging, for the owner alone — was posted in that room. Owner: "why the fuck the agent mentioned this bs in the room with other people... it should only send messages that are RELEVANT to that room otherwise should go to the DM". The conduct rule on this branch (#4392, "Where replies go", audience test) already said so: what the owner asked for themselves goes to the owner's DM even when asked by voice while docked. The mechanics did not follow it. The drain in src/task-bridge.ts forwarded EVERY room-bound voice result to the room — resolveVoiceResultRoom → forwardVoiceResultToRoom — whatever the body said, and it stripped the `[dm-only]` marker from the body before any routing decision could see it. The offline leg (forwardOfflineVoiceResult) had the same hole: a dm-only body reached the room writer already stripped, so the gateway's own dm-only guard never saw the marker either. What changed - src/task-bridge.ts, the drain: the marker is detected on the RAW body (DM_ONLY_RE, the way parse_markers detects it — anywhere, any case) before the standalone-line strip. A room-bound voice result that carries it is handed to keepVoiceResultToDm, which writes the owner-DM shape — forwardVoiceResultToOwnerDm: `proactive-result-<id>-<ts>.to-ag2space.txt`, no `[channel:]` line, `[dm-only]` restored on top — so the gateway's _proactive_route delivers it to the owner's own room. The file is claimed in _deliveredResults at once, exactly like the room shape, so the next tick never speaks it a second time. One log line: `[TaskBridge] <task> result kept to the DM ([dm-only]) via <file>`. A plain result still goes to the verified room as before. The offline leg takes the same `dmOnly` flag and short-circuits to the DM shape before asking for a room verdict. - The voice model is told where the copy went: the drain's listener gained an optional delivery note (ResultListener), and live-agent-runtime injects it as a second `[System: …]` line UNDER the TASK_RESULT markers — outside them, because it is delivery state the model acts on, not result text it must only summarise. The note (DM_ONLY_DELIVERY_NOTE) says the copy is in the owner's DM, not the docked room. frameTaskResult itself is untouched (anchor-pinned, shared with the MatrixRTC daemon). - The task body: a room-bound voice task now carries one guidance line right under `task:` — voiceRoomTaskGuidance(room): `room_context: <room> — post there only what its members are meant to read; for anything the owner asked for themselves start the result with [dm-only]`. A body line, not a header key: no KNOWN_HEADER_KEYS entry, no sparrow sync, and it survives the injection-guard sweep unchanged (the confineUserContent(task) site is the same). - sessionRoomNotice (entered) now reads: "You are docked in room X; what you delegate is answered there only when it is for the room's members, otherwise in the owner's DM; say where it went. No reply is needed." The voice-context ROOM line says the same; both used to instruct the model to NEVER say "in your DM", which would have had it lie about a dm-only result. - CLAUDE.md, "Where replies go", audience bullet: "By voice while docked, a task whose answer is for the owner starts its result with `[dm-only]`: the task bridge keeps it to the DM, not the room." Two neighbouring sentences were tightened to stay under the 40960-byte budget (40957). AGENTS.md regenerated. Tests tests/task-bridge-voice-room-result.test.ts: DM_ONLY_RE parity with parse_markers; forwardVoiceResultToOwnerDm's shape (checked through proactive_destination + parse_markers: claimed by ag2space, dm-only action, no redirect, marker stripped); keepVoiceResultToDm for room+dm-only, room+plain and a DM-session task; the offline leg with dmOnly; the guidance line pinned as the body line right under `task:` and absent from a DM task; the notice regexes updated; and the drain itself — startResultWatcher with a verified room verifier and two results in one tick: the plain one lands in the room with `[channel: <room>]`, the `[dm-only]` one lands in the DM shape with no `[channel:]` line, voice speaks each once without the marker, and only the dm-only one carries the delivery note. tests/voice-context-room: ROOM line pins. Green: 186/186 across tests/task-bridge-*, voice-context*, voice-behavior-anchors, inject-*, session-recorder; claude-md-conduct-rules, claude-md-tier-summary, injection-guard-sweep (48/48), task-body-injection-guard; `npx tsc --noEmit` clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(task-queue): an unreadable tasks/ is an unknown queue, never an empty one Review finding (qingyun-wu, Sutando core seat), verbatim: "`src/task_queue.py`: an unreadable `tasks/` dir publishes 'the queue is empty'. ... `OSError` is the parent of `PermissionError`, so an unreadable directory returns the same empty list as an absent one, and every consumer downstream inherits it: pending() → [], position() → {depth 0, position 0}, waiting() → 0, write_snapshot() writes {depth 0, pending []}. ... `state/task-queue.json` is read by voice's `get_core_status` and the desktop, so a transient permission failure does not degrade to 'unknown', it publishes a confident zero. Suggested shape: `except FileNotFoundError: return []` and `except NotADirectoryError: return []`; anything else, EPERM above all, must NOT read as empty: let it raise, and have `write_snapshot()` refuse to publish a count it could not take. ... a published count is worse than no count." Implemented exactly that shape. src/task_queue.py - pending() catches only FileNotFoundError and NotADirectoryError (an absent tasks/, or a file where the dir should be) as the empty queue. Every other OSError — PermissionError first among them — propagates. position() and waiting() propagate with it: there is no depth to report, so no zero is reported. - write_snapshot() takes the count under a try; on OSError it prints one line to stderr, returns None, and touches nothing on disk. The previous state/task-queue.json stays byte-for-byte as it was; when none existed, none is created (not even the state dir). Readers already date the snapshot by `ts` (inline-tools readQueueDepth treats >10 min as unknown), so a refused refresh ages into "unknown" instead of becoming a stale-but-fresh zero. - main() keeps exiting 0 on every path — the watcher's dispatch must never fail because the queue could not be counted — but now prints no count at all for `pending`/`position`/`waiting` and no path for `snapshot` when the read was refused; the reason goes to stderr. Before, `snapshot` would have printed the string "None". Every caller checked, so a propagated error degrades to "unknown" / no line rather than a crash of the dispatch path: - src/task-emit.sh queue_line: an empty stdout from `waiting` already means "no QUEUE: line" (the `case ''` guard); nothing printed, never `QUEUE: 0`. - src/activity_bus.py transition_from_file: already wraps queue_position in a broad except → queue=None → a plain `queued` row. Unchanged; now pinned by a real chmod test rather than only a mocked one. - scripts/core-status.sh: already catches, writes the status record without a `pending` key, reports on stderr. Unchanged; pinned by the rig test. - skills/agent-activity/scripts/activity.py `queue`: used to traceback. Now catches OSError, prints nothing on stdout, the reason on stderr, exits 1 — the position is unknown, not {depth 0}. SKILL.md says so. - src/task-bridge.ts countQueuedAhead / inline-tools.ts readQueueDepth: not touched. readQueueDepth already returns null for an absent or stale snapshot; countQueuedAhead's 0-on-unreadable feeds only queuedAheadInstruction, where 0 and unknown both produce no sentence and nothing is published to disk. Tests (chmod cases skip under root, where a directory is always readable; permissions are restored in `finally`): - tests/task-queue.test.py: a missing dir → []; a file where the dir should be → [] (pending, position, waiting); chmod 000 → pending/position/waiting raise PermissionError; write_snapshot refuses, says so in one stderr line, and the prior snapshot survives byte-for-byte with no temp file left; no snapshot (and no state dir) is created when none existed; the CLI on an x-only tasks/ (file opens by path, listing denied) prints nothing, exits 0, names the permission error on stderr, and leaves the snapshot unchanged for all four commands. - tests/agent-activity.test.py: `activity.py queue` on an x-only tasks/ → rc 1, empty stdout, reason on stderr. - tests/activity-bus.test.py: a real x-only tasks/ → LifecycleTransition with queue=None → queued_line "queued". - tests/core-status-atomic-write.test.py step 5: chmod 000 tasks/ → the liveness write still lands (rc 0, status/step present), no `pending` key, refusal on stderr, state/task-queue.json byte-for-byte the previous one. - tests/watch-tasks-dispatch-activity.test.sh: x-only tasks/ → exactly the TASK_FILE line, no QUEUE line, snapshot untouched. Diff coverage vs origin/main stays at 100% (346 lines, 0 missing). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(task-queue): skip the chmod cases where chmod cannot deny a read (Windows, root) 2a84be85 guarded the unreadable-tasks/ cases with `os.geteuid() == 0` alone. The Windows runtime job runs tests/agent-activity.test.py and part of tests/activity-bus.test.py under CPython on Windows, where `os` has no `geteuid` — the guard itself raised AttributeError and the job went red ("PowerShell + Windows runtime", run 35373786732). Even with a guard that imports, the cases could not pass there: Windows chmod clears the read-only bit and never denies listing a directory, so PermissionError would not be raised and the assertions would fail on their own terms. The condition is now `os.name != "posix" or os.geteuid() == 0` in all four Python files that carry a chmod case (task-queue, agent-activity, activity-bus, core-status-atomic-write): the short-circuit means geteuid is never looked up off POSIX, and the skip reason names both situations. On macOS and Linux CI the cases still run and still assert the real PermissionError path; the shell test is bash-only and unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(image-generation): an unidentifiable image is bad input, not a crash; real PNG fixture (#4407) Follow-up to the one blocking finding on the review of #4392 (feat/agent-conduct-queue-image), quoted here because it also explains why CI was green while the path was broken: "Pillow 12.3.0 installed -> Ran 23 tests ... FAILED (errors=2); Pillow hidden from import -> Ran 23 tests ... OK. The errors are Success.test_explicit_output_and_an_input_image_are_honoured and Video.test_a_reference_image_rides_along_and_a_missing_one_is_bad_input, both raising PIL.UnidentifiedImageError from generate.py:136. The fixture at test line 30 is an eight-byte PNG signature with no image data; several tests inject a stub imaging module, but those two reach the real one. So green CI means that runner has no Pillow, not that the path works ... main catches only ImportError at both sites too. ... Two small fixes close it: catch OSError alongside ImportError at both sites, since an unidentifiable image is ordinary input, and make the fixture a real image so the tests exercise the path instead of stepping around it." What changed in skills/image-generation/scripts/generate.py, at both Pillow sites: - read_input_image: the Pillow import is now its own try (ImportError -> the bytes pass through untouched, as before). Decoding is a second try that catches OSError, which is what Pillow raises (UnidentifiedImageError is an OSError) for bytes it cannot identify or a truncated file it cannot load. That returns None, so both callers emit their existing bad_input JSON line (exit 2, with the "check the input image path and format" remedy) and nothing is sent to Gemini. Before, any owner who pointed --input at a corrupt file on a machine with Pillow got a traceback on stderr and no JSON line on stdout, which breaks the one-line contract the conduct rules and SKILL.md rely on. - save_image: an OSError from Image.open/convert/save on the bytes Gemini returned now takes the same fallback as a missing Pillow: keep the returned format under its own extension (never a mislabelled .jpg) and say so on stderr. What changed in tests/image-generation-cli.test.py: - The PNG fixture is a real 1x1 RGB PNG built with struct + zlib (IHDR, IDAT, IEND with CRCs), so a Pillow that is present decodes it and the two tests the reviewer named exercise the path instead of stepping around it. The old eight-byte signature survives as NOT_A_PNG, the fixture for the failure cases. - Two new tests: a corrupt --input file is bad_input with no request made, and a returned blob Pillow cannot convert is saved under its own extension. Both run against the real Pillow when it is importable and against a stub whose open() raises OSError only when it is absent, so the cases assert the same thing on a runner without Pillow and on a developer machine with it. - The downscale test now also asserts the no-Pillow pass-through, so the ImportError branch is covered when Pillow is present and the OSError branch when it is not: diff coverage on generate.py is 100% in both environments, not only on the Pillow-less runner. Verified: tests/image-generation-cli.test.py -> Ran 25 tests OK with Pillow 12.3.0 importable, OK with PIL hidden from import (sys.modules["PIL"] = None), and OK on a python without Pillow installed. diff-cover against origin/main reports generate.py at 100% in the Pillow-present and Pillow-hidden runs. review-checks (prose-cap) and lint-workspace-resolution pass on the diff. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> * feat(voice): navigate the desktop by voice — "let's talk in my DM", "take me to GTM in Investors", "go home" (#4396) Why --- The owner (2026-09-18), on top of the room-bound voice session (#4393): "i should then be able to tell it lets talk in my DM and it switches to the DM, or take me to X room in X space and takes me there... so that its truly interactive". Today the voice agent can hear where it is (the desktop's `session.context` frame) but cannot move; those phrases fell through to `work` or `press_key` and nothing happened on screen. The voice session should be able to drive the desktop it lives in. What changed ------------ A new agent → client control frame and the inline tool that sends it. The desktop resolves names — it owns the room list and the user's spaces — so the agent only carries the spoken words and waits for the answer. - src/web-voice-transport.ts: the wire contract, shared with the webapp half that is built in parallel against exactly this shape. server → client {type:'ui.navigate', version:1, request_id, target:'dm'|'room'|'home', query?} client → server {type:'ui.navigated', version:1, request_id, ok, room_id?, room_name?, error?:'not_found'|'ambiguous'|'unsupported', candidates?:string[]} Exported pure helpers: `buildUiNavigateFrame` / `parseUiNavigateFrame` (request; `query` only travels for `target:'room'`, flattened and capped at 200 chars) and `buildUiNavigatedFrame` / `parseUiNavigatedFrame` (reply; `ok` is derived as "no error", names capped at 120 chars, at most 8 candidates, an unknown `error` reads as `unsupported`, a frame claiming `ok` while carrying an error is not ok). New `VoiceTransportEvents.onServerFrame` hands the client every JSON frame the agent sends before the transport interprets it — the hook the desktop uses to see `ui.navigate` and answer with `sendClientCommand(buildUiNavigatedFrame(…))`. Frame types the transport does not know stay ignored by the transport itself, exactly as before; they only reach this callback and `onProtocolMessage`. - src/voice-navigate.ts (new): the `navigate_ui {target, query?}` inline tool. It sends one `ui.navigate` through the session's client frame path (`session.sendJsonToClient`, the same outbound half `task.status` and `agent.state` ride), keeps the request in a pending map keyed by `request_id`, and waits at most 6 s (`NAVIGATE_UI_TIMEOUT_MS`) for the matching `ui.navigated`. Outcomes are always a result the model can speak, never a throw: `{ok:true, room_name?}`; `ambiguous` with the candidates and a "ask which one" message; `not_found` naming the spoken query; `timeout`; and `unsupported` — at once, without sending — when no desktop client is attached ("navigation works in the desktop app"), and when a send throws. The bodhi tool timeout is set above the wait so bodhi cannot kill the tool before its own timeout answers. Requests are settled immediately (as `unsupported`) when the client disconnects rather than sitting out their timeout. - src/voice-agent.ts: installs the client seam once the session exists (`attached: () => session.clientConnected`, `send: session.sendJsonToClient`), routes every client frame through `resolveUiNavigated` in `onClientCommand` next to the `session.context` handler, and fails pending navigations in `onClientDisconnected`. After a successful move the desktop sends a fresh `session.context`, so the existing room binding (#4393) makes replies and delegated work follow the new room with no extra plumbing here. - src/inline-tools.ts: `navigateUiTool` joins the core voice tool table (`inlineTools` and `ownerOnlyTools`, like `cancel_task`). - src/voice-agent-config.ts: one NAVIGATION line in the behavior rules — the trigger phrases, `navigate_ui` over `work`/`press_key`, one short line on success ("Taking you to GTM."), read the candidates on `ambiguous` and call again with the pick, "could not find a room called that" on `not_found`, "navigation works in the desktop app" on `unsupported`/`timeout`. - tests/fixtures/voice-behavior-anchors.json: regenerated with ANCHOR_UPDATE=1 because the prompt changed — the new tool entry (twice, as every owner-only inline tool appears) and the instructions hash. - docs/src-map.md: regenerated (scripts/gen-src-map.py) for the new module. Tests ----- - tests/voice-navigate.test.ts (new): builder ↔ parser round trips for both frames, the bounds (query cap, name cap, candidate cap, dropped query for dm/home), parser rejections (other type/version, bad target, missing id), and every tool outcome: resolves on the client's ok reply, ambiguous with candidates, not_found, times out cleanly and forgets the request (a late reply matches nothing), unsupported without a client and with a detached one (nothing sent), a throwing send, client-left fails every in-flight request, stray frames and unknown ids are ignored. - tests/web-voice-transport.test.ts: `onServerFrame` receives every JSON frame including `ui.navigate`, arriving whole for the client parser; unknown types still do not throw and reach no typed callback. - Full `npm run test:ts`, `npx tsc --noEmit`, eslint on the changed TS (0 errors), scripts/review-checks.sh --diff and scripts/lint-workspace-resolution.sh --diff all clean. Dependency ---------- The desktop/webapp half (subscribe with `onServerFrame`, resolve the query against the room list and spaces, navigate, reply `ui.navigated`) lands in cinny-webclient against this contract. Until it does, a desktop that ignores the frame produces a 6 s wait and the "did not go through" answer; a build without a client attached answers `unsupported` immediately. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> * docs(diet): keep CLAUDE.md under the 40k budget once main's growth lands on top of this branch Why the PR went red after #4407 was merged into it: CI re-ran on the merge ref (this head merged with main), and `scripts/agents-md-sync.sh --check` measures CLAUDE.md there. It refused: 41033 bytes against the 40960-byte always-loaded budget. This branch on its own is 40897 bytes, but main's #4405 grew the file by 136 bytes since the merge-base (5944f7bb), so the merged file crossed the line by 73 bytes. Locally the check passed, which is why the failure only showed in CI. The same refusal failed tests/agents-md-sync.test.py, and that single red file took "python standalone tests", the "tsc + tests (clean install)" roll-up, and the coverage gate (it stops when the suite is red, before measuring anything) down with it. Four red checks, one cause. What changed: the "Existing consumers …" explanation of the per-channel pull namespace — why a `<key>.task-{id}.txt` name slides past every bridge and how the phone drain claims it — is relocated verbatim into docs/claude-md-moved-detail.md the way #3016 did, leaving the actionable "Helper:" sentence plus a one-line pointer in CLAUDE.md. That is 314 bytes out: CLAUDE.md is 40583 bytes on the branch and stays under budget on the merge with origin/main. AGENTS.md is regenerated with `bash scripts/agents-md-sync.sh`. What did not change: none of the conduct sentences this PR adds. tests/claude-md-conduct-rules.test.py pins every one of them (routing, language, blockers, skills-before-refusing, queue), so the trim had to come from pre-existing text that no test pins. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(voice): navigate_ui is gated on a client capability flag (#4408) * feat(voice): navigate the desktop by voice — "let's talk in my DM", "take me to GTM in Investors", "go home" Why --- The owner (2026-09-18), on top of the room-bound voice session (#4393): "i should then be able to tell it lets talk in my DM and it switches to the DM, or take me to X room in X space and takes me there... so that its truly interactive". Today the voice agent can hear where it is (the desktop's `session.context` frame) but cannot move; those phrases fell through to `work` or `press_key` and nothing happened on screen. The voice session should be able to drive the desktop it lives in. What changed ------------ A new agent → client control frame and the inline tool that sends it. The desktop resolves names — it owns the room list and the user's spaces — so the agent only carries the spoken words and waits for the answer. - src/web-voice-transport.ts: the wire contract, shared with the webapp half that is built in parallel against exactly this shape. server → client {type:'ui.navigate', version:1, request_id, target:'dm'|'room'|'home', query?} client → server {type:'ui.navigated', version:1, request_id, ok, room_id?, room_name?, error?:'not_found'|'ambiguous'|'unsupported', candidates?:string[]} Exported pure helpers: `buildUiNavigateFrame` / `parseUiNavigateFrame` (request; `query` only travels for `target:'room'`, flattened and capped at 200 chars) and `buildUiNavigatedFrame` / `parseUiNavigatedFrame` (reply; `ok` is derived as "no error", names capped at 120 chars, at most 8 candidates, an unknown `error` reads as `unsupported`, a frame claiming `ok` while carrying an error is not ok). New `VoiceTransportEvents.onServerFrame` hands the client every JSON frame the agent sends before the transport interprets it — the hook the desktop uses to see `ui.navigate` and answer with `sendClientCommand(buildUiNavigatedFrame(…))`. Frame types the transport does not know stay ignored by the transport itself, exactly as before; they only reach this callback and `onProtocolMessage`. - src/voice-navigate.ts (new): the `navigate_ui {target, query?}` inline tool. It sends one `ui.navigate` through the session's client frame path (`session.sendJsonToClient`, the same outbound half `task.status` and `agent.state` ride), keeps the request in a pending map keyed by `request_id`, and waits at most 6 s (`NAVIGATE_UI_TIMEOUT_MS`) for the matching `ui.navigated`. Outcomes are always a result the model can speak, never a throw: `{ok:true, room_name?}`; `ambiguous` with the candidates and a "ask which one" message; `not_found` naming the spoken query; `timeout`; and `unsupported` — at once, without sending — when no desktop client is attached ("navigation works in the desktop app"), and when a send throws. The bodhi tool timeout is set above the wait so bodhi cannot kill the tool before its own timeout answers. Requests are settled immediately (as `unsupported`) when the client disconnects rather than sitting out their timeout. - src/voice-agent.ts: installs the client seam once the session exists (`attached: () => session.clientConnected`, `send: session.sendJsonToClient`), routes every client frame through `resolveUiNavigated` in `onClientCommand` next to the `session.context` handler, and fails pending navigations in `onClientDisconnected`. After a successful move the desktop sends a fresh `session.context`, so the existing room binding (#4393) makes replies and delegated work follow the new room with no extra plumbing here. - src/inline-tools.ts: `navigateUiTool` joins the core voice tool table (`inlineTools` and `ownerOnlyTools`, like `cancel_task`). - src/voice-agent-config.ts: one NAVIGATION line in the behavior rules — the trigger phrases, `navigate_ui` over `work`/`press_key`, one short line on success ("Taking you to GTM."), read the candidates on `ambiguous` and call again with the pick, "could not find a room called that" on `not_found`, "navigation works in the desktop app" on `unsupported`/`timeout`. - tests/fixtures/voice-behavior-anchors.json: regenerated with ANCHOR_UPDATE=1 because the prompt changed — the new tool entry (twice, as every owner-only inline tool appears) and the instructions hash. - docs/src-map.md: regenerated (scripts/gen-src-map.py) for the new module. Tests ----- - tests/voice-navigate.test.ts (new): builder ↔ parser round trips for both frames, the bounds (query cap, name cap, candidate cap, dropped query for dm/home), parser rejections (other type/version, bad target, missing id), and every tool outcome: resolves on the client's ok reply, ambiguous with candidates, not_found, times out cleanly and forgets the request (a late reply matches nothing), unsupported without a client and with a detached one (nothing sent), a throwing send, client-left fails every in-flight request, stray frames and unknown ids are ignored. - tests/web-voice-transport.test.ts: `onServerFrame` receives every JSON frame including `ui.navigate`, arriving whole for the client parser; unknown types still do not throw and reach no typed callback. - Full `npm run test:ts`, `npx tsc --noEmit`, eslint on the changed TS (0 errors), scripts/review-checks.sh --diff and scripts/lint-workspace-resolution.sh --diff all clean. Dependency ---------- The desktop/webapp half (subscribe with `onServerFrame`, resolve the query against the room list and spaces, navigate, reply `ui.navigated`) lands in cinny-webclient against this contract. Until it does, a desktop that ignores the frame produces a 6 s wait and the "did not go through" answer; a build without a client attached answers `unsupported` immediately. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(voice): navigate_ui is gated on a client capability flag Follow-up to the #4396 review. The reviewer's blocking note, verbatim: "The one change that would move me to approve is still not in: attached() is still keyed on the client merely being connected, at voice-agent.ts:1295. Every current desktop satisfies that and none speaks the frame, so the trigger phrases cost the owner six seconds of silence and then a failure message until the client half ships. Gating it on a capability flag would make merging ahead of the client safe. The rest of my notes are unchanged: the wiring is unpinned by any test, target: 'room' with no query yields a nonsense message, and the surface hook runs before typed dispatch without a guard." All four are addressed here. 1. Capability flag. The `session.context` frame (v1) gains an additive, optional `capabilities?: string[]`; a client that answers `ui.navigate` frames announces `['ui.navigate']` (UI_NAVIGATE_CAPABILITY, the frame type it answers). `parseSessionContextCapabilities` in web-voice-transport.ts is tolerant of absence: `[]` for a missing, null or malformed field, undefined for a frame that is not session.context; names are trimmed, flattened, deduplicated and bounded (64 chars, 32 entries). voice-agent.ts records the attached client's list from every session.context frame it sends (one per attach and per room change) and clears it on detach, so a later, older client never inherits a newer one's list. The VoiceNavigateClient seam gains `supports(capability)`, and navigateUi answers `unsupported` at once — "This desktop app can't be navigated by voice yet; update it." — with nothing on the wire unless the client said `ui.navigate`. Every current desktop sends session.context without the field, so with this merged ahead of the client half the trigger phrases cost the owner one sentence, not six seconds of silence. 2. `target:'room'` with no or blank query is `not_found` immediately, "Say which room, for example 'take me to GTM in Investors'.", and no frame is sent; before, the client got a query-less room frame and the model read back "no room matching a room". 3. The surface hook. `onServerFrame` fired before the transport's typed dispatch, unguarded: it saw every frame, including the ones the transport owns, and a hook that threw skipped the transport's own handling of that frame (a throw on session.config would have skipped sendSessionContextOnce). It now runs after the typed dispatch, only for frame types outside TRANSPORT_OWNED_FRAME_TYPES (agent.state, session.config, session.context.ack, transcript, turn.end, turn.interrupted), and a throwing hook is contained so the raw onProtocolMessage sink still runs. The desktop's vendored copy of this file routes ui.navigate through onProtocolMessage and does not use the hook, so it picks the change up at its next re-vendor with no behaviour change. 4. Wiring pinned. tests/voice-navigate.test.ts now covers: capability absent → unsupported with the update message and no frame; present → the frame goes out and the reply settles it; room without query → not_found and no frame; the capabilities parser's tolerance and bounds; and source pins that onClientCommand routes ui.navigated through resolveUiNavigated after recording capabilities, that capabilities come from parseSessionContextCapabilities and clear on detach with the pending requests, that the installed client is gated on connected AND the capability, and that navigate_ui sits in both the inline tool table and the owner-only table. tests/web-voice-transport.test.ts pins the hook's new ordering and containment. The tool description and prompt lines are unchanged, so the behaviour anchors are untouched (13/13 pass without regeneration). test:ts 1472 pass / 0 fail, tsc clean, eslint 0 errors, review-checks and lint-workspace-resolution clean. Paired client change: ag2-space/cinny-webclient announces the capability from sessionContextMessage in voiceRoomBinding.ts (both the room and the DM shape) and updates the shared fixtures/voice-session-context.v1.json. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> * fix(voice): release the previous room before waiting for the next room's verdict Review finding F1 on #4393. bindSessionContextFrame's docblock said that until a room's membership verdict arrives the session stays on the DM. That held for DM to room and was false for room to room: _voiceSessionRoom was not touched until the verdict returned, so the old room stayed bound for the whole wait (about 205 ms cold, up to the 6 s verdict timeout). A task spoken in that window was addressed to the room the owner had just left, and its result was posted there. The reviewer drove it at 1a6fddce: docked in a customer room, move to another room, speak during the wait, and the header carries the customer room. Now a frame naming a different room releases the bound room before the await. The released room is remembered only for change reporting, so a refused move still reads as `left` exactly once and a verified one as `entered`. A duplicate frame for the room already bound releases nothing. setVoiceSessionRoom clears the remembered room. Two tests. The room-to-room one fails without this change (36 pass, 1 fail) and passes with it (37 pass): a task written during a gated verdict carries channel_id: local-voice and the previous room nowhere in its header. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(voice): an offline room-less voice result is spoken on reconnect again The room-less offline forward writes the untagged proactive-result-<id>-<ts>.txt that any text bridge may deliver to the owner's DM. This branch had started adding that file to the drain's delivered set as it was written. On an install with no bridge to take the file (voice-only, or Telegram-only while last activity reads as voice), the drain's proactive-* fallthrough is the only delivery path, so the claim made the result undeliverable: nothing spoke it on reconnect and the file stayed in results/ for good. The untagged forward is now left unclaimed, which is main's behaviour. The two gateway-tagged shapes (.to-ag2space: the room post and the dm-only owner-DM post) keep their immediate claim, because the gateway bridge owns those and an unclaimed copy would be narrated a second time. The offline leg also scheduled its 10 s archive of the originating result before the forward had been written, so a failed write (disk full) archived a result nobody had received. The claim, forward and archive now live in _forwardOfflineThenArchive:…
What changed, and why
Stacked on #4393 (
feat/voice-session-room). The owner, 2026-09-18: "i should then be able to tell it lets talk in my DM and it switches to the DM, or take me to X room in X space and takes me there... so that its truly interactive".The room-bound voice session (#4393) can hear where it is (
session.context) but could not move: "let's talk in my DM", "take me to GTM in Investors", "go home" fell through towork/press_keyand nothing happened on screen. This PR adds one agent → client control frame, thenavigate_uiinline tool that sends it and waits for the desktop's answer, and the prompt line that routes those phrases to it. The desktop resolves room names (it owns the room list and the user's spaces); the agent only carries the spoken words.Frame contract (the webapp half is built in parallel against exactly this)
Server → client, over the voice WebSocket:
{ "type": "ui.navigate", "version": 1, "request_id": "nav-…", "target": "dm" | "room" | "home", "query": "GTM in Investors" }querytravels only fortarget: "room"(flattened to one line, capped at 200 chars).Client → server reply, same
request_id:{ "type": "ui.navigated", "version": 1, "request_id": "nav-…", "ok": true, "room_id": "!abc:ag2.space", "room_name": "GTM", "error": "not_found" | "ambiguous" | "unsupported", "candidates": ["GTM", "GTM planning"] }okis true exactly when noerroris given. Names are capped at 120 chars,candidatesat 8. On the agent side an unknownerrorvalue reads asunsupported, and a frame that claimsokwhile carrying anerroris not ok.Pure helpers exported from
src/web-voice-transport.tsfor both halves:buildUiNavigateFrame/parseUiNavigateFrame(request) andbuildUiNavigatedFrame/parseUiNavigatedFrame(reply), plusUI_NAVIGATE_TYPE,UI_NAVIGATED_TYPE,UI_NAVIGATE_TARGETS,UI_NAVIGATED_ERRORSand the caps.Where to look first
src/web-voice-transport.ts— the contract above, andVoiceTransportEvents.onServerFrame: every JSON frame the agent sends is handed to the surface before the transport interprets it. That is the hook the desktop uses to seeui.navigateand answer withsendClientCommand(buildUiNavigatedFrame(…)). Frame types the transport does not know stay ignored by the transport itself, as before.src/voice-navigate.ts(new) —navigate_ui {target, query?}. Sends oneui.navigatethroughsession.sendJsonToClient(the same outbound pathtask.statusandagent.stateuse), keeps the request in a pending map byrequest_id, waits at most 6 s for the matchingui.navigated. Every outcome is a result the model can speak, never a throw:{ok:true, room_name?};ambiguous+ candidates + "ask which one";not_foundnaming the spoken query;timeout;unsupportedimmediately, without sending, when no desktop client is attached ("navigation works in the desktop app"). The bodhi tool timeout is set above the wait so bodhi cannot kill the tool before its own timeout answers.src/voice-agent.ts— installs the client seam once the session exists (attached: () => session.clientConnected), routes client frames throughresolveUiNavigatedinonClientCommandright next to thesession.contexthandler, and fails pending navigations inonClientDisconnectedinstead of letting them sit out their timeout. After a successful move the desktop sends a freshsession.context, so the feat(voice): the voice session knows its room — tasks and results follow it, not the DM #4393 binding makes replies and delegated work follow the new room with no extra plumbing.src/inline-tools.ts—navigateUiTooljoinsinlineToolsandownerOnlyTools(likecancel_task).src/voice-agent-config.ts— one NAVIGATION rule: the trigger phrases,navigate_uioverwork/press_key, one short line on success ("Taking you to GTM."), read the candidates onambiguousand call again with the pick, "could not find a room called that" onnot_found, "navigation works in the desktop app" onunsupported/timeout.Behavior anchors regenerated (
ANCHOR_UPDATE=1 npx tsx tests/voice-behavior-anchors.test.ts) because the prompt changed: the diff is the newnavigate_uitool entry (twice — every owner-only inline tool appears in both lists) and theinstructions_hash_fixed_envline. Nothing else in the fixture moved.What user behavior this proves
Spoken "take me to GTM in Investors" →
navigate_ui {target:'room', query:'GTM in Investors'}→ui.navigateon the wire → desktop navigates and repliesui.navigated {ok:true, room_name:'GTM'}→ the model says "Taking you to GTM." → the desktop's nextsession.contextre-binds the session to that room. "Let's talk in my DM" →target:'dm'; "go home" →target:'home'. Two rooms matching → the model reads the candidates and asks; no client → "navigation works in the desktop app".Feature/documentation consistency
docs/src-map.mdregenerated withscripts/gen-src-map.pyfor the new module. No canonical doc underdocs/covers the voice client frame protocol (session.contextin #4393 added none either;docs/catalog.jsonhas no entry for it), so N/A beyond the src-map. The contract lives in the exported types and builders, which the webapp imports.Before / after
Before (parent
c7fd0df7): nonavigate_uiin the tool table, noui.navigateframe — the phrases route towork, nothing moves.After (HEAD):
Tests run
npm run test:ts(full suite):tests 1439 / suites 237 / pass 1436 / fail 0 / skipped 3, exit 0.tests/voice-navigate.test.ts(new): builder ↔ parser round trips for both frames; the bounds (query cap, name cap, candidate cap, query dropped for dm/home); parser rejections (other type, other version, bad target, missing id); tool outcomes — resolves on the client's ok reply,ambiguouswith candidates,not_found, times out cleanly and forgets the request (a late reply matches nothing),unsupportedwith no client and with a detached client (nothing sent), a throwing send, client-left fails every in-flight request, stray frames / unknown ids ignored.tests/web-voice-transport.test.ts:onServerFramereceives every JSON frame includingui.navigate, whole, ready for the client parser; unknown types still do not throw and reach no typed callback.npx tsc --noEmit: clean.npx eslinton every changed.ts: 0 errors (the warnings are the pre-existingno-explicit-anyones invoice-agent.ts/web-voice-transport.ts/ its test; none in the new files).bash scripts/review-checks.sh --diff <(git diff origin/feat/voice-session-room...HEAD):PARTIAL (hardcoded-paths + root-artifacts clean; prose-cap had no in-scope files)— no.pyin the diff.bash scripts/lint-workspace-resolution.sh --diff: clean.Live path
No live round trip yet: the desktop half that answers
ui.navigatedis in flight in cinny-webclient against this contract. Until it lands, a desktop that ignores the frame produces the 6 s wait and the "did not go through" answer; with no client attached (phone server, tests) the tool answersunsupportedimmediately and nothing is sent. I will attach the desktop round trip (spoken phrase → frame → room change →session.context) once both halves are built.Edge cases checked
Reply for an unknown or already-timed-out
request_id(ignored, returns false); asession.contextframe reaching the resolver (ignored); client disconnect with two requests in flight (both settleunsupportedat once, pending count 0);sendthrowing (settlesunsupported, no dangling timer); a reply that saysok:truewith anerror(treated as the error);candidatesthat is not an array (dropped); 20 candidates (capped at 8, blanks and non-strings dropped).Risks / rollback
Additive: a new tool, a new optional transport callback, a new frame type. Clients that do not know
ui.navigateignore it as they ignore any unknown frame today. Reverting the commit removes the tool and the prompt line; the anchors fixture reverts with it.Known gaps / follow-up
onServerFrame, resolve the query against rooms and spaces, navigate, reply) — separate PR in cinny-webclient.navigate_uiis owner-only, like the other desktop-control tools; guests in a room cannot move the owner's desktop.🤖 Generated with Claude Code