Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
a342c28
feat(pool): a follower can be installed as Codex instead of Claude
qingyun-ag2 Aug 23, 2026
6855ac6
fix(pool): kick-pool's idle guard fails closed, per runtime, from one…
qingyun-ag2 Aug 23, 2026
4329fe3
fix(pool): the codex nudge typed into any pane it did not recognise a…
qingyun-ag2 Aug 24, 2026
cba92c0
feat(pool): a completed task records how long it took
qingyun-ag2 Aug 25, 2026
278ea53
docs(pool): channel affinity's two gates, and a Server/identity desig…
qingyun-ag2 Aug 25, 2026
eaea17f
Merge remote-tracking branch 'origin/feat/pool-runtime-drive' into fe…
qingyun-ag2 Aug 25, 2026
2f58bf7
docs(pool): the docs describe the pool that is actually installed
qingyun-ag2 Aug 25, 2026
2c3625c
fix(pool): a codex follower is told where its workspace is, not left …
qingyun-ag2 Aug 25, 2026
0e3e4ca
rescue(pool): commit the live pool's uncommitted working state
qingyun-ag2 Aug 26, 2026
32d7b7b
fix(pool): rotate idle-pool tie-break + keep owner lane on claude seats
qingyun-ag2 Aug 26, 2026
c3ad3ef
feat(pool): binding room affinity — a room never switches cores while…
qingyun-ag2 Aug 26, 2026
4b737a9
fix(pool): only owner-lane assignments to non-lane seats rebind a room
qingyun-ag2 Aug 26, 2026
0e4a54d
feat(pool): change-driven liveness trace + owner-pick anomaly events
qingyun-ag2 Aug 26, 2026
602f0dd
feat(pool): explicit room bindings beat lane defaults on any alive seat
qingyun-ag2 Aug 26, 2026
0090886
fix(pool): busy cores keep unclaimed assignments — reclaim only wedge…
qingyun-ag2 Aug 26, 2026
6feeaa2
fix(pool): explicit pin to the lane core is not an anomalous pick
qingyun-ag2 Aug 26, 2026
15e44b6
fix(pool): busy deferral is bounded — a wedged core yields at the cap
qingyun-ag2 Aug 26, 2026
780bcbd
fix(pool): a sole claude seat stays owner-eligible while serving as l…
qingyun-ag2 Aug 26, 2026
1d9d996
docs: index lead-follower-pool.md and server-and-identity.md
qingyun-ag2 Aug 26, 2026
49013bc
fix(hooks): activity hook rides the canonical settings builder and re…
qingyun-ag2 Aug 26, 2026
0310be5
Merge feat/pool-operability into the rescue line — union of both pool…
qingyun-ag2 Aug 26, 2026
c6cd046
style: trim merged comment blocks to the 2-line cap
qingyun-ag2 Aug 26, 2026
760673e
fix(pool): a busy spell returns a claim grace instead of an expired c…
qingyun-ag2 Aug 26, 2026
3c4a238
test(hooks): resolved-workspace case runs in a fixture repo, not the …
qingyun-ag2 Aug 26, 2026
6423136
fix(pool): busy cap measures from assignment, not arrival
qingyun-ag2 Aug 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ Machine-readable ownership and lifecycle metadata lives in

## Start here

- [Lead-follower agent pool](lead-follower-pool.md) — multi-core pool design:
lanes, binding room affinity, reclaim semantics.
- [Sutando Server & the Agent ID card](server-and-identity.md) — server v0
composition and the identity-card contract.
- [Configure a workspace](workspace-config.md) — defaults, overrides, and
resolver APIs.
- [Run Codex as the core](codex-core.md) — core selection, setup, and rollback.
Expand Down
20 changes: 15 additions & 5 deletions docs/catalog.json
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
"last_verified": "2026-08-24"
},
"docs/benchmark-reports/2026-08-24-3a73e03.md": {
"title": "Sutando comprehensive benchmark \u2014 2026-08-24",
"title": "Sutando comprehensive benchmark — 2026-08-24",
"audience": [
"users",
"operators",
Expand All @@ -79,7 +79,7 @@
"last_verified": "2026-08-13"
},
"docs/census/d1-identity-census.md": {
"title": "D1 identity/state census \u2014 Discord / AG2 Space / Slack",
"title": "D1 identity/state census — Discord / AG2 Space / Slack",
"audience": [
"contributors",
"maintainers"
Expand Down Expand Up @@ -251,7 +251,7 @@
"last_verified": "2026-08-04"
},
"docs/runtime/claude-hook-contract-v1.md": {
"title": "Claude Code Hook Contract \u2014 v1",
"title": "Claude Code Hook Contract — v1",
"audience": [
"contributors",
"maintainers"
Expand All @@ -271,7 +271,7 @@
"last_verified": null
},
"docs/sparrow-v1-contract.md": {
"title": "ag2-sparrow v1 \u2014 the frozen delivery contract",
"title": "ag2-sparrow v1 — the frozen delivery contract",
"audience": [
"contributors",
"maintainers"
Expand Down Expand Up @@ -405,14 +405,24 @@
"last_verified": null
},
"docs/lead-follower-pool.md": {
"title": "Lead-follower agent pool \u2014 design",
"title": "Lead-follower agent pool — design",
"audience": [
"contributors",
"maintainers"
],
"status": "canonical",
"canonical_for": "lead-follower multi-core pool architecture (assignment, claim, recovery)",
"last_verified": "2026-08-26"
},
"docs/server-and-identity.md": {
"title": "Sutando Server & the Agent ID card — design",
"audience": [
"contributors",
"maintainers"
],
"status": "draft",
"canonical_for": "the server v0 composition (daemon, CLI, registry) and the agent identity-card contract it exposes",
"last_verified": "2026-08-26"
}
}
}
100 changes: 99 additions & 1 deletion docs/lead-follower-pool.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,49 @@ behavior, verbatim — and return to assignment-only the moment the lead's beat
is fresh again. Availability degrades to "no policy" rather than "no service",
and no consensus protocol enters the system. launchd restarts the lead.

## Channel affinity, as actually implemented (#884, `src/claim_task.py`)

The lead will subsume this policy, but the shipped mechanism is race-side and
worth stating exactly, because two of its properties are commonly misread.

**State.** One file per channel, `state/cores/channel-<id>.handler`, written
atomically (temp + rename) as `{"core_id": ..., "last_handled_at": <epoch>}`.

**Two independent gates.** A follower defers to the recorded handler only when
BOTH hold; they fail for different reasons and protect different things:

| gate | test | protects against |
|---|---|---|
| freshness | `now - last_handled_at < IDLE_THRESHOLD` (30 min) | a live core hoarding a channel it has stopped serving |
| liveness | `.alive` mtime `< ALIVE_THRESHOLD` (90 s) | a **dead** core stranding the channel |

Crash recovery therefore does **not** depend on the idle threshold: a dead
handler releases its channel in 90 seconds no matter how long that window is.
The idle threshold governs only rebalancing among cores that are still alive —
so raising it is far cheaper than it first appears.

**`last_handled_at` is claim-stamped, not activity-stamped.** It is written in
exactly one place — on a successful claim — and never refreshed while the task
runs or when it completes. The value means "when this core last *started* work
on this channel," which has a consequence worth designing around: a core that
spends longer than `IDLE_THRESHOLD` on a single task goes stale *while actively
working on that channel's task*, and the next message races freely to a core
with none of the conversation. The busy core stays perfectly **alive**
throughout — the heartbeat is a sidecar and keeps beating — so the failure is
not "busy mistaken for idle"; activity is never measured at all.

The minimal correction is to stamp completion as well as claim, converting the
window from "since work started" to "since work ended". Making the number
larger treats the symptom and leaves the wrong event being measured.

`SUTANDO_CORE_IDLE_THRESHOLD_SEC` overrides the window; non-numeric and
non-positive values fall back to the default rather than disabling affinity.

## What the lead's policy module owns (one place, finally)

- **Channel affinity:** sticky follower per channel, idle-timeout rebalance
(the #884 semantics, now a table in one process instead of race-side state).
(the #884 semantics above, now a table in one process instead of race-side
state — which is also the natural place to fix the claim-stamp defect).
- **Priority:** `urgent > normal > low` before mtime FIFO — the contract
`src/task_priority.py` documents but leaderless claiming could not honor.
- **Thread consolidation:** burst detection assigns the whole burst to one
Expand Down Expand Up @@ -162,3 +201,62 @@ claims. `SUTANDO_AFFINITY_BUSY_MAX` (default 3) sets how backlogged a channel's
handler must be before affinity yields — lower favors latency, higher favors
conversational continuity; `continuity_breaks` in the pool metrics measures
what that choice costs.

### Turning on a Codex follower

The runtime dimension (`--runtime` / `--core-runtime`) declares which CLI a core
runs. Two things make it usable on a running pool:

# convert core 2 to codex, in place — the lead and cores 1,3 keep working
bash scripts/install-core-pool.sh --only-core=2 --core-runtime=2:codex

# remove it again — plist, tmux session AND the stale beat, in one step
bash scripts/uninstall-core-pool.sh --only-core=2

`--only-core` converts a core in place but **cannot resize the pool** — it
refuses when the N given differs from the installed size. Adding a fourth core
is therefore a full install, which boots out and re-bootstraps every core and
the lead. That is less disruptive than it sounds: the tmux sessions outlive the
launchd jobs, so running sessions keep their context and the job is only a
supervisor. Plan for the churn anyway; nothing guarantees that ordering.

### Measuring the pool

`finish_task` appends one line per completed task to `data/pool-metrics.jsonl`:
`task_id`, `core`, `source`, `arrived_at`, `finished_at`, `duration_s`. Arrival
comes from the claimed file's mtime, which survives the assign/claim renames, so
the duration is measured rather than inferred.

Two fields are deliberately absent. `assigned_at` and `claimed_at` would need
the lead to stamp them — the renames preserve mtime, so those instants do not
exist by the time a follower finishes, and a field that cannot be populated
honestly is worse than no field.

This is the only durable record of pool timing. Result files are deleted once a
bridge delivers them, and archive mtimes are arrival times, not completion
times — so without this file, questions like "did anything wait on a busy core"
and "what does N=3 buy over N=1" are unanswerable after the fact.

Without `--only-core` the installer boots out the lead and every core, so
changing one follower restarts the whole pool. Without the teardown path,
`launchctl bootout` alone leaves the plist behind — the recovery sweep revives
any installed plist whose session is gone — and a left-behind
`state/cores/core-N.alive` keeps the lead assigning to a core that no longer
exists.

How the session is driven is owned by `scripts/pool-runtime-drive.sh`, sourced
by both `pool-core-wrapper.sh` (the in-session sweep) and `kick-pool.sh` (the
watchdog). Recognition is positive and fails closed: a session is typed into
only when the pane shows *that* runtime's idle prompt. A pane nobody recognizes,
a plist whose runtime is unreadable or unknown, and a codex startup dialog are
all skipped and logged — never typed into with the other runtime's text.

**Known gap — a Codex follower can look alive while missing work.** Its beat is
a sidecar bound to the pane pid, so `.alive` stays fresh regardless of whether
the model is reading anything. It has no in-session task watcher and
`task-notifier.sh` has no pool mode, so it learns about an assignment only when
something types the pool entry at it: the wrapper's own sweep (300s) or the
watchdog (180s on this host). Worst-case assignment latency is therefore the
sweep interval, not the sub-second watcher latency a Claude follower gets, and
a Codex core wedged *inside* a turn is invisible to both — `esc to interrupt`
reads as healthy work. Use assigned-but-unclaimed age, not `.alive`, to judge it.
180 changes: 180 additions & 0 deletions docs/server-and-identity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
# Sutando Server & the Agent ID card — design

**Status:** Design note, written 2026-08-24 from the shipped modules under
`src/runtime-api/`. Every motivation below is quoted or paraphrased from those
modules' own contracts, not reconstructed after the fact.

> **Naming caveat.** The phrase "ID card" appears nowhere in this repository.
> This document maps it to the **read-only identity surface** in
> `identity_view.py` — `sutando.info` / `status` / `owner` / `allowlist`, which
> that module calls "the Sutando Server *smallest slice*". If you meant
> something else by ID card, the second half of this doc is aimed at the wrong
> target and should be corrected before anyone builds from it.

---

## Part 1 — Sutando Server

### Motivation: an unattended agent still needs a human sometimes

A Sutando core runs long, alone, and with real capability. Two things follow
that a single process cannot solve for itself:

1. **It must be able to ask.** Approval before a governed action, elicitation
when input is missing. An agent that cannot ask either stalls forever or
proceeds without consent; both are failures.
2. **The asking must survive a crash.** If the process dies between "approval
granted" and "action performed", the system must know which side of that
line it was on. Ephemeral in-memory state cannot answer that question.

The Server exists to be the component that owns those two problems: a local
JSON-RPC daemon over a private Unix socket, with approval, elicitation and
capability execution as **durable** requests (SQLite), while discovery and
read traffic stay deliberately **ephemeral**.

### The load-bearing split: transport vs. request domain

`server.py` owns the daemon — socket ownership and permissions, framing,
connection lifecycle, supervision. `dispatcher.py` owns the request domain —
method dispatch, approval and elicitation validation, governed-capability
authorization, idempotency, durable state transitions, crash recovery.

The split is not tidiness. Its stated purpose:

> so the security policy is directly testable without a socket, and so a future
> transport cannot reimplement (or quietly diverge from) approval and
> capability behavior.

That second clause is the real motivation. A second transport — a different
socket, a remote bridge, a test harness — is exactly the kind of thing that
grows its own slightly-wrong copy of an authorization check. Putting policy
somewhere a transport cannot reach makes divergence structurally impossible
rather than merely discouraged.

### Security contracts that must not be "simplified"

The dispatcher's policies encode prior review findings and are marked as
contracts:

- Governed actions **fail closed** without an approved, unconsumed approval.
- An approval is bound to an exact **action + resource + input fingerprint**
and authorizes **exactly one** execution.
- Approval consumption and capability creation are **one transaction**.
- Recovery **never claims a side effect did not run**.

The last one deserves emphasis: after a crash the honest report is often
"unknown", and a recovery path that resolves ambiguity toward "didn't happen"
will eventually double-fire a real side effect.

### Identity is resolved daemon-side

> the actor is resolved DAEMON-SIDE from the environment
> (`SUTANDO_AGENT_ID` > `AGENT_MXID` > `AGENT_ID`), never from CLI-supplied
> params — a client cannot self-report who it is.

A client that can name itself can name someone else. Resolving identity at the
daemon removes the option rather than validating it.

### Why discovery must stay ephemeral

`capability_registry.py` is provider-neutral, receives read-only callables at
composition time, and **never receives a RequestStore** — "discovery/read
traffic must not become durable request state."

Motivation: durability is expensive and it is also a liability. A listing that
persists becomes a record to reconcile, expire, and leak. Only the things that
need to survive a crash — approvals, elicitations, executions — earn durability.

### The instance manifest: existence ≠ process

`instance_registry.py` states the distinction its whole design serves:

> **Agent existence ≠ agent process existence.**

One small, versioned, human-readable JSON per instance, which **never carries
tokens, keys, or memory**. The Server is its single writer: atomic write at
boot, `mark_stopped` on clean shutdown.

The deliberate part is the crash behaviour: a crash **leaves `status: running`
behind on purpose**, because *manifest-says-running + dead socket* is precisely
the `stale_or_crashed` signal. A manifest that tidied itself up on crash would
destroy the only evidence that a crash occurred.

And discovery reads must work **with no daemon running** — otherwise "is
anything installed here?" would be unanswerable exactly when you most need to
ask it.

---

## Part 2 — The Agent ID card

### Motivation: an agent has to be able to say who it is — and be doubted

Once agents talk to other agents, every interesting question is an identity
question. Who owns this thing? Who is allowed to address it? Is it even alive?
Is what it says about itself true?

The ID card is the answer to the first three and, crucially, is designed around
the fourth being **no**.

### What the card carries

Four fields, from `identity_view.py`, each sourced from existing workspace
records — "**nothing is invented here**":

| Field | Source | Meaning |
|---|---|---|
| `info` | daemon-resolved actor id + self descriptors | Who this agent is |
| `status` | `state/core-status.json` + own heartbeat age | What it is doing, how recently |
| `owner` | explicit ownership metadata per channel `access.json` | Who it answers to |
| `allowlist` | `channels/<source>/access.json` allowFrom, verbatim | Who may address it |

### Two design rules that carry the whole thing

**1. Absent fields stay absent — and an allowlist entry is NOT assumed to be
the owner.**

This is the card's most important line. Ownership and permission-to-speak are
different relations, and the cheap move — treating "is allowed to message it"
as "owns it" — manufactures authority out of access. The card refuses to
infer. A missing owner is reported missing, not guessed.

**2. Self-reported metadata is passed through; mtime is the only trust
signal.**

From `agents_view.py`: liveness is the heartbeat file's mtime, and the payload
is "passed through as self-reported metadata — mtime stays the only trust
signal."

So the card is explicitly **two-tier**: claims the agent makes about itself,
and one fact the filesystem asserts on its behalf. A graceful shutdown unlinks
the heartbeat, so absence means offline; a crash leaves it to age out.

### The failure this prevents

Without the separation, an agent's self-description and its provable state
merge into one blob that consumers treat uniformly — and every consumer
independently decides how much to trust it, usually by not thinking about it.
Naming the trust boundary in the data model means a reader cannot accidentally
promote a claim into a fact.

### Known sharp edge

Liveness proves the *beat*, not the *drivability*. The heartbeat is written by
a sidecar bound to the process, so a session wedged at its input layer
heartbeats perfectly while accepting no work. Anything gating on "is this agent
usable?" needs a work-derived signal — assigned-but-unclaimed age is the one
the pool uses — not the card's `status` field alone.

---

## Open questions

1. **Naming.** Confirm or correct the ID-card → `identity_view` mapping above.
2. **Card as a wire format?** Today these are RPC methods on a local socket.
If the card is meant to be presentable *between* agents, it needs a
serialization, a freshness rule, and an answer to "who vouches for it" —
none of which exist yet.
3. **Owner migration.** The `owner` field is a single scalar identity. An owner
who changes accounts has no supported rebind at the card level; see the
registry-side discussion for the same problem one layer down.
Loading
Loading