Skip to content

feat(mcp): inline email auth — sign in from an MCP client with no API key (#848) - #1707

Merged
obasilakis merged 16 commits into
devfrom
feature/848-mcp-inline-email-auth
Aug 5, 2026
Merged

obasilakis merged 16 commits into
devfrom
feature/848-mcp-inline-email-auth

Conversation

@obasilakis

Copy link
Copy Markdown
Contributor

Summary

Lets an external user sign in to Trinity from inside their MCP client with the existing 6-digit email code — no pre-minted API key, no web-UI visit. They install a keyless connector config, call request_login(email), then verify_login(code), and can use the exposed playbooks of every agent shared with that address.

Flag-gated MCP_INLINE_AUTH_ENABLED, default OFF. With it off, behaviour is byte-identical to today.

Implements #848 (Part B of abilityai/trinity-enterprise#118). Design signed off by Eugene on the issue, 2026-06-28.

Commits (reviewable in order)

87b2abb3 Prereq: tool-visibility gate → allow-list. Independently valuable
a22be001 Anonymous session tier + request_login/verify_login
24538559 Connector tools serve a key-bound or email-verified caller
d9722a8b Backend internal surface + docs

Please look hardest at these three things

1. INTERNAL_API_SECRET can now act as any verified email. This is the mechanism the "session, not a minted key" design requires — an email-verified session holds no credential, so the MCP server relays over /api/internal/mcp-auth/*. Every data call re-gates on db.email_has_agent_access(agent, email) + connector-enabled, so a compromised MCP server still cannot reach an agent the asserted email cannot. But this is a genuine widening of that secret's power and deserves a decision, not just a read.

2. A latent fail-open in the tool gate (87b2abb3). The operator gate was auth?.scope !== "connector" — it admitted every scope it had not heard of, including a null context. Not exploitable today (our authenticate throws rather than returning undefined), but it is a trap door sitting exactly where this feature steps, because fastmcp@4.4.0 #createSession does:

const allowedTools = auth ? this.#tools.filter(...canAccess...) : this.#tools;

Falsy auth skips filtering entirely. And only the stateless httpStream branch rejects an authenticate() returning undefined; the stateful branch we run does not. Now an explicit OPERATOR_SCOPES = {user, agent, system} allow-list.

3. Enumeration safety on request_login. One byte-identical body across known / unknown / malformed / rate-limited / backend-threw, and no audit row — wording, status, latency or an audit entry would each be an oracle for "is this address registered" (#186). /request is also not an open email relay: a code is only generated for an address Trinity already knows, and that lookup fails closed.

Design decisions worth knowing

  • Session, not a minted key, per the sign-off. Cost: FastMCP sessions are per-connection, so a client restart requires signing in again. Documented in requirements §7.6 rather than hidden.
  • The tool list is identical before and after login. A session's tools are resolved once at construction with no per-session refresh API, so gating visibility on login state would need a client reconnect. Login flips behaviour, not visibility.
  • scope stays "anonymous" after login — the session still holds no key and must never satisfy operatorOnly. Pinned by test.
  • The agent argument is a selector, not a grant. It picks among already-authorized agents; the backend re-gates every call, so a tampered session.agents cannot widen access.
  • A denial is a flat 403, not an access_requests write — diverging from the channel gate deliberately, because this gate runs per tool call and would be a spam vector. An explicit request-access affordance is noted as deferred.

Test plan

  • MCP server: 120/120 pass (npm test), tsc --noEmit clean. 30 new cases across tool-visibility.test.ts + inline-auth.test.ts
  • Backend: 33/33 new (pytest tests/unit/test_848_mcp_inline_auth.py)
  • Full backend unit suite deterministic (-p no:randomly): 4635 passed, 1 failed
  • That 1 failure (test_1474_read_boundary_z.py::test_schedules_summary_last_run_at_normalized) verified pre-existing — fails identically in a clean worktree at HEAD with zero changes from this branch
  • Randomized-order flake investigated and shown pre-existing: three runs on a clean tree produced 1 / 3 / 1+2-errors across test_1081_physical_meter.py and test_subscription_auto_switch_pingpong.py. This branch does not introduce it
  • Manual: set MCP_INLINE_AUTH_ENABLED=true, connect a keyless client, sign in end-to-end

Not run

/review and /cso --diff were not run as separate passes. Given this is auth code on a network-exposed surface that widens the internal secret, both are worth running before merge.

Closes #848

🤖 Generated with Claude Code

…eck (#848 prereq)

The operator-tool gate was `auth?.scope !== "connector"` — it admitted every
scope it had not heard of, including a context with no auth at all. Replace it
with an explicit `OPERATOR_SCOPES = {user, agent, system}` allow-list that
fails closed.

Not currently exploitable: our `authenticate` callback throws on a missing or
invalid key rather than returning undefined, so no falsy-auth session exists
today. It is a trap door sitting exactly where #848 must step — the natural way
to let an unauthenticated caller reach `request_login` is to return undefined
instead of throwing, and fastmcp@4.4.0 turns that into full operator exposure
via two independent behaviours:

  1. `FastMCP#createSession` skips filtering entirely for falsy auth:
     `auth ? this.#tools.filter(...canAccess...) : this.#tools`
     (dist/chunk-MDIESGNI.js:1762) — every registered tool is advertised and
     `canAccess` never runs.
  2. The stateful httpStream branch we run does NOT reject an `authenticate()`
     returning undefined; only the stateless branch guards it (:1640 vs :1690).

Blast radius had it been tripped: disclosure of the full operator tool catalog
(create_agent, delete_agent, inject_credentials, export_credentials,
get_credential_encryption_key, get_agent_ssh_access, ...). Execution of most
would still fail in `getClient()` for want of an mcpApiKey — but that is a
client-construction guard, not an authorization check, so any tool not routed
through it would genuinely run.

Renames `connectorDenied` -> `operatorOnly` across server/index/dynamic-agents;
the old name would actively misdescribe the new semantics on a security
predicate. ent#46 connector isolation is unchanged (`connectorOnly` untouched).

Adds src/tool-visibility.test.ts (9 cases) pinning: the three operator scopes
admitted, connector denied, an anonymous pre-login sentinel denied, unknown and
future scopes denied, absent auth denied under MCP_REQUIRE_API_KEY, dev-mode
semantics preserved, the two gates mutually exclusive, and the scope set itself
pinned so widening it must be deliberate. Each fail-open case also asserts the
legacy predicate admitted it, so the tests prove the fix changes behaviour.

99/99 mcp-server tests pass; tsc --noEmit clean.

Refs #848
First half of inline email auth. Flag-gated OFF (MCP_INLINE_AUTH_ENABLED),
so this commit changes no runtime behaviour.

- requirements/mcp.md §7.6 written first per RoE #1: credential model
  (session not key, and the per-connection lifetime that implies), the
  anonymous tier, why the tool surface is static across login, the internal
  backend path, whitelist bypass + why, and enumeration safety.

- McpAuthContext gains `anonymous` scope plus verifiedEmail / pendingEmail /
  sessionId. verify_login upgrades the session by mutating this object IN
  PLACE — FastMCP hands every tool the same reference, so the upgrade needs no
  library support. `scope` deliberately STAYS "anonymous" after login: the
  session still holds no API key and must never satisfy operatorOnly.

- authenticate(): an ABSENT Authorization header opens an anonymous sentinel;
  an INVALID key still throws. The sentinel is a truthy object with no
  `authenticated` key — fastmcp@4.4.0 skips canAccess entirely for falsy auth
  and rejects `{authenticated:false}` outright.

- Gates: `anonymousOnly` for the login tools, `connectorOrAnonymous` for the
  connector tools. The anonymous tool list is deliberately IDENTICAL before and
  after login — a session's tools are resolved once at construction with no
  per-session refresh API, so gating visibility on login state would need a
  client reconnect to take effect. Login flips behaviour, not visibility.

- tools/auth.ts: request_login / verify_login. request_login returns one
  constant body on every path (unknown address, malformed input, per-session
  limit, backend error) and emits no audit event — wording, timing or an audit
  row would each be an enumeration oracle (#186). Backend errors are swallowed
  for the same reason. verify_login failures are uniform (never wrong-code vs
  unknown-email). Per-session attempt caps sit on top of the backend limiters,
  which Telegram's inline /login lacks entirely.

- client.ts: requestInlineLoginCode / verifyInlineLoginCode bypass _fetch (an
  anonymous session has no bearer token) and authenticate the CALLER with
  X-Internal-Secret. The secret proves who is asking, never what they may do —
  the backend gates on email_has_agent_access.

Still to come: connector tools acting for a verified email, the four internal
backend endpoints, tests both sides, feature-flow doc.

tsc --noEmit clean; 99/99 mcp-server tests pass.

Refs #848
Completes the MCP-server half. Connector tools now handle two caller kinds
that resolve BOTH the agent and the backend credential differently:

  - connector key (ent#46): bound agent from the auth context, backend reached
    with the key. Unchanged.
  - email-verified anonymous session (#848): no key at all; the agent is chosen
    from the set shared with the verified email, and the backend is reached
    over the internal surface carrying that email.

The tools gain an OPTIONAL `agent` argument — unambiguous when one agent is
available, required when several. It selects among ALREADY-authorized agents
and is never a way to reach an unauthorized one: `session.agents` is a
convenience for defaulting and error messages, and the backend re-gates every
call on `email_has_agent_access`, so a stale or tampered list cannot widen
access. For a connector key the bound agent stays authoritative and a
disagreeing `agent` argument is REFUSED rather than ignored — silently reaching
a different agent than the client named is the worse failure.

A pre-login anonymous session is advertised these tools (the list is frozen at
session construction) and refuses to act, returning a structured
`login_required` rather than throwing.

Adds src/inline-auth.test.ts (21 cases) pinning the security-relevant
properties, not the cosmetic ones:
  - request_login is byte-identical across known / unknown / malformed /
    rate-limited / backend-threw — every enumeration path asserted equal
  - a malformed address is never relayed to the backend
  - verify_login upgrades in place but scope STAYS "anonymous" and no key is
    attached (it must never satisfy operatorOnly)
  - the verify response leaks no credential (asserts absence of
    trinity_mcp_/api_key/token/secret anywhere in the payload)
  - failures are uniform; a failed verify never binds the session
  - pre-login refusal on all three connector tools
  - an agent outside the authorized set dispatches nothing
  - the exposed-playbook allow-list still holds on the inline path
  - connector-key behaviour unchanged, incl. bound-agent mismatch refused

tsc --noEmit clean; 120/120 mcp-server tests pass.

Refs #848
Backend half. Four endpoints under /api/internal/mcp-auth (X-Internal-Secret,
reusing routers/internal.py's C-003 dependency verbatim), router → service → db
per Invariant #1, models in models.py per Invariant #14. No new tables — reuses
email_login_codes. No migration.

  /request    email a 6-digit code, iff the address is already known
  /verify     check the code, resolve reachable agents, audit the outcome
  /playbooks  exposed playbooks for a verified email
  /chat       one turn via TaskExecutionService, attributed to that email

The internal secret authenticates the CALLER, never the action. /playbooks and
/chat re-gate on email_has_agent_access + connector-enabled per call, so a
compromised MCP server still cannot reach an agent the asserted email cannot,
and nothing here ever returns a credential.

Security properties, each pinned by test:

- /request is not an open email relay. A code is generated only for an address
  with a users row or an agent_sharing entry. That lookup FAILS CLOSED — being
  unable to answer "do we know this address" must not degrade into "email
  anyone who asks".
- Every /request branch is byte-identical: 202 + {"status":"ok"} for known,
  unknown and rate-limited, send dispatched fire-and-forget (strong-ref set,
  the asyncio GC footgun), and NO audit row — an audit entry is itself an
  enumeration oracle, which is why routers/auth.py emits none either. A test
  asserts known.content == unknown.content and pins the key set to {"status"},
  so a future expires_in_seconds or per-branch message fails loudly.
- Rate limiting is keyed on EMAIL, not IP: every call arrives from the MCP
  server, so one IP bucket would be fleet-shared — useless as a limit and a
  trivial fleet-wide DoS. session_id is logged only, never a limiter key (a
  client picks its own session ids and could rotate past any cap).
- The data gate returns ONE 403 body for no-access / connector-disabled /
  no-such-agent; splitting them enumerates the fleet (Invariant #8).
- The gate runs BEFORE the idempotency claim on /chat, so an unauthorized
  caller cannot occupy a key slot for an agent it cannot reach.
- get_or_create_email_user is called with no role argument; a test asserts the
  created role is `user` (the #314 silent-promotion regression).
- The verify response is scanned whole for token/api_key/secret/bearer and its
  key set asserted exhaustively — a nested credential would fail, not just a
  top-level one.

Shared rather than duplicated: _fetch_live_playbooks moves out of
routers/connector.py into connector_service.fetch_live_playbooks, used by both
the owner route and the inline route. Verbatim move, no behaviour change; the
module docstring now admits it is no longer purely pure.

New db accessors: get_agents_shared_with_email (the existing get_shared_agents
is username-keyed, and inline auth must answer "is this address known" BEFORE
any user row exists) and list_connector_enabled_agents.

Docs: requirements §7.6 updated to match what shipped — the per-call denial is
a flat 403 and deliberately does NOT write an access_requests row, because that
gate runs per tool call and would be a spam vector (the channel gate runs once
per conversation, which is why it can). An explicit request-access affordance
is noted as deferred. feature-flows/mcp-connector.md gains the end-to-end Part B
flow. learnings.md records the two fastmcp findings from this work, including a
correction to a wrong claim in a dated security report.

.env.example documents MCP_INLINE_AUTH_ENABLED with the posture warning.

Tests: tests/unit/test_848_mcp_inline_auth.py, 33 cases.
Full backend unit suite deterministic (-p no:randomly): 4635 passed, 1 failed —
test_1474_read_boundary_z.py::test_schedules_summary_last_run_at_normalized,
verified PRE-EXISTING by running it in a clean worktree at HEAD with zero
backend changes, where it fails identically.

Refs #848
…cket DoS

/review with two independent adversarial passes (backend + mcp-server). Seven
findings fixed; two were confirmed empirically by the reviewers, not just read.

CRITICAL — cross-user disclosure on /chat
  The idempotency scope was `agent:{name}` with a caller-supplied key, so two
  different verified users of the same shared agent produced the SAME
  (scope, key): the second was served the first's stored response snapshot and
  execution_id. Reachable by accident as well as malice — MCP clients derive
  deterministic keys from call args, so two users asking one agent the same
  question collide by design. New `make_inline_auth_scope(agent, email)` folds
  the identity in. Every other `make_agent_scope` caller sits behind a per-user
  auth dependency; inline auth is the first where identity arrives in the BODY.
  The old test replayed with ONE identity, proving caching but never isolation —
  the new test asserts a different principal with the same key does NOT replay.

CRITICAL — the backend was not gated at all
  MCP_INLINE_AUTH_ENABLED existed only in the mcp-server. `include_router` was
  unconditional, so a surface that bypasses the email whitelist, creates
  accounts and dispatches chat answered on EVERY install. My own PR description
  claimed "flag-gated, default OFF" — false for the backend half. Now
  config.MCP_INLINE_AUTH_ENABLED + a router dependency that 404s the surface
  (404 not 403, so a disabled deploy does not advertise it), with a
  parametrized test over all four endpoints.

HIGH — /verify poisoned a shared per-IP lockout bucket
  client_ip is always the MCP server, so all users collapsed into one bucket.
  At 30 fails/5min, ONE anonymous client could lock inline login out fleet-wide
  and burn real web logins from the same egress — the platform-wide DoS #591
  removed, reintroduced structurally. Added account-only limiter variants; the
  IP bucket is never written from this path.

HIGH — /request timing oracle (measured 1.89x)
  The known branch did a committing INSERT the unknown branch did not, so
  identical bytes still leaked membership. All branch-dependent work moved
  behind Starlette BackgroundTasks (after the response is flushed).
  NOT asyncio.to_thread: that was tried and silently broke every send — SQLite
  is thread-affine, `_email_is_known` raised in the worker and its fail-closed
  handler swallowed it into "unknown address". Caught only because a test
  asserts the send fires.

HIGH — /request had no bound on the unknown-address branch
  The per-address cap sits behind the known-check, so unknown addresses were
  never counted. Added a coarse global ceiling, checked BEFORE the branch so it
  cannot itself become a differential; over-limit is a silent skip, same 202.

MEDIUM — mcp-server
  * `available.length > 0 &&` let ANY requested agent through when nothing was
    shared — the exact state meaning "you have nothing". Guard removed.
  * verify_login echoed upstream errors to an unauthenticated caller (raw HTTP
    statuses; the INTERNAL_API_SECRET env-var name when unset) and created a
    second distinguishable failure shape. Now one constant body, cause logged.
  * The per-session counter Map had no eviction and no session-close hook —
    unbounded growth from anonymous connections. Bounded, oldest-first.
  * Four new fetch calls had no timeout while the key-based path bounds itself,
    so a hung agent pinned an anonymous tool call forever. All bounded.
  * Audit rows for inline calls were unattributable: the new `agent` param was
    invisible to resolveTargetId, and run_playbook's `name` (a PLAYBOOK) was
    being stamped as target_type:"agent". Fixed, plus actor_email threaded
    end-to-end — note InternalAuditRequest would have SILENTLY DROPPED it
    (Pydantic extra='ignore'), so the field was added to the model, the router
    and platform_audit_service.log as a resolver fallback.

Also fixed in my own work, before the reviewers ran:
  tool-visibility.test.ts hand-copied OPERATOR_SCOPES from server.ts with a
  comment claiming it was "kept in sync" — it pinned its own copy, so widening
  the real set would have left it green. That is the drift trap already in
  learnings.md (2026-07-16). Predicates now exported and imported; guard proven
  to fire by temporarily widening the real set.

CORRECTED DOCS — a load-bearing claim of mine was wrong
  I wrote that FastMCP has "no per-session refresh API", and built the static
  tool-surface rationale on it. False: `toolsListChanged` re-filters LIVE
  sessions against current auth (dist:548-553), fanned by addTool/removeTool
  (:2202-2206) — which Trinity's own #846 reconciler fires every ~20s. The
  design stands, for the OPPOSITE reason: a login-keyed gate would flip
  non-deterministically at reconciler timing. Corrected in requirements §7.6,
  the feature-flow doc, server.ts, types.ts and connector.ts. Also noted:
  `updateAuth` REPLACES the auth object, which would discard the in-place
  upgrade entirely.

learnings.md: three entries — idempotency scope vs. caller identity,
to_thread-breaks-SQLite hidden by a fail-closed handler, and per-IP limiters
behind a single-egress proxy.

Tests: backend 41 (was 33), mcp-server 125 (was 120).
Full backend suite -p no:randomly: 4643 passed, 1 failed — the pre-existing
test_1474 failure, previously verified in a clean worktree at HEAD.

Refs #848
@obasilakis

Copy link
Copy Markdown
Contributor Author

/review complete — 7 findings fixed (50c3fd11)

Two independent adversarial passes (backend + mcp-server). Two findings were confirmed empirically by the reviewers, not just by reading.

Corrections to this PR's original description

Two things I asserted above were wrong. Both are now fixed, but the claims themselves were false when I wrote them:

  1. "Flag-gated MCP_INLINE_AUTH_ENABLED, default OFF" — true only of the mcp-server. MCP_INLINE_AUTH_ENABLED did not exist in the backend at all and include_router was unconditional, so a surface that bypasses the email whitelist, creates accounts and dispatches agent chat answered on every install regardless of the flag. Now genuinely gated (404, so a disabled deploy doesn't advertise it), with a parametrized test over all four endpoints.

  2. "A session's tool list is frozen at construction / no per-session refresh API" — false. FastMCPSession.toolsListChanged re-filters live sessions against current auth (dist/chunk-MDIESGNI.js:548-553), fanned by addTool/removeTool (:2202-2206) — which Trinity's own feat: per-agent MCP exposure flag — dynamic dedicated tool per exposed agent #846 reconciler fires every ~20s. The static-surface design stands, but for the opposite reason: a login-keyed gate would flip non-deterministically at reconciler timing. Corrected in requirements §7.6, the feature-flow doc, and three source files.

Critical

Finding
Cross-user disclosure /chat scoped idempotency as agent:{name} with a caller-supplied key, so two verified users of the same shared agent shared one (scope, key) — the second got the first's response snapshot and execution_id. Reachable by accident: MCP clients derive deterministic keys from call args. The old test replayed with one identity, proving caching but never isolation.
Ungated backend Above.

High

  • /verify poisoned a shared per-IP bucket. client_ip is always the MCP server, so all users collapsed into one bucket — 30 wrong codes from one anonymous client would lock inline login out fleet-wide and burn real web logins from the same egress. Exactly the DoS SEC: /api/token rate limiter is a platform-wide DoS primitive — 4 bad attempts locks out all users for 10 minutes (AISEC-H2) #591 removed. Now account-scoped only.
  • /request timing oracle, measured 1.89×. The known branch did a committing INSERT the unknown branch didn't, so identical bytes still leaked membership. Moved behind BackgroundTasks.
  • /request unbounded on the unknown branch — the per-address cap sits behind the known-check. Added a global ceiling, checked before the branch so it can't become its own differential.

Medium (mcp-server)

Empty-available agent passthrough · verify_login echoing backend errors (incl. the INTERNAL_API_SECRET env-var name) to unauthenticated callers · unbounded per-session Map · no timeouts on four fetch calls · unattributable audit rows (run_playbook's playbook name was being stamped as target_type:"agent").

Worth flagging: one fix broke the feature, silently

My first attempt at the timing oracle used asyncio.to_thread. Every code send then stopped: SQLite is thread-affine, _email_is_known raised in the worker, and its fail-closed handler swallowed it into "unknown address" — a total outage presenting as "the email never arrives", with a 202 still returned because the endpoint is deliberately non-signalling. Caught only because a test asserts the send actually fires. BackgroundTasks was the right primitive.

Tests

Backend 41 (was 33) · mcp-server 125 (was 120) · full backend suite -p no:randomly: 4643 passed, 1 failed (the pre-existing test_1474, previously verified in a clean worktree).

Three learnings.md entries: idempotency scope vs. caller identity; to_thread breaking SQLite hidden by a fail-closed handler; per-IP limiters behind a single-egress proxy.

Still open

AC6 is not implemented — "the default .mcp.json snippet in docs / UI works without a pre-filled API key." build_snippets requires a key and always embeds it; no keyless variant exists, and no UI surface hands one over. The config a collaborator needs is trivial ({"mcpServers":{"trinity":{"type":"http","url":"<mcp_url>"}}}) but is documented nowhere. Every mechanism behind it works — an owner just has no supported way to hand someone a working config. Needs a decision: build the snippet variant, document it only, or split to a follow-up.

/cso --diff still not run.

@github-actions

Copy link
Copy Markdown

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

AC6: "the default .mcp.json snippet in docs / UI works without a pre-filled API
key." Delivered in both surfaces.

- connector_service: `build_keyless_snippets` + a shared `_client_snippets`
  builder that `build_snippets` now also uses, so the keyed and keyless
  variants cannot drift. The keyless config is the keyed block minus the
  Authorization header, with a note pointing at request_login/verify_login.
- ConnectorStatus gains `inline_auth_available` + `keyless_snippets`, populated
  by the connector router ONLY when config.MCP_INLINE_AUTH_ENABLED is on — an
  anonymous MCP session is rejected otherwise, so offering a keyless config on a
  disabled install would be dead setup instructions. Independent of has_key:
  the keyless flow is an ALTERNATIVE to minting a key, so it shows with or
  without one.
- ConnectorChannelPanel.vue: a "Share without a key — sign in by email" block,
  gated on status.inline_auth_available, reusing the existing copy affordance.
  The UI reads the flag off the connector status it already fetches — no new
  endpoint, and NO feature-flags entry (an earlier draft added one; it was
  redundant with inline_auth_available and out of the issue's scope, so it was
  dropped).
- docs: the keyless config block + flow in feature-flows/mcp-connector.md and
  requirements §7.6.

Tests: keyless snippets carry no Authorization and keep client parity with the
keyed set; the status endpoint offers keyless ONLY when the flag is on, and does
so even with no key. Frontend builds clean.

Closes the last open box on #848.

Refs #848
7 attack axes independently refuted (default-safe posture, internal-secret
authz, cross-user replay, enumeration/relay, anon->operator reach, verify_login
disclosure, audit integrity). Verifies the 7 /review fixes hold. Two
informational accepted-risk notes (internal-secret widening; pre-existing
per-account bucket). Report: docs/security-reports/cso-2026-07-21.{json,md}.

Refs #848
…o merge/848-v0.8.5

# Conflicts:
#	docs/memory/learnings.md

@AndriiPasternak31 AndriiPasternak31 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated via /validate-pr. Requesting changes — the blockers are mergeability and review coverage, not defects I found in the code. To be explicit about scope: I ran the process gate (traceability, docs tiering, security/packaging scans, migration parity, test presence), not a line-by-line read of the auth path, so treat the absence of code findings below as "not yet reviewed", not "reviewed and clean".

What passes

Closes #848 resolves to an open P2 type-feature status-in-progress issue, so the merge-time promotion will fire. Docs are updated on both required surfaces — docs/memory/requirements/mcp.md and docs/memory/feature-flows/mcp-connector.md. A named test file (tests/unit/test_848_mcp_inline_auth.py) is present. No schema change, so the dual-track migration rule (#1183) doesn't apply. Secrets/email/IP scans clean; no new top-level backend module or os.getenv(), so neither packaging gap applies.

Blocking

1. Conflicting with dev. Good news: the only conflict is docs/memory/learnings.md — everything else auto-merges. That makes this the cheapest of the three conflicting PRs to unblock. Worth doing promptly, because until it rebases GitHub reports no checks at all (a conflicting PR has no refs/pull/N/merge, so CI has nothing to build), and right now there is zero CI signal on a new authentication path.

2. Needs /cso --diff before it lands, not just /review. The pipeline table puts a P2 feature at review + validate, CSO recommended — but the recommendation there is scoped by risk, and this PR adds a new sign-in path reachable from an MCP client with no pre-existing API key. That is precisely the "unless auth/security" carve-out. At +3538/−129 across 34 files it's also the largest auth change on the board; under the 50-file split threshold, but large enough that a structural pass is worth its cost.

3. No reviews at all yet. This has been open since 20 Jul with nothing recorded against it.

Sequence I'd suggest: rebase (one docs file) → let CI run → /review + /cso --diff → re-request. Happy to take the re-review.

@obasilakis

Copy link
Copy Markdown
Contributor Author

Unblocked — merged, green, re-requesting review

Thanks for the process pass, and for being explicit about its scope — that distinction is what makes blocker 3 actionable rather than decorative.

1. Conflict — fixed

Merged dev in at f03cd5cb. Note this is the current dev tip, not the v0.8.5 point the branch had been sitting behind — so what CI just built is this feature against everything that has landed since 20 Jul, not against a snapshot. As you predicted, docs/memory/learnings.md was the only conflict.

23/23 checks pass (2 skipping: auto-merge, report). That is the first CI signal this branch has ever had, so the "zero CI on a new authentication path" gap is closed — including all six pytest shards, prod-image-smoke, pg-migrations and schema-parity.

Post-merge verification beyond CI: tests/unit/test_848_mcp_inline_auth.py 41/41, mcp-server suite 125/125.

2. /cso --diff — already done, six days before your review

This one was already satisfied when you wrote the review — e0bb25a5, 21 Jul, "CSO --diff audit of #848 — no findings above gate". The report is in this PR's own diff: docs/security-reports/cso-2026-07-21.md + .json. Easy to miss among 34 files, and I'd rather point at it than have you spend the cycle re-running it.

To be clear about what that audit covered, so you can judge whether it's still sufficient: it ran against e0bb25a5, which is pre-merge. If you want it re-run against the post-merge tree before you sign off, say so and I'll do it — but nothing in the merge touched the auth path, so I don't expect a delta.

I'm not arguing the carve-out — you're right that a new sign-in path reachable with no pre-existing API key is exactly the "unless auth/security" case. It just happens to have been paid already.

3. No reviews — still open, and it's the real one

Agreed, and this is the item I can't close myself. The only code review on record is my own /review (7 findings, fixed in 50c3fd11 — the two that matter are a cross-user idempotency replay on shared agents and a backend surface that was ungated despite the PR description claiming otherwise; both are called out in the comment above, including the two false claims in my original description).

Re-requesting your review. Taking you up on the offer.

Worth your attention in particular, since they're where the design load sits:

  • services/mcp_auth_service.py — the enumeration-safety contract. /request answers one constant 202 with no audit row and, deliberately, no branch-dependent work on the request path at all (the known-check, the cap read and the code INSERT all run in a BackgroundTasks task after the response is flushed, because a committing write on only one branch measured ~1.9x even on in-process SQLite). If that reasoning is wrong, it's wrong in a way I can't see from inside it.
  • assert_email_may_reach_agent — the per-call gate. The internal secret authenticates the caller, never the action; every data call re-checks the asserted email's own standing. This is the load-bearing claim of the whole design and deserves an adversarial read.
  • src/mcp-server/src/tools/auth.ts — session upgrade is an in-place mutation of the object FastMCP hands every tool, and scope deliberately stays "anonymous" so an inline-verified session can never satisfy operatorOnly.

@AndriiPasternak31 AndriiPasternak31 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-validated via /validate-pr. One blocker, four lines. Prior blockers (conflict,
missing /cso pass) are cleared — and the CSO commit e0bb25a5 is the last
substantive one, so it covers the whole final diff.

Verified by execution, not by reading: MCP npm test 125/125 + tsc --noEmit
clean; backend pytest unit/test_848_mcp_inline_auth.py unit/test_118_mcp_connector_oss.py
56 passed; all 23 CI checks green including the base/head regression diff
across 3 seeds.

Blocking

  • Wire MCP_INLINE_AUTH_ENABLED into backend.environment AND
    mcp-server.environment in BOTH docker-compose.yml and
    docker-compose.prod.yml
    — four wirings, because two processes read it
    (config.py:173, server.ts:201).

    Not a grep result — this is compose's own resolution with the flag
    explicitly set by the operator:
    
    ```
    $ MCP_INLINE_AUTH_ENABLED=true docker compose -f <file> config
    docker-compose.yml       backend      MCP_INLINE_AUTH_ENABLED     = <<ABSENT>>
                             mcp-server   MCP_INLINE_AUTH_ENABLED     = <<ABSENT>>
    docker-compose.prod.yml  backend      MCP_INLINE_AUTH_ENABLED     = <<ABSENT>>
                             mcp-server   MCP_INLINE_AUTH_ENABLED     = <<ABSENT>>
       (sibling MCP_AGENT_CHAT_PULL_ENABLED = false in all four, i.e. wired)
    ```
    
    Compose reads `.env` only for `${...}` interpolation, never to inject into a
    container; there is no `env_file:` and no Dockerfile `ENV` for it either. So
    `require_inline_auth_enabled` 404s the whole `/api/internal/mcp-auth/*`
    surface, the MCP server rejects keyless connections and registers no auth
    tools, and `keyless_snippets` never appears — permanently, with no operator
    lever. It fails *safe* (it makes the CSO report's axis-1 default-safe posture
    unconditional rather than opt-in), so this is not a security issue. It's that
    3.5k lines ship unable to be switched on. Copy the sibling one line up:
    `- MCP_AGENT_CHAT_PULL_ENABLED=${MCP_AGENT_CHAT_PULL_ENABLED:-false}`.
    
    Corroboration: the one unchecked box in the test plan is
    `[ ] Manual: set MCP_INLINE_AUTH_ENABLED=true, connect a keyless client` —
    the only step that would have caught this. Neither `/verify-local` nor CI can:
    both boot at defaults, so a flag that can never be turned on boots clean and
    goes green.
    

Should land with it

  • architecture.md entries for the new surface: routers/mcp_auth.py in the
    router catalog, services/mcp_auth_service.py in the service catalog,
    tools/auth.ts + a request_login/verify_login row in the MCP tools table
    (Invariant #13 makes that table the canonical third-surface record), and the 4
    /api/internal/mcp-auth/{request,verify,playbooks,chat} endpoints.

    **Do not hand-edit the module counts.** They are already stale independent of
    this PR — the doc says 63 routers / 66 services / 22 tool modules; the tree has
    68 / 96 / 29. Reconciling those is `/validate-architecture` drift and belongs in
    its own pass, not here. (I got this wrong in my first read of this PR and asked
    for `63→64`; that would have written a still-wrong number.)
    
  • MCP_INLINE_AUTH_TIMEOUT_MS → .env.example (+ compose for consistency).
    Functional via its 15000 default, so lower severity — same class as the blocker.

  • feature-flows.md gets no Recent Updates row despite a substantial
    mcp-connector.md rewrite.

  • Correct the fastmcp version citation, and test call-time enforcement.
    The code comments in server.ts and the CSO report both reason about
    fastmcp@4.4.0 internals with line citations (dist/chunk-MDIESGNI.js:548-553,
    :2202-2206, :572-574) — but package.json pins ^4.8.0 and
    package-lock.json resolves 4.8.0. The reasoning was verified against a
    version the project does not install.

    I re-verified both load-bearing behaviours on 4.8.0, and **the conclusion is
    unchanged**:
    - The falsy-auth filter-skip is still there (`chunk-JEOCXSB4.js:1857`
      `const allowedTools = auth ? this.#tools.filter(...) : this.#tools`), so the
      `87b2abb3` hardening is genuinely warranted on the shipped version.
    - Enforcement is real, not advertisement-only — though via a different
      mechanism than "checked at `tools/call`": `setupToolHandlers(tools)` closes
      over `toolsMap = new Map(tools.map(...))` built from the *filtered* list, and
      the `CallToolRequestSchema` handler does `toolsMap.get(name)` →
      `throw McpError(MethodNotFound)`. A filtered-out tool is absent from the call
      map, not merely hidden. `canAccess` itself is never re-invoked per call.
    
    Worth a test: `tool-visibility.test.ts` exercises the predicates in isolation
    only, so nothing in the suite would catch a future fastmcp change to
    advertisement-only filtering — the exact regression the allow-list defends
    against.
    

Non-blocking

  • Surface mcp_inline_auth_enabled on GET /api/settings/feature-flags; both its
    closest siblings (mcp_agent_chat_pull_enabled:183, redelivery_governor_enabled:188)
    are there as observability-only entries — useful while a posture-changing flag soaks.
  • CSO report filename is cso-2026-07-21.md; convention is cso-diff-YYYY-MM-DD.md.
  • Carry the report's N1 note (set INTERNAL_API_SECRET explicitly in prod rather than
    the → SECRET_KEY fallback) into the release notes now that this surface exists.
  • The anonymous scope is missing from the scope table at architecture.md:1967 — but
    so is connector already, so this is pre-existing practice, not a gap this PR
    introduced. Worth adding since anonymous is the security-relevant tier.
  • The /request constant-time claim holds for time-to-response, but a Starlette
    BackgroundTask runs before the connection is released for keep-alive reuse, so with
    a pooled client the known/unknown delta is in principle observable on the next
    request over the same connection. Very low signal and the global cap bounds probing —
    not worth changing, noted so the claim's boundary is on record.

Scope of this review

Process gate + a targeted read of the auth path + the library-behaviour checks above.
I did not run an independent adversarial security pass — the CSO artifact in this PR
is the author's own, and my agreement with it is read-level. If independent assurance is
wanted on a keyless auth surface, a fresh-context /cso --diff is the instrument;
/verify-local is not (nothing here is in its blast radius: no base-image, no deps, no
migrations, no new top-level backend module).

The hardening in 87b2abb3 is correct and independently valuable.
make_inline_auth_scope folding the verified email in is right — MCP clients derive
deterministic keys from call args, so two users of one shared agent collide by design,
not just by malice. resolveAgent's refusal to guard on available.length > 0 is right
for the same reason. Test coverage is the strong part: seven non-happy-path classes
(enumeration safety, verify failure modes, access gate, cross-user replay, flag gate,
internal-secret gate, global rate limit) — no surface left happy-path-only.

…citations (#848)

Addresses the /validate-pr review on #1707.

BLOCKING — the flag reached neither container. MCP_INLINE_AUTH_ENABLED was
defined in config.py, read in server.ts, documented in .env.example, gated at
the router and covered by a parametrized endpoint test — and wired into zero
compose services. Compose reads .env only for ${...} interpolation, never to
inject into a container, and there is no env_file: or Dockerfile ENV, so the
whole feature was permanently off with no operator lever. Now wired in all four
places (backend + mcp-server x dev + prod), verified with compose's own
resolution rather than grep.

It fails safe, which is why nothing caught it: CI and /verify-local both boot at
defaults, so a flag that can never be enabled boots clean and goes green. The
durable fix is tests/unit/test_848_inline_auth_compose_wiring.py — a static
packaging guard (precedent: test_1489_vite_bug_build_args.py) asserting all four
wirings, the ${VAR:-false} shape (a hardcoded value is the same un-switchable
bug with extra steps; a non-false default would make a network-exposed keyless
path opt-out), and the reader set itself, so a third process reading the flag
without wiring it fails here. Mutation-checked: removing one wiring fails 2.

fastmcp citations were wrong, twice over. The reviewer caught 4.4.0 vs the
pinned 4.8.0; the dev merge has since bumped to 4.12.1, so 4.4.0 was only what a
stale local node_modules held. Re-verified both load-bearing behaviours against
the version package-lock actually resolves — the falsy-auth filter-skip is still
present, so the 87b2abb allow-list hardening is warranted — and corrected every
citation to symbol-first with the line as an "as of" locator, since the minified
chunk filename moves between releases. Also corrected the mechanism: canAccess is
never re-invoked per call; enforcement is by absence from the toolsMap
setupToolHandlers builds from the filtered list.

Per the review, that assumption is now pinned by a test instead of prose:
tool-visibility.test.ts boots a real FastMCP server and drives it with a real MCP
client, asserting a filtered-out tool is not merely hidden from tools/list but
rejected on call AND that its body never executes. Behavioural, so it survives
chunk renames — the churn that made the citations wrong. Mutation-checked
against the pre-#848 deny-check.

Corrected a live error in learnings.md: the entry claiming canAccess filtering
happens "once at session construction, so any log-in-and-new-tools-appear design
needs a client reconnect". toolsListChanged re-filters live sessions and the #846
reconciler fires it every ~20s. The static-surface design stands for the opposite
reason — a login-keyed gate would flip non-deterministically at reconciler
timing. New entry for the compose-wiring class.

Docs (Invariant #13 third-surface record): architecture.md gains routers/mcp_auth.py,
services/mcp_auth_service.py, the tools/auth.ts row, the four /api/internal/mcp-auth/*
endpoints, and the missing `connector` + `anonymous` scope rows. Module counts left
alone — already stale independent of this PR, and reconciling them is
/validate-architecture drift. feature-flows.md gains its Recent Updates row.

Also: MCP_INLINE_AUTH_TIMEOUT_MS documented and wired (mcp-server only — the
backend never reads it, asserted); mcp_inline_auth_enabled surfaced on
GET /api/settings/feature-flags as observability, mirroring its two siblings and
giving operators a post-deploy check that the two halves agree; CSO report
renamed to the cso-diff-DATE-issue convention; CSO N1 (set INTERNAL_API_SECRET
explicitly rather than relying on the SECRET_KEY fallback) carried into
.env.example and requirements §7.6 — there is no unreleased-notes file, and
.env.example is where an operator looks when enabling the flag.

Verified: backend unit 5620 passed / 0 failed (the previously-known pre-existing
test_1474 failure now passes, fixed by the dev merge); mcp-server 127/127 (was
125) and tsc --noEmit clean, both against the real 4.12.1 after npm ci.
…-email-auth

# Conflicts:
#	docs/memory/architecture.md
#	docs/memory/feature-flows.md
#	docs/memory/learnings.md
…a known landmine

CSO --diff pass over the post-review branch. 0 findings above the daily 8/10
gate; 3 informational, 1 fixed here.

The re-run is not a formality. The 2026-07-21 audit assessed gates that were
correct but UNREACHABLE — MCP_INLINE_AUTH_ENABLED reached zero containers, so
its "default-safe posture" was unconditional rather than opt-in. With the four
wirings in place the surface is genuinely reachable on an opt-in deploy, so the
live controls were re-verified against code rather than re-read from the prior
report: router-level gating of all 4 routes, assert_email_may_reach_agent
running before the idempotency claim and failing closed, uniform 403 on both
denial reasons, account-only limiters that never touch _ip_key, constant-time
/request with everything branch-dependent in BackgroundTasks, name-only
build_tool_description (#846 precedent), and the OPERATOR_SCOPES allow-list.
Parity/uniformity guards: 38 passed.

N2, found and fixed: routers/mcp_auth.py claimed the branch-dependent work runs
"in a worker thread". It does not and must not — asyncio.to_thread there
silently stops every code send, because SQLite connections are thread-affine so
_email_is_known raises in the worker and its own fail-closed handler swallows
that into "unknown address" while the endpoint keeps answering 202. The router
is the file a maintainer opens first, so the natural "make these consistent"
edit was to reintroduce a total silent outage. Not a security finding — the
invited failure is availability, not a bypass — but a real defect sitting on the
enumeration-safety contract. Now names BackgroundTasks and says why a thread is
wrong.

N1 (INTERNAL_API_SECRET can act as any verified email) carried forward
unchanged and still accepted; its operational recommendation is no longer
theoretical now the surface is reachable, and was carried into .env.example and
requirements §7.6 in the previous commit. N3: 4 transitive npm vulns in
src/mcp-server, all isDirect:false, pre-existing on dev and untouched by this
PR — noted because package.json already overrides fast-uri, so the pin is
evidently not resolving the advisory.

Stated plainly in the report: this is the author auditing their own change on a
keyless auth surface, same caveat as the prior audit. Independent assurance
needs a fresh-context run by someone who did not write it.
@obasilakis

Copy link
Copy Markdown
Contributor Author

Feedback addressed — blocker fixed, /cso --diff re-run, and one thing you should know about the fastmcp citation

Thanks for both passes, and specifically for measuring the flag rather than grepping it. That blocker was real and I would not have found it — every signal I had was green, and the one step that would have caught it is the box I left unchecked.

Blocking: MCP_INLINE_AUTH_ENABLED — fixed, all four wirings

Verified the way you found it, with the flag explicitly set:

docker-compose.yml       backend      MCP_INLINE_AUTH_ENABLED = true
docker-compose.yml       mcp-server   MCP_INLINE_AUTH_ENABLED = true
docker-compose.prod.yml  backend      MCP_INLINE_AUTH_ENABLED = true
docker-compose.prod.yml  mcp-server   MCP_INLINE_AUTH_ENABLED = true

You were also right that neither CI nor /verify-local can catch this class — both boot at defaults, so a flag that can never be turned on boots clean and goes green. So the fix isn't just the four lines: tests/unit/test_848_inline_auth_compose_wiring.py (11 cases) asserts all four wirings, the ${VAR:-false} shape (a hardcoded value would be the same un-switchable bug with extra steps; a non-false default would make a network-exposed keyless path opt-out), and the reader set itself — so a third process reading the flag without wiring it fails there. Mutation-checked: removing one wiring fails 2 tests.

Worth flagging for the team beyond this PR: #1876 landed on dev this same week for exactly this class (CANARY_ENABLED/CANARY_SLACK_WEBHOOK_URL never reaching the prod backend), joining #1039, #1056 and ent#31. Four occurrences says the per-PR /validate-pr grep isn't catching it — and I think I see why: that check greps for new backend os.getenv(), so it structurally misses a flag whose gate lives in the mcp-server (process.env). That's how this one passed the gate you ran. Recorded in learnings.md; the generalized guard is worth its own issue rather than more per-PR greps.

The fastmcp citation — you were right, and it had rotted one version further than you found

You caught 4.4.0-vs-4.8.0. By the time I looked, the dev merge had bumped the lock to 4.12.1 — and 4.4.0 turned out to be what my stale local node_modules held, not any version the project ever resolved. So the reasoning was against a version nobody installs, twice over.

I re-verified both load-bearing behaviours on the actually-installed 4.12.1 after npm ci, and your conclusion holds exactly: the falsy-auth filter-skip is still there (chunk-5BQXF2VT.js:1998-2000), so the 87b2abb3 hardening is warranted on the shipped version. Your mechanism correction was also right and I've adopted it — canAccess is never re-invoked per call; enforcement is by absence from the toolsMap that setupToolHandlers builds from the filtered list (:1181 → :1214-1220). All citations now lead with the symbol and treat the line as an "as of" locator, since the chunk filename moves between releases.

On your "worth a test" — agreed, and that's the actual fix here. Prose citations rot at the next dependabot bump; this one rotted twice before anyone noticed. tool-visibility.test.ts now boots a real FastMCP server and drives it with a real MCP client, asserting a filtered-out tool is rejected on call and that its body never executes. Behavioural, so it survives chunk renames. Mutation-checked against the pre-#848 deny-check.

That investigation also turned up a live error in learnings.md — the entry asserting canAccess filtering happens "once at session construction, so any log-in-and-new-tools-appear design needs a client reconnect." That's false (toolsListChanged re-filters live sessions; the #846 reconciler fires it every ~20s), and it's the second false claim this same investigation produced. The static-surface design stands, for the opposite reason. Corrected, with the generalizable trap written up: a design justified by an impossibility keeps looking correct long after the impossibility is disproved.

Should-land items

  • architecture.md — routers/mcp_auth.py, services/mcp_auth_service.py, the tools/auth.ts row in the MCP tools table (Invariant feat: SMARTS trading pipeline with Telegram notifications and Miro visualization #13), all 4 /api/internal/mcp-auth/* endpoints, and the scope table gains connector + anonymous. Module counts untouched per your instruction. portal_delegate (ent#163) landed on dev mid-merge so the union carries all three — and it's the cleanest illustration of why the deny-check shape was untenable: not an MCP tool principal at all, yet !== "connector" would have advertised it every operator tool the day it appeared.
  • MCP_INLINE_AUTH_TIMEOUT_MS — documented + wired, mcp-server only. The backend never reads it, so wiring it there would encode a wiring that shouldn't exist; the test asserts that asymmetry holds.
  • feature-flows.md — Recent Updates row added.
  • mcp_inline_auth_enabled on feature-flags — added, mirroring both siblings. It's also the cheap post-deploy check that the two halves agree, which is the confirmation that was missing here.
  • CSO filename → cso-diff-2026-07-21-848.md (-848 matches cso-diff-2026-05-12-678.md precedent).
  • N1 → carried into .env.example and requirements §7.6. There's no unreleased-notes file (/release generates at cut time), and .env.example is where an operator actually looks when enabling the flag.

/cso --diff re-run: cso-diff-2026-07-30-848.md

0 findings above the 8/10 gate. Not a formality — as you implied, the prior audit's default-safe posture was unconditional rather than opt-in because the flag couldn't be turned on. Now it can, so every gate previously "correct but unreachable" is live and I re-verified it against code: router-level gating of all 4 routes, assert_email_may_reach_agent before the idempotency claim and fail-closed, uniform 403 on both denial reasons, account-only limiters that provably never touch _ip_key, constant-time /request, name-only build_tool_description (#846), OPERATOR_SCOPES. Guards: 38 passed.

One real defect found and fixed (N2). routers/mcp_auth.py claimed the branch-dependent work runs "in a worker thread." It does not and must not — asyncio.to_thread there silently stops every code send (SQLite is thread-affine, _email_is_known raises in the worker, its own fail-closed handler swallows it into "unknown address") while the endpoint keeps returning 202. The router is the file you open first, so the natural "make these consistent" edit was to reintroduce a total silent outage. Not a security finding — the invited failure is availability, not a bypass — but a landmine on the enumeration-safety contract, and it was there because I wrote the router docstring before reverting the to_thread attempt.

Stated plainly in the report: this is me auditing my own change on a keyless auth surface. Same caveat as the 07-21 artifact. You named this as the outstanding gap and it remains outstanding — a fresh-context /cso --diff by someone who didn't write it is the instrument, and I can't self-serve it.

Also

Re-merged dev (it had moved 11 commits and gone CONFLICTING again). Three doc conflicts, all append-point collisions, resolved in date order.

Also closing your /review follow-up: AC6 is implemented — build_keyless_snippets + the ConnectorChannelPanel.vue surface exist and are flag-gated. My earlier "AC6 is not implemented / needs a decision" note is stale; no decision needed.

Verified: backend unit 5795 passed / 0 failed (the pre-existing test_1474 failure I'd flagged now passes — fixed by the dev merge, so that caveat is retired). mcp-server 127/127 (was 125) + tsc --noEmit clean, both against the real 4.12.1 after npm ci.

One note on the #1360 "newest ~20" cap in feature-flows.md: it's at 57 rows. Pre-existing drift, same category as the stale module counts — left alone.

Re-requesting review.

…-email-auth

Conflicts were both append-at-same-position collisions in the memory docs,
resolved by keeping both entries:

- docs/memory/feature-flows.md — #848 and #1880 both added a 2026-07-30 row to
  the newest-first "Recent Updates" table. #848 placed above #1880.
- docs/memory/learnings.md — #848 and #1880 both appended a 2026-07-30 pitfall
  section to the newest-last ledger. #848 placed after #1880.

No code conflicts; src/backend/services/canary_alerts.py and the #1880 tests
auto-merged from dev.
…-email-auth

# Conflicts:
#	docs/memory/architecture.md
#	docs/memory/feature-flows.md
#	docs/memory/learnings.md
#	src/backend/main.py
#	src/backend/models.py
#	src/mcp-server/src/types.ts
@obasilakis

Copy link
Copy Markdown
Contributor Author

Re-conflicted against 78 new dev commits since your last pass, so this is a re-merge plus a walk of the checklist. Everything below is verified, not asserted.

The blocker

MCP_INLINE_AUTH_ENABLED is wired in all four places. Verified with your own instrument — compose's resolution with the flag explicitly set, not a grep:

docker-compose.yml       backend      MCP_INLINE_AUTH_ENABLED = "true"
docker-compose.yml       mcp-server   MCP_INLINE_AUTH_ENABLED = "true"
docker-compose.prod.yml  backend      MCP_INLINE_AUTH_ENABLED = "true"
docker-compose.prod.yml  mcp-server   MCP_INLINE_AUTH_ENABLED = "true"

tests/unit/test_848_inline_auth_compose_wiring.py is the guard, so a future compose edit that drops a wiring fails CI rather than shipping an un-switchable flag again.

Should-land items

  • architecture.md — routers/mcp_auth.py in the router catalog, services/mcp_auth_service.py in the service catalog, auth.ts + a request_login/verify_login row in the MCP tools table, the four /api/internal/mcp-auth/* endpoints, and the anonymous scope row. Module counts left alone per your correction.
  • MCP_INLINE_AUTH_TIMEOUT_MS → .env.example + both compose files.
  • feature-flows.md — Recent Updates row added.
  • fastmcp citations — corrected, and dev has since moved the pin again: package.json is now ^4.12.1 and the lock resolves 4.12.1, which is what the comments in server.ts and tool-visibility.test.ts cite. Both load-bearing behaviours re-checked there.
  • Call-time enforcement is now tested. tool-visibility.test.ts boots a real FastMCP httpStream server with the real makeOperatorOnly/anonymousOnly predicates and drives it with a real MCP client, asserting a filtered-out tool is not merely hidden but uninvokable. Per your note it encodes the observable outcome, not the toolsMap mechanism — a fastmcp refactor that preserves enforcement stays green, one that drops it goes red.

Non-blocking

  • mcp_inline_auth_enabled surfaced on GET /api/settings/feature-flags alongside its two siblings.
  • CSO reports renamed to convention: cso-diff-2026-07-21-848.{md,json}, plus cso-diff-2026-07-30-848.{md,json} from the re-audit.
  • N1 carried into docs/memory/requirements/mcp.md as an operational prerequisite (there is no open release-notes file — 0.8.5 is already cut — so it lands at the next cut from there).
  • Your /request keep-alive note is on record and unchanged; no code change.

Merge resolutions

Six conflicts, all additive on both sides, both kept:

Verification

  • All 23 CI checks green on the merge commit, including the base/head regression diff across 3 seeds.
  • Local pre-push: MCP tsc --noEmit clean, npm test 140/140; backend test_848_mcp_inline_auth + test_848_inline_auth_compose_wiring + test_118_mcp_connector_oss + test_186_enumeration_uniformity + test_models_centralized 100 passed; and your side of the merge — test_1854_agent_mcp_key + test_1310_auth_wiring + test_1310_auth_consolidation + test_1483_route_order — 95 passed, 1 skipped.

Re-requesting. The one thing this pass does not add is the independent adversarial security pass you flagged as optional — the CSO artifact in the PR is still the author's own.

…-email-auth

# Conflicts:
#	.env.example
#	docker-compose.prod.yml
#	docker-compose.yml
…-email-auth

# Conflicts:
#	docs/memory/learnings.md

@AndriiPasternak31 AndriiPasternak31 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Both prior blockers are cleared — verified by execution, not by reading the comment. This round I did the thing I owed you: the adversarial read of assert_email_may_reach_agent you asked for. It found two gaps. Neither is a blocker on this PR, and the reason why is the substantive part of this review.

Prior blockers — cleared

Compose resolution with the flag explicitly set, all four wirings:

docker-compose.yml       backend     ENABLED=true  TIMEOUT=<<ABSENT>>
docker-compose.yml       mcp-server  ENABLED=true  TIMEOUT=9999
docker-compose.prod.yml  backend     ENABLED=true  TIMEOUT=<<ABSENT>>
docker-compose.prod.yml  mcp-server  ENABLED=true  TIMEOUT=9999

The TIMEOUT asymmetry is correct and I checked it rather than assumed: no backend source reads MCP_INLINE_AUTH_TIMEOUT_MS, and test_848_inline_auth_compose_wiring.py pins that. Wiring it backend-side would encode a wiring that shouldn't exist.

Should-land items all present: architecture.md router catalog (:96), service catalog (:164), MCP tools table (:319), all four endpoints (:1181-1184), anonymous scope row (:2191); .env.example both vars + the N1 note; feature-flows.md row (:38); tool-visibility.test.ts boots a real FastMCP server and drives it with a real MCP client. Local: 96 passed across the four backend suites; 23/23 CI green.

The adversarial read: what "the email's own standing" does not cover

Your framing is that the internal secret authenticates the caller and the asserted email's own standing in Trinity authorizes the action. That claim holds for the grant — revoke a share and the next call fails, because assert_email_may_reach_agent re-reads it. It does not hold for account state. Two gates that every other sign-in path applies are absent here, and I confirmed both by execution rather than inference.

1. users.suspended_at (#995) does not reach this path. Seeded a suspended user on an agent's sharing allow-list, connector enabled, and drove the real flow:

[1] code row issued to SUSPENDED address: True
[2] /verify status=200 {"verified":true,"username":"suspended@example.com","agents":[{"name":"agent-a",...}]}
[3] users.suspended_at = '2026-02-01T00:00:00Z'
[GATE] assert_email_may_reach_agent ADMITTED suspended user

#995 is enforced only in dependencies.py:438 (JWT) and :460 (MCP key). This surface authenticates via verify_internal_secret and never reaches get_current_user; email_has_agent_access (db/agent_settings/sharing.py:235) resolves role/owner/sharing and never reads suspended_at.

2. The #5 MFA gate is not called. With a provider that requires a second factor:

[A] mfa_gate.gate_login -> CHALLENGE (2FA required)
[B] inline /verify status=200 {"verified":true,...,"agents":[...]}

gate_login has exactly two callers — routers/auth.py:374 (admin) and :649 (email). mcp_auth_service.verify_login_code reimplements the post-code half of the login path (get_or_create_email_user → update_last_login → return) and stops one step before the gate. So the inline path accepts the same first factor the web path deems insufficient on its own.

Why neither blocks

I went looking for the precedent before writing this up, and it changes the verdict. Telegram (telegram_adapter.py:825) and WhatsApp (whatsapp_adapter.py:654) already have the identical email-code inline login: same db.verify_login_code, same email_has_agent_access gate, no mfa_gate, no suspended_at. Both are worse than this PR on persistence — they write the verified email to telegram_chat_links / whatsapp_chat_links, so the binding survives restarts indefinitely, whereas #848's dies with the connection.

So this is a pre-existing platform class, not a defect this PR introduces. The PR docs explicitly cite Telegram's inline /login as the model (tools/auth.ts:25-30), and it followed that model faithfully and in a more conservative form. Holding it to a standard three existing surfaces don't meet would be the wrong call.

Worth a follow-up issue covering all four surfaces at once. The natural fix is one predicate — email_has_agent_access rejecting a suspended principal — which closes channels, public links and inline auth together; the MFA half is a separate decision, since an inline caller can't complete a challenge and the honest behaviour is probably to refuse and redirect to the web flow. Both are cheap, and neither belongs in this PR. Happy to file it if you'd rather not.

One thing to note for that issue: an inline session is never re-validated after verify_login (verifiedEmail is written once at tools/auth.ts:269 and has no TTL), so the per-call backend gate is the only revocation point. That's a good design — it just means whatever check gets added has to live in the gate, not only at verify, or suspension won't terminate an established session.

Non-blocking

architecture.md:1183 documents /api/internal/mcp-auth/playbooks as GET; routers/mcp_auth.py:190 defines it as POST. The three siblings are right.

What I checked that's correct

I went hunting for the fail-open you'd expect in this shape and found you'd already closed it: dynamic-agents.ts moved the #846 dedicated chat_with_<slug> tools from connectorDenied to operatorOnly. A deny-check there would have advertised every exposed agent's chat tool to the new anonymous tier — the same trap as the original operator gate, one layer down. That's the catch I'd have led this review with.

Also verified rather than assumed: authenticate mints an anonymous context only on an absent credential and still throws on an invalid one (server.ts:279-299); makeOperatorOnly is a genuine allow-list and requireAnonymous re-checks at execute time, so canAccess being advertisement-shaped doesn't matter; verifiedEmail() is gated on isAnonymous, so a connector key can't spoof it; unverified anonymous sessions hit NeedsLoginError rather than falling through; make_inline_auth_scope folds the email in, which is the right fix for the cross-user snapshot replay and matches derive_payment_key's shape.

The /request enumeration reasoning holds. Deferring all branch-dependent work rather than just the send is the correct reading of #186 — the oracle is the timing, not the body — and BackgroundTasks-not-to_thread is right for the reason you documented.

obasilakis added a commit that referenced this pull request Aug 27, 2026
…2390)

* feat(hosting): publish prebuilt images and install from them (#2280)

A fresh VM was spending 5-10 minutes compiling the agent base image
(Python + Node + Go + Claude Code, ~1.9 GB) before it served anything.
That fails the "one click" bar for every marketplace channel and is
rejected outright by managed hosts and template catalogues, which accept
pull-only compose files only. This is the gate the whole hosting arc
(#2281 DigitalOcean, #2282 Vultr, #2283 Hostinger/Dokploy, #835 Packer)
sits behind.

- `.github/workflows/publish-images.yml` pushes five images to GHCR on
  every `v*` tag — the four platform services and the agent base image.
  `latest` is applied only on a real release tag, so a dispatch smoke
  build can never become what every hosted install pulls. The checkout
  sets `submodules: false`, so the published backend is OSS-only by
  construction rather than by a .dockerignore rule someone can edit.

- `docker-compose.hosted.yml` is `docker-compose.prod.yml` with the
  `build:` blocks replaced by GHCR `image:` refs and nothing else
  changed. It is generated from prod, not hand-written.

- `start.sh --hosted` installs from those images. A flag, not a second
  script: secret generation, the ADMIN_PASSWORD contract #2381 made
  honest, DOCKER_GID detection, the health poll and the next-steps card
  are shared. A parallel installer is the same bug one layer up.

Two things that are easy to get wrong, so they are stated:

The agent base image is not a compose service and never will be. The
backend creates agent containers through the Docker SDK from the literal
local tag `trinity-agent-base:latest`, hardcoded in lifecycle.py and
allowlisted as `trinity-agent-base:*` by SEC-172, and compose cannot
retag — so `--hosted` pulls the GHCR copy and tags it locally. Retagging
rather than repointing keeps SEC-172 and the #1809 drift check untouched.
A bare `docker compose -f docker-compose.hosted.yml up -d` starts a
platform that cannot create a single agent, and fails later at
agent-create time rather than at install time. Hence "re-run start.sh
--hosted" as the upgrade instruction, never "pull".

Two compose files describing one platform is exactly the shape of a bug
this repo has shipped five times and named: LOG_* (#1039), the VoIP
master switch (#1056), AGENT_AUTH_SECRET (#1707), the container log caps
(#1871), and ADMIN_USERNAME as recently as #2381. So
tests/unit/test_2280_hosted_compose_parity.py fails CI when the two files
disagree on the service set, any third-party pin, the top-level
volumes/networks, or any per-service environment/ports/volumes/networks/
depends_on/container_name/restart/cap_*/security_opt/healthcheck value.
It compares the raw YAML, not `docker compose config` output: the raw
form still holds the unexpanded ${VAR:-default} strings, so a changed
default is a diff, where the resolved form would agree whenever the local
.env happens to match.

Also here:

- Build args are per image. Only the backend declares the six provenance
  ARGs; the scheduler and mcp-server declare none, and an undeclared
  --build-arg warns on every build. The frontend takes none deliberately
  — every VITE_* var is inlined into the world-readable bundle, so a
  published image may only carry values correct for every install, which
  works only because VITE_API_URL is inert (#722).
- The docker/* actions are SHA-pinned, matching this repo's posture for
  every other third-party action. One of them holds a packages:write
  token.
- Provenance values reach the build-args step through `env:`, never
  through ${{ }} inside a shell body. The commit subject is chosen by
  whoever lands the commit, so interpolating it into `run:` is a
  command-injection sink on a runner holding that token.
- TLS on a bare VM is decided and written down (tunnel / private network
  / operator-run proxy), with plain HTTP on a public IPv4 called out as
  the one combination to avoid.
- The 8 GB floor is stated in the compose header and both install docs,
  and pinned by the parity test so it cannot quietly drop out.

AC4 (the cloud-init user-data example) lives in trinity-ops-public, so
#2280 stays open until that lands.

Refs #2280

* fix(hosting): address review on #2390 — all ten findings (#2280)

dolho's review, in order.

High:

1. A `TRINITY_IMAGE_TAG` pinned in `.env` was silently replaced with
   `latest` on every run. The script exported an unconditional default
   and never read the file, and compose gives the shell environment
   precedence over `.env` — so the operator's pin lost, and the summary
   printed `Currently pinned to: latest` as though they had chosen it.
   That is exactly the unscheduled upgrade the rest of the script warns
   about. Now resolved shell/CI > `.env` > `latest`, after `.env` exists,
   mirroring the FRONTEND_PORT read. Documented in `.env.example`, which
   is where an unattended or marketplace install can persist it.

2. Every documented pin example was a tag that is never published.
   `{{version}}` strips the leading `v`, so git tag `v0.9.0` published
   `0.9.0` — while four places told operators to set `v0.9.0`, a pull
   that fails `manifest unknown`, which start.sh treats as fatal with a
   message blaming their spelling. Publishes the v-prefixed alias too:
   one extra tag ref, and the release name and the image tag become the
   same string.

Medium:

3. `FRONTEND_PORT` was inert — hardcoded `"80:8080"` — while start.sh
   offers it as the fix for a port conflict and prints the resulting URL.
   Fixed in prod as well as hosted: same inert knob, and diverging the
   two files would defeat the parity guard.

4. The documented tunnel never started. `cloudflared` is profile-gated,
   so setting `TUNNEL_TOKEN` produced no container and no error, leaving
   the instance in the plain-HTTP-on-a-public-IPv4 state the same table
   says to avoid. `--hosted` now appends `--profile tunnel` when the
   token is set. Documenting the flag instead was rejected: a default
   posture that silently no-ops is the finding.

5. Converting a source install in place started from an empty database.
   Dev keeps `/data` in a named volume, hosted binds a directory — and
   Redis is shared, so the result is half-migrated, not merely fresh.
   Refuses with the copy command rather than warning: the failure is
   silent, and by the time it is noticed the new DB may be written to.

Low:

6. The compose header pointed twice at `scripts/deploy/hosted-up.sh`,
   which does not exist. It is `start.sh --hosted`.

7. The header oversold the file as standalone. It has six relative bind
   mounts; ingested without the repo, Docker creates them empty and the
   first-run seed finds no manifests, with no error. Pull-only means
   "does not build", not "needs nothing on disk" — now stated.

8. A `workflow_dispatch` build stamped provenance from the dispatching
   branch, not the tree it built. `type=sha` and `GITHUB_REF_NAME` both
   ignore `inputs.ref`. Both now come from the checked-out HEAD.

9. The parity guard was an 18-key allowlist while claiming to cover any
   value — it omitted `profiles`, `env_file`, `logging`, `labels`,
   `deploy`, `stop_grace_period`, and `profiles` is live on cloudflared
   today. Inverted to wholesale: prod minus {build,image} vs hosted minus
   {image}. An allowlist only watches keys someone remembered; the drift
   that matters is the key nobody thought of.

10. `--hosted` disabled the #1432 Docker Desktop Vector fix outright. The
    explicit `-f` defeats override AUTO-merge, but the override is still
    applicable — it is now appended by name. The vector service is
    byte-identical across both files.

The wholesale guard earned itself twice while writing this: it caught a
`profiles` key the old allowlist ignored, and then caught me reverting
the prod half of finding 3 with a stray `git checkout --`.

Verified with a stubbed docker on PATH: `.env` pin honoured and shell
override still wins; `--profile tunnel` appears only with a token set;
the override file is appended only on a VM-backed runtime; the dev→hosted
guard exits 1; and the dev path still issues a bare `compose up -d` with
no -f, no pull and no profile.

Refs #2280

* fix(hosting): start.sh reads .env and names the compose project the way Compose does (#2280)

Second review round on #2390. Nine findings, and the two that matter share one
root cause: start.sh had grown its own approximate copies of rules Compose
already owns, and both copies were wrong in the direction that fails silently.

**The guard failed open on the directory names most likely to be in use.**
The dev->hosted data guard derived the compose project name with
`tr -cd '[:alnum:]'`, which strips `_` and `-`. Compose keeps them. So in a
checkout named `project_trinity` (the layout CLAUDE.md documents), `trinity-dev`,
or a worktree like `trinity-2280`, `docker volume inspect` looked up a name no
volume has ever had, missed, and the guard passed — producing exactly the
empty-DB-plus-shared-Redis half-migration it exists to prevent. It also ignored
COMPOSE_PROJECT_NAME entirely, and was the third spelling of the project name in
the file (`volume_exists()` had a fourth: `basename "$PWD"`, unlowercased).

There is now one `compose_project_name()`, and its output is asserted equal to
`docker compose config`'s own `name:` for every case that broke.

**The .env readers overrode Compose's correct parse.** Four call sites ran
`grep ... | cut -d'=' -f2- | tr -d '[:space:]'`, which keeps surrounding quotes,
swallows an inline ` # comment` into the value, and destroys internal spaces.
Compose does the opposite on all three counts. That was not a cosmetic
disagreement: the results are `export`ed, and Compose gives the shell environment
precedence over `.env` — so `TRINITY_IMAGE_TAG="v0.9.0"` became `"v0.9.0"`, the
pull failed, and the error message blamed the operator's spelling for a line that
was valid. One `env_value()` now mirrors compose-go's dotenv rules, verified
against `docker compose config` rather than against a reading of the source.

Also fixed, in order of how they were found:

- **The reverse switch is guarded too.** An operator who installed with
  `--hosted` and later runs the script without the flag gets `docker-compose.yml`,
  an empty `trinity-data` volume and the same shared `redis-data` — identical
  half-migrated state, with no warning at all. It is the likelier of the two
  crossings, because it needs no new flag, just a forgotten one. Both refusals
  now branch off one read of the state, so they cannot drift.
- **The guard's copy command handles an absolute TRINITY_DATA_PATH.**
  `"$(pwd)/${_data_path#./}"` strips only a leading `./`, so
  `TRINITY_DATA_PATH=/srv/trinity-data` emitted `/repo//srv/trinity-data` and
  would have copied the database somewhere nothing reads. A path containing
  spaces also survives the read now that `tr -d '[:space:]'` is gone.
- **The tunnel profile is persisted to `.env`.** Compose acts only on services in
  the active profile set, so the summary's own `docker compose ... stop` left
  `trinity-cloudflared` running and the instance publicly reachable after the
  operator had been told the stack was down. Persisting `COMPOSE_PROFILES=tunnel`
  makes every later bare command correct, including ones typed weeks from now.
  Appended rather than rewritten in place: Compose resolves a duplicated key
  last-wins, which keeps an operator-authored value out of a `sed` replacement.
- **A failed platform pull is explained.** It died bare on `set -e`, unlike the
  agent-base pull one screen above, on precisely the install least equipped to
  read a raw Compose error — while its two likeliest causes (an unpublished tag,
  a private GHCR package) name neither.
- **The port pre-flight probes the port the Web UI will actually bind.** It
  hardcoded 80 while both compose files honour `${FRONTEND_PORT:-80}`, so an
  operator who took the message's own advice kept being warned about 80 and was
  never warned about their real port.
- **The publish workflow proves anonymous pullability.** A container package
  first created by GITHUB_TOKEN is not reliably public; if one lands private,
  `docker pull` on a fresh VM fails with `denied` and the failure lands on an
  operator who can do nothing about it. A post-push step now fetches the manifest
  with a scratch DOCKER_CONFIG — the operator's exact path, with the runner's own
  login unable to mask the result — and fails the job with the visibility fix in
  the error.
- **The agent install guide copies `.env.example` before appending to it.**
  `start.sh` seeds `.env` only `if [ ! -f .env ]`, so writing the image pin first
  created the file and suppressed the seed; the install then proceeded on a
  single-key `.env`. It still boots, which is what makes it worth calling out.

tests/unit/test_2390_start_sh_env_and_project_name.py executes both helpers under
bash rather than grepping for the fix, with the expected values taken from
`docker compose config`. Two tail guards fail the build if a third hand-rolled
`.env` reader or project-name derivation reappears — both verified to fail when
the old forms are put back.

Verified: 1739 passed across the parity/packaging/guard/wiring/compose/env
selection (the 5 TestCgnatSsrfGuard failures reproduce on an untouched dev
checkout — macOS IPv6/DNS). End-to-end smoke of both install paths against a
stubbed docker covers every finding: the guard now refuses in `trinity-dev` and
`project_trinity`, refuses the reverse switch, prints an absolute copy path,
resolves `TRINITY_IMAGE_TAG="v0.9.0"  # pinned` to `v0.9.0`, persists
`COMPOSE_PROFILES` additively and idempotently, explains a compose-pull failure,
and warns about a bound 9481 while staying silent about a listening 80.

Refs #2280

* fix(hosting): the publish verification can run, a dispatch can't move `latest`, and stop.sh knows which stack it is stopping (#2280)

Addresses the second review round on #2390. Three defects, each silent at
authoring time and only visible on a release cut or on a hosted server, plus
the docs the capability was missing.

C1 — the anonymous-pull verification failed on EVERY publish, before any
network call. The step built `ghcr.io/${{ github.repository_owner }}/…` and
this owner is literally `Abilityai`; a GHCR repository name must be lowercase,
so docker rejects the reference locally ("repository name must be lowercase",
reproduced). All five matrix jobs would burn 5x15s retries and exit 1 with
`::error title=GHCR package is not public` even when the packages ARE public —
destroying exactly the signal the step exists for and, per the workflow's own
comment, training everyone to ignore the warnings that matter. The push itself
was never affected (docker/metadata-action lowercases its `images:` input) and
start.sh already hardcodes `ghcr.io/abilityai/`. Lowercased in the shell body.

I1 — a `workflow_dispatch` from a tag ref republished that tag's version tags
and repointed `latest` BACKWARDS. `startsWith(github.ref, 'refs/tags/v')`
distinguishes tag from branch, not push from dispatch, and dispatching an old
tag (the natural way to smoke-test the workflow) rebuilds at a new digest — the
agent base bakes "Claude Code (latest)", so the content genuinely differs —
silently downgrading every unpinned hosted install on its next
`start.sh --hosted`. Gating the explicit `type=raw,value=latest` line alone
would have been inert: metadata-action's default `flavor: latest=auto` applies
`latest` on any semver tag ref by itself. So: `flavor: latest=false`, and every
mutable/version tag now requires `github.event_name == 'push'`. A dispatch
publishes `sha-<short>` and nothing else, which is what a smoke build needs;
re-cutting a release is re-pushing the git tag, which fires `push`.

I2 — stop.sh was not hosted-aware and ran the verb start.sh's own summary
forbids. Hosted passes an explicit `-f`, which disables compose's file
auto-merge, so a bare `docker compose` in the checkout loads the DEV file. Same
project name, so it acts on the same containers — but it knows nothing about
`cloudflared`, which the dev file does not define: the tunnel keeps running and
the instance stays publicly reachable after "All services stopped". That is the
hazard `persist_compose_profile` was added to close, re-entering through the one
entry point that change did not touch. The file is now selected from the running
stack's own `com.docker.compose.project.config_files` label — compose's record
of what created the project, which cannot drift the way a marker file or a
`trinity.db`-at-the-bind-path heuristic can — degrading to the dev default when
there is no container or no label. And the verb is `stop`, not `down`: `down`
removes the platform containers and tears down `trinity-agent-network`, and is
what start.sh tells operators not to run in both branches. A script named
stop.sh running the forbidden verb was a standing contradiction.

Docs: new `docs/memory/feature-flows/hosted-install.md` (publish workflow ->
GHCR -> `--hosted` pull/retag -> data-switch guards -> upgrade path, with a
Testing section) + the index row and a Recent Updates entry. Fixed three stale
"both compose files" claims in architecture.md — there are three now (#2216 DB
backup, #1159 agent auth, #1871 x-logging). requirements 8.9 gains HOST-017 and
HOST-018 for the two defects above, the corrected HOST-001 tag-gating rule, and
the two test files its Tests row omitted.

Tests: `tests/unit/test_2280_publish_workflow_and_stop.py` (7) pins all three —
no raw mixed-case owner in an image reference and the built one is lowercased;
`latest=false` present; every non-sha tag gated on push and the sha tag gated on
nothing; stop.sh never runs `compose down` and selects its file from the label.
40 passed across the three #2280/#2390 files.
vybe pushed a commit that referenced this pull request Sep 2, 2026
… on it (#2380) (#2431)

* feat(hosting): record install provenance, and show a hardening guide only where it belongs (#2380)

A marketplace droplet is the one install where Trinity knows, at boot, that it
is on a public IPv4 with no domain, with zero network configuration, and an
operator who has read no deployment docs. It is therefore the one install that
should be prompted to add a real name or a VPN. This records how an instance was
installed and gates that prompt on it.

The obvious predicate is unusable, which is the whole design. Measured across
all 16 managed instances: every one serves plain HTTP with no DOMAIN, no
HTTPS_ENABLED, on a 100.x Tailscale CGNAT address — structurally identical to an
unhardened droplet, and already correct, because HTTP over a WireGuard tunnel is
encrypted transport (HOST-010). A "no TLS configured, so warn" rule fires
permanently on every paying client. No environmental signal separates the two
cases; the install channel does, and it is knowable only at provisioning time.

TRINITY_INSTALL_SOURCE (.env) is read once at boot and recorded into
system_settings by database._record_install_source{,_engine}, on both backends.
An env var rather than /etc/trinity/*, which the issue also offered: config.py
reads zero files today and a marker file needs a read-only bind mount added to
all three compose files — the packaging class this codebase has shipped
repeatedly (#1039, #1056, #1707), where the value never reaches the container and
the feature is silently inert forever. A boot recorder rather than a migration,
for #2381's reason: a migration answers once and records itself, so it can never
reach an instance provisioned before it, nor one whose marker was corrected
afterwards.

Three properties are security rather than hygiene:

- Write-once. Provenance is a fact about an installation *event*. If a later
  .env edit could rewrite it, it would answer "what does this box currently
  claim" instead, and the marketplace gate would be self-assertable by anyone who
  can edit a file. Enforced at the PRIMARY KEY on both arms (INSERT OR IGNORE /
  ON CONFLICT DO NOTHING) rather than by the SELECT that precedes it, which is a
  separate statement and therefore a check-then-act.
- No env fallback on read. get_install_source reads the row only. Without this,
  write-once is decorative — an unrecorded install could be talked into a
  marketplace verdict just by exporting the variable.
- Blocked on PUT *and* DELETE, and refused at the db sink as well. The DELETE
  block is not symmetry: because the recorder is write-once, a delete is
  precisely the move that unlocks a rewrite. The sink guard follows
  db/settings.py's own stated rule — boundary AND sink, because the generic
  catch-all can write any key, which is how the same door was found open by
  #506, #1609, ent#12, #1644, ent#14 and ent#346 in turn.

An unrecognised marker records nothing — not the value, and not `unknown`
either, which would combine with write-once to freeze a typo permanently. An
absent row already reads `unknown`, so leaving it absent costs nothing and lets a
corrected marker land on the next boot. Absent, empty, unrecognised and
unreadable all resolve to `unknown`, never toward a marketplace value: the
failure direction is always to hide the guide.

Surfaced on GET /api/settings/feature-flags (install_source, the server-resolved
marketplace_install gate, install_tls_posture) — no new endpoint, per the AC.
marketplace_install is resolved server-side so the browser holds no second copy
of which channels count. /api/version carries install_source too, threaded into
_build_version_payload as a parameter because that function is exec-sliced by its
own tests and must stay stdlib-only (#1443's constraint on `edition`).

Honest state by construction. Nothing probes a socket or reads a certificate —
TLS terminates outside the backend, so no in-process check can — and the posture
is derived by pure string parsing of the URL the instance *advertises*, resolved
through the canonical get_public_chat_url() so it cannot disagree with public
links and the webhook registrations about what this instance is called. The card
says "advertises", never "secure", and https-ip reads as working-but-upgradeable
rather than broken: LE IP certificates went GA 2026-01-15 and DigitalOcean's own
1-Click rules ship Caddy with them, so a droplet can boot on genuinely trusted
HTTPS — just on a ~6-day renewal at an unmemorable address. The card is also
explicit that Trinity issues no certificate itself; nothing in the tree reads
public_chat_url and reconfigures a proxy.

The card is admin-gated (its only remediation is an adminOnly settings tab, and
its copy discloses the box's network posture to every user of it), renders only
after the flags resolve, and retires when a domain is configured — with
Settings.vue force-refreshing the flags on save so it clears in-session rather
than on the next hard reload.

Nothing in the OSS tree writes the marker; #2281's Packer snapshot does. Until
then provenance reads `unknown` everywhere and the guide renders nowhere, which
is the contract working and is what makes shipping this half first safe.

test_2217_canary_status.py's hand-rolled settings_service stub gains the three
new reads it now has to answer.

Fixes #2380

* test(2380): stop the marker-normalisation tests reloading the shared config module

`TestMarkerNormalisation` called `importlib.reload(config)` to exercise the real
`.strip().lower()` at import. Reload mutates the shared module object in place,
and `config.py` mints a random `SECRET_KEY` at import whenever the env var is
unset — which it is under the unit suite. Modules that bound
`from config import SECRET_KEY` at their own import kept the old value while
anything reading `config.SECRET_KEY` per request got the new one, so a JWT signed
on one side failed verification on the other. Order-dependent, hence the varying
failure counts across seeds in the regression diff:

    test_187_jwt_revocation::test_minted_token_carries_jti
    test_voice_auth::TestVoiceWebSocketAuth::test_owner_passes_auth_gate
    test_voice_auth::TestVoiceWebSocketAuth::test_other_user_rejected_4003
    test_voice_auth::TestVoiceWebSocketAuth::test_admin_bypasses_ownership

This is the #1895 divergence, and the conftest hook written for it structurally
cannot catch this shape: it restores `sys.modules["config"]` to the same object a
reload has already mutated.

The intent behind the reload was right — the `marker` fixture rebinds
`cfg.TRINITY_INSTALL_SOURCE` directly and so proves nothing about the real boot
path — so it is kept, and only the mechanism changes: `config.py` is executed
under a throwaway module name that is never registered in `sys.modules`. That
runs the identical module-level statement with none of the reach. Pinning
`SECRET_KEY` across a reload, or snapshotting and restoring the module dict,
would both have worked, but each leaves a cleanup step that can be forgotten and
would need revisiting the day config grows another import-time-derived value;
isolation has nothing to restore.

Verified: the four tests pass when run directly after the new module, and a full
unit run with the new module forced first has a failure set identical to the
branch parent's (23 pre-existing IPv6-mapped-address failures local to this
machine, unchanged on both sides).
obasilakis added a commit that referenced this pull request Sep 2, 2026
…2281)

Addresses dolho's review. C1 is the one that shipped a security hole; the
other four are the silent-failure class this bundle is otherwise careful
about.

C1 — the SPA was reachable on plain HTTP, past Caddy. First boot moves the
frontend to FRONTEND_PORT=8081 so Caddy can own 80/443, and compose publishes
it with the short syntax `"${FRONTEND_PORT:-80}:8080"`, which binds 0.0.0.0.
The DOCKER-USER DROP list covered 8000/8080/8686 and the OTel ports but not
8081, so the login page answered on http://<ip>:8081 — past the certificate
and past the http->https redirect the whole image is built around. 8081 added,
and the probe and the insert now read ONE variable: two hand-kept lists that
disagree means the -C probe never matches its own rule and every re-run of
first boot stacks another DROP.

The list is hand-maintained against a compose file in another directory, which
is this repo's most-shipped bug (#1039, #1056, #1707, #1871, #2381) — so
tests/unit/test_2281_firstboot_port_exposure.py asserts every host port the
hosted compose publishes on an external interface appears in DROP_PORTS,
resolving ${FRONTEND_PORT} from first boot's own `set_env` line. Verified it
fails on the pre-fix list naming exactly {'frontend': [8081]}. `packer build`
could not have caught this even once it runs: the rule is installed correctly,
it simply does not cover the port.

I1 — the DROP rules could evaporate at the first reboot. `apt-get install
iptables-persistent` ran at first boot as `>/dev/null 2>&1 || true`, at the
busiest apt moment a droplet ever has; a transient mirror failure then made
the save impossible, silently, and the ports reopened at the next reboot with
no symptom anywhere. The install moves to build time (deterministic, baked
once, debconf preseeded so a non-interactive Packer run cannot hang on the
autosave prompt) and first boot keeps only the save — whose failure now says
so on the tee'd log, naming the re-run.

I2 — the OTel collector was not pre-pulled. It is not profile-gated in
docker-compose.hosted.yml, unlike cloudflared, so start.sh --hosted starts it
on every droplet and first boot was fetching a few hundred MB — against the
one property this snapshot exists to have. Pulled at build time, with the tag
read from the checkout rather than hardcoded so a bump in compose cannot leave
this line pinning a stale one.

I3 — a user-supplied password containing | & or \ was silently mangled.
set_env substituted user-controlled text into the replacement half of
`sed "s|^k=.*|k=$value|"`, where & expands to the whole match, \ escapes and |
ends the expression. It now rewrites the line instead, which needs no escaping
at all; verified byte-exact round-trip for `p|a&s\sw#ord`. Only the user-data
path was exposed — the generated password is A-Za-z0-9.

I4 — marketplace-partners was cloned at HEAD and its two scripts run as root
as the last thing to touch the image a vendor review approves. Pinned to
b708788 (master @ 2026-07-16) via fetch-by-SHA, since a shallow clone can only
take a ref.

Verified: bash -n and shellcheck -S warning clean on all five scripts; the new
guard fails on the old list and passes on the new one; 16 tests green across
the two #2281 guards and the #2280 compose parity suite.

Refs #2281
vybe pushed a commit that referenced this pull request Sep 2, 2026
…#2281) (#2471)

* feat(hosting): Packer bundle for the DigitalOcean Marketplace 1-Click (#2281)

Builds an Ubuntu 24.04 snapshot from the prebuilt GHCR images (#2280) rather
than from source: a 1-Click that compiles the agent base image on first boot
fails the one-click bar, which is why #2280 gates this.

Build time bakes Docker, Caddy, ufw, a pinned Trinity checkout at /opt/trinity,
and all five images with the agent base retagged locally. First boot does only
what is droplet-specific: resolve the admin password, write .env, close the
Docker/ufw gap, issue a certificate for the droplet's own IP, and run
`start.sh --hosted --unattended`.

Four decisions worth stating, each with a failure mode behind it:

- `image_tag` is required and rejects `latest`. A snapshot resolving `latest` at
  first boot would serve a different Trinity on every droplet created from one
  reviewed image, and Marketplace review approves a specific artifact.

- The admin password comes from the MOTD, read via the control panel's browser
  Console — no SSH client needed. DigitalOcean 1-Clicks have no vendor-defined
  input form at deploy time (only the Managed Database checkbox), so it cannot be
  collected in the UI. An operator may override it through Additional Options ->
  Startup scripts, but only as `#cloud-config` write_files: 1-Click per-instance
  code runs from cloud-init's `scripts-per-instance`, which runs BEFORE
  `scripts-user`, so a user-data shell script would land after first boot had
  already generated a password and started Trinity.

- Docker publishes past ufw. docker-compose.hosted.yml exposes 8000, 8080, 8686
  and the OTel ports on 0.0.0.0, and Docker's iptables rules are consulted before
  ufw's chain, so `ufw deny 8000` on such a droplet is inert. First boot inserts a
  DOCKER-USER DROP for those ports on eth0 — the one chain Docker leaves to the
  operator and evaluates first. Host and inter-container access is unchanged.

- First boot calls the same scripts/deploy/start.sh every other install uses. A
  marketplace-specific copy of the installer is the shape that has gone stale
  here before (#1039, #1056, #1707, #1871).

The build's last step is DigitalOcean's own 90-cleanup.sh then 99-img-check.sh,
in that order and last, so the snapshot is never created from a droplet that
would be rejected at review, and nothing runs after the check to reintroduce
what it verified was gone.

Not yet exercised: `packer build` and img-check need the GHCR images, which do
not exist until the first `v*` tag publishes them.

Refs #2281, #2280

* fix(packer): satisfy HCL's validation-message grammar so the template validates

`packer validate` refused the whole template:

  Error: Invalid validation error message
    on trinity.pkr.hcl line 45, in variable "image_tag":
  Validation error message must be at least one full sentence starting
  with an uppercase letter and ending with a period or question mark.

The message began with the lowercase variable name. Reworded to lead with
"The image_tag variable ...".

This is not cosmetic: the failure is at the template level, so it fired
before any builder ran and made the image_tag guard — the thing that stops
a snapshot being built from a moving `latest` — unreachable. Both paths
are now exercised: a pinned tag validates, and `-var image_tag=latest` is
refused by the rule at trinity.pkr.hcl:43.

* fix(packer): pin Caddy to a version that can issue IP certificates, and verify one was

The whole no-domain HTTPS story rests on Caddy obtaining a Let's Encrypt
certificate for the droplet's bare IP. Two things made that a hope rather
than a property.

**Caddy was unpinned.** `apt-get install caddy` took whatever the stable
repo happened to hold. Caddy could not issue IP certificates at all until
recently — caddyserver/caddy#7399, where v2.10.0 fails outright with
"subject '<ip>' cannot have public IP certificate"; the IPv4 fix landed via
mholt/acmez#47 (2025-12-17) and IPv6 was not resolved until 2026-04-25, so
v2.11.3 (2026-05-12) is the first stable release carrying both. It worked
only because today's stable is new enough: a silent dependency on a moving
upstream for the property this image is built around. Now pinned to 2.11.4
with a 2.11.3 floor asserted at build time — the assertion survives an
edited pin, a version dropped from the repo, and a `caddy upgrade` on a
running droplet, none of which the pin alone covers. Deliberately not
`apt-mark hold`: forward versions carry the fix and holding would block
security updates for the life of the droplet.

**Nothing checked whether issuance succeeded.** The failure was silent and
worse than silent: Caddy serves a TLS error, first boot still exits 0, and
the MOTD prints a confident `https://<ip>` URL — so the user's first contact
with Trinity is a browser warning on a box that reported success. First boot
now polls its own public IP with normal certificate verification for ~150s
and records the verdict to `/etc/trinity/tls-status`.

The probe deliberately does not use `curl -f`. Trinity has not started yet
(that is step 6), so Caddy answers 502 — which is the point: without `-f`,
curl treats an HTTP error as success, so a zero exit means the handshake
completed and the chain validated against the system trust store. That is
exactly the claim under test, and nothing weaker separates a real Let's
Encrypt certificate from Caddy falling back to its internal CA.

Never fatal. A droplet serving HTTP with an honest banner is far better than
one that refuses to finish booting, so the MOTD derives its scheme from the
verified status rather than from what the Caddyfile intended, and says
plainly that traffic is unencrypted when it is.

Also updates the MOTD's next-steps line from "locking the instance to a VPN"
to the Cloudflare Tunnel path, per the #2380 decision: a VPN breaks every
inbound integration (Telegram, WhatsApp, VoIP, public links, x402, inbound
A2A, webhook triggers — everything that calls us; only Slack survives, on
Socket Mode), while a tunnel reaches the same "nothing listening on the
public interface" outcome with all of them intact.

Verified: `packer validate` passes; the version comparison accepts
2.11.3/2.11.4/2.12.0 and rejects 2.10.0/2.11.2; the MOTD renders correctly
for both tls-status values; bash -n and shellcheck -S warning clean on all
five scripts.

Refs #2281, #2380

* fix(hosting): commit the cloud-init boot hook .gitignore was swallowing (#2281)

`packer/digitalocean/files/var/lib/cloud/scripts/per-instance/001-trinity` — the
hook that makes a droplet do anything on first boot — was written locally and
never committed. `.gitignore` carries the bare setuptools patterns `var/` and
`lib/`, which git applies at every depth, so `git add packer/` skipped it and
`git status` stayed clean.

The bundle under `files/` mirrors a target filesystem, so the collision is
structural rather than unlucky: a future `files/usr/lib/...` would go the same
way. The whole payload tree is therefore re-included, following the `parts/`
precedent already in the file — directory first, since git cannot re-include a
file whose parent directory is excluded, then `**` to beat both bare patterns
for everything below. It sits inside that block deliberately, so the later
`.env` / `*.log` rules still apply underneath and a stray secret in the bundle
stays ignored.

Nothing that ran on the PR could have caught this. `packer validate` never reads
the payload tree, `shellcheck` only sees the files it is handed, and `packer
build` — which would have died at `install: cannot stat` — is deferred to the
first release with GHCR images, i.e. to the vendor-submission moment.

So `tests/unit/test_2281_packer_bundle_tracked.py` asserts every
`install ... /tmp/trinity-files/<path>` source in 01-provision.sh is tracked, via
`git ls-files` rather than `os.path.exists` — so it also fails on the developer's
machine where the file is present-but-untracked, which is the state the bug was
written in.

* fix(hosting): close the :8081 TLS bypass and four first-boot hazards (#2281)

Addresses dolho's review. C1 is the one that shipped a security hole; the
other four are the silent-failure class this bundle is otherwise careful
about.

C1 — the SPA was reachable on plain HTTP, past Caddy. First boot moves the
frontend to FRONTEND_PORT=8081 so Caddy can own 80/443, and compose publishes
it with the short syntax `"${FRONTEND_PORT:-80}:8080"`, which binds 0.0.0.0.
The DOCKER-USER DROP list covered 8000/8080/8686 and the OTel ports but not
8081, so the login page answered on http://<ip>:8081 — past the certificate
and past the http->https redirect the whole image is built around. 8081 added,
and the probe and the insert now read ONE variable: two hand-kept lists that
disagree means the -C probe never matches its own rule and every re-run of
first boot stacks another DROP.

The list is hand-maintained against a compose file in another directory, which
is this repo's most-shipped bug (#1039, #1056, #1707, #1871, #2381) — so
tests/unit/test_2281_firstboot_port_exposure.py asserts every host port the
hosted compose publishes on an external interface appears in DROP_PORTS,
resolving ${FRONTEND_PORT} from first boot's own `set_env` line. Verified it
fails on the pre-fix list naming exactly {'frontend': [8081]}. `packer build`
could not have caught this even once it runs: the rule is installed correctly,
it simply does not cover the port.

I1 — the DROP rules could evaporate at the first reboot. `apt-get install
iptables-persistent` ran at first boot as `>/dev/null 2>&1 || true`, at the
busiest apt moment a droplet ever has; a transient mirror failure then made
the save impossible, silently, and the ports reopened at the next reboot with
no symptom anywhere. The install moves to build time (deterministic, baked
once, debconf preseeded so a non-interactive Packer run cannot hang on the
autosave prompt) and first boot keeps only the save — whose failure now says
so on the tee'd log, naming the re-run.

I2 — the OTel collector was not pre-pulled. It is not profile-gated in
docker-compose.hosted.yml, unlike cloudflared, so start.sh --hosted starts it
on every droplet and first boot was fetching a few hundred MB — against the
one property this snapshot exists to have. Pulled at build time, with the tag
read from the checkout rather than hardcoded so a bump in compose cannot leave
this line pinning a stale one.

I3 — a user-supplied password containing | & or \ was silently mangled.
set_env substituted user-controlled text into the replacement half of
`sed "s|^k=.*|k=$value|"`, where & expands to the whole match, \ escapes and |
ends the expression. It now rewrites the line instead, which needs no escaping
at all; verified byte-exact round-trip for `p|a&s\sw#ord`. Only the user-data
path was exposed — the generated password is A-Za-z0-9.

I4 — marketplace-partners was cloned at HEAD and its two scripts run as root
as the last thing to touch the image a vendor review approves. Pinned to
b708788 (master @ 2026-07-16) via fetch-by-SHA, since a shallow clone can only
take a ref.

Verified: bash -n and shellcheck -S warning clean on all five scripts; the new
guard fails on the old list and passes on the new one; 16 tests green across
the two #2281 guards and the #2280 compose parity suite.

Refs #2281
AndriiPasternak31 added a commit that referenced this pull request Sep 14, 2026
…2742)

`b6757ba66` wired `SYNC_HEALTH_POLL_INTERVAL_SECONDS` into `docker-compose.yml`,
`docker-compose.prod.yml` and `.env.example` — and missed
`docker-compose.hosted.yml`, the pull-only twin every marketplace and managed-host
channel actually runs. `test_2280_hosted_compose_parity` caught it in CI:
deterministic, all three head seeds, clean in all three base seeds.

That is the bug reproducing inside its own repair. The commit being fixed exists
because a knob never reached the container; it then failed to reach the container
that ships to marketplace installs. The hosted file's own header names the class
and lists its five prior victims (#1039, #1056, #1707, #1871, #2381) — this is
the sixth, and the first to be caught by the guard rather than by an operator.

The line is byte-identical to prod's, since #2280 compares the backend
`environment` list wholesale. No hosted-specific assertion is added here: that
guard is strictly stronger than anything this file could restate, and two guards
over one fact drift apart. The reasoning is recorded in the test's docstring so
the next reader does not "helpfully" add the redundant one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
vybe pushed a commit that referenced this pull request Sep 15, 2026
… the agent's index.lock (#2742) (#2797)

* docs(git-sync): requirements, architecture and flow for lock-free sync-health polling (#2742)

Trinity Rule #1 — the requirements/architecture delta lands before the code.

What the docs now say:

- requirements/github.md §11.9 gains "Lock-free status read" and "Stuck-lock
  report, never a runtime delete", and the stale-lock hygiene bullet records
  that the boot reap now speaks. §11.8's SyncHealthService bullet names the
  leader lease AND its honest side effect (time-to-sync_failing ~90s → ~180s).
- architecture/agent-lifecycle.md is the single home: why the status read took
  index.lock ~2x/min in every workspace, what replaced it, and why the runtime
  delete was cut on measurement rather than hardened.
- architecture/agent-runtime.md gets a one-line pointer (no second home).
- architecture/background-services.md's Sync Health row states the lease, its
  fail direction and the reason (a feed that raises sync_failing must not go
  dark when Redis does) — per that file's header rule.
- feature-flows/git-sync-health.md: §1a announced boot reap + the observe-only
  runtime report with the three measurements that decide it; a new §2a for the
  agent status handler with the two distinct bounds (35s caller / 90s
  computation) and the ~130s child-budget arithmetic that replaces the doc's
  wrong "~30s worst case"; Files Touched, Testing, Operator Controls; and a
  Known Limitations rewrite that retires the "still blocks the event loop"
  bullet and replaces it with the residuals this change does NOT close.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(sync-health): leader lease across uvicorn workers, fail-open (#2742)

main.py starts SyncHealthService in EVERY uvicorn worker and prod runs
`--workers 2`, with no lease — unlike opqueue:leader (#1632), monitoring:leader
(#1464), skills:sync:leader and canary:leader. So every git-enabled agent was
asked for /api/git/status twice a minute, and each ask runs a 30s credentialed
`git fetch origin` inside the container.

`synchealth:leader` in the #1464 shape: SET NX EX, own-lease-only refresh,
release from stop(), acquire + transition logs at the top of _poll_cycle with an
early return for non-leaders.

Two deliberate divergences from the copied shape:

- Release is a single compare-and-delete EVAL, not GET-then-DEL. The non-atomic
  form lets a worker whose lease lapsed between the two calls delete a sibling's
  FRESH grant — two pollers for a whole cycle. EVAL is outside the -@dangerous
  categories denied to the backend/scheduler ACL users.
- Fail direction is OPEN and stated in the docstring: Redis down means every
  worker polls, i.e. today's behaviour. Failing closed would darken the only
  feed that ever raises sync_failing, exactly when infra is already degraded.

The lease has an honest side effect and it is named, not buried.
db.upsert_sync_state INCREMENTS consecutive_failures per failed upsert, so it is
NOT idempotent: two unleased workers reached sync_failing in ~90s, one leader
takes ~180s. Asserted in TestLeaderLeaseAlertTiming — including the load-bearing
fact itself (the same payload polled twice increments twice), so the test cannot
pass against an idempotent upsert and prove nothing.

Also here (same file, same subject): SYNC_HEALTH_POLL_INTERVAL_SECONDS, read at
CALL time in the _maintenance_timeout_seconds shape, parse-guarded and
positive-clamped, with the default DELIBERATELY unchanged at 60s — the AC is
written as "the 60s poll" and the sampler evidence was taken there. poll_interval
becomes a property so the module-level singleton (built at import) still honours
the env; `is not None` rather than truthiness so the tests' poll_interval=0 keeps
meaning "one cycle then exit".

The `service` fixture now stubs get_breaker_redis -> None: with a local Redis it
would otherwise leave a real 30s lease behind and the NEXT test's fresh service
would lose the election and silently poll nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(agent-server): status read is lock-free and sweep-registered (#2742)

`git status --porcelain` takes .git/index.lock on EVERY invocation to refresh
the index — whether or not a rewrite follows — for a window that scales with
index size (0.6ms at one file, 12.8-15.4ms at 20k; 334-473ms measured in a real
container). The backend polls /api/git/status every 60s for every git-enabled
agent, so the platform was taking that lock ~2x/min in every workspace, outside
_REPO_LOCK, racing the agent's own `git add`.

`git --no-optional-locks status --porcelain` on exactly that one call site.
Scoped deliberately: the auto-sync cycle's status (under the repo lock, right
after `git add -A`, proceeding to commit) and the sync/pull bodies keep the plain
form — they are lock-serialized and they WANT the refreshed stat cache. Flag
form rather than GIT_OPTIONAL_LOCKS=0 because run_registered has no env= kwarg,
the env form would silently change the mutating sites too, and only the argv is
assertable in a test. Verified across a clean index, a stat-dirty index,
core.untrackedCache, --untracked-files=all, an fsmonitor hook and a repo with no
.git/index: zero lock sightings and zero rewrites in every case.

The flag closes only the index-lock half, so every status child now routes
through run_registered (#1595's seam): a sweep tick straddling the 30s
`git fetch` still SIGKILLs it, and a killed fetch orphans FETCH_HEAD.lock /
packed-refs.lock — which NO reaper covers, not even startup.sh's (its find is
scoped to refs/ and logs/). run_registered accepts neither capture_output nor
text, so both kwargs are DELETED at all ten sites; a name-only substitution
would be a TypeError ten times over.

Three of those ten are the shared helpers _compute_ahead_behind,
_get_pull_branch and _persist_last_remote_sha, which are also reached from
_conflict_response (the 409 arm of every locked endpoint), sync_to_github and
pull_from_github. So the conversion sweep-registers three children on the
MUTATING paths too. That is desirable — those are the children a repack-length
operation exposes longest — but it is stated and pinned by a test rather than
left for a reviewer to find.

Also here, because the diff was already moving these lines:

- remote_url is now unconditionally redact_url_userinfo'd. The old shape
  special-cased @github.com and returned everything else VERBATIM, so any
  non-github.life-white.uk remote (GHES, GitLab, a host-rewritten origin) put a live
  `https://oauth2:<PAT>@host/...` in the response body — proxied unmodified by
  git_service.get_git_status to the UI and the MCP tool. ent#615 owns the
  broader class; this is the one line of it in this diff.
- _read_sync_state_file gates on st_size (64 KiB) BEFORE reading. That file is
  fully agent-authored and merged wholesale, and the backend reads every agent
  concurrently once a minute, so an unbounded read_text() OOMs both sides.
- The comment beside _REPO_LOCK now says what it does NOT exclude: the agent's
  own git and the backend's docker exec sites. Misreading that is what produced
  a plan to treat a successful non-blocking acquire as evidence of quiescence.

_compute_git_status is extracted verbatim as one blocking callable (the handler
calls it directly for now) so the next commit can put the coalescing in front of
it without also moving the body.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(agent-server): coalesce /api/git/status on the event loop (#2742)

Three callers converge on this one route — the backend's 60s sync-health poll,
the UI git panel's own 60s poll while open, and the MCP get_git_status tool —
and each computation runs a 30s `git fetch origin` against one repo. The issue
observed two overlapping fetches 0.9s apart. A guard on the poller alone would
not have satisfied AC2.

Now the first caller starts the computation and every other caller awaits the
SAME future. The slot is keyed on the resolved home path, check-and-set is
atomic because there is no `await` between the test and the assignment, and the
agent server is single-process (agent_server/main.py calls uvicorn.run(app, ...)
with no workers=) — the BACKEND is not, which is what the Redis lease is for.

Coalescing happens ON THE LOOP, and only the computation takes a thread. This
is the load-bearing shape choice, not a style preference: asyncio.to_thread uses
the loop's DEFAULT executor — min(32, cpu+4) = 6 threads on a 2-vCPU agent — and
services/headless_executor.py records that ctx.terminate, auto-sync and
pipe-close deliberately stay on that same pool. Parking followers there is the
#2433 starvation class, where a burst of status callers stalls EXECUTION
TERMINATION. Five callers must cost exactly one to_thread, and a test asserts it
so the rejected shape cannot come back.

Two bounds, deliberately distinct numbers for distinct things:

- _STATUS_FOLLOWER_WAIT_SECONDS = 35 is a CALLER bound, at or just past the
  point every real client has already given up (poller 10s, git_service 30s);
  a longer wait can only produce work nobody awaits. Timeout is a 504.
- _STATUS_LEADER_DEADLINE_SECONDS = 90 is a COMPUTATION bound. The child
  timeouts sum to ~130s nominal before run_registered's post-killpg drain, and
  a slow leader costs no follower threads but DOES hold the slot, so every
  caller in that window 504s. The docstring carries the arithmetic; the flow
  doc's old "~30s worst case" (which is where a 60s follower bound came from)
  is corrected in the docs commit.

asyncio.shield is load-bearing: a follower that times out or disconnects must
not cancel the computation everyone else is waiting on. And the future is
ALWAYS resolved — to_thread -> run_in_executor -> _WorkItem.run catches
BaseException and calls set_exception — so there is no hand-rolled
set_result/set_exception pair and no "leader vanished" fallback. The comment
says not to add one.

`computed_at` admits what coalescing costs: a follower arriving at t=29s of a
30s leader run is served a 29-second-old snapshot, visible as "1 ahead" right
after a successful push. Stamping the age is cheaper and more honest than
claiming coalescing changes nothing. TTL caching was rejected — serving late
followers a fresh run reintroduces the overlapping fetch AC2 forbids.

No 409 on status by design: it is a read, and _with_repo_lock on it would make
every poll a contended write and flap the agent `unreachable`.

D5: test_1920's walk now covers docker/base-image/agent_server/ as well as
src/backend — Invariant #5, "a guard that walks only one of the two trees is
not a guard" (ent#314). Proven by planting an nx=True in an agent-server module
and watching it fail. That tree gets no allowlist row on purpose: it issues no
nx=True set and cannot (agents are not on the platform network, so Redis is
unreachable from one), and #2742's coalescing is a different class entirely.
Its one-home property is asserted directly instead, beside a meta-assertion
that fails loudly if the agent-server root ever moves out from under the walk.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(agent-server): announce the startup lock reap and report a stuck lock (#2742)

AC3 has two halves and only one of them was missing. The RECOVERY already
existed and is provably safe: startup.sh reaps index.lock at container start,
where "no git process is running" is definitional because the PID namespace is
empty. What was missing was the OBSERVABLE half — the block was `rm -f`, silent
whether or not it removed anything, so the one moment the platform reliably
heals a wedge produced no evidence that it had.

startup.sh now tests-then-removes, echoes a line naming each lock it cleared
(Vector captures it), and drops ~/.trinity/lock-recovery.json. It also reaps
index.lock under .git/modules/* and .git/worktrees/*, which NOTHING covered
before — its find is scoped to refs/ and logs/ — so a submodule or
linked-worktree wedge used to survive every restart. Proven by extracting the
shipped block and running it against fixtures: /verify-local's agent stage boots
local:test-echo, so startup.sh never executes there and trusting the stage would
prove nothing.

_record_lock_recovery folds that marker into sync_state.last_lock_recovery on
the next status read, then deletes it so the episode is reported once rather
than every minute forever. It goes through a dedicated metrics-only writer:
_write_sync_state_file unconditionally stamps last_sync_at = now, which would
mark a never-synced agent as freshly synced and turn its dashboard dot green.

_index_lock_stuck REPORTS a currently-present lock and never unlinks it. The
runtime delete was cut on measurement, not on taste:

- st_size == 0 is the signature of a LIVE writer, not an abandoned one — git
  creates the lock with O_EXCL before walking the worktree and writes the new
  index into it only at the very end. Measured 100% of a healthy `git add -A`'s
  life: 3.9s on 450MB, 29s at 60k files, 155s under a clean filter.
- st_mtime is stamped at create and never advances, so age measures the
  IN-FLIGHT OPERATION. At t=+130s a healthy add reads size=0 age=130s — both
  naive gates satisfied.
- A wrong unlink is permanent and strictly worse than the wedge: git renames by
  PATH, so a second git's in-flight file gets promoted onto .git/index, the
  corrupting process exits rc=0 with empty stderr, and the 0-byte index is
  cleared by nothing — not the boot reap, not _reap_stale_git_litter, not
  `git reset`.
- _REPO_LOCK would not have helped: it excludes this server's own auto-sync
  cycle and nothing else, not the agent's own `git add`, which is the premise
  of this issue.

So detection is two-point inode stability — the same (st_ino, st_mtime_ns,
st_size) unchanged across >=3 status reads spanning >=15 min — on
time.monotonic(), which a forward NTP step or a live migration cannot move.
The tunable is the sighting COUNT, not a wall-clock age.

Three properties that are each a defect if dropped:

- It resolves the REAL gitdir. `.git` is a FILE for a linked worktree and for a
  submodule, both creatable by the agent in one command, and assuming a
  directory makes the observer look where the lock provably is not. Candidates
  cover <gitdir>/index.lock, modules/*/index.lock and worktrees/*/index.lock.
  A symlinked .git is skipped outright.
- It takes NO repo lock. An lstat needs no mutual exclusion, and holding
  _REPO_LOCK across the observation would make a status poll a brand-new source
  of 409 agent_busy on an operator's POST /api/git/sync.
- It is wrapped end to end in its own except OSError. It runs inside
  _compute_git_status's try, whose tail is HTTPException(500), and
  _fetch_git_status treats any non-200 as None and writes nothing — so one
  EACCES would silently stop the agent's sync-health row advancing and
  sync_failing would never fire either. An observability path must never be
  able to darken the feed it feeds.

The sighting ledger is in memory, deliberately NOT in sync-state (a deviation
from the plan, on safety grounds): a per-tick read-modify-write into the
agent-authored document would race the auto-sync writer and could drop
consecutive_failures or last_sync_status — the observability path corrupting
the feed. It needs no durability either, since the only thing that clears a
genuinely wedged lock is the container restart that also clears the ledger.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(sync-health): log self-healed index.lock recoveries once (#2742)

The backend half of AC3, plus the AC4 concurrency proofs.

_coerce_lock_recovery / _coerce_lock_stuck rebuild both new fields from values
the backend has checked, and never pass the agent's dict through. That is not
belt-and-braces: sync-state.json is agent-authored, the agent server merges it
wholesale (merged.update(data)), so last_lock_recovery is fully agent-controlled
even on an agent where nothing ever reaped anything — and
git_service.get_git_status proxies response.json() UNMODIFIED to the UI and the
MCP tool.

Three guards, each for a named failure:

- isinstance(value, str) before parsing: parse_iso_timestamp raises
  AttributeError, not ValueError, on a non-str.
- except (ValueError, TypeError) around the parse AND the window comparison: a
  valid-but-naive ISO parses cleanly and then raises TypeError on aware > naive,
  and that raise lands in _sync_agent AFTER the upsert where
  gather(return_exceptions=True) swallows it — so the symptom is a lost alert
  with no traceback. The same guard with the same comment already exists in the
  file this record comes from.
- An explicit UTC offset is REQUIRED. startup.sh always stamps Z, so a naive
  value did not come from us, and the house posture over agent-authored JSON is
  to reject rather than guess.

The window is [now - 1h, now + 60s], with the hour deliberately a floor rather
than 2x the poll interval: startup.sh stamps `at` at container boot and the
agent server folds it in on its FIRST status read, and a cold boot can put
several minutes between those two. The clamp's job is to reject nonsense — the
far-future `at` that would otherwise be "newer" forever — and the DEDUP is what
stops a real record repeating.

Dedup is against the last OBSERVED value. The rejected alternative ("at newer
than the prior row's last_check_at") is not a dedup at all: last_check_at is
re-stamped to now on EVERY upsert, so 9999-01-01 would be newer on every tick,
per agent, for the life of the process — a WARNING asserting a platform action
that never happened. Both halves are tested, so neither can pass by rejecting
everything.

index_lock_stuck is edge-triggered and re-arms after the lock clears. No
operator-queue item and no DB column: the report is a diagnosis, not a decision.
The agent-supplied `path` is dropped at the boundary — it is composed from a
.git the agent can point anywhere and adds nothing to a fleet-level WARNING.

AC4 ships in three phases, strongest first:

- The GATE is deterministic AND genuinely concurrent: a real
  `git status --porcelain` child is SIGSTOP-frozen the instant it takes
  .git/index.lock, and the agent's own `git add` — a second real process — then
  fails rc!=0 with index.lock in stderr. 10/10 locally. SIGSTOP is the only way
  to hold that window open: git runs hooks and filters OUTSIDE the index lock
  (an fsmonitor hook sleeping 1s stretches status to 1279ms while the lock
  window stays 0.8ms). SIGCONT is in a `finally` — --timeout-method=signal
  raises inside the test and a leaked frozen child would hold the lock for the
  rest of the session. Its twin proves the flagged argv cannot be caught at all.
- The lock-sighting sampler (AC1, earlier commit) is the property.
- The threaded witness through the real get_git_status() route is
  self-validating and non-gating: its control arm must reproduce an index.lock
  failure IN THIS RUN or the test skips. Measured through this route the duty
  cycle is ~0.7% (about 95% of each iteration is git fetch), so the margin is
  roughly one failure per run — it skips ~2 runs in 5 here, which is exactly the
  point: it can never pass without having demonstrated it can fail.

Writer rounds write time.time_ns() and failures are classified by
"index.lock" in stderr, never by return code: `git commit` exits 1 with EMPTY
stderr when a round writes content identical to the previous one, which under
check=True is indistinguishable from a lock failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(git-sync): sync the adjacent flows and bound the leader (#2742)

Tail steps: /update-tests and /sync-feature-flows.

Two gaps the plan's test matrix listed and the implementation had not yet
covered, both now closed:

- test_leader_deadline_releases_the_slot — a wedged leader must release the
  in-flight slot rather than monopolise the read. It holds no follower threads
  on this design, but it does hold the slot, so every caller in that window
  504s.
- An AST guard pinning _STATUS_LEADER_DEADLINE_SECONDS to the status body's
  REAL child-timeout budget. They are both 90s today (10 rev-parse + 10 status
  + 10 log + 30 fetch + 10 merge-base + 10 log + 10 remote get-url). Adding a
  child, or widening one, now fails here instead of showing up as an
  unexplained 504 in production — and the flow doc's arithmetic is guarded with
  it. Plus the ordering invariant between the two bounds (caller < computation).

Flow-doc sync — one home per feature, so the two adjacent docs get pointers,
not copies:

- github-sync.md documents GET /api/git/status's own contract, so its endpoint
  row, its _persist_last_remote_sha row (whose child is now sweep-registered on
  the LOCKED sync path too), its stale "line 249-250" call-site reference, and
  its revision history all needed the #2742 delta.
- mcp-git-tools.md: the MCP tool proxies the agent payload verbatim, so it
  gains computed_at / lock_recovery / index_lock_stuck with no signature change
  — and its callers now coalesce with the poller and the UI panel rather than
  stacking a third overlapping git fetch.
- feature-flows.md's Git Sync Health category row and git-sync-health.md's
  index_lock_stuck field list corrected to match what the code actually emits.

The test-runner catalog entry lives in the private .claude submodule and is
committed THERE, on its own branch, deliberately WITHOUT bumping this repo's
gitlink — `git add -A` stages that gitlink silently under
`diff.ignoresubmodules=all`, which is what ejected #2606 from a merge train.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(git-sync): drop the ghp_-shaped placeholder from the redaction test (#2742)

Public repo. The fixture token was never real and is too short to match
GitHub's own secret-scanning format, but a `ghp_` prefix is exactly what the
repo's pre-commit checklist tells a reviewer to grep for, and the test's point
is that URL userinfo is stripped — not that it is a GitHub PAT specifically.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(learnings): record the to_thread-deletes-the-accidental-exclusion pitfall (#2742)

Moving the status handler off the event loop removed a mutual exclusion that
sync_to_github/pull_from_github were providing by never awaiting. _REPO_LOCK
was added for the cycle, so it does not cover a newly threaded path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(git-sync): replace the 29x flag-cost figure with fleet measurements (#2742)

Re-measured inside a real agent container on the ext4 workspace volume. The
penalty tracks content bytes re-hashed, not index size: ~1x on 42491 files of
~1 MB, ~390x on 1500 files of 294 MB. The single 29x figure was taken outside
the fleet and sat between the two regimes, describing neither.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(sync-health): wire the poll-cadence knob into both compose files (#2742)

`SYNC_HEALTH_POLL_INTERVAL_SECONDS` was read by `sync_health_service` at call
time — a property rather than an import-time copy, specifically so an operator
override is honoured — and then reached no container: it was absent from
`docker-compose.yml`, `docker-compose.prod.yml` and `.env.example` alike. The
knob shipped inert. Proven by rendering rather than grepping: before, the var
does not appear in `docker compose config`'s backend environment at all.

This is the #1056 packaging class (`VOIP_*`), which
`test_ent237_skill_source_env_packaging.py` records as having already recurred
seven times. Prod compose launches standalone — no base-compose merge and no
`env_file:` on the backend service — so the explicit `environment:` list is the
only route in, and wiring dev alone would not have carried over.

The `${VAR:-60}` pass-through form is the correct one here: a cadence has no
"disable" sentinel, so unset and empty must both land on the unchanged 60 s
default. `TestPollIntervalReachesTheContainer` asserts the FORM, not mere
presence, and pins the compose default against `DEFAULT_POLL_INTERVAL` rather
than a literal so the two cannot drift into a container that polls at a
different rate from a laptop. Verified to have teeth by deleting the prod
wiring and watching it go red.

Found by /validate-pr §4.9 on this branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(sync-health): the cadence knob must reach the hosted compose too (#2742)

`b6757ba66` wired `SYNC_HEALTH_POLL_INTERVAL_SECONDS` into `docker-compose.yml`,
`docker-compose.prod.yml` and `.env.example` — and missed
`docker-compose.hosted.yml`, the pull-only twin every marketplace and managed-host
channel actually runs. `test_2280_hosted_compose_parity` caught it in CI:
deterministic, all three head seeds, clean in all three base seeds.

That is the bug reproducing inside its own repair. The commit being fixed exists
because a knob never reached the container; it then failed to reach the container
that ships to marketplace installs. The hosted file's own header names the class
and lists its five prior victims (#1039, #1056, #1707, #1871, #2381) — this is
the sixth, and the first to be caught by the guard rather than by an operator.

The line is byte-identical to prod's, since #2280 compares the backend
`environment` list wholesale. No hosted-specific assertion is added here: that
guard is strictly stronger than anything this file could restate, and two guards
over one fact drift apart. The reasoning is recorded in the test's docstring so
the next reader does not "helpfully" add the redundant one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: sim <eugene@beingluminous.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants