feat(operator-queue): ask endings — one sink, endings ledger, person-only endings, wake on any ending, self-readback (abilityai/trinity-enterprise#611, PR A of 2)

[webmixgamer]

Description

An ask ends in exactly one of three ways — answered, cancelled or expired. This PR records every ending, audits it once, delivers it to the agent that raised it, and shows it with who and when. It is PR A of the ent#611 ask object; PR B adds the native create path (ask_operator over MCP).

• One sink. Every ending goes through services/ask_service.py: the operator answer, the Workspace answer, single cancel, bulk cancel and the poller's expiry. Before this there were five write sites, each doing its own subset: single cancel did none of audit, broadcast or wake; respond audited nothing; expiry never broadcast. Each ending now runs the same four steps:
  1. a compare-and-set UPDATE that also writes the endings ledger (disposition, disposed_at, disposed_by ∈ {person, timeout}, disposed_by_email, disposition_reason, batch_id);
  2. one audit row per transition, ids and enums only;
  3. one thin /ws trigger;
  4. the registered ending observers, which receive only the rows this call won.
• Only a person ends an ask. respond, cancel and bulk-cancel, and the Workspace answer, refuse agent-, system- and every other non-person key with 403 person_required. It is an allowlist over mcp_scope, and it runs before the row is read. Before this, an agent's own key resolved to its owner and could answer its own approval.
• No answer after the deadline: 409 expired, even before the poller sweeps the row. expires_at is stored ISO-Z at ingest (Invariant #16).
• Wake on any ending (the ent#329 opt-in).
  • An answer keeps its operator_response resume.
  • A cancel or an expiry wakes the filer once per agent per event (trigger operator_ending). An expiry carries the rider verbatim: "Denied by timeout; do not re-ask the same action without new information." The operator's reason is framed as data.
  • Platform alarms, rows the agent already closed, and stopped agents are skipped; the skip is audited.
• The agent reads its ending back.
  • GET /api/agents/{name}/operator-queue/{request_id} and MCP get_my_ask: self-only (identity comes from the key), a redacted projection, and still readable after Clear All.
  • An "Ended asks (last 24 h)" line in every composed Execution Context, carrying ids, disposition and time only.
• Surfaces.
  • The Operations Resolved card shows who ended the ask and when; the feed sorts by ending time (Credential-sync workflow missing for existing agents #627 AC6).
  • /m gets a "Recently ended" strip.
  • The Workspace lists asks that ended in the last 7 days, with a coarse who (you / the operator / timeout) and never an operator's email or reason. The sidebar count and "Waiting on you" stay pending-only.
  • The Clear-All and wake opt-in copy say only what the platform delivers.
• Migration pair: operator_queue_ask_object (SQLite) and Alembic 0076_operator_queue_ask_object on 0075_auto_sync_enabled_backfill. Twelve nullable TEXT columns, no backfill. Six serve the endings ledger; six serve the agent-raised ask that PR B writes.

Design decisions beyond the approved plan

• Opt-in pre-read in the default observer (ask_service._opted_in, one flag read per agent per event). An agent that has not opted in is never handed to a spawner, which keeps ent#430's rule that an opted-out agent is not dispatched at all. The spawned work re-reads the flag at spend time, and that read is the authority.
• The bug: POST /api/operator-queue/{id}/respond accepts any string as an approval decision — no layer checks response ∈ options #2376 offered-option check moved into the sink (answer() validates first). The sink is the one writer of an answer, so no entry point can skip the check. The routes map ResponseNotOfferedError to their existing 422 shapes, and the bug: POST /api/operator-queue/{id}/respond accepts any string as an approval decision — no layer checks response ∈ options #2376 AST guard now names services/ask_service.py.
• Thin triggers.
  • operator_queue_responded and the new operator_queue_cancelled carry {id, agent_name} only. operator_queue_responded used to broadcast the answer text and the responder's email.
  • The sink's bulk operator_queue_cleared drops cleared_by; it is allowlisted in the ent#467 guard with its reason.
  • Expiry sends no trigger of its own: the poll cycle's one operator_queue_sync announces it, even when Docker is unreadable.
  • _broadcast_payload is a named identity function so that the ent#467 guard, which follows a *broadcast*(…) call back to its dict literal, can read every payload.
• Workspace.
  • answerAsk replaces the row with the server's answered projection instead of removing it.
  • PortalAsks hides the answer controls and the sync badge on ended asks.
  • PortalWork passes pending-only.
• Trigger operator_ending is registered wherever operator_response is: the Executions filter, the analytics bucket, _AUTONOMOUS_TRIGGERS, PULL_REACHABLE_TRIGGERS, canvas_service._OPERATOR_SIDE_TRIGGERS, and the frontend KNOWN_TRIGGERS. operator_response was also missing from KNOWN_TRIGGERS: at 17 characters it exceeds the 16-character fallback and was folding into "other".
• The wake setting was renamed to "Wake this agent when its asks end". The user-doc anchor moved to #wake-when-an-ask-ends, and every live link was updated.

Fixed during review and /cso --diff (each with a failing test first and a red mutation)

• An extreme-offset deadline (past year 9999 once moved to UTC) raised at ingest, so the poller quarantined the ask instead of showing it. It is now kept verbatim, like unparseable text.
• The wake's container-state read was a blocking Docker call on the event loop. It now goes through the bug: Workspace roster lists agents with no container (DB-sourced roster vs Docker-as-truth) #2196 executor wrapper.
• An operator's Clear All made a client's ended ask vanish from the Workspace inside the 7-day window. The Workspace list now reads past cleared_at, which is the operator's list hygiene, just as the agent's readback does.
• The Clear-All confirmation promised a wake to opted-in agents that are stopped. It now says running agents.
• CSO F1 (MEDIUM, independently verified): a system-scoped key could still answer an ask through the Workspace answer route, recorded as disposed_by='person'. PortalPrincipal now carries is_person, and the answer route refuses a non-person with the same 403. The Workspace's read breadth for system keys (bug: Agent Detail and Workspace over-fetch on load (21 redundant requests, N+1 roster) #2198) is unchanged. Report: docs/security-reports/cso-diff-2026-09-25-ent611-ask-endings.md.

Related Issue

Part of abilityai/trinity-enterprise#611 (PR A of 2). The issue moves to status-in-dev only when PR B lands.

This PR has no closing keyword, on purpose. It is PR A of 2, and the issue closes with PR B. Adding one at merge time would mark the issue done before its second half ships.

Journey Impact

Journey Impact: extends: J05

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Behaviour change worth naming: agent- and system-scoped keys can no longer answer or cancel asks. An agent answering its own approval was self-approval; the MCP respond_to_operator_queue description says so.

Testing

• I have tested this locally
• New tests added (if applicable)
• All existing tests pass
• Every new test executes the changed path

Backend. tests/unit/test_ent611_ask_endings.py, 162 tests, real SQLite for every ledger writer. Neighbour suites were updated where the sink moved a call site: ent#329, ent#430 ×2, #2915, #2376, ent#467, ent#499, #2048, #2198. The full unit island ran sequentially on the merged tree (669f585a1): 18,209 passed and 40 failed. The 40 are the same order-dependent set as a clean cec2a64be run: test_ent666_objective_join ×21, test_ent477_* ×15 and test_retention_floor ×4. Each passes in isolation. They predate this PR and are now registered as debt.

Frontend. operatorQueueEnding.spec.js, portalAskEndings.spec.js, mobileAdminRecentlyEnded.spec.js (mounted, jsdom) and workspaceAsks.spec.js. The raw-colour, loading-gate and source-text ratchets are green with no baseline edits: the new UI reuses each file's existing class strings.

MCP. operator_queue.test.ts (the get_my_ask stub client) and access-wiring.test.ts (transport: identity from the key; a user-scoped key refused).

Guards. check_alembic_heads (one head, 0076) and check_alembic_parity both PASS. The enterprise-docs guard has 0 hits, with a positive control.

Mutation:

• answer() drops its _status_conflict refusal → TestSink lost/late-answer tests and the tightened G2 guard (test_ent329_operator_resume.py) go red.
• bulk_cancel hands observers the REQUESTED rows → TestTheWakeGetsOnlyWhatEnded::test_pending_a_and_already_cancelled_b_wake_for_a_only goes red.
• The wake drops the platform-alarm skip → TestWakeObserverRouting and test_ent499_problem_report.py go red.
• The get_my_ask call site uses a fixed agent → operator_queue.test.ts and access-wiring.test.ts go red.
• The ent#467 allowlist entry is removed → the guard names services/ask_service.py.
• Without OverflowError → TestCreate::test_expires_at_is_stored_as_iso_z[9999…] and test_the_stored_deadline_never_reads_as_a_rewrite[9999…] go red.
• A sync Docker read on the loop → TestEndingWake::test_the_running_check_never_blocks_the_event_loop goes red.
• include_cleared dropped → TestWorkspaceListing::test_an_operator_clear_all_never_hides_an_ending_from_the_person_it_was_for goes red.
• The Workspace refusal removed → TestWorkspaceAnswerIsPersonOnly::test_a_non_person_platform_principal_is_refused_before_the_row_is_read goes red.
• is_person never computed → TestWorkspaceAnswerIsPersonOnly::test_the_platform_path_carries_whether_the_caller_is_a_person[system-False] goes red.

Each file was restored byte-identically.

What CI does not run

• The live-stack walk-through, done on a local instance before this PR:
  • answer, cancel with a reason, bulk cancel (2 cancelled and 1 skipped, one batch_id on exactly the flipped rows), and an expiry swept about 2 s after its deadline, each ledgered;
  • audit rows answered / cancelled (has_reason, no text) / bulk_cancel / expired;
  • on an opted-in agent: 1 operator_response and 2 operator_ending wakes, all successful, with the expiry wake carrying the rider verbatim. A not-opted-in agent was never woken;
  • the agent's own key refused on respond (403 person_required);
  • get_my_ask over MCP returning the redacted projection;
  • the surfaces checked in both themes: the Resolved card, the Clear-All and Reliability copy, the /m strip, and the Workspace flow as a client.
• PostgreSQL. The 0076 upgrade on real PostgreSQL runs only in the path-filtered pg-migrations job.
• The pull-worker path does not compose the platform prompt (bug: pull-claimed turns bypass platform system prompt composition — platform contract never delivered on the pull path #1629), so pull-claimed turns do not get the "Ended asks" line.

Merge notes

• Migration. 0076_operator_queue_ask_object parents on 0075_auto_sync_enabled_backfill. It was 0075 on 0074 until fix(git-sync): the auto-sync toggle is authoritative and live (#3010) #3017 landed its own 0075 on the same parent; it was re-parented and renumbered then (the two touch disjoint tables). If another revision lands on dev first, re-parent down_revision onto the new head at merge; alembic-head-watch will comment.
• Lane C (dependencies.py, client_portal/portal_auth.py): journey-smoke and frontend-e2e must be green on this PR.
• Registered, not fixed here:
  • agent keys still read the pre-existing responded_by_email / addressed_to_email on queue reads;
  • the wake is not durable across a crash;
  • the expiry wake is an uncapped agent-driven self-trigger (independent verification: no worse than the uncapped self-scheduling that already exists; bounded by max_parallel_tasks).
• Separate PR: the pre-existing bug(operator-queue): a pending approval can sit unseen for days — the human's card goes stale while the container↔platform sync stays silent #2915 stale_id mis-flag, where every ended ask reads "Re-used id" on the Resolved card.

Checklist

• My code follows the project's style guidelines
• I have updated the documentation (if applicable)
  • requirements security.md §26.6 and new §26.9 OPS-001-ENDINGS;
  • feature flows operating-room.md (Endings), execution-context-injection.md, mobile-admin-pwa.md, workspace-work.md, workspace-agents-at-the-centre.md, workspace-rail.md;
  • architecture api-endpoints.md, database.md, mcp-server.md, background-services.md, workspace.md;
  • user docs (approvals, the wake setting, the FAQs, executions);
  • three learnings fragments.
• I have not committed any sensitive data (API keys, credentials, etc.)
• I have added appropriate logging for new functionality

🤖 Generated with Claude Code

[webmixgamer]

The two red checks (pg-migrations, schema-parity) are the SQLAlchemy 2.1.0 test-environment breakage, not this PR.

• tests/requirements-test.txt sets only a floor, sqlalchemy>=2.0.0, so CI resolved the new 2.1.0.
• 2.1.0 makes psycopg 3 the default driver for postgresql://, but these jobs install psycopg2.
• So alembic upgrade head fails at import_dbapi (ModuleNotFoundError: No module named 'psycopg') before any revision runs.

The same failure appears on other open PRs; #3015 caps the test pin. Production images pin sqlalchemy==2.0.36 and are unaffected.

I'll re-run once #3015 lands. Everything else is green, including journey-smoke, e2e, the regression diff and pytest (head).

[webmixgamer]

/review Report

Branch: feature/ent611-ask-object → dev. Merge-base is cec2a64be; dev's one later commit, #3012, came in by the merge 669f585a1.
Files Changed: 82 (+4903 / −520)
Scope: CLEAN. The intent is PR A of abilityai/trinity-enterprise#611: how an ask ends. The diff delivers one transition sink, the endings ledger, person-only endings, a wake on any ending, the agent's own readback, the Execution Context line and the surfaces. PR B's create path is untouched.
Plan Completion: every PR A item of the approved plan is done, with no gaps.

Execution coverage

changed symbol	executed by	live consumer	verdict
ask_service.answer / cancel / bulk_cancel / expire	TestSink, TestEndingRoutes (real SQLite)	the operator routes, the Workspace answer_ask, the poll cycle	✅ executed
_wake_filer → maybe_dispatch_ending	TestEndingWake, TestWakeObserverRouting, route-level TestTheWakeGetsOnlyWhatEnded	the sink's _ended	✅ executed
is_person_principal, reject_non_person_principal, get_self_acting_agent, PortalPrincipal.is_person	TestPersonGate*, TestSelfReadback, TestWorkspaceAnswerIsPersonOnly, each principal kind separately	respond, cancel and bulk-cancel; the readback route; the Workspace answer	✅ executed
the G2, #2376, ent#430 and ent#467 AST guards	CI	route and sink source	🛡 guard, backed by the behavioural tests above

Fix mutations: 10 in total, all red, each on the line that applies the fix and restored byte-identically. They are listed in the PR body under Mutation.

Critical Findings

None.

Informational Findings — all fixed on this branch, each with a failing test first

• [I1] Error handling. db/operator_queue.py::_iso_z_deadline let an OverflowError escape for a deadline past year 9999 once moved to UTC. The create then raised, and the poller quarantined the ask instead of showing it. Its fingerprint twin, _normalise_expires, already caught that error. The value is now kept verbatim.
• [I2] Event-loop blocking. maybe_dispatch_ending called the synchronous Docker read inside a coroutine. It now uses docker_utils.agent_container_state_async, the bug: Workspace roster lists agents with no container (DB-sourced roster vs Docker-as-truth) #2196 executor wrapper, and still never raises.
• [I3] Requirement gap (C2). The Workspace list now carries ended asks, but it had inherited include_cleared=False. So an operator's Clear All hid a client's ended ask inside the 7-day window. It is now include_cleared=include_ended, the rule the agent's own readback already follows.
• [I4] Copy vs. the issue's AC. The Clear-All confirmation promised a wake to opted-in agents that are stopped, but those are skipped. It now says "running agents".

Security (/cso --diff)

One MEDIUM finding, independently verified and fixed here. A system-scoped key could answer an ask through the Workspace answer route, and the answer was recorded as disposed_by='person'. The report is in docs/security-reports/cso-diff-2026-09-25-ent611-ask-endings.md.

Clean categories

• SQL: parameterised throughout. The only f-string SQL is the migrations' constant column tuple.
• Races: every ending is a compare-and-set. Bulk cancel re-selects its winners by batch_id, and expiry is a per-id compare-and-set.
• Auth: the gate is an allowlist over mcp_scope and fails closed on a missing attribute. The readback checks identity first and answers with a uniform 403.
• Credentials: none found in the added lines.
• Enterprise-docs guard: 0 hits, and the positive control fires.
• Enum completeness: every reader of operator_response also knows operator_ending, and derive_mode maps it to task.
• Docs: requirements, feature flows and the architecture area files are updated to match.

Residuals registered, not changed here

• Agent keys still read the pre-existing responded_by_email / addressed_to_email on queue reads.
• The wake is not durable across a crash. The readback and the context line still carry the ending.
• The expiry wake is an uncapped agent-driven self-trigger. It is no worse than the self-scheduling that already exists, and max_parallel_tasks bounds both.

Summary

• Critical: 0
• Informational: 4, all fixed on this branch
• Scope: clean
• CI: all green except pg-migrations and schema-parity. Both fail on the test environment's SQLAlchemy 2.1.0 driver change before any revision runs; fix(ci): cap test SQLAlchemy below 2.1 — pg-migrations broke on the new default driver #3015 caps the version. A re-run is still needed after fix(ci): cap test SQLAlchemy below 2.1 — pg-migrations broke on the new default driver #3015 lands, because 0075 has not yet run on PostgreSQL.

[github-actions]

✅ Alembic head check clear — merging this PR into dev leaves one head (0075_auto_sync_enabled_backfill).

Previously flagged; resolved.

Advisory — this check does not block merge. · head_sha: dfcb7c07bfddb434cf62887d3d74a10d5e71121e · run

[vybe]

Validated at lane C + schema, head dfcb7c0: READY. No criticals. All checks green on this head. The four terminal writes are CAS-gated on status == 'pending' with the ledger in the same UPDATE; the person gate is an allowlist that runs before the row read; new /ws triggers carry identifiers only; the three schema sources agree column for column and the tree resolves to one Alembic head (0076).

Follow-ups, none blocking, all for your call:

1. Ended asks above the composer are unbounded. PortalConversation.vue:435-443 mounts PortalAsks without pending-only, and fetchAsks now sends include_ended: true, so the shrink-0 block above the composer renders every ask that ended in the last 7 days (server limit 200) with no max-height or scroll. On a phone a handful of ended asks takes most of the screen. It needs a bound or a collapsed form; which one is a design choice.
2. Cancel reason has no in-product producer: stores/operatorQueue.js:209 sends { ids } only, so the field is reachable by raw API alone.
3. to_role, resolved_to, proposal and supersedes_expired have no writer until #3028 lands.
4. The expiry wake is agent-drivable: an opted-in agent can file short-deadline asks and be woken each poll cycle (operator_resume_service.py, maybe_dispatch_ending). Bounded by the opt-in and max_parallel_tasks, but uncapped.
5. Machine keys are denied disposed_by_email but still receive responded_by_email, which holds the same value on answered rows.
6. client_portal/asks/service.py answer_ask reports resume_requested from the opt-in alone, so a platform-minted row whose observer skips dispatch still reports true.

The branch is left in place because #3028 is stacked on it.