Replace spawn-on-demand pool servicing with a persistent scheduler actor

Summary

Replace the pool scheduler's Arc<Mutex<SchedulerState>> plus atomic service guard and transient tokio::spawn calls with one long-lived scheduler task that owns the fairness and waiter state.

This is the final structural implementation slice for ADR 013 proposal #638. It follows the stable slot graph in #645 and should consume the scoped permit-selection API from #646.

Current behaviour

The current scheduler:

shares SchedulerState through a synchronous mutex;
uses AtomicBool::is_servicing as a distributed task-lifecycle guard;
calls kick from acquire and lease-drop paths;
spawns service_waiters when the guard transitions from idle;
may spawn a task that immediately discovers no waiter and exits;
divides shutdown, waiter selection, restart recovery, and capacity notification among several caller contexts.

This topology makes one conceptual state machine look like shared mutable data. It also complicates #550's bounded-waiter admission and leaves adversarial interleavings from #593/#535 difficult to test directly.

Proposed architecture

Create one scheduler task during pool construction. It owns SchedulerState directly and receives commands through a Tokio channel.

A representative protocol is:

enum SchedulerCommand<S, P, C> {
    Acquire {
        handle_id: u64,
        reply: oneshot::Sender<Result<PooledClientLease<S, P, C>, ClientError>>,
    },
    DeregisterHandle {
        handle_id: u64,
    },
    CapacityAvailable,
    Shutdown {
        reply: oneshot::Sender<()>,
    },
}

The exact types may differ, but the protocol and ownership must remain explicit.

Ownership split

Keep scheduler control and physical slot storage separate:

Arc<PoolCore> from Introduce PoolCore and index-based pooled leases #645 owns fixed slots and shared physical resources;
a cloneable SchedulerHandle owns only the command sender and synchronous handle-ID allocation if retained;
SchedulerTask owns the receiver, SchedulerState, fairness rotation, waiter counts, and shutdown state;
the task may own Arc<PoolCore>, but PoolCore must not own the command sender in a way that creates a strong cycle.

Dropping the final public scheduler sender must eventually close the mailbox, terminate the task, and release the task's pool-core reference.

Responsibilities of the actor

Admission

reject new acquire commands after shutdown begins;
enforce the waiter bound and/or acquisition deadline required by [security][low] Client pool admits unbounded blocked waiters under backpressure #550;
return a typed error when admission fails;
prune cancelled oneshot replies so abandoned callers do not consume waiter capacity indefinitely.

Fairness

preserve FIFO arrival order for live admitted waiters;
preserve round-robin turns among logical handles;
handle deregistration idempotently;
never grant a lease to a cancelled or deregistered waiter.

Capacity

attempt an immediate grant when an acquire command arrives and capacity exists;
wait for capacity without spawning a task per lease release;
use Race borrowed pool-slot permits without per-slot Arc cloning #646's slot-index/permit selection;
treat duplicate CapacityAvailable messages as harmless hints, not extra permits.

Shutdown

reject later acquires;
resolve every queued waiter with ClientError::disconnected or the chosen typed shutdown error;
cancel or drop any pending permit-selection future safely;
acknowledge explicit close() only after scheduler state has drained and the actor can terminate;
terminate on mailbox closure even when callers omit explicit close.

Public API behaviour

Preserve existing WireframeClientPool, PoolHandle, and PooledClientLease APIs where practical.

WireframeClientPool::handle() may remain synchronous by allocating a logical ID outside the actor. The implementation must prove command ordering for acquire and deregistration from one logical handle. A registration acknowledgement is acceptable if the public API change is justified, but do not quietly assume cross-sender global ordering that Tokio does not promise.

WireframeClientPool::close(self) becomes an awaitable scheduler shutdown rather than shutdown(); yield_now(); drop(self).

Drop behaviour

PooledClientLease::drop cannot await. It should:

release its OwnedSemaphorePermit by ordinary drop;
send or coalesce a non-blocking capacity hint to the actor;
tolerate an already-closed scheduler mailbox during shutdown;
avoid cloning a large ownership graph;
never spawn a task.

If permit release can be observed without a per-drop command, for example through a shared Notify polled by the actor, that is acceptable if it preserves the single-owner scheduler state and has no lost-wakeup race.

Integration with existing issues

Simplify client pool waiter state transitions #535: reuse its state-transition simplification inside the actor or close it as superseded with an explicit explanation.
Add pool poison recovery integration coverage #539: reassess whether scheduler poison-recovery integration remains relevant once scheduler state is no longer behind std::sync::Mutex; retain slot-bookkeeping scope if still applicable.
[security][medium] Cancelled pooled client operations can return dirty sockets as reusable #548: do not grant reuse of a dirty physical connection merely because capacity returned. Cancellation safety remains a separate invariant.
[security][low] Client pool admits unbounded blocked waiters under backpressure #550: implement bounded waiter admission in the actor rather than adding another shared counter outside it.
Mutation testing: client pool lifecycle and slot rotation survivors #593: preserve and extend shutdown, rotation, and waiter behavioural tests.

Acceptance criteria

Tests

Add deterministic state-machine and runtime tests covering:

immediate uncontended grant;
FIFO ordering under saturation;
round-robin ordering across several logical handles;
cancelled waiter before and during grant;
handle deregistration with queued waiters;
bounded waiter rejection;
capacity release racing with acquire and shutdown;
explicit close with blocked waiters;
implicit shutdown when all senders drop;
actor task count remaining one under a long acquire/drop loop;
no PoolCore retention after final handle/lease drop.

Use loom or a small pure scheduler-state model for event-order races that cannot be forced reliably with wall-clock Tokio tests.

Non-goals

Moving socket request/response I/O into the scheduler actor.
Dynamically resizing the slot collection.
Replacing Tokio channels or semaphores.
Weakening cancellation safety to simplify capacity accounting.

Dependencies

ADR 013 proposal ADR 013 proposal: Single-owner client-pool scheduler and slot graph #638.
Baseline issue Establish runtime ownership and task-churn baselines #639.
Pool core Introduce PoolCore and index-based pooled leases #645.
Scoped permit selection Race borrowed pool-slot permits without per-slot Arc cloning #646.

References

Epic Epic: Align Tokio task boundaries with ownership boundaries #635
ADR 011 proposal ADR 011 proposal: Runtime ownership and task-lifetime boundaries #636
ADR 013 proposal ADR 013 proposal: Single-owner client-pool scheduler and slot graph #638
Simplify client pool waiter state transitions #535
Add pool poison recovery integration coverage #539
[security][medium] Cancelled pooled client operations can return dirty sockets as reusable #548
[security][low] Client pool admits unbounded blocked waiters under backpressure #550
Mutation testing: client pool lifecycle and slot rotation survivors #593
Establish runtime ownership and task-churn baselines #639
Introduce PoolCore and index-based pooled leases #645
Race borrowed pool-slot permits without per-slot Arc cloning #646
src/client/pool/scheduler.rs
src/client/pool/client_pool.rs
src/client/pool/handle.rs
src/client/pool/lease.rs

🔗 Related PRs

#540 - Approve the actor and codec-driver boundary (10.1.3) [merged]
leynos/pg-embed-setup-unpriv#157 - Add verification tests for lifecycle invariants (#152) [merged]
#552 - Align roadmap footnotes with the roadmap grammar [open]
#612 - Kill mutation-testing survivors and widen scaffolding excludes (#566, #567, #569, #570, #593, #595, #598) [merged]
leynos/cuprum#232 - docs: mark #125 FD-ownership hazard resolved in roadmap 8.1 (#125) [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

Record the pool acquire-path defects instead of patching them (#639)

Report contended pool allocations as an observation

Separate the two pool acquire-path defects in the recorded baselines

Record the pool acquire-path defects instead of patching them (#639)

Separate the two pool acquire-path defects in the recorded baselines

Record the pool acquire-path defects instead of patching them (#639)

Report contended pool allocations as an observation

Separate the two pool acquire-path defects in the recorded baselines

Establish runtime ownership and task-churn baselines (#639) (#756)

Document borrowed permit selection and isolate flaky size-16 case (#646)

Race borrowed pool-slot permits without per-slot Arc cloning (#646) (#…

@coderabbitai plan

Coding Plan

Summary

Move SchedulerState into one persistent SchedulerTask spawned at pool construction. The task receives Acquire and DeregisterHandle through an unbounded mailbox. It enforces the max_waiters admission bound itself.
Observe returned capacity through one RAII-guarded pending acquire_slot_permit race. Leases then hold no scheduler reference. Lease drop releases only its permit and never sends or spawns.
Deliver shutdown through the existing PoolCore shutdown Notify. Await the actor JoinHandle in close(). Report PoolClosed for closure and SchedulerFailed for a supervised actor panic. Terminate the actor when the final sender drops, so that PoolCore is released.
Prove the event-order races with synchronous handler models. Prove the lifecycle and task-count criteria with current-thread runtime tests. Record the #639 baselines with updated actor-era assertions.

Design Choices

Design Choice 1: Bounded or unbounded command mailbox

Options Considered:

Use a bounded mpsc mailbox. Acquire uses send().await and DeregisterHandle uses try_send. A lost deregistration needs delayed cleanup through sweep pruning.
Use an unbounded mpsc mailbox. The actor enforces the max_waiters admission limit separately. Drop-path sends never fail while the actor is alive.

Chosen Option: 2

Rationale: Deregistration cannot be lost and per-sender order is preserved. Admission stays bounded inside the actor, which is the exception that ADR 013 accepts.

Design Choice 2: How the actor learns that capacity returned

Options Considered:

PooledClientLease::drop sends a coalesced CapacityAvailable hint through a sender that the lease holds.
The actor keeps one pending PoolCore::acquire_slot_permit race while waiters are queued. The semaphore then signals capacity at level, so no lost wakeup can occur.

Chosen Option: 2

Rationale: It removes release_control from leases and removes every send from lease drop. Tokio semaphore fairness delivers the next released permit to the queued race.

Design Choice 3: Shutdown delivery mechanism

Options Considered:

Send a Shutdown { reply } command through the mailbox.
Use the existing PoolCore::shutdown flag and its Notify as a dedicated channel. Set a shared status first. The close owner then awaits the actor JoinHandle.
Add a CancellationToken from tokio-util.

Chosen Option: 2

Rationale: It reuses the lost-wakeup-safe shutdown_notified_after_check path. It adds no dependency. Awaiting the JoinHandle is the acknowledgement that ADR 013 pins.

Design Choice 4: Error reported for shutdown

Options Considered:

Keep ClientError::disconnected().
Add ClientError::PoolClosed and ClientError::SchedulerFailed. Neither variant is recyclable.

Chosen Option: 2

Rationale: Scheduler closure is not a broken socket, and the ADR verification requires SchedulerFailed for an actor panic. Document the visible behaviour change.

Design Choice 5: Handle registration for round-robin

Options Considered:

Send a RegisterHandle command from handle(). This depends on cross-clone ordering.
Allocate the ID synchronously. Register the ID in the actor on the first Acquire for it, and remove it on DeregisterHandle. This relies only on the per-sender order of each handle's own sender.

Chosen Option: 2

Rationale: It needs no API change and no ordering assumption across clones. It replaces the UnregisteredHandle rejection.

Design Choice 6: Uncontended acquisition path

Options Considered:

Route every acquisition through the actor, which then tries an immediate grant.
Keep a caller-side try_acquire_permit fast path that bypasses the actor.

Chosen Option: 1

Rationale: Admission and fairness then stay with one owner, as the ticket and #550 require. Record the latency cost in the #639 benchmarks.

Design Choice 7: Acquisition deadline (`#550`)

Options Considered:

Move the existing max_waiters bound into the actor only.
Add AcquireTimeout and deadline configuration as well.

Chosen Option: 1

Rationale: The waiter bound satisfies the acceptance criterion. Leave the deadline as follow-up work under #550.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Typed errors and an actor-owned scheduler model

Define the error vocabulary. Turn the scheduler state into a plain struct with synchronous transition handlers. Define the command protocol and the cloneable scheduler handle. These tasks fix the protocol and the pure model that Phase 2 drives and Phase 3 tests.

Task 1: Task 1.1: Add `PoolClosed` and `SchedulerFailed` errors

Give callers typed, non-recyclable errors for pool closure and actor failure.

Update src/client/error.rs:
- Add ClientError::PoolClosed.
- Add ClientError::SchedulerFailed.
- Add constructor helpers next to pool_saturated.
Update ClientError::should_recycle_connection to return false for both new variants.
Update the close doc example in src/client/pool/client_pool.rs and the related doc text. Show PoolClosed instead of the disconnected I/O error.

Task 2: Task 1.2: Convert `SchedulerState` into an actor-owned model

Remove the shared-mutex model. Keep the FIFO and round-robin semantics. Make every transition a synchronous method.

Move SchedulerState out of scheduler/mod.rs into a new src/client/pool/scheduler/state.rs. Store it by value. Do not wrap it in Mutex or Arc.
Remove servicing_waiters. A waiter stays queued until the actor holds a permit for it. The waiter count is therefore the FIFO length plus the per-handle queue lengths. This matches the old bound, which also counted the waiter held by the worker.
Replace the UnregisteredHandle rejection with lazy registration:
- The first round-robin Acquire for an ID inserts the ID into round_robin_order and handle_waiters.
- DeregisterHandle removes the ID and drops its queued senders.
- DeregisterHandle is idempotent. An unknown ID is a no-op.
Keep prune_cancelled_waiters and run it before every admission check.
Make take_next_waiter skip senders whose is_closed() is true, for both policies. This grant-sweep pruning means a cancelled waiter never receives a lease.
Keep the round-robin rotation rule unchanged: pop the visited ID from the front, push it to the back, then pop its oldest live sender.
Keep WaiterEnqueueResult::Saturated and the max_waiters limit. Remove the UnregisteredHandle variant.

Task 3: Task 1.3: Define the command protocol, shared status and `SchedulerHandle`

Make the actor protocol and its ownership explicit.

Create src/client/pool/scheduler/command.rs with a crate-private SchedulerCommand<S, P, C> enum:
- Acquire { handle_id, reply }, where reply is the existing oneshot lease reply type.
- DeregisterHandle { handle_id }.
- A test-only (cfg(test)) variant that injects a panic for failure tests.
- Do not add CapacityAvailable or Shutdown variants. Assumptions 2 and 3 give the reasons.
Create src/client/pool/scheduler/handle.rs with a cloneable crate-private SchedulerHandle. It holds:
- the unbounded mpsc command sender;
- an Arc to a small shared status record.
The shared status record holds:
- the AtomicU64 handle-ID counter, which starts at 1;
- the lifecycle state (Running, Closed, Failed);
- the churn counters;
- a per-pool identifier for logs and metrics. Allocate a process-local pool ID at construction if no pool identifier exists.
- It holds no sender, no receiver and no PoolCore.
Give SchedulerHandle these operations:
- register_handle allocates the next ID synchronously.
- acquire sends Acquire and awaits the reply.
- deregister sends DeregisterHandle without blocking and ignores a closed mailbox.
- close_error maps the shared status to SchedulerFailed or PoolClosed.
Map a send failure or a dropped reply sender through close_error. Do not map it to disconnected().
Fail early in acquire if the status is not Running.

🤖 Prompt for AI agents

Implement Phase 1: add typed errors, convert the scheduler state into an owned
model, and define the command protocol and `SchedulerHandle`.

- In `src/client/error.rs`, add `ClientError::PoolClosed` and
`ClientError::SchedulerFailed` with constructor helpers next to
`pool_saturated`. Make `should_recycle_connection` return `false` for both.
Update the `close` doc example in `client_pool.rs` to show `PoolClosed`.
- Move `SchedulerState` into `scheduler/state.rs` as a plain owned struct with
no `Mutex` or `Arc`.
  - Remove `servicing_waiters` and the `UnregisteredHandle` variant.
  - Count waiters as FIFO length plus per-handle queue lengths.
  - Register round-robin handle IDs lazily on the first `Acquire`.
- Make `DeregisterHandle` remove the ID and drop its queued senders. Make it
idempotent.
  - Keep `prune_cancelled_waiters` and run it before every admission check.
  - Make `take_next_waiter` skip senders with `is_closed()` for both policies.
- Keep the round-robin rotation rule and `WaiterEnqueueResult::Saturated` with
`max_waiters`.
- Create `scheduler/command.rs` with crate-private `SchedulerCommand<S, P, C>`:
`Acquire { handle_id, reply }`, `DeregisterHandle { handle_id }` and a
`cfg(test)` panic-injection variant. Add no `CapacityAvailable` or `Shutdown`
variant.
- Create `scheduler/handle.rs` with a cloneable crate-private `SchedulerHandle`
holding an unbounded `mpsc` sender and an `Arc` to a shared status record.
- The status holds an `AtomicU64` ID counter starting at 1, the lifecycle state
(`Running`, `Closed`, `Failed`), the churn counters and a pool identifier.
Allocate a process-local pool ID if none exists.
  - The status holds no sender, receiver or `PoolCore`.
- Implement `register_handle`, `acquire`, a non-blocking `deregister` that
ignores a closed mailbox, and `close_error`.
- Map send failure or a dropped reply through `close_error`, never
`disconnected()`.
  - Fail early in `acquire` if the status is not `Running`.
- Keep every file under 400 lines. Do not use `.expect()` in production code.
Use en-GB-oxendict spelling.

Phase 2: Persistent actor task and lifecycle wiring

Run one actor task per pool. Wire construction, acquisition, deregistration, close and implicit termination to it. Remove the spawn-on-demand machinery. These tasks change the runtime ownership graph together.

Task 1: Task 2.1: Implement the scheduler actor loop with a pending-grant guard

Give one task ownership of the receiver, the state and the permit race.

Create src/client/pool/scheduler/actor.rs with a SchedulerTask that owns:
- the command receiver;
- SchedulerState;
- an Arc<PoolCore>;
- the Arc to the shared status.
Keep the async loop thin. Put all decisions in synchronous handlers so tests can drive them without a runtime race:
- handle_command handles Acquire and DeregisterHandle.
- grant_sweep handles immediate grants.
- on_permit_won handles race results.
- drain_for_shutdown handles shutdown.
grant_sweep loops while live waiters exist and PoolCore::try_acquire_permit succeeds. For each permit it:
- pops the next live waiter by policy;
- builds the lease with PooledClientLease::new;
- sends the lease to the waiter.
- If the send fails, it drops the lease, which releases the permit. Then it continues.
- If no live waiter exists, it drops the permit.
On Acquire, the actor runs these steps:
- prune cancelled waiters;
- check admission and reply PoolSaturated { limit } when the queue is full;
- enqueue the waiter;
- run grant_sweep;
- if the waiter is still queued, call record_capacity_wait.
Hold the PoolCore::acquire_slot_permit race in a PendingGrant RAII guard:
- Start the race only while live waiters remain after grant_sweep.
- Dropping the guard cancels the race. This holds for shutdown, mailbox closure and an empty queue.
- Store the race future in a pinned slot that the loop reuses. Do not allocate a new box for each race where practical.
Use one biased tokio::select! with these branches in this order:
1. shutdown (core.shutdown_notified_after_check);
2. the pending race, which calls on_permit_won and then grant_sweep;
3. recv().
Exit the loop when recv() returns None. All senders are then gone. Drop the state, drop the guard and return, which releases the actor's Arc<PoolCore>.
On shutdown, the actor runs these steps:
- set the status to Closed;
- drop the guard;
- reply PoolClosed to every queued waiter;
- call receiver.close();
- drain the buffered commands, replying PoolClosed to each Acquire;
- return.
Delete src/client/pool/scheduler/service.rs.
Remove is_servicing, try_begin_servicing, stop_servicing, restart_if_waiters, kick, notify_capacity_available, resolve_immediate_lease and PoolScheduler from scheduler/mod.rs. Keep scheduler/mod.rs as the module root and the spawn entry point.

Task 2: Task 2.2: Add termination supervision and failure reporting

Make abnormal actor termination permanent and visible, and keep it distinct from closure.

Add a termination guard in scheduler/actor.rs.
- Declare it after the receiver and the state, so it drops first during unwinding.
- If std::thread::panicking() is true, set the status to Failed.
- Then emit an error-level tracing event with the pool identifier.
- Then increment a failure metric.
- Publishing Failed before the reply senders drop lets waiting callers observe SchedulerFailed.
Add wireframe_pool_scheduler_failures_total in src/metrics/client_pool.rs. Label it with the pool identifier. Use the existing cfg(all(feature = "metrics", feature = "pool", not(loom))) gate and the existing no-op fallback.
Do not restart the actor. Later acquisitions return SchedulerFailed from the status check.
Do not add queued-waiter or liveness gauges in this ticket. The acceptance criteria do not require them. Record them in ADR 013 as remaining observability work.

Task 3: Task 2.3: Wire pool construction and an awaitable `close`

Create exactly one actor per pool. Hold the JoinHandle only in the non-shared close owner.

Update WireframeClientPool::connect in src/client/pool/client_pool.rs:
- Build PoolCore.
- Build the shared status and the unbounded channel.
- Spawn SchedulerTask once.
- Record record_scheduler_spawn once.
- Store the SchedulerHandle and the JoinHandle in the pool.
Replace PoolSchedulerControl with SchedulerHandle in the pool struct. Do not give PoolCore any sender or JoinHandle.
Rewrite close(self) with these steps:
1. Set the status to Closed.
2. Call core.shutdown(). This wakes the actor and any permit waits.
3. Drop the pool's own sender and its core reference.
4. Await the JoinHandle.
5. If the JoinError reports a panic, log it.
Remove yield_now() from close.
Update the test-only TestPool constructor used by client_pool_tests.rs so that it spawns the actor in the same way. Make sure its callers run inside a Tokio runtime.

Task 4: Task 2.4: Update `PoolHandle` and `PooledClientLease`

Route acquisition and deregistration through the actor. Strip the scheduler from leases.

Update src/client/pool/handle.rs:
- PoolHandle holds its own SchedulerHandle clone and its handle_id.
- Remove the core field if no other handle method reads it.
- acquire(&mut self) delegates to SchedulerHandle::acquire.
- Drop calls SchedulerHandle::deregister. The handle's own sender carries both commands, so per-sender FIFO places DeregisterHandle after that handle's Acquire.
Keep pool.handle() synchronous. Keep pool.acquire() as a temporary handle.
Update src/client/pool/lease.rs:
- Remove release_control and the notify_capacity_available call from Drop.
- Remove the control argument from PooledClientLease::new.
- Keep the pending_responses > 0 discard before _permit drops, which preserves #548.
- Make sure lease drop performs no send and no tokio::spawn.
Remove the obsolete try_acquire_immediately and acquire_for_handle paths.

Task 5: Task 2.5: Re-point task-churn counters at the actor model

Keep PoolTaskChurn meaningful for a persistent task.

Update src/client/pool/churn.rs:
- Change the module and field docs to state that scheduler_spawns is 1 per pool, recorded at construction.
- State that acquisition adds 0 to scheduler_spawns.
- State that capacity_waits counts admitted waiters that the actor could not grant at once.
Keep PoolTaskChurnObserver unchanged.
Remove the scheduler's use of lock_or_recover in src/client/pool/sync.rs. Keep lock_or_recover for last_returned_at, and update its doc comment to list only the remaining callers.

🤖 Prompt for AI agents

Implement Phase 2: run one persistent actor per pool and remove the
spawn-on-demand machinery.

- Create `scheduler/actor.rs` with `SchedulerTask` owning the command receiver,
`SchedulerState`, `Arc<PoolCore>` and the shared status `Arc`.
- Keep the async loop thin. Put decisions in synchronous handlers:
`handle_command`, `grant_sweep`, `on_permit_won`, `drain_for_shutdown`.
- `grant_sweep` loops while live waiters exist and
`PoolCore::try_acquire_permit` succeeds. Pop the next live waiter by policy,
build the lease with `PooledClientLease::new` and send it. If the send fails,
drop the lease and continue. If no live waiter exists, drop the permit.
- On `Acquire`: prune cancelled waiters, check admission and reply
`PoolSaturated { limit }` when full, enqueue, run `grant_sweep`, and call
`record_capacity_wait` if the waiter is still queued.
- Hold the `PoolCore::acquire_slot_permit` race in a `PendingGrant` RAII guard.
Start it only while live waiters remain after `grant_sweep`. Dropping it cancels
the race. Reuse a pinned slot where practical.
- Use one biased `tokio::select!` in this order: shutdown
(`core.shutdown_notified_after_check`), the pending race (call `on_permit_won`
then `grant_sweep`), `recv()`.
- When `recv()` returns `None`, drop the state and the guard and return, so the
actor releases its `Arc<PoolCore>`.
- On shutdown: set status `Closed`, drop the guard, reply `PoolClosed` to every
queued waiter, call `receiver.close()`, drain buffered commands replying
`PoolClosed` to each `Acquire`, and return.
- Add a termination guard declared after the receiver and state. If
`std::thread::panicking()` is true, set status `Failed`, emit an error-level
`tracing` event with the pool identifier, then increment
`wireframe_pool_scheduler_failures_total`. Add that metric in
`src/metrics/client_pool.rs`, labelled by pool identifier, behind the existing
`cfg(all(feature = "metrics", feature = "pool", not(loom)))` gate with the
existing no-op fallback. Do not restart the actor. Add no gauges.
- Delete `scheduler/service.rs`. Remove `is_servicing`, `try_begin_servicing`,
`stop_servicing`, `restart_if_waiters`, `kick`, `notify_capacity_available`,
`resolve_immediate_lease` and `PoolScheduler` from `scheduler/mod.rs`. Keep
`mod.rs` as module root and spawn entry point.
- In `WireframeClientPool::connect`: build `PoolCore`, the shared status and the
unbounded channel; spawn `SchedulerTask` once; call `record_scheduler_spawn`
once; store `SchedulerHandle` and `JoinHandle` in the pool. Replace
`PoolSchedulerControl` with `SchedulerHandle`. Give `PoolCore` no sender or
`JoinHandle`.
- Rewrite `close(self)`: set status `Closed`, call `core.shutdown()`, drop the
pool's sender and core reference, await the `JoinHandle`, log a `JoinError`
panic. Remove `yield_now()`.
- Update the test-only `TestPool` constructor to spawn the actor. Run its
callers inside a Tokio runtime.
- In `handle.rs`: `PoolHandle` holds a `SchedulerHandle` clone and its
`handle_id`. Remove the `core` field if unused. `acquire(&mut self)` delegates
to `SchedulerHandle::acquire`. `Drop` calls `SchedulerHandle::deregister`. Keep
`pool.handle()` synchronous and `pool.acquire()` as a temporary handle.
- In `lease.rs`: remove `release_control`, the `notify_capacity_available` call
and the control argument of `PooledClientLease::new`. Keep the
`pending_responses > 0` discard before `_permit` drops (#548). Lease drop must
not send or spawn. Remove `try_acquire_immediately` and `acquire_for_handle`.
- In `churn.rs`: document that `scheduler_spawns` is 1 per pool at construction,
acquisition adds 0, and `capacity_waits` counts admitted waiters not granted at
once. Keep `PoolTaskChurnObserver` unchanged.
- In `sync.rs`: remove the scheduler's `lock_or_recover` use. Keep it for
`last_returned_at` and update its doc comment.

Phase 3: Deterministic tests for the actor protocol

Prove the state machine, the runtime lifecycle and the ownership release. Adapt existing tests to the new errors and the new task model.

Task 1: Task 3.1: Pure state and handler-model tests

Cover event-order races by driving the synchronous handlers in chosen orders.

Update src/client/pool/scheduler/tests.rs:
- Remove the servicing-owner tests.
- Replace round_robin_enqueue_rejects_unknown_handles with lazy-registration and idempotent-deregistration tests.
- Keep the admission-bound proptest and keep pruning in it.
- Add a test that take_next_waiter skips closed senders under both policies.
Add a handler-level test file next to scheduler/actor.rs, for example scheduler/actor_tests.rs. Use the TestCore alias and test_pool_slot. Use oneshot receivers and call the handlers with no spawned task. Cover these cases:
- an immediate uncontended grant;
- FIFO order under saturation;
- round-robin turns across several handle IDs with several waiters each;
- a waiter cancelled before the grant;
- a waiter cancelled between selection and grant, where the reply closes after the permit is won;
- deregistration with queued waiters;
- PoolSaturated at max_waiters;
- a permit win with no live waiter, which drops the permit;
- shutdown that drains every queued waiter with PoolClosed.
Extend src/client/pool/core_tests/transition_model.rs with actor event steps: Acquire, Cancel, Deregister, Release, Shutdown and SenderClose. Use them to enumerate orders in which capacity release races with acquisition and shutdown. Assert that capacity is preserved and that no cancelled or deregistered waiter receives a lease.

Task 2: Task 3.2: Runtime lifecycle tests

Prove task count, close and implicit termination on a real current-thread runtime.

Add tests to src/client/tests/pool_lifecycle.rs or a new sibling file. Do not let any file exceed 400 lines. Cover these cases:
- The alive-task count rises by exactly one at construction and stays flat across a long acquire/drop loop. Read it with Handle::metrics().num_alive_tasks().
- scheduler_spawns stays at 1 across that loop.
- Explicit close() with blocked waiters resolves every waiter with PoolClosed. After close() returns, the actor task is gone.
- An acquisition after close(), made through a surviving PoolHandle, returns PoolClosed.
- Dropping the pool and all handles without close() terminates the actor. Prove it with the task count.
- In the same scenario, the core DropProbe fires after the final lease drops.
- A panic injected through the test-only command surfaces SchedulerFailed to the queued waiters and to later acquisitions. It does not surface PoolClosed and does not hang.
Poll with the existing wait_for_capacity_waits style helpers. Do not use sleeps.
Extend pool_and_handle_drop_keep_lease_slot_alive_until_final_lease_drop in client_pool_tests.rs to run with the actor started.

Task 3: Task 3.3: Adapt existing tests and BDD steps

Align existing coverage with the new errors and remove scheduler poison coverage.

Delete the scheduler poison-recovery tests in src/client/pool/scheduler_tests.rs. The scheduler state no longer sits behind a mutex. This retires the scheduler part of #539. Keep the slot last_returned_at poison coverage.
Update the assertions in these tests from disconnected to PoolClosed:
- close_refuses_later_handle_acquisitions;
- shutdown_drains_a_queued_waiter;
- close_rejects_new_and_queued_acquisitions_while_an_existing_lease_keeps_its_slot;
- the BDD step definitions for tests/features/client_pool*.feature.
Make sure shutdown_drains_a_queued_waiter still fails if the drain is a no-op.
Keep the #548 tests in lease_tests.rs and tests/pool_concurrency.rs unchanged. Confirm that they pass without release_control.
Run scripts/check-feature-gate.sh and the tests/ui compile-fail suite. Confirm that the pool code stays under feature = "pool".

🤖 Prompt for AI agents

Implement Phase 3: add deterministic tests for the actor protocol and adapt
existing tests.

- In `scheduler/tests.rs`: remove servicing-owner tests; replace
`round_robin_enqueue_rejects_unknown_handles` with lazy-registration and
idempotent-deregistration tests; keep the admission-bound proptest with pruning;
add a test that `take_next_waiter` skips closed senders under both policies.
- Add `scheduler/actor_tests.rs` using the `TestCore` alias, `test_pool_slot`
and oneshot receivers. Call the handlers directly with no spawned task. Cover:
immediate uncontended grant, FIFO under saturation, round-robin across several
handle IDs, cancel before grant, cancel between selection and grant,
deregistration with queued waiters, `PoolSaturated` at `max_waiters`, permit win
with no live waiter, and shutdown draining waiters with `PoolClosed`.
- Extend `core_tests/transition_model.rs` with actor event steps `Acquire`,
`Cancel`, `Deregister`, `Release`, `Shutdown`, `SenderClose`. Enumerate orders
and assert capacity is preserved and no cancelled or deregistered waiter
receives a lease.
- Add runtime lifecycle tests to `src/client/tests/pool_lifecycle.rs` or a new
sibling file, keeping each file under 400 lines:
- alive-task count (`Handle::metrics().num_alive_tasks()`) rises by one at
construction and stays flat across a long acquire/drop loop;
  - `scheduler_spawns` stays at 1;
- `close()` with blocked waiters resolves them with `PoolClosed` and the actor
task is gone afterwards;
- acquisition through a surviving `PoolHandle` after `close()` returns
`PoolClosed`;
- dropping the pool and all handles without `close()` terminates the actor,
proven by task count, and the core `DropProbe` fires after the final lease
drops;
- an injected panic surfaces `SchedulerFailed` to queued waiters and later
acquisitions, not `PoolClosed`, and does not hang.
- Poll with `wait_for_capacity_waits` style helpers. Do not use sleeps.
- Extend `pool_and_handle_drop_keep_lease_slot_alive_until_final_lease_drop` in
`client_pool_tests.rs` to run with the actor started.
- Delete the scheduler poison-recovery tests in `scheduler_tests.rs`. Keep the
slot `last_returned_at` poison coverage.
- Change assertions from `disconnected` to `PoolClosed` in
`close_refuses_later_handle_acquisitions`, `shutdown_drains_a_queued_waiter`,
`close_rejects_new_and_queued_acquisitions_while_an_existing_lease_keeps_its_slot`
and the BDD steps for `tests/features/client_pool*.feature`. Ensure
`shutdown_drains_a_queued_waiter` fails if the drain is a no-op.
- Keep the #548 tests in `lease_tests.rs` and `tests/pool_concurrency.rs`
unchanged and confirm they pass.
- Run `scripts/check-feature-gate.sh` and the `tests/ui` suite. Confirm the pool
code stays under `feature = "pool"`.

Phase 4: Benchmarks and documentation

Record the #639 evidence and update the architecture records.

Task 1: Task 4.1: Update `#639` benchmark assertions and capture new baselines

Replace the spawn-on-demand witnesses with persistent-task evidence.

Update these files:
- wireframe_testing/src/runtime_ownership_benchmarks/client_pool_benchmark_support/ (measurement/harness.rs, tests/uncontended.rs, tests/contended.rs, tests/sampling.rs, measurement/distribution/tests.rs);
- tests/runtime_ownership_benchmark_helpers/pool.rs.
Make these changes in those files:
- Assert that acquisition adds exactly 0 scheduler spawns.
- Replace the >= 1 spawn witnesses with capacity_waits witnesses.
- Rename tests such as uncontended_acquisition_spawns_at_least_one_scheduler_task and contended_acquisition_does_spawn_pool_scheduler_tasks to match the new meaning.
Separate the construction delta from the acquisition delta in the harness:
- Read alive tasks before construction, and assert a construction delta of exactly 1.
- Assert an acquisition-phase delta of <= 0.
- Assert a delta of 0 after close().
Keep the labels in benches/runtime_ownership_baselines/driver.rs and observe.rs consistent with the new counter meaning.
Run make bench-runtime-ownership. Add uncontended and contended latency, allocation and task-count captures to docs/runtime-ownership-baselines.md, labelled with the commit and toolchain. State that the workload parameters did not change.

Task 2: Task 4.2: Update the architecture and developer documentation

Record the implemented protocol and the resolved decisions.

Update docs/adr-013-client-pool-scheduler-and-slot-ownership.md. Record these items:
- Phase 3 (#647) is implemented.
- The decision for an unbounded mailbox with a separate admission limit.
- The dedicated shutdown channel and the JoinHandle acknowledgement.
- Level-triggered capacity through the pending-grant race.
- Lazy handle registration.
- That #535 is superseded by the actor's synchronous handlers.
- That #539 has slot-only scope.
- That the queued-waiter gauges remain open work.
- That the AcquireTimeout deadline remains open work.
Update docs/wireframe-client-design.md sections 17.2.1 and 17.2.2. Describe the PoolClosed and SchedulerFailed semantics and the actor ownership graph.
Add the actor-handler testing convention to docs/developers-guide.md: tests drive synchronous handlers for event-order races and use runtime tests for lifecycle.
Run make fmt, make markdownlint and make nixie.

🤖 Prompt for AI agents

Implement Phase 4: update the #639 benchmarks and the documentation.

- Update the benchmark support files under
`wireframe_testing/src/runtime_ownership_benchmarks/client_pool_benchmark_support/`
(`measurement/harness.rs`, `tests/uncontended.rs`, `tests/contended.rs`,
`tests/sampling.rs`, `measurement/distribution/tests.rs`) and
`tests/runtime_ownership_benchmark_helpers/pool.rs`.
  - Assert that acquisition adds exactly 0 scheduler spawns.
  - Replace `>= 1` spawn witnesses with `capacity_waits` witnesses.
- Rename `uncontended_acquisition_spawns_at_least_one_scheduler_task` and
`contended_acquisition_does_spawn_pool_scheduler_tasks` to match the new
meaning.
- In the harness, read alive tasks before construction and assert a construction
delta of exactly 1. Assert an acquisition-phase delta of `<= 0`. Assert a delta
of 0 after `close()`.
- Keep labels in `benches/runtime_ownership_baselines/driver.rs` and
`observe.rs` consistent with the new counter meaning.
- Run `make bench-runtime-ownership`. Add uncontended and contended latency,
allocation and task-count captures to `docs/runtime-ownership-baselines.md`,
labelled with commit and toolchain. State that workload parameters did not
change.
- Update `docs/adr-013-client-pool-scheduler-and-slot-ownership.md` to record:
Phase 3 (#647) implemented; unbounded mailbox with separate admission limit;
dedicated shutdown channel and `JoinHandle` acknowledgement; level-triggered
capacity through the pending-grant race; lazy handle registration; #535
superseded by synchronous handlers; #539 slot-only scope; queued-waiter gauges
open work; `AcquireTimeout` deadline open work.
- Update `docs/wireframe-client-design.md` sections 17.2.1 and 17.2.2 with
`PoolClosed` and `SchedulerFailed` semantics and the actor ownership graph.
- Add the actor-handler testing convention to `docs/developers-guide.md`: drive
synchronous handlers for event-order races, use runtime tests for lifecycle.
- Run `make fmt`, `make markdownlint` and `make nixie`. Use en-GB-oxendict
spelling.

Research

Ticket Summary

Replace the spawn-on-demand pool scheduler with one persistent scheduler actor per client pool. The actor owns all waiter and fairness state. It receives commands through a Tokio channel and observes slot capacity directly. Remove the shared Mutex<SchedulerState>, the is_servicing guard and every lease-drop tokio::spawn. Make close() await actor termination. Make implicit sender loss terminate the actor and release PoolCore. Preserve FIFO and round-robin order, bounded admission (#550) and the dirty-socket invariant (#548).

Codebase Summary

The pool lives under src/client/pool/. WireframeClientPool::connect (client_pool.rs) builds two independent roots:

Arc<PoolCore> (core.rs) owns Box<[PoolSlot]>, the shutdown flag and its Notify.
Arc<PoolSchedulerControl> holds Arc<PoolScheduler> and the churn counters.

The scheduler code is in scheduler/mod.rs and scheduler/service.rs. PoolScheduler keeps SchedulerState behind std::sync::Mutex (recovered by sync.rs::lock_or_recover). SchedulerState holds the FIFO queue, the round-robin order, the per-handle queues and servicing_waiters.

The current servicing flow:

kick uses a compare-and-swap on is_servicing to spawn a transient service_waiters worker.
Acquire calls kick.
PooledClientLease::drop (lease.rs) calls kick through release_control.

PoolCore exposes the #646 permit-selection API:

try_acquire_permit is the synchronous rotation probe.
acquire_slot_permit is the async race. It uses FuturesUnordered only when every slot is blocked.

Dirty-socket safety does not depend on admission permits. It depends on four mechanisms:

the lease drop order, which runs the discard before the permit is released;
the protocol_gate semaphore;
the armed SlotConnection;
the bb8 broken checks.

PoolHandle (handle.rs) is not Clone. Its acquire takes &mut self, and its Drop deregisters the handle ID. ClientError (error.rs) is #[non_exhaustive] and has PoolSaturated { limit }. It has no PoolClosed or SchedulerFailed variant. Shutdown currently surfaces as the recyclable disconnected() error.

Test layers:

pure state tests in scheduler/tests.rs;
scheduler poison tests in scheduler_tests.rs;
core permit-race and transition models in core_tests/;
lifecycle tests in src/client/tests/pool_lifecycle.rs;
drop-probe tests in client_pool_tests.rs;
BDD features in tests/features/.

The #639 benchmarks and PoolTaskChurn counters (churn.rs) assume spawn-on-demand servicing. ADR 013 pins these requirements:

RAII for the pending grant;
non-blocking shutdown delivery;
the PoolClosed and SchedulerFailed errors;
a JoinHandle held only by the close owner;
no command sender in PoolCore or in leases.

Repository rules:

No file may exceed 400 lines.
Do not use .expect() in production code.
Use en-GB-oxendict spelling.
Run make check-fmt, make lint and make test before committing.

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement Phase 1: add typed errors, convert the scheduler state into an owned
model, and define the command protocol and `SchedulerHandle`.

- In `src/client/error.rs`, add `ClientError::PoolClosed` and
`ClientError::SchedulerFailed` with constructor helpers next to
`pool_saturated`. Make `should_recycle_connection` return `false` for both.
Update the `close` doc example in `client_pool.rs` to show `PoolClosed`.
- Move `SchedulerState` into `scheduler/state.rs` as a plain owned struct with
no `Mutex` or `Arc`.
  - Remove `servicing_waiters` and the `UnregisteredHandle` variant.
  - Count waiters as FIFO length plus per-handle queue lengths.
  - Register round-robin handle IDs lazily on the first `Acquire`.
- Make `DeregisterHandle` remove the ID and drop its queued senders. Make it
idempotent.
  - Keep `prune_cancelled_waiters` and run it before every admission check.
  - Make `take_next_waiter` skip senders with `is_closed()` for both policies.
- Keep the round-robin rotation rule and `WaiterEnqueueResult::Saturated` with
`max_waiters`.
- Create `scheduler/command.rs` with crate-private `SchedulerCommand<S, P, C>`:
`Acquire { handle_id, reply }`, `DeregisterHandle { handle_id }` and a
`cfg(test)` panic-injection variant. Add no `CapacityAvailable` or `Shutdown`
variant.
- Create `scheduler/handle.rs` with a cloneable crate-private `SchedulerHandle`
holding an unbounded `mpsc` sender and an `Arc` to a shared status record.
- The status holds an `AtomicU64` ID counter starting at 1, the lifecycle state
(`Running`, `Closed`, `Failed`), the churn counters and a pool identifier.
Allocate a process-local pool ID if none exists.
  - The status holds no sender, receiver or `PoolCore`.
- Implement `register_handle`, `acquire`, a non-blocking `deregister` that
ignores a closed mailbox, and `close_error`.
- Map send failure or a dropped reply through `close_error`, never
`disconnected()`.
  - Fail early in `acquire` if the status is not `Running`.
- Keep every file under 400 lines. Do not use `.expect()` in production code.
Use en-GB-oxendict spelling.
===============================================================================

Task: 2

Implement Phase 2: run one persistent actor per pool and remove the
spawn-on-demand machinery.

- Create `scheduler/actor.rs` with `SchedulerTask` owning the command receiver,
`SchedulerState`, `Arc<PoolCore>` and the shared status `Arc`.
- Keep the async loop thin. Put decisions in synchronous handlers:
`handle_command`, `grant_sweep`, `on_permit_won`, `drain_for_shutdown`.
- `grant_sweep` loops while live waiters exist and
`PoolCore::try_acquire_permit` succeeds. Pop the next live waiter by policy,
build the lease with `PooledClientLease::new` and send it. If the send fails,
drop the lease and continue. If no live waiter exists, drop the permit.
- On `Acquire`: prune cancelled waiters, check admission and reply
`PoolSaturated { limit }` when full, enqueue, run `grant_sweep`, and call
`record_capacity_wait` if the waiter is still queued.
- Hold the `PoolCore::acquire_slot_permit` race in a `PendingGrant` RAII guard.
Start it only while live waiters remain after `grant_sweep`. Dropping it cancels
the race. Reuse a pinned slot where practical.
- Use one biased `tokio::select!` in this order: shutdown
(`core.shutdown_notified_after_check`), the pending race (call `on_permit_won`
then `grant_sweep`), `recv()`.
- When `recv()` returns `None`, drop the state and the guard and return, so the
actor releases its `Arc<PoolCore>`.
- On shutdown: set status `Closed`, drop the guard, reply `PoolClosed` to every
queued waiter, call `receiver.close()`, drain buffered commands replying
`PoolClosed` to each `Acquire`, and return.
- Add a termination guard declared after the receiver and state. If
`std::thread::panicking()` is true, set status `Failed`, emit an error-level
`tracing` event with the pool identifier, then increment
`wireframe_pool_scheduler_failures_total`. Add that metric in
`src/metrics/client_pool.rs`, labelled by pool identifier, behind the existing
`cfg(all(feature = "metrics", feature = "pool", not(loom)))` gate with the
existing no-op fallback. Do not restart the actor. Add no gauges.
- Delete `scheduler/service.rs`. Remove `is_servicing`, `try_begin_servicing`,
`stop_servicing`, `restart_if_waiters`, `kick`, `notify_capacity_available`,
`resolve_immediate_lease` and `PoolScheduler` from `scheduler/mod.rs`. Keep
`mod.rs` as module root and spawn entry point.
- In `WireframeClientPool::connect`: build `PoolCore`, the shared status and the
unbounded channel; spawn `SchedulerTask` once; call `record_scheduler_spawn`
once; store `SchedulerHandle` and `JoinHandle` in the pool. Replace
`PoolSchedulerControl` with `SchedulerHandle`. Give `PoolCore` no sender or
`JoinHandle`.
- Rewrite `close(self)`: set status `Closed`, call `core.shutdown()`, drop the
pool's sender and core reference, await the `JoinHandle`, log a `JoinError`
panic. Remove `yield_now()`.
- Update the test-only `TestPool` constructor to spawn the actor. Run its
callers inside a Tokio runtime.
- In `handle.rs`: `PoolHandle` holds a `SchedulerHandle` clone and its
`handle_id`. Remove the `core` field if unused. `acquire(&mut self)` delegates
to `SchedulerHandle::acquire`. `Drop` calls `SchedulerHandle::deregister`. Keep
`pool.handle()` synchronous and `pool.acquire()` as a temporary handle.
- In `lease.rs`: remove `release_control`, the `notify_capacity_available` call
and the control argument of `PooledClientLease::new`. Keep the
`pending_responses > 0` discard before `_permit` drops (#548). Lease drop must
not send or spawn. Remove `try_acquire_immediately` and `acquire_for_handle`.
- In `churn.rs`: document that `scheduler_spawns` is 1 per pool at construction,
acquisition adds 0, and `capacity_waits` counts admitted waiters not granted at
once. Keep `PoolTaskChurnObserver` unchanged.
- In `sync.rs`: remove the scheduler's `lock_or_recover` use. Keep it for
`last_returned_at` and update its doc comment.
===============================================================================

Task: 3

Implement Phase 3: add deterministic tests for the actor protocol and adapt
existing tests.

- In `scheduler/tests.rs`: remove servicing-owner tests; replace
`round_robin_enqueue_rejects_unknown_handles` with lazy-registration and
idempotent-deregistration tests; keep the admission-bound proptest with pruning;
add a test that `take_next_waiter` skips closed senders under both policies.
- Add `scheduler/actor_tests.rs` using the `TestCore` alias, `test_pool_slot`
and oneshot receivers. Call the handlers directly with no spawned task. Cover:
immediate uncontended grant, FIFO under saturation, round-robin across several
handle IDs, cancel before grant, cancel between selection and grant,
deregistration with queued waiters, `PoolSaturated` at `max_waiters`, permit win
with no live waiter, and shutdown draining waiters with `PoolClosed`.
- Extend `core_tests/transition_model.rs` with actor event steps `Acquire`,
`Cancel`, `Deregister`, `Release`, `Shutdown`, `SenderClose`. Enumerate orders
and assert capacity is preserved and no cancelled or deregistered waiter
receives a lease.
- Add runtime lifecycle tests to `src/client/tests/pool_lifecycle.rs` or a new
sibling file, keeping each file under 400 lines:
- alive-task count (`Handle::metrics().num_alive_tasks()`) rises by one at
construction and stays flat across a long acquire/drop loop;
  - `scheduler_spawns` stays at 1;
- `close()` with blocked waiters resolves them with `PoolClosed` and the actor
task is gone afterwards;
- acquisition through a surviving `PoolHandle` after `close()` returns
`PoolClosed`;
- dropping the pool and all handles without `close()` terminates the actor,
proven by task count, and the core `DropProbe` fires after the final lease
drops;
- an injected panic surfaces `SchedulerFailed` to queued waiters and later
acquisitions, not `PoolClosed`, and does not hang.
- Poll with `wait_for_capacity_waits` style helpers. Do not use sleeps.
- Extend `pool_and_handle_drop_keep_lease_slot_alive_until_final_lease_drop` in
`client_pool_tests.rs` to run with the actor started.
- Delete the scheduler poison-recovery tests in `scheduler_tests.rs`. Keep the
slot `last_returned_at` poison coverage.
- Change assertions from `disconnected` to `PoolClosed` in
`close_refuses_later_handle_acquisitions`, `shutdown_drains_a_queued_waiter`,
`close_rejects_new_and_queued_acquisitions_while_an_existing_lease_keeps_its_slot`
and the BDD steps for `tests/features/client_pool*.feature`. Ensure
`shutdown_drains_a_queued_waiter` fails if the drain is a no-op.
- Keep the #548 tests in `lease_tests.rs` and `tests/pool_concurrency.rs`
unchanged and confirm they pass.
- Run `scripts/check-feature-gate.sh` and the `tests/ui` suite. Confirm the pool
code stays under `feature = "pool"`.
===============================================================================

Task: 4

Implement Phase 4: update the #639 benchmarks and the documentation.

- Update the benchmark support files under
`wireframe_testing/src/runtime_ownership_benchmarks/client_pool_benchmark_support/`
(`measurement/harness.rs`, `tests/uncontended.rs`, `tests/contended.rs`,
`tests/sampling.rs`, `measurement/distribution/tests.rs`) and
`tests/runtime_ownership_benchmark_helpers/pool.rs`.
  - Assert that acquisition adds exactly 0 scheduler spawns.
  - Replace `>= 1` spawn witnesses with `capacity_waits` witnesses.
- Rename `uncontended_acquisition_spawns_at_least_one_scheduler_task` and
`contended_acquisition_does_spawn_pool_scheduler_tasks` to match the new
meaning.
- In the harness, read alive tasks before construction and assert a construction
delta of exactly 1. Assert an acquisition-phase delta of `<= 0`. Assert a delta
of 0 after `close()`.
- Keep labels in `benches/runtime_ownership_baselines/driver.rs` and
`observe.rs` consistent with the new counter meaning.
- Run `make bench-runtime-ownership`. Add uncontended and contended latency,
allocation and task-count captures to `docs/runtime-ownership-baselines.md`,
labelled with commit and toolchain. State that workload parameters did not
change.
- Update `docs/adr-013-client-pool-scheduler-and-slot-ownership.md` to record:
Phase 3 (#647) implemented; unbounded mailbox with separate admission limit;
dedicated shutdown channel and `JoinHandle` acknowledgement; level-triggered
capacity through the pending-grant race; lazy handle registration; #535
superseded by synchronous handlers; #539 slot-only scope; queued-waiter gauges
open work; `AcquireTimeout` deadline open work.
- Update `docs/wireframe-client-design.md` sections 17.2.1 and 17.2.2 with
`PoolClosed` and `SchedulerFailed` semantics and the actor ownership graph.
- Add the actor-handler testing convention to `docs/developers-guide.md`: drive
synchronous handlers for event-order races, use runtime tests for lifecycle.
- Run `make fmt`, `make markdownlint` and `make nixie`. Use en-GB-oxendict
spelling.

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For design choice 1 go ahead with option 3 and replan.

Start Coding →

Give each client pool a persistent scheduler task

Address review findings on the pool scheduler

Navigation Menu

Repository navigation

Replace spawn-on-demand pool servicing with a persistent scheduler actor #647

Summary

Current behaviour

Proposed architecture

Ownership split

Responsibilities of the actor

Admission

Fairness

Capacity

Shutdown

Public API behaviour

Drop behaviour

Integration with existing issues

Acceptance criteria

Tests

Non-goals

Dependencies

References

1 remaining item

Coding Plan

Summary

Implementation Steps

Phase 1: Typed errors and an actor-owned scheduler model

Phase 2: Persistent actor task and lifecycle wiring

Phase 3: Deterministic tests for the actor protocol

Phase 4: Benchmarks and documentation

Ticket Summary

Codebase Summary

🚀 Next Steps

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Replace spawn-on-demand pool servicing with a persistent scheduler actor #647

Description

Summary

Current behaviour

Proposed architecture

Ownership split

Responsibilities of the actor

Admission

Fairness

Capacity

Shutdown

Public API behaviour

Drop behaviour

Integration with existing issues

Acceptance criteria

Tests

Non-goals

Dependencies

References

Activity

coderabbitai commented on Aug 8, 2026

1 remaining item

leynos commented on Oct 30, 2026

coderabbitai commented on Oct 30, 2026

Coding Plan

Summary

Implementation Steps

Phase 1: Typed errors and an actor-owned scheduler model

Phase 2: Persistent actor task and lifecycle wiring

Phase 3: Deterministic tests for the actor protocol

Phase 4: Benchmarks and documentation

Ticket Summary

Codebase Summary

🚀 Next Steps

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions