[ishandhanani]

POC for ai-dynamo#12803: the Cold Pool policy from ai-dynamo#12806, rebuilt as a request-classifier plugin on top of the dispatch gate (ai-dynamo#18443) instead of ~700 lines of core queue changes. Stacked on ai-dynamo#18443's branch.

Summary

Core (generic, no Cold Pool knowledge; +180/-20):

• ClassifyRequest::set_lane(name): a sessionless request waits in a classifier-named lane instead of the class's shared lane, so a held or constrained head blocks only its own lane. Session lanes take precedence.
• ClassifyRequest::set_allowed_workers(set): narrows the request's allowed workers (intersected with the caller's); the queue's capacity check and selection both honor it.
• ClassifyRequest::cached_tokens(): best cached prefix, for effective-prefill decisions.

Plugin request_classifier: cold_pool (lib/router-plugins/builtin/src/cold_pool.rs):

• Warm / Cold / unmanaged classification by effective prefill (same rule as feat(kv-router): add cold pool routing ai-dynamo/dynamo#12806).
• Pool = top-N workers by rendezvous hash of worker ID, capped at a third of the workers.
• Cold: allowed workers = pool, own lane; the gate holds it until a member has no active Warm work, then Go { target: that member }; after soft_drain_timeout_ms, any member.
• Warm, while Cold is pending: allowed workers exclude the pool; while Cold only runs: exclude just the workers running it (decided at classify time).
• workers: 0 keeps classification and the lane without containment, for baselines.

default_policy_class: default
policy_classes:
  - {name: default, quantum: 4096, queue_policy: fcfs, prefill_busy_threshold_frac: 1.0}
request_classifier:
  type: cold_pool
  parameters: {request_threshold: 98304, workers: 2, soft_drain_timeout_ms: 20000}

Validation

• cargo test -p dynamo-kv-router --lib: 1241 passed, including a_classifier_lane_keeps_a_held_head_out_of_the_shared_lane (negative control: without a lane, a held sessionless head blocks the shared lane).
• cargo test -p dynamo-router-policies: all suites, including 7 Cold Pool unit tests and tests/cold_pool_gate.rs (one policy class through LocalScheduler: Cold held until a member drains while Warm avoids the pool; Cold takes a busy member after the timeout).
• cargo test -p dynamo-llm --lib -- routing_host::tests: 80 passed.
• cargo clippy -D warnings on kv-router, router-policies, llm; cargo fmt.

GPU pilot (earlier revision with a gate-side worker-set decision, same policy semantics)

8x H200, 8 SGLang TP1 workers of GLM-4.7-Flash, agg, KV router; Warm = agent-loadgen SWE-bench, 320 agents, 15 min; Cold = 160K-token random (cache-miss) prompt every 20 s from minute 2. Baseline = same classifier with workers: 0.

Warm	baseline	Cold Pool
ITL p99 (per-request avg)	795 ms	248 ms (-69%)
ITL mean	116 ms	102 ms (-13%)
TTFT mean / p95	0.67 / 2.37 s	0.65 / 2.11 s
E2E mean / p99	20.9 / 159 s	19.2 / 143 s
input tok/s	219K	241K (+10%)

Cold	baseline	Cold Pool
workers used	8	2 (pool)
TTFT median	47.5 s	129 s
E2E median	54 s	378 s

The Cold arrival rate exceeded the 2-worker pool's capacity (~2.4 workers of Cold prefill demand), so Cold queued without bound. Follow-up runs at sustainable Cold rates are pending.

Gaps found

• Membership hashes the worker instance ID, which changes on restart; stable_routing_id() exists and should be used (same issue in feat(kv-router): add cold pool routing ai-dynamo/dynamo#12806).
• No overflow when the Cold arrival rate exceeds pool capacity (same in feat(kv-router): add cold pool routing ai-dynamo/dynamo#12806).
• The gate has no load view: the clean-member target may be prefill-busy; plugin accounting follows Sent/Completed events with a small lag.
• One classifier per router: Cold Pool and ThunderAgent cannot both be installed without composition.