Skip to content

feat(router): add live worker loads to the request classifier context - #94

Draft
ishandhanani wants to merge 1 commit into
idhanani/dyn-admission-worker-inputsfrom
idhanani/dyn-admission-live-worker-loads
Draft

ishandhanani wants to merge 1 commit into
idhanani/dyn-admission-worker-inputsfrom
idhanani/dyn-admission-live-worker-loads

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Summary

Stacked on #93. That PR gives classifiers a per-request snapshot of worker inputs, taken before classify. This PR adds the live view it deferred: RequestClassifierContext::worker_loads() reads the router's current load on every registered worker rank each time it is called. A classifier that holds requests, such as ThunderAgent, can call it when it decides to release one.

  • worker_loads() returns Option<Vec<(WorkerWithDpRank, WorkerLoadInput)>>. It is None when the host attaches no load reader, for example in contexts built only with RequestClassifierContext::new.
  • The KV router attaches the reader in KvRouter::request_classifier_context() through LocalScheduler::worker_load_reader(). The reader projects load with project_worker_loads(None, now), the same projection the scheduler actor uses.
  • WorkerLoadInput::from_projection is now the single conversion that worker selection, the feat(router): expose worker inputs to request classifiers #93 snapshot, and this live view all use.

Meaning: no request is attached, so decode_cost_blocks is the current decode footprint. In read_worker_inputs it also includes that request's uncached prompt blocks. Values come from the router's own bookings, not engine KV occupancy. Each call is one pass over registered ranks under the load table's read lock, with no prefix-trie walk.

ThunderAgent is unchanged. Moving its release loop to host load is a separate change. It would need to replace its normal_usage estimate, not add to it, because host load already counts the requests it admitted.

Validation

  • cargo test -p dynamo-kv-router --lib: 1082 passed. The new worker_load_reader_reads_current_load creates one reader, books a request after that, and checks that the same reader reports the booking (active_requests 0 to 1).
  • cargo test -p dynamo-llm --lib -- kv_router::tests::classifier_context_tracks_capacity_and_registered_ranks: passed, with a new assertion that the router context exposes worker_loads().
  • cargo test -p dynamo-custom-policy-builtin: all passed.
  • cargo clippy -p dynamo-kv-router -p dynamo-custom-policy-builtin --all-targets --features dynamo-kv-router/bench -- -D warnings and cargo clippy -p dynamo-llm --lib -- -D warnings: clean.
  • pre-commit run on the changed files: passed.

`RequestClassifierContext::worker_loads()` reads the router's current load
on every registered worker rank at call time. A classifier that holds
requests can use it when it decides to release one, instead of the
snapshot that `ClassifyRequest::read_worker_inputs` takes before
`classify`.

Rows reuse `WorkerLoadInput`, built by the same conversion worker
selection uses. No request is attached, so `decode_cost_blocks` is the
current decode footprint.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@github-actions github-actions Bot added feat documentation Improvements or additions to documentation labels Oct 14, 2026

This branch had an error being deployed

1 failed deployment
external_collaborator — 87070d0e Deployed Oct 14, 2026 by ishandhanani via ok-to-test #304
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant