Repository navigation
feat(router): add live worker loads to the request classifier context - #94
Draft
ishandhanani wants to merge 1 commit into
Draft
ishandhanani wants to merge 1 commit into
ishandhanani wants to merge 1 commit into
Conversation
`RequestClassifierContext::worker_loads()` reads the router's current load on every registered worker rank at call time. A classifier that holds requests can use it when it decides to release one, instead of the snapshot that `ClassifyRequest::read_worker_inputs` takes before `classify`. Rows reuse `WorkerLoadInput`, built by the same conversion worker selection uses. No request is attached, so `decode_cost_blocks` is the current decode footprint. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
had a problem deploying
to
external_collaborator
October 14, 2026 15:11 — with
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #93. That PR gives classifiers a per-request snapshot of worker inputs, taken before
classify. This PR adds the live view it deferred:RequestClassifierContext::worker_loads()reads the router's current load on every registered worker rank each time it is called. A classifier that holds requests, such as ThunderAgent, can call it when it decides to release one.worker_loads()returnsOption<Vec<(WorkerWithDpRank, WorkerLoadInput)>>. It isNonewhen the host attaches no load reader, for example in contexts built only withRequestClassifierContext::new.KvRouter::request_classifier_context()throughLocalScheduler::worker_load_reader(). The reader projects load withproject_worker_loads(None, now), the same projection the scheduler actor uses.WorkerLoadInput::from_projectionis now the single conversion that worker selection, the feat(router): expose worker inputs to request classifiers #93 snapshot, and this live view all use.Meaning: no request is attached, so
decode_cost_blocksis the current decode footprint. Inread_worker_inputsit also includes that request's uncached prompt blocks. Values come from the router's own bookings, not engine KV occupancy. Each call is one pass over registered ranks under the load table's read lock, with no prefix-trie walk.ThunderAgent is unchanged. Moving its release loop to host load is a separate change. It would need to replace its
normal_usageestimate, not add to it, because host load already counts the requests it admitted.Validation
cargo test -p dynamo-kv-router --lib: 1082 passed. The newworker_load_reader_reads_current_loadcreates one reader, books a request after that, and checks that the same reader reports the booking (active_requests0 to 1).cargo test -p dynamo-llm --lib -- kv_router::tests::classifier_context_tracks_capacity_and_registered_ranks: passed, with a new assertion that the router context exposesworker_loads().cargo test -p dynamo-custom-policy-builtin: all passed.cargo clippy -p dynamo-kv-router -p dynamo-custom-policy-builtin --all-targets --features dynamo-kv-router/bench -- -D warningsandcargo clippy -p dynamo-llm --lib -- -D warnings: clean.pre-commit runon the changed files: passed.