Repository navigation
feat(frontend): direct-mode infrastructure for stock engine gRPC servers - #102
Draft
ishandhanani wants to merge 3 commits into
Draft
ishandhanani wants to merge 3 commits into
ishandhanani wants to merge 3 commits into
Conversation
Add TransportType::Grpc for instances the frontend dials over an engine's native gRPC API, and DistributedRuntime::register_static_endpoint_instances so an Endpoint's client can read a caller-owned instance list instead of discovery. The static source synthesizes Added and Removed events from diffs of the watched list, so PushRouter, prefill binding and affinity see the same event stream they see from etcd. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
EngineConfig::Direct carries a fixed worker list and a StreamingDispatch that is the final hop under PushRouter. register_direct_model installs the workers as static instances, puts prefill engines on a committed prefill target, and builds the usual routed pipeline (preprocessor, migration, prefill hop, KV routing) on top. ModelManager::set_direct_dispatch injects the dispatch where the pipeline builds its PushRouter, and KvRouter reads KV events over ZmqDirectIngress from the endpoints the worker card advertises under direct_kv_events_endpoints and direct_kv_replay_endpoint. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
dynamo-sidecar-common::direct holds what every direct backend shares: DirectBackend (probe one URL, dispatch to it), DirectFleet (probe a URL list, assign roles from the operator's prefill list when an engine cannot report one, build EngineConfig::Direct), the transport-fault remap that lets the router mark an instance down and migrate, stable instance ids from the URL, and the KV-event bookkeeping including the replay socket. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
had a problem deploying
to
external_collaborator
October 23, 2026 07:04 — with 
GitHub Actions
Failure
GitHub Actions
Failure
This was referenced Oct 23, 2026
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of a three-PR stack: this PR (infrastructure), then #99 (SGLang backend and CLI), then #100 (vLLM backend). Nothing here is reachable from the CLI yet; the backends in the next two PRs are what dial engines.
Summary
Lets the frontend route to inference engines it dials itself over their native gRPC API, with no Dynamo sidecar beside the engine and no discovery. The runtime stays; this changes only how the frontend talks to an engine. Everything above the dispatch seam (preprocessor, migration, prefill/decode hop, KV-aware routing, affinity) is unchanged.
TransportType::Grpc;DistributedRuntime::register_static_endpoint_instances(endpoint, watch<Vec<Instance>>)makesEndpoint::client()read a caller-owned instance list through a staticEndpointDiscoverySourcethat synthesizes Added/Removed events from diffs, soPushRouter, prefill binding and affinity see the stream they see from etcd.EngineConfig::Direct { model, workers, dispatch };entrypoint::input::direct::register_direct_modelinstalls the workers as static instances, puts prefill engines on a committedprefilltarget, and builds the routed pipeline on a static client.ModelManager::set_direct_dispatchinjects the final hop wherebuild_preprocessed_routingandPrefillBinding::activatebuild theirPushRouter.KvRoutertakesZmqDirectIngressin direct mode and reads the engines' KV-event and replay sockets from the worker card (direct_kv_events_endpoints,direct_kv_replay_endpoint).directmodule with what every backend shares:DirectBackend(probe one URL, dispatch to it),DirectFleet(probe a URL list, assign roles from the operator's prefill list when an engine cannot report one, buildEngineConfig::Direct),remap_transport_error(lifts transport faults to the top-level kinds that firereport_instance_downand migration), stable instance ids from the URL, and KV-event bookkeeping.Trade-offs accepted up front: no snapshot recovery, no request-plane drain, no follower KV relay. This is the same shape as llm-d and SGLang's model gateway (static list or pod watch plus engine probe, engine events only).
Validation
cargo clippy -D warnings --testsclean ondynamo-runtime,dynamo-kv-router,dynamo-llm,dynamo-sidecar-common; Python bindingscargo checkclean;cargo fmt --checkclean.dynamo-sidecar-common22 (fleet role assignment, duplicate URLs, fault remap, KV-event bookkeeping),dynamo-runtimeclient 101 (static discovery source among them),dynamo-llmkv_router::447 andentrypoint::6.End to end behavior is exercised by the mocker integration tests in #99 and the live SGLang and vLLM runs recorded on #99 and #100.