[nnshah1]

Stacked on #102. Adds the Kubernetes pod feeder from #33 (phase 4) for direct mode: engines can join and leave a serving direct frontend, and a label-selector pod watch drives that.

Summary

Today register_direct_model installs a constant list through keep_alive, so the engine set is fixed at startup. The runtime half already handles a changing list: register_static_endpoint_instances takes a watch<Vec<Instance>>, and the static discovery source diffs each update into Added/Removed. This PR makes the rest follow.

• lib/llm
  • EngineConfig::Direct.workers is a watch::Receiver<Vec<DirectWorker>>.
  • register_direct_model derives three views of the list:
    • the decode/aggregated instances;
    • the prefill instances;
    • the runtime-config map, which drives the selection catalog and so the per-engine ZMQ KV subscriptions.
  • It also derives the prefill target's admitted ids.
  • Each derived watch publishes only when its projection changes.
  • If the list's owner drops its sender, the last value stays until shutdown. The static path therefore behaves exactly as before.
  • Whether the deployment is disaggregated is still fixed by the list at startup.
• sidecar-common direct
  • DirectFleet::into_membership(model) returns the config plus a DirectMembership<B>. into_engine_config is unchanged for callers.
  • DirectMembership::add(url, role) probes the engine and checks it serves the fleet's model. It applies the operator role hint for backends that can't report roles, and rejects a prefill engine on an aggregated frontend. It registers the engine with the backend before publishing it, so the router never selects an unregistered instance.
  • remove(url) withdraws the engine. The dispatch state is dropped through the existing StreamingDispatch::on_instance_removed.
• sidecar-common direct::pods
  • A pod serves once it is Ready, has no deletion timestamp, and has an IP.
  • Its URL is http://<pod IP>:<port>. The port comes from the pod's nvidia.com/dynamo-engine-grpc-port annotation, or else the selector's pool-wide port.
  • Each pod gets its own probe task that retries with backoff, so an engine that is Ready but not yet answering never blocks the others.
  • Not-ready, deleted, or missing-on-relist pods are withdrawn.
  • probe_first_pods waits for the first serving pod of each selector, which seeds the model card. PodEngines::adopting hands those pods to the watcher, which withdraws them at the first relist if they are gone.
  • Prefill and serving pools are two selectors over one membership. This also gives vLLM per-pod roles without --prefill-engine-url.

Not in this PR: the CLI wiring for pod selectors in make_engine, which belongs on #99, plus RBAC and manifests.

Validation

• cargo test -p dynamo-sidecar-common --lib: 37/37. New tests:
  • membership add/remove/idempotent add/model mismatch/role shape;
  • pod_url rules;
  • join/leave on readiness, retry of an engine that isn't answering, relist withdrawal, adopted-engine reconciliation.
• cargo test -p dynamo-llm --lib entrypoint::input::direct: 2/2. Covers per-role projection and holding the last list when the owner goes away.
• End to end over the SGLang mocker, with this branch applied on top of feat(frontend): direct mode for stock SGLang gRPC engines #99's direct_mocker_integration.rs plus two local tests that are not in this PR: 5/5.
  • The existing round-robin, KV-events and P/D tests pass unchanged through the watch path.
  • Membership tests under round-robin and under KV with engine events:
    • start with engine A;
    • add(B), after which B serves;
    • remove(A), after which A receives none of the next 4 requests.
• cargo fmt, cargo clippy -D warnings on both crates.

Known edges

• Fast remove then re-add of the same URL. on_instance_removed runs asynchronously after the runtime broadcasts Removed. If add re-registers the same id before that callback runs, the callback drops the fresh registration. A pod's readiness round trip takes seconds, so this is unlikely, but a generation check in the backend's targets map would close it.
• Ids are still URL hashes. A pod that comes back on a new IP is a new engine, and the old id is withdrawn with it. An in-place container restart keeps the IP, but the not-Ready → Ready cycle removes and re-adds the engine, which clears its index entries.
• feat(frontend): direct mode for stock SGLang gRPC engines #99/feat(frontend): direct mode for stock vLLM gRPC engines #100 compatibility. Code that destructures EngineConfig::Direct { workers, .. } and only passes workers to register_direct_model compiles unchanged.

🤖 Generated with Claude Code