[ishandhanani]

Stacked on #102 (direct-mode infrastructure). Only the last commits are this PR's. #100 (vLLM) stacks on this one.

Summary

python -m dynamo.frontend --direct --engine-url http://host:30001 ... routes to stock sglang.launch_server --grpc-port servers over their native gRPC API with no sidecar.

• dynamo-sglang-sidecar::direct: probe(url) runs the sidecar's discovery RPCs (GetModelInfo, GetServerInfo, KV-event source discovery) and produces what the sidecar would have registered (role, block size, context length, DP ranks, bootstrap host/port, ZMQ sources); SglangDirectDispatch implements DirectBackend and StreamingDispatch with a lazy per-instance channel pool. The stream body moved out of LLMEngine::generate into grpc_generate_stream so both paths share one wire contract; the dispatcher variant sends Abort on cancellation, deferring the decode-leg abort until the first token so the bootstrap handoff completes.
• SGLang reports its own role, so --prefill-engine-url must agree with what the engine says. Engines must run with --incremental-streaming-output, as with the sidecar.
• CLI and bindings: EngineType.Direct, --direct, --direct-backend sglang, repeatable --engine-url and --prefill-engine-url, and --dyn-tool-call-parser / --dyn-reasoning-parser (named after the sidecar flags; they override the engine-reported parsers). Model path, served name and KV block size default from the probe.
• Docs page docs/fern/pages/cli/model-deployment/direct-engines.md.

Validation

• cargo clippy -D warnings --tests clean on dynamo-sidecar-common, dynamo-sglang-sidecar, dynamo-sidecar-testkit; bindings cargo check; cargo fmt --check; pre-commit clean on the Python and docs files.
• dynamo-sglang-sidecar 122 unit tests. lib/sidecar/testkit/tests/direct_mocker_integration.rs 3/3 against the SGLang mocker: round-robin over two engines, KV routing from the engines' ZMQ events, prefill/decode handoff with the bootstrap rendezvous held open. test_frontend_direct_args.py 6/6.

Live on try6767 (SGLang 0.5.22, Qwen3-0.6B, 2x H100): aggregated KV routing with observed prefix overlap, and disaggregated prefill/decode with distinct worker ids per request, TTFT ~39 ms. Evidence in the comment below (recorded on the pre-split branch; the code is the same).

[ishandhanani]

Live validation on H100 (SGLang 0.5.22)

Host: try6767, stock sglang.launch_server from an existing SGLang 0.5.22 venv (torch 2.14.1+cu130), one GPU per engine, Qwen3-0.6B. The frontend is this branch's bindings built in a separate venv. No sidecar, no etcd, no NATS (--discovery-backend mem).

Engine flags: --grpc-port <port> --incremental-streaming-output --kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:<port>","topic":"kv-events"}' --host 0.0.0.0.

Aggregated, two engines, --router-mode kv

python -m dynamo.frontend --direct --engine-url http://127.0.0.1:18701 --engine-url http://127.0.0.1:18711 \
  --router-mode kv --discovery-backend mem --http-port 18650

• Probe: both engines reported mode=Aggregated, model /data/models/Qwen/Qwen3-0.6B, one KV-event source each; the frontend served the model under the engine's reported name.
• Four chat completions: finish_reason=length, usage 19 prompt / 40 completion tokens, real model text.
• Streaming: 33 SSE chunks for a 32-token request.
• KV routing: two requests sharing a long system prompt; the second was routed Best: worker_3458080756468540 dp_rank=0 with 3/4 blocks overlap to the worker that served the first, so the engine's own ZMQ KV events reached the frontend's index.
• Warning worth knowing: Gap detected: expected seq 0, got 1 ... No replay endpoint configured. SGLang numbers its first batch 1, and this run set no replay_endpoint; harmless here, and the probe does not yet forward a replay endpoint.

Disaggregated, prefill on GPU 2 and decode on GPU 3, --router-mode kv

Engines: --disaggregation-mode prefill --disaggregation-transfer-backend nixl --disaggregation-bootstrap-port 8997 and --disaggregation-mode decode --disaggregation-transfer-backend nixl. Same frontend command as above; roles came from GetServerInfo, no flags on the frontend.

• Probe: endpoint=http://127.0.0.1:18701 mode=Prefill, endpoint=http://127.0.0.1:18711 mode=Decode; the prefill router activated from the probed card.
• Every request logged prefill_worker_id=3458080756468540 decode_worker_id=3627132050719741: prefill ran on the prefill engine over gRPC, the bootstrap tuple from its first chunk went to the decode engine, decode streamed the tokens back.
• Four chat completions, 40 tokens each, finish_reason=length; streaming 33 chunks; TTFT 190 ms on the first request and about 39 ms after; ITL about 1.4 ms.
• No frontend or engine errors.

Session artifacts (scripts, logs) are under /data/agent-sessions/20261022T1600Z-direct-sglang/ on the host.