[ishandhanani]

Replaces #202. Rebuilt on current main in dynamo-kv-hints, at the minimum that works end to end (about 550 lines, no tests).

What

Two hints, both fire and forget.

Wake-up prefetch. A chat request carrying x-dynamo-experimental-wakeup is a wake-up call ("the user opened the thread"), not a turn.

1. The router selects the worker this prompt would get. It uses the query-only path: no admission and no booking.
2. It sends that worker's scheduler a KvHint with one kv.prefetch@1.0 action over ZMQ.
3. It answers the call immediately without running it.
4. The worker loads the prompt from Mooncake into its device cache and radix tree. The router learns of the pages from the worker's KV events, and the real request is KV-routed to them.

A code comment at the hook records why there is no booking, and the burst risk: concurrent wake-ups see the same loads and can all pick one worker.

Pin (DYN_KVM_PIN_ENDPOINT). When a response ends with a tool call, or the request announces one with x-dynamo-experimental-sim-tool-call, the prompt and response pages are pinned in the Mooncake master.

• TTL: the expected tool time × 1.5 + 5 s, or 2 min when unknown.
• Priority: log2 of the tokens a miss would recompute.

Where the code lives

• dynamo-kv-hints:
  • The kv.prefetch action.
  • pin: the slot, the Mooncake object keys, and the TTL and priority rule.
  • sender (feature zmq): PUSH sockets to scheduler hint sockets and to the Mooncake master.
• dynamo-llm:
  • The wake-up branch in the routing host.
  • Arming the pin slot at routing and carrying it through migration.
  • The stream taps.
• SGLang sidecar: publishes hint_zmq_endpoint and, for the direct Mooncake linker, the object_layout from SGLang's server info ([Feat] linker: Describe the direct linker's Mooncake objects in the L3 layout sgl-project/sglang#49434).
• Layering gate: dynamo-kv-hints may reach zmq through its feature (HOST_ALLOWED).

Related

• SGLang: [Feat] linker: Prefetch on an out-of-band KV hint and export L3 load telemetry sglang#18 (the hint socket and kv.prefetch on the linker's prefetch path).
• Mooncake: [Store] Pin hints: pin keys now or when written, priority eviction, ZMQ hint socket, pin metrics Mooncake#1
• agent-loadgen: https://github.com/NVIDIA-dev/agent-loadgen/pull/17 ([away] profile table with wakeup = true)

Validation

• cargo fmt --check passes.
• cargo clippy -p dynamo-kv-hints -p dynamo-llm -p dynamo-sglang-sidecar --all-targets -D warnings passes.
• bash scripts/router_layering_check.sh --host: 55 checks, 0 failures.
• A KvHint encoded here decodes with msgspec into SGLang's KvHintsEnvelope.
• Not yet run on GPUs.