[moraxu]

Description

Two trtllm-serve configurations for agentic workloads with Muse Spark Open on one GB300 NVL node (TP4/EP4), and a deployment-guide section that documents them:

• examples/musespark_open/serve_config_agentx_max_throughput.yaml: 48 concurrent sessions;
• examples/musespark_open/serve_config_agentx_balanced.yaml: 24 concurrent sessions.

They start from aggregated AgentX serving recipes:

• full 1M-token context;
• KV cache manager v2 with all_reusable block reuse and swa_endpoint_rewind_tokens, and a 160 GiB-per-GPU host KV tier;
• breakable prefill CUDA graphs, the MAX_UTILIZATION scheduler, postprocess workers, stream_interval 20 and the tokenization cache.

They differ from those recipes in three ways, to run on this branch as committed:

• No speculative decoding. This branch runs the DFlash2 drafter's non-causal sliding-window layers only on the VANILLA context-attention backend: the TRTLLM backend refuses them, and FA4 is SM90-only. VANILLA keeps a dense context buffer per batch slot, sized for max_seq_len: about 5 GiB per GPU per slot at 1M tokens.
• Engine defaults only. No model-specific environment opt-ins.
• Text only (disable_mm_encoder: true).

Without speculation, the recipes' 72- and 80-session points (max_num_tokens 16384) are past the knee: 14.7k and 11.5k tokens/s/GPU, with a p90 TTFT of 98 s and 191 s. A sweep from 16 to 64 sessions (max_batch_size = concurrency, max_num_tokens 8192, KV fraction 0.8) picked the two points:

• Max-throughput: 48 sessions. It is the knee: 56 and 64 sessions add 2 % and 5 % throughput while the p90 TTFT rises from 2.9 s to 8.8 s and 11.0 s.
• Balanced: 24 sessions. It is the highest-throughput point whose p90 output speed per session is at least 100 tokens/s.

The guide section covers:

• what the configurations change;
• the host-memory requirement (640 GiB per node for the KV tier);
• the sweep;
• the AgentX benchmark command: agentx-harness pinned at 104c7c8, aiperf 0.12.0, the public Apache-2.0 dataset semianalysisai/cc-traces-weka-062126;
• the full-hour results.

The example README lists the two files.

The numbers were measured at 0e868637c8. The preview container pinned in the guide is built from the earlier 79e45c5d, so the container pin moves when a newer preview tag from this branch exists.

Test Coverage

On one GB300 NVL node, feat/musespark-open at 0e868637c8:

• AgentX, full protocol (3600 s measured, 1800 s warm-up grace), with each server started from the YAML file in this PR:

  Configuration	Sessions	Total (tokens/s/GPU)	Output (tokens/s/GPU)	Output speed per session (tokens/s, p50 / p90)	TTFT (s, p50 / p90 / p99)	Inter-token latency p50 (ms)	Prefix-cache hits	Failed requests
  Max-throughput	48	26,169	175	24.1 / 64.6	1.3 / 2.7 / 9.7	41.3	96.0 %	6 of 2,969 (0.2 %)
  Balanced	24	19,306	143	41.9 / 108.6	0.9 / 2.2 / 6.7	23.4	96.6 %	5 of 2,239 (0.2 %)

• Serve smoke on both files: all 11 hard checks pass (text, JSON object and schema, streaming, 8 concurrent requests, long-prompt chunked prefill and block reuse). The weight accounting confirms 1,479 tensors consumed and the 881 vision and audio tensors skipped, as disable_mm_encoder intends.

• GSM8K through trtllm-eval, same engine settings without the serving-side ones (num_postprocess_workers, stream_interval, enable_tokenization_cache): 94.47 (max-throughput) and 95.22 (balanced). For reference, the TP4/EP4 settings of serve_config.yaml, without guided decoding and partial reuse, score 94.73 the same way.

• pre-commit on the changed files.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

[moraxu]

/bot skip --comment "documentation-only change: deployment guide section, example README and two example serving YAMLs; no code paths"

[tensorrt-cicd]

PR_Github #84345 [ skip ] triggered by Bot. Commit: 1487b5a Link to invocation

[tensorrt-cicd]

PR_Github #84345 [ skip ] completed with state SUCCESS. Commit: 1487b5a
Skipping testing for commit 1487b5a

Link to invocation