You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] trtllm-bench KV-cache sizing and reporting ignore the selected modeling_v2 target #21889
A code-inspection finding from PR #21585: the engine and bench can resolve different KV-cache dtypes for the same deployment.
The PR adds target-dependent NVFP4 capability validation. The engine selects its modeling_v2 target before validating KV precision, but bench reconstructs a fresh ModelConfig without that target in two places:
The shared validator consequently sees no target capability in bench and applies the SM107 NVFP4-to-FP8 fallback.
For an eligible GLM-5.2 NVFP4 Rubin StairCase deployment (SM107, DEP4/DEP16 with attention DP, MEGAMOE_CUTEDSL, kv_cache_config.dtype=nvfp4, modeling_v2=auto or require), the engine retains NVFP4 while bench reports FP8. Its sizing heuristic also uses FP8's one byte per KV element instead of NVFP4's half byte, which can undersize the limits. This mismatch is introduced by adding target-dependent validation without adapting these bench callers; it is not a claim that every modeling_v2 target is affected.
The design requirement is one consistent effective configuration for engine construction, capacity planning and reporting. Bench should reuse the resolved engine configuration where available, and sizing should use the same target-aware resolution policy before engine construction. The effective dtype must reflect the selected implementation and its deployment constraints. Neither an architecture-wide GLM exemption nor simply reporting the requested dtype preserves correct fallback behavior.
Workaround: pass both CLI flags --max_batch_size and --max_num_tokens to bypass the sizing heuristic. Supplying only one, or supplying them only in the extra LLM configuration YAML, does not bypass that branch. The FP8 report label remains incorrect.
Acceptance:
Engine, automatic sizing and reporting agree on the effective KV dtype for the affected deployment.
Supported NVFP4 targets retain NVFP4; rejected targets and built-in/modeling-v1 paths retain their existing validation/fallback behavior.
Resolve implementation capabilities through target selection, without a model-name exception or unconditional trust in the requested dtype.
I can take this if you're not already on it: pass the resolved modeling_v2 target into both bench call sites (automatic sizing and reporting) and cover the SM107 NVFP4→FP8 fallback with a CPU unit test. If you already have a branch, say so and I'll defer.
Activity
Ethereal49 commentedon Nov 10, 2026
I can take this if you're not already on it: pass the resolved modeling_v2 target into both bench call sites (automatic sizing and reporting) and cover the SM107 NVFP4→FP8 fallback with a CPU unit test. If you already have a branch, say so and I'll defer.