Skip to content

[Bug] trtllm-bench KV-cache sizing and reporting ignore the selected modeling_v2 target #21889

Open

Description

@dongfengy

A code-inspection finding from PR #21585: the engine and bench can resolve different KV-cache dtypes for the same deployment.

The PR adds target-dependent NVFP4 capability validation. The engine selects its modeling_v2 target before validating KV precision, but bench reconstructs a fresh ModelConfig without that target in two places:

The shared validator consequently sees no target capability in bench and applies the SM107 NVFP4-to-FP8 fallback.

For an eligible GLM-5.2 NVFP4 Rubin StairCase deployment (SM107, DEP4/DEP16 with attention DP, MEGAMOE_CUTEDSL, kv_cache_config.dtype=nvfp4, modeling_v2=auto or require), the engine retains NVFP4 while bench reports FP8. Its sizing heuristic also uses FP8's one byte per KV element instead of NVFP4's half byte, which can undersize the limits. This mismatch is introduced by adding target-dependent validation without adapting these bench callers; it is not a claim that every modeling_v2 target is affected.

The design requirement is one consistent effective configuration for engine construction, capacity planning and reporting. Bench should reuse the resolved engine configuration where available, and sizing should use the same target-aware resolution policy before engine construction. The effective dtype must reflect the selected implementation and its deployment constraints. Neither an architecture-wide GLM exemption nor simply reporting the requested dtype preserves correct fallback behavior.

Workaround: pass both CLI flags --max_batch_size and --max_num_tokens to bypass the sizing heuristic. Supplying only one, or supplying them only in the extra LLM configuration YAML, does not bypass that branch. The FP8 report label remains incorrect.

Acceptance:

  • Engine, automatic sizing and reporting agree on the effective KV dtype for the affected deployment.
  • Supported NVFP4 targets retain NVFP4; rejected targets and built-in/modeling-v1 paths retain their existing validation/fallback behavior.
  • Resolve implementation capabilities through target selection, without a model-name exception or unconditional trust in the requested dtype.

Activity

  1. Ethereal49 commented on Nov 10, 2026

    @Ethereal49

    I can take this if you're not already on it: pass the resolved modeling_v2 target into both bench call sites (automatic sizing and reporting) and cover the SM107 NVFP4→FP8 fallback with a CPU unit test. If you already have a branch, say so and I'll defer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    KV-Cache Managementkv-cache management for efficient LLM inference

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions