[Performance]: SM107 trtllm-gen MLA decode: the pre-selection cap mMaxNumCtasPerSeqKv = 16 / max(clusterDimX, 2) limits split-KV at low batch #20378

Proposal to improve performance

On SM107, TllmGenFmhaKernel::selectKernelAutotuned (fmhaKernels.h) caps options.mMaxNumCtasPerSeqKv at 16 / max(clusterDimX, 2) = 8 for every GmemReduction + Static request before the autotuner runs, so that a later promotion to CgaSmemReduction stays within the 16-CTA cluster limit. That is the generation path. #18945 removed the same cap on SM100 because it took most of the split-KV parallelism away from low-batch MLA decode; #19038 kept it for SM107. On SM107 we see the effect that #18945 describes.

Kimi K3 MLA decode, per-rank TP8 shape (16 heads, one latent KV head, HQk 576 / HV 512, BF16, 64 tokens per page, 8 query tokens from MTP, batch 1, multi-CTA KV with the static scheduler, as AttentionOp issues it). TllmGenFmhaRunner microbenchmark on SM107, us per call, with the pre-selection cap replaced by the value in the first column:

From a cap of about 24, the autotuner leaves CGA reduction and picks the gmem-reduction kernel with more KV splits. Above 32 nothing changes. Against the current cap, 2K output is bit-identical (same kernel), and 32K / 128K differ by at most 1 bf16 ulp (different split), with no NaN/Inf.

End to end (Kimi K3 + MTP, 8 GPUs over 2 nodes, single request), cap 32 vs the current cap: decode step unchanged at 2K, -0.64 ms at 32K, -2.51 ms at 128K. Cap 16 gives -0.35 / -1.67 ms. Accuracy and long-context needle checks were unchanged.

One side effect: with a cap above 16 / clusterDimX, the post-selection limitCgaSmemReductionCtasKv clamps the shapes where the autotuner still picks CGA reduction. It logs "CGA reduction is not supported when numCtasPerSeqKv * clusterDimX > 16 ..." for each of them, which comes to many thousands of lines per rank during warmup and graph capture. Nothing is logged during graph replay.

Questions / proposal:

• Is the pre-selection cap still needed on SM107, or is the post-selection limitCgaSmemReductionCtasKv enough? If the cap only guards a CGA promotion that would otherwise be illegal, raising it on SM107 to 32 (or removing it, as [https://nvbugs/6737351][fix] Remove CGA smem-reduction clamp from trtllm-gen FMHA kernel selection #18945 did on SM100) keeps the clamp where it matters and lets the gmem-reduction kernels get their split-KV.
• If maintainers prefer a knob for now, the change is a few lines: an override of the pre-selection cap, with the current value as the default. I can send either version, plus a downgrade of that warning to debug level.

I only measured MLA decode; the cap also applies to SM107 GQA generation, so a check there would be good too.

cc @yifeizhang-c (#19038) @kris1025 (#18945)