[Performance]: FORCE_CHUNK prefills concurrent prompts in lockstep, so every TTFT waits for the last prompt #21891

Proposal to improve performance

With block reuse on, a hybrid linear-attention model prefills with FORCE_CHUNK
(py_executor_creator.py), and each request advances at most to its next expected snapshot point per
step (setCtxRequestsChunkSize<kFORCE_CHUNK> in microBatchScheduler.cpp). When the snapshot interval is
smaller than max_num_tokens, every running prompt gets one interval per step, so concurrent prompts
finish prefill together instead of in arrival order.

Measured on 1.4.0rc2, Qwen3.8-Flash-Next-FP8 (36 GDN + 12 attention layers), TP 2 + EP 2 on 2× RTX PRO
6000, periodic_snapshot_interval: 2048, additional_snapshot_offsets_from_end: [0], default
max_num_tokens (8192), 32k-token prompts sent at once through trtllm-serve:

At the same throughput, serving the 4 prompts in arrival order would give TTFTs of about 1/4, 2/4, 3/4 and
4/4 of the total, a mean of about 8.7 s instead of 13.9 s.

No setting changes this while block reuse is on:

• scheduler_config.context_chunking_policy is replaced by FORCE_CHUNK without a warning ([#21891][feat] Warn when block reuse forces FORCE_CHUNK on hybrid models #21878 adds
  one).
• Turning block reuse off restores FIRST_COME_FIRST_SERVED, but loses prefix caching.
• From the scheduler code (not measured): an interval of at least max_num_tokens lets the first prompt
  take the whole budget, which restores arrival order at the cost of coarser reuse.

The underlying limit is one snapshot per request per step: a chunk cannot cover several snapshot points, so
the first prompt cannot run ahead. Possible directions:

• Let a chunk span several expected snapshot points and save the recurrent state at each of them from
  within the forward pass. The chunked GDN and Mamba kernels already compute the state at every internal
  chunk boundary.
• Until then, document that an interval below max_num_tokens makes concurrent prefills advance in
  lockstep.

Related: #20516 reorders admission by uncached length, which does not separate equal-length prompts that
share a step. #18486 (closed) placed snapshots at shared-prefix boundaries.

Your current environment

• TensorRT-LLM 1.4.0rc2 (nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc2); the scheduler code is unchanged
  on main at 73874e1.
• 2× NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120), TP 2, EP 2, max_batch_size: 4, KV cache
  manager V2.