Hybrid GDN (Qwen3.5-class) at high concurrency: mixed prefill+decode iterations run the decode portion eagerly (~4x attention cost, ~42ms/chunk overhead) — can decode stay graphed? #16976

⚠️ Revised 2026-07-29. Our original report claimed the GPU was only ~35% busy. That was a
measurement artifact: our kernel-time sum excluded CUDA-graph replays (they appear as CUPTI
GRAPH_TRACE rows, not kernels). Correct accounting shows GPU ~86.5% busy, and the bottleneck
analysis below has been rewritten accordingly. The original text is preserved in the edit history;
the API-level data (launch/sync counts) from the original report remains valid.

Question

We serve a Qwen3.5-0.8B-architecture hybrid model (24 layers = 18 gated-deltanet + 6 full-attention, bf16 weights, fp8 KV) with trtllm-serve (PyTorch backend, 1× H100 SXM, TP1) at concurrency 384 (~19K prompt / ~3.1K decode tokens, greedy). After correcting our GPU accounting, throughput appears bounded by removable GPU work in mixed (prefill+decode) iterations rather than by host overhead. Is there a recommended path — existing or planned — to avoid running the decode portion of a mixed batch eagerly?

Setup

• TensorRT-LLM v1.3.0rc20 and v1.3.0rc22 (both measured; +2% end-to-end difference, same profile shape)
• max_batch_size=384, max_num_tokens tested at 32768 and 8192 (no throughput difference), max_seq_len=65536, chunked prefill, overlap scheduler, cuda_graph_config: {enable_padding: true, max_batch_size: 384}, fp8 KV, block reuse off

Corrected measurements (Nsight, graph-trace-aware; 15s steady-state, 396 iterations, 26.4 iter/s)

Global GPU busy = 86.5% of wall (kernel intervals ∪ GRAPH_TRACE intervals; nvidia-smi agrees at 85%). Graphed decode replays run ~22.2 ms each on-GPU.

Mixed (ctx) iterations: 118 of 396 = 46.8% of wall, mean 59.5 ms at max_num_tokens=8192, 86% genuine GPU time inside the range. Host Python (~25 ms/iter incl. update_requests ~9.4 ms) exists but is almost fully hidden behind the busy GPU — we estimate ≤ ~6–14% total upside from all host-side fixes combined.

The interesting structure is inside the mixed iteration cost, measured across chunk sizes:

i.e. slope ≈ 3.7 µs/token with a ~42 ms intercept that does not scale with prefill size. The intercept is the cost of dragging the 383 generation requests through the eager mixed forward:

• generation attention (kernel_mha) inside the eager mixed pass: 13.7 ms vs ~3.4 ms for the equivalent work in the decode graph (~4×)
• GDN v/k state fp32 cast round-trips ≈ 14% of ctx-iteration GPU time (pure dtype-conversion overhead in the hybrid path)

At max_num_tokens=32768 the same request costs one ~178 ms mixed iteration (≈ 3 × 59.5), so chunk-size tuning cannot help — the eager-decode drag is paid per chunk.

Consistent end-to-end evidence: four materially different configs (with/without one-model MTP, and a checkpoint variant with half the full-attention layers) all converge at ~3.8 req/s — decode-side changes don't move the ceiling; the mixed-iteration structure does.

Questions

1. Mixed-iteration split: is it feasible/planned to execute the generation portion of a mixed scheduler iteration via the captured decode CUDA graph while running the context chunk in a separate (eager) forward, merging KV/scheduler state? Back-of-envelope this is worth +30–45% for small hybrid models at high concurrency. We're happy to prototype with guidance on where the invariants live.
2. GDN fp32 casts: can the hybrid GDN state path avoid the fp32 round-trips (≈14% of ctx GPU time), or is that numerically load-bearing?
3. Any existing configuration we've missed that reduces the eager-decode drag in mixed batches? (torch.compile piecewise CUDA graphs measured −10% on this model on rc22, both fullgraph variants.)
4. Lower priority (mostly hidden behind the GPU, listed for completeness): batched update_requests bindings; extending [TRTLLM-14054][perf] Reduce per-step host preparation overhead in the PyTorch executor decode path #16313's steady-state fast path to one-model MTP.

We can share all .nsys-rep captures (including sync backtraces and the graph-trace-aware analysis scripts) and can test prototype branches on our rig within a day. We've also verified a small fix in this area (redundant host-syncing boolean-index in gdn_mixer state zeroing, computed 42×/step; dedup = +2% end-to-end) that we're happy to submit as a PR.