[None][fix] Pad INT8-woq MoE intermediate to 64 for tensor-parallel sharding by Dorijan10 · Pull Request #16198 · NVIDIA/TensorRT-LLM

Dorijan10 · 2026-07-09T17:38:09Z

#15550 has merged, so this now stands alone on main: the padding fix, non-aligned-shape test coverage, and a C++ change addressing a CodeRabbit finding from that review. The other finding, defaulting unpadded_hidden_size after the INT8-woq dim swap, is now on main via #19753.

Problem

INT8WoqPerChannelFusedMoEMethod does not pad its per-partition intermediate dimension after tensor-parallel sharding, so any model whose moe_intermediate_size / tp_size is not a multiple of 64 fails to load at TP > 1. The INT8 mixed-GEMM preprocessor (preprocess_weights_for_mixed_gemm) requires 64-aligned rows (rows_per_tile = 128 * 8 // BITS_PER_ELT_A, BITS_PER_ELT_A = 16 for W8A16). The FP8, NVFP4, MXFP4 and WFP4A16 fused-MoE methods all round the sharded intermediate up and F.pad; INT8-woq was the only one that did not.

This limitation is already noted in-tree: should_skip_cutlass in tests/unittest/_torch/moe/moe_test_utils.py records that W8A16 "fails in preprocess_weights_for_mixed_gemm (num_rows % rows_per_tile != 0)" and skips the affected TP configurations rather than exercising them.

Reproduction

nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B has moe_intermediate_size = 1856 = 64 × 29. Since 29 is prime, 1856 / tp is a multiple of 64 only for tp ∈ {1, 29}; every power-of-2 TP leaves a non-aligned shard (1856/8 = 232, not ÷64). The native W8A16 checkpoint at --tensor_parallel_size {2,4,8} fails during weight load:

File ".../fused_moe/quantization.py", in load_expert_w3_w1_weight
RuntimeError: The size of tensor a (464) must match the size of tensor b (232) at non-singleton dimension 1

TP = 1 loads (1856 is already 64-aligned) but the checkpoint does not fit a single 32 GB card, so TP > 1 is required in practice.

This is not specific to one checkpoint. Across the current Nemotron 3 MoE lineup, Nano, Lightning, Elastic, TwoTower and Cascade-2 all have moe_intermediate_size = 1856 and are misaligned at every TP ≥ 2, and Super (2688 = 64 × 42) is misaligned at TP ≥ 4. Ultra (5120 = 64 × 80) is aligned at every TP ≤ 8 and pays no padding cost, which also shows the change is a no-op wherever alignment already holds.

Fix

Round the per-partition intermediate up to a multiple of 64 in create_weights, and F.pad each shard up to it in both weight loaders and the fc31 scales. Zero-padding is numerically inert (Relu2(0) = 0 for squared-ReLU, silu(0)·0 = 0 for gated), meaning it is lossless.

Test Coverage

Adds a W8A16 CUTLASS parameter set with intermediate_size = 200 (a multiple of neither 16 nor 64, rounding up to 256), covering both the non-gated (squared-ReLU) and gated (SwiGLU) layouts; the gated case additionally exercises padding of the w3 gate projection before concatenation. Every intermediate_size in MOE_MODEL_CONFIGS is already a multiple of 64, so this path previously had no coverage. Without the fix all eight cases fail with AssertionError in preprocess_weights_for_mixed_gemm (num_rows % B_ROWS_PER_MMA == 0 in tensorrt_llm/quantization/functional.py); with the fix all eight pass against the dequantized reference on A100 for float16 and bfloat16.

Note that the new parameter set bypasses the 128-alignment gate in get_quick_skip_reason() for this case only, since that gate is precisely what the padding makes unnecessary for W8A16; all other skip reasons are honoured. Happy to instead relax that gate for W8A16 generally if maintainers prefer.

On SM80+ runners the generator raises ValueError if it produces no cases, so an upstream rewording of the skip message cannot silently drop them. CPU-only runners collect none.

Validation

Consumer Blackwell (sm_120), TP = 8: native W8A16 checkpoint loads and serves; GSM8K 5-shot strict-match 0.8385 ± 0.0101 vs BF16 0.8393 ± 0.0101 (Δ 0.0008, within stderr). Also loads and serves coherently at TP = 2 and TP = 4. All 40 existing W8A16 cases in test_moe_backend.py pass on A100 (gated and non-gated, float16 and bfloat16, CUTLASS). The branch merges cleanly against current main.

Dev Engineer Review

INT8 weight-only MoE intermediate dimensions are padded to a multiple of 64 after tensor-parallel sharding.
Weight shards receive zero padding. FC1 scales receive unit-scale padding.
Shape validation now runs after activation and layout resolution.
Gated and non-gated activation paths use the correct intermediate-dimension rules.
No public API or configuration changes were found.
The implementation is consistent with the stated padding requirement.

QA Engineer Review

Added generated W8A16 coverage for:
- Non-gated Relu2 MoE with intermediate size 200.
- Gated SwiGLU MoE with intermediate size 200.
The tests validate padding from 200 to 256.
The existing backend test consumes the generated parameters.
No files under tests/integration/test_lists/ were modified.
CI or manual-QA test-list coverage is not shown.

Verdict: needs follow-up.

coderabbitai · 2026-07-23T13:45:00Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Last reviewed commit: 2ba620c
Latest commit, not reviewed: ccf446e

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

Changes

INT8 weight-only MoE loading now pads sharded intermediate dimensions to 64-aligned sizes. Runtime validation accounts for activation type and transposed INT8-WOQ layouts. CUTLASS W8A16 tests cover non-aligned Relu2 and SwiGLU configurations.

INT8 MoE support

Layer / File(s)	Summary
Padded INT8 weight and scale loading `tensorrt_llm/_torch/moe/fused_moe/quantization.py`	Weight buffers and FC1 scales use 64-aligned intermediate dimensions. Weight padding uses zeros, and scale padding uses unit values.
Gated-aware MoE shape validation `cpp/tensorrt_llm/thop/moeOp.cpp`	`runMoe` and `runMoeMinLantency` resolve activation parameters and INT8-WOQ layouts before setting defaults and validating FC1/FC2 dimensions.
Non-aligned W8A16 backend coverage `tests/unittest/_torch/moe/test_moe_backend.py`	CUTLASS W8A16 parameters cover intermediate size 200 with Relu2 and SwiGLU activations. The generator verifies both activation families.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: bowenfu, lori-ren

Merge Risk: 🔵 Low · up to 2ba62

This change enables previously failing INT8-WOQ MoE models to load at non-aligned tensor-parallel sizes, with padding intended to preserve numerical behavior. It is mergeable with owner awareness because an interrupted multi-step load could leave partially updated model state until the module is rebuilt or invalidated.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 5 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title follows the required [None][fix] format and clearly describes the primary change: padding INT8-WOQ MoE intermediate dimensions for tensor-parallel sharding.
Description check		The description clearly explains the problem, root cause, fix, test coverage, validation results, and relevant limitations. It does not reproduce the full checklist, but the required technical informa…

Full details: Description check

Explanation

The description clearly explains the problem, root cause, fix, test coverage, validation results, and relevant limitations. It does not reproduce the full checklist, but the required technical information is substantially complete.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (2)

tensorrt_llm/_torch/modules/fused_moe/quantization.py (1)
1444-1446: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Annotate the new scale-padding helper.

_pad_scale is newly defined without parameter or return annotations. Use torch.Tensor -> torch.Tensor.
Proposed fix
-        def _pad_scale(t):
+        def _pad_scale(t: torch.Tensor) -> torch.Tensor:
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/_torch/modules/fused_moe/quantization.py` around lines 1444 -
1446, Update the nested _pad_scale helper to annotate its parameter and return
value as torch.Tensor, preserving its existing padding behavior.
Source: Coding guidelines
cpp/tensorrt_llm/thop/moeOp.cpp (1)
483-483: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Replace tabs with spaces.

These changed lines contain tab indentation, contrary to the required four-space style.

Also applies to: 834-834
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/tensorrt_llm/thop/moeOp.cpp` at line 483, Replace the tab indentation in
the affected comment lines near the non-WOQ activation branches with the
required four-space indentation, including the additional occurrence, without
changing the comment text or surrounding logic.
Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cpp/tensorrt_llm/thop/moeOp.cpp`:
- Around line 795-801: Move the defaulting of unpadded_hidden_size_val in
cpp/tensorrt_llm/thop/moeOp.cpp around lines 795-801 until after the INT8-WOQ
dimension swap, using the resolved hidden_size. Add a min-latency regression in
tests/unittest/_torch/modules/moe/test_moe_backend.py around lines 489-546 where
hidden size differs from padded intermediate size and unpadded_hidden_size is
omitted; retain coverage of the existing case.
- Around line 483-495: Move the fc1/fc2 intermediate-dimension validation in the
relevant MoE setup flow to after swiglu_alpha, swiglu_beta, and swiglu_limit
finalize base_activation_type, matching runMoeMinLantency. Ensure
isGatedActivation(base_activation_type) evaluates the finalized activation so
SwigluBias uses the gated shape rule.

---

Nitpick comments:
In `@cpp/tensorrt_llm/thop/moeOp.cpp`:
- Line 483: Replace the tab indentation in the affected comment lines near the
non-WOQ activation branches with the required four-space indentation, including
the additional occurrence, without changing the comment text or surrounding
logic.

In `@tensorrt_llm/_torch/modules/fused_moe/quantization.py`:
- Around line 1444-1446: Update the nested _pad_scale helper to annotate its
parameter and return value as torch.Tensor, preserving its existing padding
behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 46f7b912-8649-4ef1-bfe3-58ad266624da

📥 Commits

Reviewing files that changed from the base of the PR and between 83733c4 and eb79f4f.

📒 Files selected for processing (4)

cpp/tensorrt_llm/thop/moeOp.cpp
tensorrt_llm/_torch/modules/fused_moe/quantization.py
tests/unittest/_torch/modules/moe/quantize_utils.py
tests/unittest/_torch/modules/moe/test_moe_backend.py

INT8WoqPerChannelFusedMoEMethod did not pad the per-partition intermediate dimension after tensor-parallel sharding, so any model whose moe_intermediate_size / tp_size is not a multiple of 64 failed to load at TP>1. The INT8 mixed-GEMM preprocessor (preprocess_weights_for_mixed_gemm) requires 64-aligned rows (rows_per_tile = 128 * 8 // BITS_PER_ELT_A, BITS_PER_ELT_A = 16 for W8A16). The FP8/NVFP4/MXFP4/WFP4A16 fused-MoE methods round the sharded intermediate up and F.pad; the INT8-woq method did not. Mirror that pattern: round the per-partition intermediate up to a multiple of 64 in create_weights and F.pad each shard up to it in both weight loaders and the fc31 scales. Zero-padding is numerically inert (Relu2(0) = 0 for squared-ReLU, silu(0) * 0 = 0 for gated), so lossless. Reproduces on Nemotron-3-Nano-30B-A3B (moe_intermediate_size = 1856 = 64 * 29) at TP={2,4,8}. Validated on consumer Blackwell (sm_120) at TP=8: native W8A16 checkpoint loads and serves; GSM8K 5-shot strict-match 0.8385 vs BF16 0.8393 (within stderr); also TP=2/4. Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai>

The mixed-GEMM preprocessor requires the packed row count to be a multiple of 64 for W8A16 (rows_per_tile = 128 * 8 // BITS_PER_ELT_A) and of B_ROWS_PER_MMA = 16. Every intermediate_size in MOE_MODEL_CONFIGS is already a multiple of 64, so the path that pads a non-aligned per-partition intermediate had no coverage. Add a W8A16 CUTLASS-only parameter set with intermediate_size = 200 (a multiple of neither 16 nor 64, rounding up to 256), covering both the non-gated (squared-ReLU) and gated (SwiGLU) layouts; the gated case also exercises padding of the w3 gate projection before concatenation. The 128-alignment gate in get_quick_skip_reason() is bypassed for this case only, since that gate is precisely what the padding makes unnecessary; all other skip reasons are honoured. Fails without the padding fix (AssertionError in preprocess_weights_for_mixed_gemm at num_rows % B_ROWS_PER_MMA == 0) and passes with it, validated against the dequantized reference on A100 for float16 and bfloat16. Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai>

The fc1/fc2 intermediate-size validation in runMoe ran before the swiglu_alpha/beta/limit handling, which can promote base_activation_type to SwigluBias (a gated activation). A caller supplying those parameters with an initially non-gated activation was therefore validated under the non-gated shape rule while the kernel ran gated. Move the validation below the activation-parameter handling so it sees the finalized activation type, matching the ordering already used in runMoeMinLatency. This also places it after the INT8-woq hidden/inter size swap, as in runMoeMinLatency. Also drops a stray tab left in runMoeMinLatency by an earlier commit in this stack. Addresses review feedback on the gated/non-gated validation introduced earlier in this stack. Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai>

unpadded_hidden_size_val was initialized from fc2_expert_weights.sizes()[1] before the INT8 weight-only per-channel path reinterprets fc2 as [num_experts, inter_size, hidden_size]. When the optional argument is absent, the output buffer was therefore sized from the intermediate dimension rather than the hidden dimension. Compute the default after the swap in both runMoe and runMoeMinLatency. Not reachable through the current call sites, since CutlassFusedMoE always passes unpadded_hidden_size, but the defaulting path was wrong for a direct caller that omits it. Also widen the non-aligned MoE test config to hidden_size=512 with intermediate_size=200, so the hidden size and the padded intermediate (256) are distinct and a confusion between them is not masked. Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai>

coderabbitai · 2026-08-11T11:44:32Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/unittest/_torch/modules/moe/test_moe_backend.py`:
- Around line 921-924: Update the W8A16 coverage guard before returning params
to track the activation types represented in the generated cases and raise
ValueError unless both Relu2 and Swiglu are present; replace the current assert
params check while preserving the existing empty-case protection.
- Around line 866-897: Add multi-GPU configurations to
W8A16_NON_ALIGNED_MOE_MODEL_CONFIGS and parameter generation so test_moe_backend
exercises post-sharding intermediate sizes: TP2 with 100→128, TP4 with 50→64,
and TP8 with 25→64, while preserving the existing single-GPU 200→256 case.
Ensure the generated cases provide the corresponding world_size, tp_size, and
moe_tp_size values through the existing test configuration flow.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 13a8ef67-097f-4482-8ee8-240c3c722dfa

📥 Commits

Reviewing files that changed from the base of the PR and between 26842d1 and 70f5d9b.

📒 Files selected for processing (3)

cpp/tensorrt_llm/thop/moeOp.cpp
tensorrt_llm/_torch/modules/fused_moe/quantization.py
tests/unittest/_torch/modules/moe/test_moe_backend.py

🚧 Files skipped from review as they are similar to previous changes (2)

tensorrt_llm/_torch/modules/fused_moe/quantization.py
cpp/tensorrt_llm/thop/moeOp.cpp

…6 guard The previous assert only proved the parameter list was non-empty, so a reworded upstream skip message could drop one activation family while the other kept it populated. Track the generated activation types and raise ValueError unless both Relu2 and Swiglu are present. Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai>

Dorijan10 · 2026-08-12T12:35:49Z

@leslie-fang25 thanks for merging #15550. This PR stacks on that one. All the GitHub checks are green and both CodeRabbit review threads are addressed. Could you please trigger CI when convenient?

leslie-fang25 · 2026-08-19T01:42:27Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-19T01:48:56Z

PR_Github #67212 [ run ] triggered by Bot. Commit: 6fa214d Link to invocation

tensorrt-cicd · 2026-08-19T05:28:39Z

PR_Github #67212 [ run ] completed with state SUCCESS. Commit: 6fa214d
/LLM/main/L0_MergeRequest_PR pipeline #54739 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Dorijan10 · 2026-08-19T09:39:27Z

Thanks for the approval and the CI run @leslie-fang25 . I cannot open the CI report or the failure analysis from outside. Could you paste the failed test list from #54739?

I have checked the obvious candidates on my side and found nothing, so the main thing I need to know is whether the failures sit in the MoE tests or somewhere unrelated. The failure analysis page would be ideal.

leslie-fang25 · 2026-08-21T01:40:18Z

Here is the failure analysis page

TL;DR
PR likely to blame?: No — PR #16198 only changes INT8-WOQ MoE padding/test code, while the hard failures are in unrelated sampler.py mypy checks and Qwen3.5 DFlash draft-weight loading.
Failure category: Mixed — unrelated static-analysis failures + unrelated DFlash test failure + one transient infra retry.
Action for PR author: Re-run after main/base-branch fixes land; no evidence points to this PR’s MoE changes.
Build Summary
Job: LLM/job/main/job/L0_MergeRequest_PR #54739
Result: FAILURE
Note: the orchestrator was still running its “AI Failure Analysis” stage when queried, but downstream failing jobs had completed.
PR / Commit: PR #16198, head 6fa214d6f80eab52ca3522a040d8fed7e7d5ecc2
Downstream routing:
Build-x86_64 → Jenkins-Sub-4 LLM/job/helpers/job/Build-x86_64 #1391
Build-SBSA → Jenkins-Sub-3 LLM/job/helpers/job/Build-SBSA #6984
L0_Test-x86_64-Single-GPU → Jenkins-Sub-2 LLM/job/main/job/L0_Test-x86_64-Single-GPU #7100 — FAILURE
L0_Test-SBSA-Single-GPU → Jenkins-Sub-3 LLM/job/main/job/L0_Test-SBSA-Single-GPU #5689 — FAILURE
x86_64/SBSA multi-GPU stages were blocked/skipped after upstream failures.
Failed downstream jobs:
L0_Test-x86_64-Single-GPU #7100
L0_Test-SBSA-Single-GPU #5689
Root Cause(s)
RC-1: Qwen3.5 DFlash test fails during draft model weight loading
Category: Test failure
Failing job: LLM/job/main/job/L0_Test-x86_64-Single-GPU #7100 on Jenkins-Sub-2
Failed test(s):
DGX_H100-PyTorch-3/accuracy/test_llm_api_pytorch.py::TestQwen3_5_4B::test_dflash
Error:
FAILED DGX_H100-PyTorch-3/accuracy/test_llm_api_pytorch.py::TestQwen3_5_4B::test_dflash
- RuntimeError: Executor worker returned error

[TRT-LLM] [E] [_torch] Error executing load_single_module with args
('model.layers.0.mlp.gate_up_proj', Linear()):
The size of tensor a (9216) must match the size of tensor b (9728)
at non-singleton dimension 0

RuntimeError: The size of tensor a (9216) must match the size of tensor b (9728)
at non-singleton dimension 0
Likely cause: The Qwen3.5 DFlash draft checkpoint shape does not match the initialized gate_up_proj module shape. The failure occurs while loading /scratch.trt_llm_data/llm-models/Qwen3.5-4B-DFlash/model.safetensors into the DFlash draft model, before generation.
Related source files:
Runtime stack references:
tensorrt_llm/_torch/models/modeling_speculative.py
tensorrt_llm/_torch/models/modeling_qwen3vl.py
tensorrt_llm/_torch/modules/linear.py
PR-changed files do not overlap this code path:
cpp/tensorrt_llm/thop/moeOp.cpp
tensorrt_llm/_torch/modules/fused_moe/quantization.py
tests/unittest/_torch/modules/moe/test_moe_backend.py
Is waived?: No — code search found no test_dflash waiver in tests/integration/test_lists/waives.txt.
RC-2: Static type-check failure in sampler.py blocks package sanity stages
Category: Static analysis / base-branch failure, not PR-caused
Failing jobs:
LLM/job/main/job/L0_Test-x86_64-Single-GPU #7100
[B200_PCIe-PackageSanityCheck-PY312-DLFW] Run type check
LLM/job/main/job/L0_Test-SBSA-Single-GPU #5689
[GH200-PackageSanityCheck-PY312-DLFW] Run type check
[GH200-PackageSanityCheck-PY312-UB2404] Run type check
Error:
Run static analysis of type correctness using mypy.......................Failed
- hook id: type-check
- exit code: 1

tensorrt_llm/_torch/pyexecutor/sampler/sampler.py:1746:
error: Returning Any from function declared to return "bool"  [no-any-return]

tensorrt_llm/_torch/pyexecutor/sampler/sampler.py:2924:
error: Non-overlapping equality check
(left operand type: Literal[...], right operand type:
Literal[LlmRequestState.GENERATION_COMPLETE])  [comparison-overlap]

Found 2 errors in 1 file (checked 8 source files)
Likely cause: A mypy/type-check regression in tensorrt_llm/_torch/pyexecutor/sampler/sampler.py. PR #16198 does not modify this file, so this is unrelated to the PR.
Related source files:
tensorrt_llm/_torch/pyexecutor/sampler/sampler.py
Is waived?: Not applicable; this is a pre-commit/type-check failure, not a pytest waiver.
Recommended Actions
Do not block PR #16198 on the observed failures as PR-caused; the changed files are MoE INT8-WOQ focused and do not match the failing DFlash/sampler paths.
Ask the owner of sampler.py / static-analysis health to fix:
sampler.py:1746 returning Any from a bool function.
sampler.py:2924 invalid/non-overlapping enum comparison.
Ask the Qwen/DFlash owner to investigate the draft-model shape mismatch:
model.layers.0.mlp.gate_up_proj
expected destination shape includes dimension 9216, checkpoint tensor has 9728.
Re-run CI after the unrelated main/base issues are resolved.

leslie-fang25 · 2026-08-21T01:40:38Z

/bot run --disable-fail-fast

coderabbitai · 2026-08-21T01:40:40Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

tensorrt-cicd · 2026-08-21T01:48:15Z

PR_Github #68054 [ run ] triggered by Bot. Commit: d9f2919 Link to invocation

tensorrt-cicd · 2026-08-21T05:31:13Z

PR_Github #68054 [ run ] completed with state SUCCESS. Commit: d9f2919
/LLM/main/L0_MergeRequest_PR pipeline #55512 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tensorrt-cicd · 2026-10-14T23:59:25Z

PR_Github #77931 [ run ] triggered by Bot. Commit: 290f7c3 Link to invocation

tensorrt-cicd · 2026-10-15T06:40:32Z

PR_Github #77931 [ run ] completed with state FAILURE. Commit: 290f7c3
/LLM/main/L0_MergeRequest_PR pipeline #64235 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Dorijan10 · 2026-10-15T09:15:29Z

@leslie-fang25 Thanks for triggering the CI, can you please send across the CI report from #64235, so we can triage the failures.

leslie-fang25 · 2026-10-16T01:59:57Z

There is a real failure, here is the CI failure analysis

TL;DR
PR likely to blame?: Partially — the PR directly added generate_int8_woq_non_aligned_test_params() in tests/unittest/_torch/moe/test_moe_backend.py, and that new import-time guard is failing test collection on both x86 and ARM CPU stages. There are also unrelated-looking failures in test_tracing.py and speculative decoding.
Failure category: Mixed — 1 PR-caused test collection failure + 2 unrelated/non-PR test failures.
Action for PR author: Fix the new MoE test-parameter generator so it does not raise ValueError during collection when platform/test-db skip reasons prevent generating both activation cases.
Confidence: Medium — the PR-caused MoE failure is clear; the unrelated GPU failures were not exhaustively root-caused because the PR-caused CPU failure is sufficient to fail CI.
Build Summary
Job: LLM/job/main/job/L0_MergeRequest_PR #64235
Result: FAILURE
Note: Jenkins still showed building: true because the final AI analysis/reporting stage was still in progress, but downstream test failures had already completed.
PR: #16198, head SHA 290f7c31fcb30c435fd6394b133ebb745266a115
PR title: [None][fix] Pad INT8-woq MoE intermediate to 64 for tensor-parallel sharding
Changed files:
cpp/tensorrt_llm/thop/moe/cutlass/moeOp.cpp
tensorrt_llm/_torch/moe/fused_moe/quantization.py
tests/unittest/_torch/moe/test_moe_backend.py
Downstream routing:
Build-SBSA → Jenkins-Sub-1 #10680
Build-x86_64 → Jenkins-Sub-1 #10786
L0_Test-SBSA-Single-GPU → Jenkins-Sub-3 #7988 — FAILURE
L0_Test-x86_64-Single-GPU → Jenkins-Sub-4 #3457 — FAILURE
SBSA/x86_64 Multi-GPU stages were blocked after single-GPU failures.
Root Cause(s)
RC-1: New MoE test generator raises ValueError during pytest collection
Category: Test failure / PR-caused
Failing jobs:
LLM/job/main/job/L0_Test-SBSA-Single-GPU #7988, stage [CPU-Generic-arm-1] Run Pytest
LLM/job/main/job/L0_Test-x86_64-Single-GPU #3457, stage [CPU-Generic-x86-1] Run Pytest
Failed test(s):
Collection of unittest/_torch/moe/test_moe_backend.py
Wrapper test:
CPU-Generic-arm-1/test_unittests.py::test_unittests_v2[unittest/_torch/moe]
CPU-Generic-x86-1/test_unittests.py::test_unittests_v2[unittest/_torch/moe]
Error:
ERROR collecting _torch/moe/test_moe_backend.py
unittest/_torch/moe/test_moe_backend.py:2246: in <module>
    TEST_PARAMS += generate_int8_woq_non_aligned_test_params()
unittest/_torch/moe/test_moe_backend.py:2242: in generate_int8_woq_non_aligned_test_params
    raise ValueError("non-aligned W8A16 coverage must include both Relu2 and Swiglu cases")
E   ValueError: non-aligned W8A16 coverage must include both Relu2 and Swiglu cases
Likely cause: PR #16198 added generate_int8_woq_non_aligned_test_params() and an import-time assertion that both Relu2 and Swiglu cases must be generated. In CI CPU contexts, iter_base_test_configs(...) appears to skip one or both of those W8A16 CUTLASS configs for reasons other than the "Non-128-aligned" skip reason, so the generator raises at module import time and breaks pytest collection entirely.
Related source files:
tests/unittest/_torch/moe/test_moe_backend.py
Is waived?: No evidence of a waiver found for this failure.
RC-2: A10 PyTorch tracing tests failed after rerun
Category: Test failure / likely unrelated to this PR
Failing job: LLM/job/main/job/L0_Test-x86_64-Single-GPU #3457
Failing stage: [A10-PyTorch-2] Run Pytest
Failed test(s):
A10-PyTorch-2.unittest.others.test_tracing::test_tracing[pytorch-disable_processpool]
A10-PyTorch-2.unittest.others.test_tracing::test_tracing[pytorch-enable_processpool]
Error:
AssertionError: failure reported in unittests; culprit test(s):
FAILED A10-PyTorch-2.unittest.others.test_tracing::test_tracing[pytorch-disable_processpool],
FAILED A10-PyTorch-2.unittest.others.test_tracing::test_tracing[pytorch-enable_processpool]
Likely cause: This is outside the MoE/INT8-WOQ files changed by the PR and persisted after rerun. It looks like an independent test failure rather than a direct regression from PR #16198.
Related source files:
unittest/others/test_tracing.py
Is waived?: No waiver found.
RC-3: H100 speculative decoding test left in-flight / timeout-like failure
Category: Test failure / inconclusive, likely unrelated to this PR
Failing job: LLM/job/main/job/L0_Test-x86_64-Single-GPU #3457
Failing stage: [DGX_H100-PyTorch-5] Run Pytest
Failed test(s):
DGX_H100-PyTorch-5/unittest/_torch/speculative/hw_agnostic/test_ngram.py::test_llama_ngram[True-True-TRTLLM]
Error:
FAILED DGX_H100-PyTorch-5/test_unittests.py::test_unittests_v2[unittest/_torch/speculative/hw_agnostic]
- AssertionError: failure reported in unittests; culprit test(s):
IN-FLIGHT DGX_H100-PyTorch-5/unittest/_torch/speculative/hw_agnostic/test_ngram.py::test_llama_ngram[True-True-TRTLLM]
Likely cause: The test was still marked IN-FLIGHT when the stage failed after ~1h38m, suggesting a timeout/hang-style failure. The failing area is speculative decoding, unrelated to the PR’s INT8-WOQ MoE padding changes.
Related source files:
unittest/_torch/speculative/hw_agnostic/test_ngram.py
Is waived?: No waiver found.
Recommended Actions
Fix the PR-caused MoE collection failure first. In tests/unittest/_torch/moe/test_moe_backend.py, avoid raising ValueError at module import time when CI/platform skip logic removes the new W8A16 non-aligned configs.
Recommended approaches:
Convert the guard into a test-time assertion only in environments where those configs are expected to be generated.
Or relax the skip handling so CPU/aarch64 collection can proceed.
Or explicitly skip the new generated cases when CUTLASS/W8A16/GPU prerequisites are unavailable, without raising during collection.
Re-run at least:
CPU x86 unittest collection for unittest/_torch/moe
CPU ARM unittest collection for unittest/_torch/moe
The intended GPU coverage for the new W8A16 non-aligned MoE cases.
Treat A10 test_tracing.py and H100 speculative test_ngram as separate follow-ups unless they reproduce after the MoE collection issue is fixed.
If the PR author wants to validate locally, the key failure mode is simply importing/collecting:
tests/unittest/_torch/moe/test_moe_backend.py
under CPU-only or otherwise skipped W8A16/CUTLASS environments.

…ners Runners without a GPU reject W8A16 CUTLASS in can_implement, so the generator yields no cases and the guard raised during collection. Raise only on SM80+ runners, where an empty list means the alignment skip message changed upstream. Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai>

Dorijan10 · 2026-10-16T09:40:11Z

Thanks for the report @leslie-fang25. RC-1 was mine: CPU runners reject W8A16 in can_implement, so the generator produced no cases and the guard raised at collection. Fixed in 5249ba3; the guard now only fires on SM80+ runners.

RC-2 (test_tracing, A10) and RC-3 (test_ngram, H100) are outside this PR's files and not waived on main. Could you trigger /bot run again, and waive them if they recur?

leslie-fang25 · 2026-10-19T06:56:32Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-19T07:04:03Z

PR_Github #78835 [ run ] triggered by Bot. Commit: 5249ba3 Link to invocation

tensorrt-cicd · 2026-10-19T11:21:04Z

PR_Github #78835 [ run ] completed with state SUCCESS. Commit: 5249ba3
/LLM/main/L0_MergeRequest_PR pipeline #64954 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…-padding Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai>

Dorijan10 · 2026-10-19T11:56:00Z

@leslie-fang25 I merged current main (3c93414). Our branch still had the old test_tracing.py from before #19178, which may explain the A10 failure, but please check or share the CI report from the last run. Could you re-trigger /bot run again? If test_ngram (H100) hangs again, could it be waived? Main has no change to it.

leslie-fang25 · 2026-10-19T23:37:13Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-19T23:43:45Z

PR_Github #79053 [ run ] triggered by Bot. Commit: 3c93414 Link to invocation

tensorrt-cicd · 2026-10-20T03:57:58Z

PR_Github #79053 [ run ] completed with state FAILURE. Commit: 3c93414
/LLM/main/L0_MergeRequest_PR pipeline #65123 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Dorijan10 · 2026-10-20T09:16:00Z

Thanks for running it again @leslie-fang25, can you please share the CI report? Especially if there are new failures.

leslie-fang25 · 2026-10-21T01:38:38Z

Here is the CI analysis, the failures should be unrelated. Could you resolve the conflict and I will trigger the CI again?

TL;DR
PR likely to blame?: No — PR #16198 only changes INT8-WOQ MoE padding and MoE unit coverage; the failures are in unrelated LLMAPI MPI session cleanup, H100 executor/KV-cache tests, and A10 test harness unfinished tests.
Failure category: Mixed — unrelated test failure/flakiness plus likely infrastructure/GPU-context failure.
Action for PR author: Re-run CI; there is no evidence that the MoE padding changes caused these failures.
Build Summary
Job: LLM/job/main/job/L0_MergeRequest_PR #65123
Result: FAILURE
Note: orchestrator was still finalizing the AI failure analysis stage when queried, but downstream test jobs had already failed.
PR: NVIDIA/TensorRT-LLM #16198
Head SHA: 3c93414943e7bc3fda1e766e5134bc3578f6fc34
Title: [None][fix] Pad INT8-woq MoE intermediate to 64 for tensor-parallel sharding
Downstream routing:
Build-SBSA #11642 on Jenkins-Sub-2: SUCCESS
Build-x86_64 #10212 on Jenkins-Sub-3: SUCCESS
L0_Test-SBSA-Single-GPU #9432 on Jenkins-Sub-2: FAILURE
L0_Test-x86_64-Single-GPU #3683 on Jenkins-Sub-4: FAILURE
Multi-GPU test stages were blocked/cascaded after single-GPU failures.
Changed files in PR:

cpp/tensorrt_llm/thop/moe/cutlass/moeOp.cpp
tensorrt_llm/_torch/moe/fused_moe/quantization.py
tests/unittest/_torch/moe/test_moe_backend.py
No jenkins/ files were modified.

Root Cause(s)
RC-1: SBSA CPU LLMAPI MPI session timed out in FlashInfer temporary cleanup
Category: Test failure / likely flake
Failing job: LLM/job/main/job/L0_Test-SBSA-Single-GPU #9432
Failed test(s):
CPU-Generic-arm-1.unittest.llmapi.test_mpi_session::test_remote_mpi_session[flashinfer_temporary_cleanup]
Error:
FAILED CPU-Generic-arm-1/test_unittests.py::test_unittests_v2[unittest/llmapi/test_mpi_session.py]
AssertionError: failure reported in unittests; culprit test(s):
FAILED CPU-Generic-arm-1.unittest.llmapi.test_mpi_session::test_remote_mpi_session[flashinfer_temporary_cleanup]

subprocess.CalledProcessError:
Command '['bash', '.../_test_remote_mpi_session.sh', 'flashinfer_temporary_cleanup']'
returned non-zero exit status 124.
Relevant stderr:

+ timeout 60 mpirun --allow-run-as-root ... trtllm-llmapi-launch python3 _run_mpi_comm_task.py --task_type flashinfer_temporary_cleanup
mkdir: cannot create directory ‘.../home-file’: Not a directory
Failed to create the persistent FlashInfer JIT workspace root: .../home-file/.cache/tensorrt_llm/flashinfer
Using temporary FlashInfer JIT workspace after persistent workspace isolation failed: .../trtllm-flashinfer-rank-0...
Using temporary FlashInfer JIT workspace after persistent workspace isolation failed: .../trtllm-flashinfer-rank-1...
...
returned non-zero exit status 124
Likely cause: The remote MPI test hung until its 60-second timeout after falling back to a temporary FlashInfer JIT workspace. This is unrelated to the PR’s MoE INT8-WOQ changes.
Related source files:
tests/unittest/llmapi/test_mpi_session.py
tests/unittest/llmapi/_test_remote_mpi_session.sh
Is waived?: No — this test is not listed in tests/integration/test_lists/waives.txt.
RC-2: H100 executor tests failed due to CUDA invalid device context / unusable KV cache manager
Category: Infrastructure/Transient or pre-existing GPU-context failure
Failing job: LLM/job/main/job/L0_Test-x86_64-Single-GPU #3683
Failing stage: [DGX_H100-PyTorch-4] Run Pytest
Failed test(s): Multiple tests under:
unittest/_torch/executor/test_pytorch_model_engine.py
unittest/_torch/executor/test_pytorch_model_engine_warmup.py
Representative failures include:

PyTorchModelEngineTestCase::test_forward_pass_callable_on_cuda_graph_on

PyTorchModelEngineTestCase::test_foward_pass_callable_off

PyTorchModelEngineTestCase::test_kv_cache_manager_with_execution_stream

PyTorchModelEngineTestCase::test_warmup

test_warmup_builders_resynchronize_stale_draft_buffers[...]

Error:

tensorrt_llm.bindings.internal.batch_manager.kv_cache_manager_v2.CorruptedError:
std::shared_ptr<...::KvCache> KvCacheManager::createKvCache(...):
KV cache manager is unusable.

CachedCudaEvent::CachedCudaEvent(CudaStream):
CUDA driver error: 201 (invalid device context)
Likely cause: The H100 runner’s CUDA context became invalid, causing many unrelated executor/KV-cache tests to fail with the same CUDA driver error: 201 (invalid device context). The failures are clustered around CUDA/KV-cache initialization and do not involve the MoE files changed by this PR.
Related source files:
Runtime area: _torch/executor, KV cache manager bindings
Not related to changed PR files
Is waived?: No — the specific failing test_pytorch_model_engine / test_pytorch_model_engine_warmup cases are not waived. Some nearby executor/kv-cache tests are waived, but not these failing cases.
RC-3: A10 stage terminated with unfinished tests
Category: Infrastructure/Test harness failure
Failing job: LLM/job/main/job/L0_Test-x86_64-Single-GPU #3683
Failing stage: [A10-PyTorch-2] Run Pytest
Failed test(s): No specific assertion failure was found; the stage reported unfinished tests.
Error:
unittest/_torch/sampler/test_torch_sampler.py will rerun 1 time, because it did not run
unittest/_torch/test_mnnvl_alltoall_workspace.py will rerun 1 time, because it did not run
...
[INFRA-RETRY] A10-PyTorch-2: stage failed with no infra pattern matched
(classified as user failure); not retrying.
Exception: hudson.AbortException: Some tests terminated unexpectedly, please check the test report.
Likely cause: Test harness/runner termination left a set of A10 tests unfinished. The log did not show a concrete code assertion failure for the A10 stage, and the unfinished tests are unrelated to the PR’s MoE INT8-WOQ changes.
Related source files: Not identifiable from the log; affected tests are broad A10 unit test shards.
Is waived?: Not applicable — this was an unfinished-test/harness termination, not a single waived test failure.
Recommended Actions
Re-run CI. The observed failures are unrelated to the PR’s MoE INT8-WOQ changes.
If the SBSA failure repeats, route to the LLMAPI/MPI owners to investigate test_remote_mpi_session[flashinfer_temporary_cleanup] timeout behavior on aarch64 CPU.
If the H100 failures repeat on the same runner class, route to infra/GPU test owners; the common symptom is CUDA driver error: 201 (invalid device context), suggesting runner/GPU-context corruption rather than this PR.
If the A10 unfinished-test failure repeats, inspect the generated unfinished_test.txt and runner health; no actionable PR-side code failure was identified.

…-padding Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai> # Conflicts: # cpp/tensorrt_llm/thop/moe/cutlass/moeOp.cpp

Dorijan10 · 2026-10-21T10:01:33Z

Thanks for the report @leslie-fang25. I resolved the conflict with main in moeOp.cpp (ccf446e). #19753 had already moved the unpadded_hidden_size default after the INT8-woq swap, so that part of our diff is gone and only the validation move remains. Could you trigger CI again?

leslie-fang25 · 2026-10-21T23:18:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-21T23:26:46Z

PR_Github #79707 [ run ] triggered by Bot. Commit: ccf446e Link to invocation

tensorrt-cicd · 2026-10-22T05:30:03Z

PR_Github #79707 [ run ] completed with state FAILURE. Commit: ccf446e
/LLM/main/L0_MergeRequest_PR pipeline #65624 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Dorijan10 · 2026-10-22T08:55:51Z

Thanks for triggering @leslie-fang25. #65624 failed again, can you send across the CI report?

leslie-fang25 · 2026-10-22T23:03:39Z

/bot run --disable-fail-fast

leslie-fang25 · 2026-10-22T23:04:42Z

The failure seems unrelated, I have triggered to rerun these tests again

TL;DR
PR likely to blame?: No — PR #16198 only changes INT8-WOQ MoE padding code/tests, while the hard failures are stale trtllm-serve JSON schemas and unrelated sparse-attention CUDA illegal-memory-access failures.
Failure category: Mixed — 2 unrelated test failures plus 1 transient infrastructure retry on H100.
Action for PR author: Re-run after main/CI is fixed; if still required, update checked-in serve schema JSONs with python3 scripts/generate_trtllm_serve_schemas.py, but this is not caused by this PR’s MoE changes.
Build Summary
Job: LLM/job/main/job/L0_MergeRequest_PR #65624
Result: FAILURE / still showing building: true because the AI analysis stage was still in progress when queried; downstream test jobs have completed.
PR / Commit: PR #16198, head ccf446ef4ca69c9ece0c3043a6f702660cd68a73
Downstream routing:
Build-x86_64 → Jenkins-Sub-4 LLM/job/helpers/job/Build-x86_64 #4865
Build-SBSA → Jenkins-Sub-1 LLM/job/helpers/job/Build-SBSA #11215
L0_Test-x86_64-Single-GPU → Jenkins-Sub-2 LLM/job/main/job/L0_Test-x86_64-Single-GPU #9763 — FAILURE
L0_Test-SBSA-Single-GPU → Jenkins-Sub-3 LLM/job/main/job/L0_Test-SBSA-Single-GPU #8336 — FAILURE
Test-x86_64-Multi-GPU / Test-SBSA-Multi-GPU — blocked/cascaded, not independent root causes
PR changed files:
cpp/tensorrt_llm/thop/moe/cutlass/moeOp.cpp
tensorrt_llm/_torch/moe/fused_moe/quantization.py
tests/unittest/_torch/moe/test_moe_backend.py
No jenkins/, schema, serve-config, or sparse-attention files changed.
Root Cause(s)
RC-1: Checked-in trtllm-serve schema JSON files are stale
Category: Test failure
Failing jobs:
LLM/job/main/job/L0_Test-x86_64-Single-GPU #9763, stage [CPU-Generic-x86-1] Run Pytest
LLM/job/main/job/L0_Test-SBSA-Single-GPU #8336, stage [CPU-Generic-arm-1] Run Pytest
Failed test(s):
unittest.llmapi.test_trtllm_serve_config_schemas::test_checked_in_schemas_are_current
Error:
FAILED unittest/CPU-Generic-x86-1/unittest/llmapi/test_trtllm_serve_config_schemas.py::test_checked_in_schemas_are_current
ValueError: Missing or stale configuration schemas:
trtllm-serve-config.schema.json,
trtllm-serve-disagg-config.schema.json,
trtllm-serve-visual-gen-config.schema.json.
Run python3 scripts/generate_trtllm_serve_schemas.py and commit the updated JSON.
Same failure repeated on SBSA/arm:

FAILED unittest/CPU-Generic-arm-1/unittest/llmapi/test_trtllm_serve_config_schemas.py::test_checked_in_schemas_are_current
ValueError: Missing or stale configuration schemas: trtllm-serve-config.schema.json,
trtllm-serve-disagg-config.schema.json, trtllm-serve-visual-gen-config.schema.json.
Likely cause: The generated serve schema JSON files in the branch do not match runtime schema generation. This failed on both x86 and SBSA, including rerun, so it is a hard test failure rather than a one-off flake.
Related source files:
unittest/llmapi/test_trtllm_serve_config_schemas.py
scripts/generate_trtllm_serve_schemas.py
generated schema JSON files listed above
PR correlation: Not traceable to PR #16198. The PR does not touch serve config/schema code or generated schema JSON files.
Is waived?: No evidence this is waived; the test failed after rerun.
RC-2: B200 sparse-attention tests hit CUDA illegal memory access
Category: Test failure, unrelated to PR
Failing job: LLM/job/main/job/L0_Test-x86_64-Single-GPU #9763, stage [DGX_B200-PyTorch-1] Run Pytest
Failed test(s): The first reported failures are in sparse attention, then thousands cascade due to CUDA sticky error:
unittest._torch.attention.sparse.csa2.test_module::test_bounded_replay_full_module_graph_changes_hit_position
unittest._torch.attention.sparse.csa2.test_module::test_native_chunked_compression[...]
unittest._torch.attention.sparse.deepseek_v4.test_compressor_kernel::test_prefill_corner_cases[...]
plus many subsequent failures after CUDA entered a sticky error state
Error:
FAILED unittest/DGX_B200-PyTorch-1/unittest/_torch/attention/sparse/csa2/test_module.py::test_bounded_replay_full_module_graph_changes_hit_position
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Returning 700 (CUDA_ERROR_ILLEGAL_ADDRESS) from cuStreamSynchronize

FAILED unittest/DGX_B200-PyTorch-1/unittest/_torch/attention/sparse/csa2/test_module.py::test_native_chunked_compression[False-128-2]
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Sticky error detected
Returning 700 (CUDA_ERROR_ILLEGAL_ADDRESS) from cuStreamGetCaptureInfo_v3
Likely cause: A sparse-attention CUDA/kernel issue or B200-specific runtime regression caused an illegal memory access. After the first CUDA illegal address, the CUDA context became poisoned, producing thousands of cascading failures.
Related source files:
unittest/_torch/attention/sparse/csa2/test_module.py
unittest/_torch/attention/sparse/deepseek_v4/test_compressor_kernel.py
PR correlation: Not traceable to PR #16198. The PR only modifies MoE INT8-WOQ padding paths and MoE backend tests; it does not modify sparse attention code/tests.
Is waived?: No evidence this is waived.
RC-3: H100 executor failures were classified as transient Slurm/monitor infrastructure
Category: Infrastructure/Transient
Failing job: LLM/job/main/job/L0_Test-x86_64-Single-GPU #9763, stage [DGX_H100-PyTorch-2] Run Pytest
Initial symptom:
FAILED unittest/DGX_H100-PyTorch-2/unittest/_torch/executor/test_pytorch_model_engine.py::...
tensorrt_llm.bindings.internal.batch_manager.kv_cache_manager_v2.CorruptedError:
KV cache manager is unusable.
CachedCudaEvent(...): CUDA driver error: 201 (invalid device context)
Infrastructure classification in log:
[INFRA-RETRY] DGX_H100-PyTorch-2: Infrastructure failure detected on attempt 1: <typed:slurm-job-still-running>
[INFRA-RETRY] DGX_H100-PyTorch-2: Exception: trtllm.exceptions.InfraFailure:
SLURM job 20648596 for DGX_H100-PyTorch-2 is still in non-terminal state RUNNING
(97min of 240min walltime); the monitor lost contact while the job was alive
(transient infra), so this is not a test failure.
Likely cause: Jenkins/Slurm monitor lost contact with a still-running Slurm job. The pipeline correctly classified this attempt as infrastructure and retried. This is not a PR regression.
Related source files: None.
Is waived?: Not applicable.
Recommended Actions
Do not attribute this CI failure to PR #16198’s MoE changes. The observed failures are in serve schema validation and sparse-attention CUDA tests, not INT8-WOQ MoE.
For the schema failure, either:
rebase/re-run after main is updated, or
if the branch is expected to carry generated artifacts, run:
python3 scripts/generate_trtllm_serve_schemas.py
and commit updated:
trtllm-serve-config.schema.json
trtllm-serve-disagg-config.schema.json
trtllm-serve-visual-gen-config.schema.json
Report or track the B200 sparse-attention CUDA_ERROR_ILLEGAL_ADDRESS separately with the sparse-attention owners. This appears unrelated to the PR and poisons the CUDA context, causing many cascading failures.
Ignore the H100 executor failures from the first attempt as infrastructure noise; the log explicitly classified it as a Slurm monitor/contact loss while the job was still running.

tensorrt-cicd · 2026-10-22T23:11:42Z

PR_Github #79987 [ run ] triggered by Bot. Commit: ccf446e Link to invocation

tensorrt-cicd · 2026-10-23T08:07:48Z

PR_Github #79987 [ run ] completed with state SUCCESS. Commit: ccf446e
/LLM/main/L0_MergeRequest_PR pipeline #65839 completed with status: 'SUCCESS'

CI Report

⚠️ Resource Maintenance — Test Coverage Reduced:

Resource maintenance in LLM/main/L0_Test-x86_64-Multi-GPU #2913: B200 multi-GPU stages are paused to relieve the CI queue backlog (109 running / 459 pending on 2026-10-22). Owner: @sharabeshj-nvda. The CI team decides when to lift it.
Affected stages:
DGX_B200-2_GPUs-PyTorch-1
DGX_B200-4_GPUs-PyTorch-1
DGX_B200-4_GPUs-PyTorch-2
DGX_B200-4_GPUs-PyTorch-3
DGX_B200-4_GPUs-PyTorch-Ray-1
DGX_B200-8_GPUs-PyTorch-1
DGX_B200-8_GPUs-PyTorch-2
DGX_B200-8_GPUs-PyTorch-3
DGX_B200-8_GPUs-PyTorch-4
DGX_B200-8_GPUs-PyTorch-Ray-1

Link to invocation

Dorijan10 · 2026-10-23T10:20:52Z

Thanks @leslie-fang25 for re-running. CI passed on ccf446e! Could one of you merge it when convenient?

Move the W4A16 wiring onto the cutlass/ leaf package as cutlass/w4a16.py, rename FusedMoeOpFlags.use_int8_woq_per_channel to use_woq_per_channel, and keep the INT8 padding from NVIDIA#16198 in WoqPerChannelFusedMoEMethodBase, gated to INT8. Signed-off-by: Dorian Magasic <dorian.magasic@turintech.ai>

github-actions Bot assigned Dorijan10 Jul 9, 2026

Dorijan10 force-pushed the fix/int8-woq-moe-tp-padding branch 2 times, most recently from 4db3246 to eb79f4f Compare July 23, 2026 13:21

Dorijan10 marked this pull request as ready for review July 23, 2026 13:34

Dorijan10 requested a review from a team as a code owner July 23, 2026 13:34

Dorijan10 requested review from leslie-fang25 and rosong11 July 23, 2026 13:34

Dorijan10 mentioned this pull request Jul 23, 2026

[None][fix] Enable INT8 weight-only (W8A16) MoE for non-gated activations #15550

Merged

coderabbitai Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread cpp/tensorrt_llm/thop/moeOp.cpp Resolved

Comment thread cpp/tensorrt_llm/thop/moeOp.cpp Resolved

Dorijan10 added 4 commits August 11, 2026 12:11

Dorijan10 force-pushed the fix/int8-woq-moe-tp-padding branch from 5c4edb4 to 70f5d9b Compare August 11, 2026 11:44

coderabbitai Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread tests/unittest/_torch/moe/test_moe_backend.py Resolved

Comment thread tests/unittest/_torch/moe/test_moe_backend.py Resolved

leslie-fang25 approved these changes Aug 17, 2026

View reviewed changes

Merge branch 'main' into fix/int8-woq-moe-tp-padding

d9f2919

rosong11 approved these changes Oct 19, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into fix/int8-woq-moe-tp…

3c93414

…-padding Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai>

Merge remote-tracking branch 'upstream/main' into fix/int8-woq-moe-tp…

ccf446e

…-padding Signed-off-by: Dorijan10 <dorian.magasic@turintech.ai> # Conflicts: # cpp/tensorrt_llm/thop/moe/cutlass/moeOp.cpp

leslie-fang25 merged commit 64ef54c into NVIDIA:main Oct 23, 2026

Conversation

Dorijan10 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Reproduction

Fix

Test Coverage

Validation

Dev Engineer Review

QA Engineer Review

Uh oh!

coderabbitai Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 11, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Dorijan10 commented Aug 12, 2026

Uh oh!

leslie-fang25 commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 19, 2026

Uh oh!

Dorijan10 commented Aug 19, 2026

Uh oh!

leslie-fang25 commented Aug 21, 2026

Uh oh!

leslie-fang25 commented Aug 21, 2026

Uh oh!

coderabbitai Bot commented Aug 21, 2026

Uh oh!

tensorrt-cicd commented Aug 21, 2026

Uh oh!

tensorrt-cicd commented Aug 21, 2026

Uh oh!

tensorrt-cicd commented Oct 14, 2026

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

Dorijan10 commented Oct 15, 2026

Uh oh!

leslie-fang25 commented Oct 16, 2026

Uh oh!

Dorijan10 commented Oct 16, 2026

Uh oh!

leslie-fang25 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

Dorijan10 commented Oct 19, 2026

Uh oh!

leslie-fang25 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

Dorijan10 commented Oct 20, 2026

Uh oh!

leslie-fang25 commented Oct 21, 2026

Uh oh!

Dorijan10 commented Oct 21, 2026

Uh oh!

leslie-fang25 commented Oct 21, 2026

Dorijan10 commented Jul 9, 2026 •

edited

Loading

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading