Skip to content

[https://nvbugs/6672360][test] Unwaive GPT-OSS accuracy cases for release 1.3 - #20632

Merged
dongfengy merged 1 commit into
NVIDIA:release/1.3from
dongfengy:release13-unwaive-gptoss-six-20261019
Oct 20, 2026
Merged

dongfengy merged 1 commit into
NVIDIA:release/1.3from
dongfengy:release13-unwaive-gptoss-six-20261019

Conversation

@dongfengy

@dongfengy dongfengy commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

The change removes seven GPT-OSS accuracy waivers from tests/integration/test_lists/waives.txt: six platform-specific entries and the global CUTLASS Eagle3 2-GPU entry. The global Triton Eagle3 waiver remains, so that variant stays skipped. Removing the global CUTLASS waiver enables the matching case on other configured platforms as well as RTX PRO 6000. No test code or assertions changed.

The provided test results report that all six selected cases passed on the release branch. The reported runs do not establish coverage for every platform and case combination enabled by the waiver changes.

QA Engineer Review

The waiver list changes enable existing GPT-OSS accuracy cases; they do not add or alter test functions, IDs, or selectors. The available results report six passing cases with no skips, errors, or failures. Release CI does not run all six hardware and case combinations in its current premerge configuration, so broader coverage needs follow-up in release CI. Coverage verdict: needs follow-up.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Removes six platform-specific GPT-OSS accuracy waivers and the global CUTLASS Eagle3 2-GPU waiver. The corresponding cases are present in the relevant test lists, but current premerge CI does not cover all enabled hardware and case combinations.

Description

Restore six GPT-OSS accuracy cases on release/1.3 by removing seven waiver entries: the six platform-specific entries and the overlapping all-platform CUTLASS Eagle3 2-GPU waiver. Removing only the RTX entry would leave that case skipped by the global rule. The global Triton variant remains waived.

Related tracking: https://nvbugs/6672360, https://nvbugs/6649739, and https://nvbugs/6762288.

This change intentionally re-enables the global CUTLASS case on other configured CI platforms. The manual validation below covers the listed hardware; broader coverage is left to release CI.

Test Coverage

All six original tests passed on release/1.3@e2be0074b6a8129486bb9c683bad54d439dcc9a2, using wheels built from that source and release-Dockerfile devel images. No test assertions, model inputs, or sample counts were changed. The PR base d40f019da251d52d783e3176f343e7b39d4d18e8 differs only by three unrelated DeepSeek waiver deletions.

Platform Original case GSM8K accuracy GPQA accuracy Result
GB200, 1 GPU test_w4_1gpu[v2_kv_cache-True-True-trtllm-fp8] 89.992% N/A PASS
GB300, 4 GPUs test_w4_4gpus[v1_kv_cache-tp4-trtllm-auto] 90.599% 63.131% PASS
RTX PRO 6000 Blackwell Server Edition, 2 GPUs test_eagle3_2gpus[cutlass-one_model-overlap_scheduler] 90.751% 64.141% PASS
RTX PRO 6000 Blackwell Server Edition, 4 GPUs test_eagle3_4gpus[v1_kv_cache-cutlass-one_model-overlap_scheduler] 89.992% 64.646% PASS
RTX PRO 6000 Blackwell Server Edition, 4 GPUs test_eagle3_4gpus[v2_kv_cache-cutlass-one_model-overlap_scheduler] 90.447% 63.131% PASS
RTX PRO 6000 Blackwell Server Edition, 4 GPUs test_eagle3_vswa_reuse_4gpus[v1_kv_cache-one_model] 90.220% 62.626% PASS

Every case used all 1,319 GSM8K samples; the five GPQA cases used all 198 samples. Every accuracy exceeded its original threshold, with exactly one expected test and no skips/errors/failures per run. VSWA also passed the original cold/warm reuse and identical 8-token output assertions.

Release CI should validate the enabled selections. Its current premerge configuration does not execute all six hardware/case combinations above.

PR Checklist

  • Description explains the seven-rule removal and the global CUTLASS scope.
  • Waiver-only change; no new code paths, APIs, dependencies, ownership, documentation, or architecture changes.
  • Existing original tests provide coverage; full manual results are listed above.
  • Please check this after reviewing the above items as appropriate for this PR.

…ease 1.3

Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@coderabbitai

coderabbitai Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 01716e48-a95d-4383-9f72-64ec52f3478b
📥 Commits

Reviewing files that changed from the base of the PR and between d40f019 and 8dec304.

📒 Files selected for processing: 1
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes: 1
  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

Seven GPTOSS integration test skip waivers and their associated NVBUG references were removed from the waiver list.

Changes

GPTOSS integration waivers

Layer / File(s) Summary
Remove GPTOSS skip waivers
tests/integration/test_lists/waives.txt
Removed waivers for one Eagle3 CUTLASS overlap-scheduler case, two W4 TRT-LLM cases, and four RTX PRO 6000 Blackwell cases using Eagle3 CUTLASS or VSWA-reuse.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~4 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 8dec3

No actionable merge-blocking issue is established. The newly enabled cases can proceed through normal release CI validation.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title clearly identifies the NVBugs issue, change type, affected GPT-OSS accuracy cases, and release branch scope.
Description check Passed The description explains the seven waiver removals, global CUTLASS scope, related tracking, manual test coverage, limitations, and checklist status. It satisfies the required template.
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #79040 [ run ] triggered by Bot. Commit: 8dec304 Link to invocation

@dongfengy
dongfengy requested a review from BowenFu October 19, 2026 22:51
@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #79040 [ run ] completed with state SUCCESS. Commit: 8dec304
/LLM/release-1.3/L0_MergeRequest_PR pipeline #373 completed with status: 'SUCCESS'

CI Report

Link to invocation

@dongfengy
dongfengy merged commit a334c95 into NVIDIA:release/1.3 Oct 20, 2026
QiJune added a commit to QiJune/TensorRT-LLM that referenced this pull request Oct 20, 2026
Remove 5 waive entries on release/1.3 whose NVBugs are closed and whose
tests passed on the waived platform in pre-merge CI (L0_MergeRequest_PR
NVIDIA#342 and NVIDIA#375, with post-merge extra stages):
6481323, 6428057, 6762288 (triton variant), 6700265, 6075431.

Handled by other release/1.3 PRs: 6893730 (NVIDIA#20592), the 6762288 cutlass
variant and 6672360 (NVIDIA#20632), 6278337 (NVIDIA#20585).

Waives for the other closed bugs (6435097, 6695515, 6812288, 6432818,
6428092, 6432826) are kept: their tests did not run on the waived
platform in that CI run, so the unwaive is not verified yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
QiJune added a commit to QiJune/TensorRT-LLM that referenced this pull request Oct 20, 2026
Remove 5 waive entries on release/1.3 whose NVBugs are closed and whose
tests passed on the waived platform in pre-merge CI (L0_MergeRequest_PR
6481323, 6428057, 6762288 (triton variant), 6700265, 6075431.

Handled by other release/1.3 PRs: 6893730 (NVIDIA#20592), the 6762288 cutlass
variant and 6672360 (NVIDIA#20632), 6278337 (NVIDIA#20585).

Waives for the other closed bugs (6435097, 6695515, 6812288, 6432818,
6428092, 6432826) are kept: their tests did not run on the waived
platform in that CI run, so the unwaive is not verified yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
QiJune added a commit to QiJune/TensorRT-LLM that referenced this pull request Oct 20, 2026
Remove the remaining 6 waive entries on release/1.3 whose NVBugs are
closed. PRs NVIDIA#20585, NVIDIA#20590, NVIDIA#20592 and NVIDIA#20632 handle the other
closed-bug entries.

- 6428092, 6432826: pipeline-parallel TorchSampler IndexError, fixed
  by NVIDIA#16163, which is on release/1.3.
- 6435097, 6432818: closed as fixed by NVIDIA#16163, but their B200 / GB300
  waivers were added later from runs on those GPUs; only QA lists run
  these tests there.
- 6695515, 6812288: closed as not reproducible.

The 6428092 entry waives the whole `-m "gpu2"` selection of
test_llm_multi_gpu_pytorch.py, which also holds test_llm_rpc_tp2,
test_llm_rpc_streaming_tp2 and test_llm_return_logprobs_streaming_tp2.
main waives those three under 6859934 because it does not have the
fix in NVIDIA#19847 yet; release/1.3 has it.

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
QiJune added a commit to QiJune/TensorRT-LLM that referenced this pull request Oct 22, 2026
Remove 5 waive entries on release/1.3 whose NVBugs are closed and whose
tests passed on the waived platform in pre-merge CI (L0_MergeRequest_PR
6481323, 6428057, 6762288 (triton variant), 6700265, 6075431.

Handled by other release/1.3 PRs: 6893730 (NVIDIA#20592), the 6762288 cutlass
variant and 6672360 (NVIDIA#20632), 6278337 (NVIDIA#20585).

Waives for the other closed bugs (6435097, 6695515, 6812288, 6432818,
6428092, 6432826) are kept: their tests did not run on the waived
platform in that CI run, so the unwaive is not verified yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants