[moraxu]

Description

Cherry-pick of #19958 (merged to main as cc63e66daf) to release/1.3.

accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=2-overlap_scheduler={False,True}] hung in the QA run on H20 at 53b392e (last pass db7161b) and was waived as full:H20/... (nvbugs/6345827, #15510). The same two cases on H100 were waived the same week (#15504) under nvbugs/6313072, which tracks an unrelated OOM in the aggregated TestDeepSeekV3Lite::test_bfloat16 and has since been verified fixed and closed. Neither GPU has run the two disaggregated cases since; in L0 the test only runs on DGX_H200-8_GPUs-PyTorch-Post-Merge-1.

At the time the test used backend: DEFAULT, i.e. the C++ transceiver over UCX. The failing run's teardown log shows the generation ranks blocked in checkGenTransferStatus -> future.get() on transfers whose KV data the context side never sent (gen rank 3 still mid-recv on one request while ranks 0-2 had already received it), with all 16 in-flight requests (the harness's max_workers=16) stuck behind it until the pytest timeout. This is consistent with the CacheSender lost-wakeup race: removeResponse() checked mReadyResponses.empty() outside the lock and then cleared mAnyReady, which could overwrite a concurrent sendAsync() and leave the responder asleep with a ready response queued; once every in-flight request is waiting on generation, no further sendAsync() arrives to wake it. That race was fixed by #15737 (nvbugs/6342844), and since #17090 the test uses the Python transceiver (NIXL), so it no longer exercises that path at all.

This PR removes the four stale QA waives (H20 and H100).

Test Coverage

• Unwaived for QA on H20 and H100 (qa/llm_function_core.txt): TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=2-overlap_scheduler=False] and [mtp_nextn=2-overlap_scheduler=True].
• Local 8xH200 (DGX) with a wheel built from main 72688933d5:
  • test_auto_dtype[mtp_nextn=2-overlap_scheduler=False]: PASSED (MMLU 71.37, GSM8K 64.48)
  • test_auto_dtype[mtp_nextn=2-overlap_scheduler=True]: PASSED (MMLU 71.37, GSM8K 64.94)
  • Extra probes of the same workload (not part of this PR):
    • the configuration the test used when it was waived (C++ transceiver over UCX, KV cache manager V1) on today's main: PASSED (MMLU 71.30, GSM8K 65.01);
    • the current configuration with every rank's KV pool capped at 160K tokens (~13x smaller than on H20): PASSED (MMLU 71.27, GSM8K 64.10), ruling out a dependence on the smaller H20/H100 memory;
    • the C++/UCX configuration with the 160K cap was ~30% through and still progressing when the node allocation ended.
• CI history: DGX_H200-8_GPUs-PyTorch-Post-Merge-1 has passed both cases on every main post-merge run since 2026-07-10 and on every release/1.3 post-merge run that reached them (one release/1.3 build failed the whole stage in setup on pip install opencv-python-headless); QA B200/B300 pass all four test_auto_dtype variants; QA H20 passes the mtp_nextn=0 variants.
• Final confirmation is the next QA run on H20/H100 (re-run requested).

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Dev Engineer Review

tests/integration/test_lists/waives.txt removes four waivers for TestDeepSeekV3Lite::test_auto_dtype with mtp_nextn=2, covering both overlap-scheduler settings on H20 and H100. No source or API changes are reported. The PR states that these waivers are stale because the implicated race was fixed and the test now uses the Python transceiver; other waivers remain.

QA Engineer Review

Only a test-list file changed. The four mtp_nextn=2 cases are no longer skipped on H20 and H100. The cases appear in tests/integration/test_lists/qa/llm_function_core.txt and llm_function_rtx6k.txt; the H200 CI list includes both cases in tests/integration/test_lists/test-db/l0_dgx_h200.yml. The PR reports passing local H200 and CI runs, but H20 and H100 confirmation is pending. Coverage verdict: needs follow-up.

Per-File QA Perspective

• tests/integration/test_lists/waives.txt: Removes the four H20/H100 waiver entries for the two mtp_nextn=2 selector variants. QA should confirm both variants pass on H20 and H100.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: c0077ff8-4f27-4b07-9643-11726f4b49ed

📥 Commits

Reviewing files that changed from the base of the PR and between fac30be and 1640b00.

📒 Files selected for processing: 1

• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes: 1

• tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

The integration waive list removes four DeepSeekV3Lite test_auto_dtype cases with mtp_nextn=2, covering H100 and H20 configurations with both overlap-scheduler settings.

Changes

DeepSeekV3Lite waive list

Layer / File(s)	Summary
Remove DeepSeekV3Lite waive entries tests/integration/test_lists/waives.txt	Removes the H100 and H20 test_auto_dtype entries with mtp_nextn=2 for both overlap-scheduler settings. Other H100 entries in the range remain.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~2 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 1640b

The tests are no longer waived. The requested H20 and H100 QA rerun remains pending, but no failure is established.

Pre-merge checks: 5

Passed checks: 5 passed

Check name	Status	Explanation
Description check		The description clearly explains the stale QA waives, the suspected resolved race condition, the scope of the cherry-pick, and the validation performed. It includes relevant test coverage and complete…
Title check		The title clearly identifies the NVBugs ticket, test-related change, and the DeepSeekV3Lite cases unwaived on H20 and H100.
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[moraxu]

/bot run --disable-fail-fast --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #77925 [ run ] triggered by Bot. Commit: 1640b00 Link to invocation

[tensorrt-cicd]

PR_Github #77925 [ run ] completed with state FAILURE. Commit: 1640b00
/LLM/release-1.3/L0_MergeRequest_PR pipeline #229 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[moraxu]

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #79586 [ run ] triggered by Bot. Commit: 1640b00 Link to invocation

[tensorrt-cicd]

PR_Github #79586 [ run ] completed with state SUCCESS. Commit: 1640b00
/LLM/release-1.3/L0_MergeRequest_PR pipeline #460 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

• Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
• If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
  Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[moraxu]

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #79694 [ run ] triggered by Bot. Commit: 1640b00 Link to invocation

[tensorrt-cicd]

PR_Github #79694 [ run ] completed with state SUCCESS. Commit: 1640b00
/LLM/release-1.3/L0_MergeRequest_PR pipeline #476 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation