Skip to content

[https://nvbugs/7051902][infra] Waive 1 failed case(s) on DGX_H100 - #21730

Merged
tburt-nv merged 2 commits into
NVIDIA:mainfrom
trtllm-agent:auto-waive/ec658127c61a66d949e1a6f776eb7b52
Nov 6, 2026
Merged

tburt-nv merged 2 commits into
NVIDIA:mainfrom
trtllm-agent:auto-waive/ec658127c61a66d949e1a6f776eb7b52

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 1 failed case(s) on DGX_H100. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports disaggregated/test_disaggregated.py::test_disaggregated_context_first_bf16_ownership broken on main for DGX_H100: failure signature runtimeerror: # worker(s) exited during startup: /path (rc=#), onset 2026-11-05T04:51:24Z, latest failure 2026-11-05T15:11:01Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7051902 tracks the fix. Scope DGX_H100 is the stage prefix that ran the failing test.

  • Bug: https://nvbugs/7051902 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7051902 is fixed or closed and the test passes on DGX_H100 post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-05T16:05:49.045376+00:00

Tests in this break event:

  • disaggregated/test_disaggregated.py::test_disaggregated_context_first_bf16_ownership on DGX_H100

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

full:DGX_H100/disaggregated/test_disaggregated.py::test_disaggregated_context_first_bf16_ownership SKIP (https://nvbugs/7051902)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope DGX_H100 matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run b2679f67-9e1c-407e-a303-e337949777f9; snapshot SHA-256 bf238dc37a1a674c5920684005308a7cfad7e1619c5a98e0bcd30b6d049c4109; config SHA-256 09cf0ee078a35cefbb2080f3d321976a2a18bc5d6e6c6b3de9d637db1c92730b
  • disaggregated/test_disaggregated.py::test_disaggregated_context_first_bf16_ownership: incident 56dc2aaef48f7e991ea73fdf1f7bc58747d9c2f8150d33576171428b4d322727, observation 52a0254ac0d92e812907a61a60a09743caa9982e5ea7671f5e5c83a51a4d6960

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

tests/integration/test_lists/waives.txt adds three DGX_H100 skip entries associated with NVBug 7051902. The entries cover one disaggregated integration test and two unit tests. This is a test-selection change; it does not fix the underlying behavior. Remove the waivers after the bug is fixed and the tests pass.

QA Engineer Review

The waiver list adds these test IDs: test_disaggregated_context_first_bf16_ownership, test_context_first_bf16_owned_transfer, and test_context_first_bf16_lifecycle_gpu. All three entries use the full:DGX_H100 selector and reference NVBug 7051902. The change skips these tests; it does not provide test results or verify a fix. Coverage verdict: needs follow-up.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Adds three DGX_H100 waivers for disaggregated ownership, KV transfer, and transfer lifecycle tests. Confirm the waivers are removed after the bug is fixed and the tests pass.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
Comment thread tests/integration/test_lists/waives.txt Resolved
Signed-off-by: tburt-nv <195370667+tburt-nv@users.noreply.github.com>
@tburt-nv
tburt-nv marked this pull request as ready for review November 6, 2026 17:12
@tburt-nv

tburt-nv commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

/bot skip --comment "waive only"

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@tburt-nv
tburt-nv enabled auto-merge (squash) November 6, 2026 17:13
@coderabbitai

coderabbitai Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Three DGX_H100 context-first BF16 tests were added to the integration skip list. Each entry references NVBug 7051902. The existing allreduce-strategy accuracy test remains skipped.

Changes

Integration test waivers

Layer / File(s) Summary
Add DGX_H100 test waivers
tests/integration/test_lists/waives.txt
The skip list now includes the context-first BF16 disaggregated ownership, owned-transfer, and GPU transfer-lifecycle tests. Each entry references NVBug 7051902.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Other

Suggested reviewers: lori-ren

Merge Risk: Minimal · up to fe45d

Three DGX_H100 tests will be temporarily skipped; no actionable merge-blocking code defect was established.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title identifies the DGX_H100 waiver, but says one case was waived; the change summary reports three added tests.
Description check Passed The description includes the required sections and explains the waiver. It describes one test, but the change summary reports three added tests. Test Coverage does not identify completed validation.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 7a06809 and fe45d5e.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83120 [ skip ] triggered by Bot. Commit: fe45d5e Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83120 [ skip ] completed with state SUCCESS. Commit: fe45d5e
Skipping testing for commit fe45d5e

Link to invocation

@tburt-nv
tburt-nv merged commit 6f6fb34 into NVIDIA:main Nov 6, 2026
15 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants