Skip to content

[https://nvbugs/7060043][infra] Waive 1 failed case(s) on A10 - #21838

Merged
chzblych merged 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/2d0f1da65dc61f728c889d592db902a9
Nov 8, 2026
Merged

chzblych merged 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/2d0f1da65dc61f728c889d592db902a9

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 1 failed case(s) on A10. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/modeling/test_modeling_apertus.py::TestApertus::test_apertus_allclose_to_hf[backend:trtllm-use_cuda_graph:True] broken on main for A10: failure signature attributeerror: tuple object has no attribute batch_size, onset 2026-11-07T00:06:28Z, latest failure 2026-11-07T01:06:18Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7060043 tracks the fix. Scope A10 is the stage prefix that ran the failing test.

  • Bug: https://nvbugs/7060043 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7060043 is fixed or closed and the test passes on A10 post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-07T01:36:04.892146+00:00

Tests in this break event:

  • unittest/_torch/modeling/test_modeling_apertus.py::TestApertus::test_apertus_allclose_to_hf[backend:trtllm-use_cuda_graph:True] on A10

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

full:A10/unittest/_torch/modeling/test_modeling_apertus.py::TestApertus::test_apertus_allclose_to_hf[backend:trtllm-use_cuda_graph:True] SKIP (https://nvbugs/7060043)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope A10 matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run 8a83e72e-bfe4-488d-9ae8-95e1349667d9; snapshot SHA-256 7f4ea50cb134560abef028c66eef60707654332a7bba7a5d9a277050915a9ae4; config SHA-256 80248ef10d45ba9db91b7f0a8258df726f95897e1dc5b17a275cf58bc305d1f5
  • unittest/_torch/modeling/test_modeling_apertus.py::TestApertus::test_apertus_allclose_to_hf[backend:trtllm-use_cuda_graph:True]: incident 7bce537e1d58910b886cf1c1e494cbed9edac3cf03e7ade6cb5b8a636a51c770, observation 25c49228ab43a10e4c8cda98772ea61080fc65c72e19d750c17f648b7097d2ca

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

The change adds one A10-only waiver for the Apertus TRT-LLM test with CUDA graphs enabled. It links the waiver to NVBUG 7060043. The change does not establish that the underlying failure is fixed; restore the test after the fix and a passing A10 run.

QA Engineer Review

Modified tests/integration/test_lists/waives.txt. The entry skips TestApertus::test_apertus_allclose_to_hf[backend:trtllm-use_cuda_graph:True] on A10 only. The waiver narrows the impact to that test selector and hardware. Coverage verdict: needs follow-up until the bug is fixed and the test passes.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Adds an A10 waiver for one Apertus selector. No test functions changed; the waived test is excluded from the applicable run while the entry remains.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@chzblych
chzblych marked this pull request as ready for review November 8, 2026 03:38
@chzblych

chzblych commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator

/bot skip --comment "Test waive"

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@chzblych
chzblych enabled auto-merge (squash) November 8, 2026 03:40
@coderabbitai

coderabbitai Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The integration test waiver list adds a full:A10 skip rule for an Apertus test using the TRT-LLM backend with CUDA graphs enabled. The rule references NVBUG 7060043.

Changes

Apertus test waiver

Layer / File(s) Summary
Add the A10 skip rule
tests/integration/test_lists/waives.txt
Adds a full:A10 skip rule for TestApertus::test_apertus_allclose_to_hf with the TRT-LLM backend and CUDA graphs enabled. The rule references NVBUG 7060043.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Other

Possibly related PRs

Suggested reviewers: lori-ren, allisonlim-nv

Merge Risk: Minimal · up to e4435

The change temporarily skips the reported failing A10 test. Restore coverage when the tracked fix is verified.

Architecture risk: Low · up to e4435

The change affects 1 system.

Changed systems: tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in tests/integration/test_lists/waives.txt: Added a full:A10 skip rule for TestApertus::test_apertus_allclose_to_hf with the TRT-LLM backend and CUDA graphs enabled, referencing NVBUG 7060043.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title uses the NVBugs ID and type format from the template and names a waiver of one failed A10 case. It matches the single-line skip entry in waives.txt. It does not name the Apertus test or the …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the required structure only partly. It includes a Description section with the waiver rationale, bug link, scope, and restoration condition, and it lists the affected PRs and t…
Full details: Title check

Explanation

The title uses the NVBugs ID and type format from the template and names a waiver of one failed A10 case. It matches the single-line skip entry in waives.txt. It does not name the Apertus test or the reason for the waiver, but it does describe the main change.

Full details: Description check

Explanation

The description follows the required structure only partly. It includes a Description section with the waiver rationale, bug link, scope, and restoration condition, and it lists the affected PRs and the waiver line. It leaves the Test Coverage section as template-style text with no test list, and the PR Checklist has only the generic final checkbox ticked. The template's title format is not shown in the description. Most required information is present, so the check passes.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between a3febfb and e4435ef.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83259 [ skip ] triggered by Bot. Commit: e4435ef Link to invocation

@sunnyqgg sunnyqgg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the waive against the test-db lists, the waive parser and the failing A10 run. The line itself is correct: it matches the exact failing node id, A10 is the only L0 platform that runs this test (l0_a10.yml:35), and the other 12 Apertus cases pass in that run. Counts: 0 blocker, 0 possibly blocking, 1 required, 0 nice-to-have. The required item is inline: the root cause is a two-line bug in the test, and that fix should land instead of this waive or right after it.

Comment thread tests/integration/test_lists/waives.txt Resolved
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83259 [ skip ] completed with state SUCCESS. Commit: e4435ef
Skipping testing for commit e4435ef

Link to invocation

@chzblych
chzblych merged commit 73874e1 into NVIDIA:main Nov 8, 2026
26 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants