Skip to content

[None][infra] Skip retired H100 perf stage through maintenance config - #21991

Merged
chzblych merged 5 commits into
NVIDIA:mainfrom
chzblych:codex/maintenance-retired-h100-perf-20261110
Nov 10, 2026
Merged

chzblych merged 5 commits into
NVIDIA:mainfrom
chzblych:codex/maintenance-retired-h100-perf-20261110

Conversation

@chzblych

@chzblych chzblych commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Problem

#21524 retired H100_PCIe-PyTorch-Perf-1 from main, but older PR revisions can still schedule the legacy benchmark and fail before producing performance results.

Solution

Add the exact stage name and retirement reason to jenkins/config/maintenance_stages.txt. The existing target-TOT merge makes the entry available to older PR revisions whose pipeline already supports maintenance. Matching stages skip agent allocation and test execution through the existing maintenance path, which marks the stage as failed for visibility without failing the overall build.

Document the configuration format, DGX B200 wildcard examples, selective pausing during severe resource scarcity, skip behavior, snapshot scope, and recovery validation with failure fixes or tracked test waivers before merging the recovery PR in jenkins/AGENTS.md.

Related change

#21524

Test Coverage

Validation

  • Passed production-parser checks for the active configuration and documentation example.
  • Passed configuration parsing and PR-addition merge checks using the production mergeWaiveList.py.
  • Passed the no-config-diff case: an older PR inherits the target-TOT maintenance entry.
  • git diff --check and DCO sign-off verified.
  • Jenkins execution has not been run. Pre-commit was unavailable in the local environment.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why.

  • PR follows the TRT-LLM coding guidelines to the best of our knowledge.

  • Configuration checks cover this change; no new code paths.

  • No API, dependency, ownership, or architecture changes.

  • Documentation updated as needed.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Dev Engineer Review

The maintenance configuration adds an exact, case-sensitive skip entry for H100_PCIe-PyTorch-Perf-1 and gives the reason. The documentation describes matching, merge behavior, skip effects, and recovery validation. The diff changes no runtime code or APIs. No material code-level risks are evident.

QA Engineer Review

No test changes.

Per-File QA Perspective

  • jenkins/AGENTS.md: Documents maintenance matching and Jenkins skip behavior, plus steps to verify recovery. No test code or runtime behavior changes.
  • jenkins/config/maintenance_stages.txt: Adds a maintenance skip for the retired stage. Verify that the exact stage is skipped and that the skip does not fail the overall build.

Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
@chzblych
chzblych marked this pull request as ready for review November 10, 2026 10:52
@chzblych
chzblych requested a review from a team as a code owner November 10, 2026 10:52
@chzblych
chzblych requested review from mlefeb01 and mzweilz November 10, 2026 10:52
@chzblych

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "Safe merge"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83781 [ skip ] triggered by Bot. Commit: f8f4439 Link to invocation

@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Walkthrough

The Jenkins documentation now describes stage maintenance configuration, matching, operational effects, and recovery. The maintenance configuration adds a reason and a skip entry for H100_PCIe-PyTorch-Perf-1.

Changes

Stage maintenance

Layer / File(s) Summary
Document and configure stage maintenance
jenkins/AGENTS.md, jenkins/config/maintenance_stages.txt
The documentation describes the required Reason: header, stage matching, skip behavior, configuration merging, and recovery checks. The configuration adds a retirement reason and skips H100_PCIe-PyTorch-Perf-1.

Estimated code review effort: 1 (Trivial) | ~4 minutes

Change: Bug fix

Merge Risk: Minimal · up to f8f44

This change marks a retired Jenkins stage to be skipped and documents the maintenance rules. No concrete runtime risk was found, and it is ready to merge after normal checks.

Architecture risk: Low · up to f8f44

The change affects 1 system.

Changed systems: jenkins

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — jenkins (service) was modified; 2 changed files map to changed impact.

Before / after behavior

  • observed — Modified behavior in jenkins/AGENTS.md: Adds Stage maintenance documentation covering the required non-empty, single-line Reason: header; case-sensitive whole-name matching for exact stage names and * patterns; illustrative DGX B200 patterns and limits; resource-scarcity selection guidance; maintenance skip and build behavior; snapshot merging; and recovery steps, including explicit stage execution checks before merging entry removals.
  • observed — Modified behavior in jenkins/config/maintenance_stages.txt: Adds a maintenance reason identifying the retired stage and lists H100_PCIe-PyTorch-Perf-1 as the stage to skip.
Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title follows the required format and clearly identifies the infrastructure change: skipping the retired H100 performance stage through maintenance configuration.
Description check Passed The description explains the problem and solution, lists validation and test coverage, notes that Jenkins execution was not run, and completes the PR checklist.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional comments: 1

jenkins/AGENTS.md-45-46: 1

45-46: Major · Stability & Availability · Quick win

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Include CBTS variants in the scarcity example.

When CBTS narrows a stage, cbtsResizeSplits adds -cbts before maintenance matching. The exact entries for stages 2–9 therefore do not pause their CBTS variants, which can continue to consume scarce capacity. Add the corresponding exact -cbts names to the example while leaving stage 1 eligible. (raw.githubusercontent.com)

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 0bcbcfe and f8f4439.

Files selected for processing: 2

  • jenkins/AGENTS.md
  • jenkins/config/maintenance_stages.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83781 [ skip ] completed with state SUCCESS. Commit: f8f4439
Skipping testing for commit f8f4439

Link to invocation

@chzblych
chzblych merged commit 6f92fc5 into NVIDIA:main Nov 10, 2026
16 checks passed
@chzblych
chzblych deleted the codex/maintenance-retired-h100-perf-20261110 branch November 10, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants