diff --git a/jenkins/AGENTS.md b/jenkins/AGENTS.md index bef85d77bcd5..be819ebca5f5 100644 --- a/jenkins/AGENTS.md +++ b/jenkins/AGENTS.md @@ -19,6 +19,50 @@ limitations under the License. Scope: this directory and its descendants. Supplements the repository-root `AGENTS.md`. +## Stage maintenance + +- Edit `jenkins/config/maintenance_stages.txt`. Blank lines and `#` comments + are ignored. The first active line must use the exact prefix `Reason:` with + a non-empty explanation on one physical line. This reason applies to all + subsequent entries; each entry is one exact runtime stage name or `*` wildcard + pattern, matched case-sensitively against the whole name. For example: + + ```text + Reason: The affected H100 resource is temporarily unavailable. + H100_PCIe-PyTorch-1 + ``` + +- The following DGX B200 names and patterns are illustrative examples, not + entries to add by default. `DGX_B200-*` matches all DGX B200 stages, including + single-GPU, multi-GPU, and post-merge variants. `DGX_B200-*_GPUs-*` matches only + multi-GPU variants, + such as `DGX_B200-2_GPUs-PyTorch-1` and `DGX_B200-4_GPUs-PyTorch-Post-Merge-1`. + `DGX_B200-PyTorch-2` matches exactly that stage; append `*` only if runtime + suffixes such as `-cbts` should also match. Only `*` is a wildcard, not `?`, + character ranges, regular expressions, or exclusion rules. +- During severe resource scarcity, temporarily list selected stages to reduce + demand while retaining representative coverage. For example, exact entries for + `DGX_B200-PyTorch-2` through `DGX_B200-PyTorch-9` leave + `DGX_B200-PyTorch-1` eligible to run. List each name on its own line; do not + write a range. Ensure no other pattern matches stages you intend to retain; + `DGX_B200-*` would disable them too. Explain the capacity constraint in the + shared reason and remove these entries when capacity recovers. +- Matching stages show `Skip - Maintenance` and the reason, without launching + a test agent. The nested stage is red to show missing coverage, but the skip + alone does not fail the overall build. Previously successful stages may show + `Skip - Reused` instead. Explicit stage selection does not override maintenance. +- Preparation merges the PR's config additions/deletions into the target branch's + latest snapshot. After merge, older PRs with maintenance support pick it up + without rebasing; builds that already took their snapshot are unaffected. +- To restore execution, remove the relevant entries in a recovery PR; dates in + the reason do not expire them. Keep the file and header when no entries remain. + Before merging, run `/bot run --stage-list ""` + on the recovery PR and verify every affected stage actually executes. The PR's + deletions take effect for its own validation before merge. Regressions may have + entered while coverage was disabled: fix failures or add tracked test + waivers through the normal waiver process, then rerun and confirm the remaining + tests pass on the latest PR head before merging the maintenance removal. + ## Path-valued pytest options - For new or modified long pytest options that take a single file or directory diff --git a/jenkins/config/maintenance_stages.txt b/jenkins/config/maintenance_stages.txt index c5ade961488b..68a99fbcb4ba 100644 --- a/jenkins/config/maintenance_stages.txt +++ b/jenkins/config/maintenance_stages.txt @@ -27,3 +27,6 @@ # Reason: The H100 cluster is under maintenance. # H100_PCIe-* # DGX_H100-* + +Reason: H100_PCIe-PyTorch-Perf-1 was retired from main; skip the legacy stage in older PR revisions. +H100_PCIe-PyTorch-Perf-1