Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions jenkins/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,50 @@ limitations under the License.
Scope: this directory and its descendants. Supplements the repository-root
`AGENTS.md`.

## Stage maintenance

- Edit `jenkins/config/maintenance_stages.txt`. Blank lines and `#` comments
are ignored. The first active line must use the exact prefix `Reason:` with
a non-empty explanation on one physical line. This reason applies to all
subsequent entries; each entry is one exact runtime stage name or `*` wildcard
pattern, matched case-sensitively against the whole name. For example:

```text
Reason: The affected H100 resource is temporarily unavailable.
H100_PCIe-PyTorch-1
```

- The following DGX B200 names and patterns are illustrative examples, not
entries to add by default. `DGX_B200-*` matches all DGX B200 stages, including
single-GPU, multi-GPU, and post-merge variants. `DGX_B200-*_GPUs-*` matches only
multi-GPU variants,
such as `DGX_B200-2_GPUs-PyTorch-1` and `DGX_B200-4_GPUs-PyTorch-Post-Merge-1`.
`DGX_B200-PyTorch-2` matches exactly that stage; append `*` only if runtime
suffixes such as `-cbts` should also match. Only `*` is a wildcard, not `?`,
character ranges, regular expressions, or exclusion rules.
- During severe resource scarcity, temporarily list selected stages to reduce
demand while retaining representative coverage. For example, exact entries for
`DGX_B200-PyTorch-2` through `DGX_B200-PyTorch-9` leave
`DGX_B200-PyTorch-1` eligible to run. List each name on its own line; do not
write a range. Ensure no other pattern matches stages you intend to retain;
`DGX_B200-*` would disable them too. Explain the capacity constraint in the
shared reason and remove these entries when capacity recovers.
- Matching stages show `Skip - Maintenance` and the reason, without launching
a test agent. The nested stage is red to show missing coverage, but the skip
alone does not fail the overall build. Previously successful stages may show
`Skip - Reused` instead. Explicit stage selection does not override maintenance.
- Preparation merges the PR's config additions/deletions into the target branch's
latest snapshot. After merge, older PRs with maintenance support pick it up
without rebasing; builds that already took their snapshot are unaffected.
- To restore execution, remove the relevant entries in a recovery PR; dates in
the reason do not expire them. Keep the file and header when no entries remain.
Before merging, run `/bot run --stage-list "<affected-stage-names-or-patterns>"`
on the recovery PR and verify every affected stage actually executes. The PR's
deletions take effect for its own validation before merge. Regressions may have
entered while coverage was disabled: fix failures or add tracked test
waivers through the normal waiver process, then rerun and confirm the remaining
tests pass on the latest PR head before merging the maintenance removal.

## Path-valued pytest options

- For new or modified long pytest options that take a single file or directory
Expand Down
3 changes: 3 additions & 0 deletions jenkins/config/maintenance_stages.txt
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,6 @@
# Reason: The H100 cluster is under maintenance.
# H100_PCIe-*
# DGX_H100-*

Reason: H100_PCIe-PyTorch-Perf-1 was retired from main; skip the legacy stage in older PR revisions.
H100_PCIe-PyTorch-Perf-1
Loading