Skip to content

pixi: add root clean task for stale build artifacts - #2443

Open
rparolin wants to merge 2 commits into
NVIDIA:mainfrom
rparolin:pixi/root-clean-task
Open

pixi: add root clean task for stale build artifacts#2443
rparolin wants to merge 2 commits into
NVIDIA:mainfrom
rparolin:pixi/root-clean-task

Conversation

@rparolin

Copy link
Copy Markdown
Collaborator

What

Moving a checkout between the cu12 and cu13 pixi environments leaves compiled Cython extensions, generated .pyx/.cpp sources, and the cache_driver/cache_runtime/cache_nvrtc parser caches from the previous CUDA major behind. The next build then fails with errors that point at code which is perfectly fine, and the fix — knowing which of the many ignored paths to delete — is folklore.

Adds toolshed/clean.py plus two tasks:

pixi run clean-dry-run   # list what would be removed
pixi run clean           # remove it

Design

The artifact list comes from git status --ignored, not a hand-maintained glob list. Anything git reports as ignored is by definition generated, so tracked files are never at risk — notably the handful of *.cpp files that are checked in despite the blanket *.cpp ignore rule (param_packer.cpp, loader.cpp, *_impl.cpp). Keeping a parallel glob list in sync with .gitignore would be exactly the kind of drift this repo already has enough of.

Deletion is then restricted by an allowlist of repository-owned paths (CLEAN_TOP_LEVEL_DIRS / CLEAN_TOP_LEVEL_ARTIFACTS), so an unrecognized ignored directory at the top level is preserved rather than removed just for being ignored. .pixi and hand-made virtualenvs are protected at any depth: they are expensive or developer-local, and are never the cause of a stale-artifact build failure.

Rejected alternative

The first attempt was a one-liner: git clean -Xdf with :(exclude) pathspecs and no script at all. Dropped, because on git 2.43 the presence of an exclude pathspec changes matching for the positive pathspecs too — the command silently listed paths outside the directories it was given (ci/tools/__pycache__, .mypy_cache) while failing to actually exclude .pixi. Not something to build a destructive command on.

Verification

Dry run against a dirty worktree: 212 paths, 698 MiB, with .pixi and .venv correctly preserved. Unit-checked the path-classification rules across 14 cases (package artifacts, root artifacts, protected dirs at depth, unknown root dot-dirs).


PLC Local Security Evidence (Advisory)

Local advisory checks only. Authoritative release gates remain Pulse, SonarQube, Coverity, BlackDuck, nSpect, ScanSpect, OSRB, and Anchore — none of the rows below are release-gate results.

Check Status Tool Details
Secrets SKIP Pulse Secret Scanner Image pull denied — not authenticated to gitlab-master.nvidia.com:5005 (needs a PAT with read_registry)
SAST PASS Semgrep 1.157.0 0 high / 0 medium / 0 low over the changed files (bundled ruleset, severity filter ERROR)
License SKIP pip-licenses pixi.lock is a conda+PyPI lockfile; pip-licenses reads only installed Python distribution metadata and has no pixi.lock parser. BlackDuck + OSRB remain authoritative
Dependencies WARN osv-scanner / pip-audit Source in a dependency-bearing ecosystem changed (toolshed/clean.py) but no manifest or lockfile is in scope. The script imports only the Python standard library, so there is nothing to resolve; recorded rather than suppressed
Container N/A No container files in this change
Security Review PASS inline Reviewed the destructive path: no shell invocation, no user-controlled input, deletion targets come from git status and are filtered through a path allowlist before shutil.rmtree

Moving a checkout between the cu12 and cu13 pixi environments leaves
compiled Cython extensions, generated .pyx/.cpp sources, and the
cache_driver/cache_runtime/cache_nvrtc parser caches from the previous
CUDA major behind. The next build then fails with errors that point at
code which is fine, and the fix -- knowing which of the many ignored
paths to delete -- is folklore.

Add `toolshed/clean.py` plus two tasks:

  pixi run clean-dry-run   # list what would be removed
  pixi run clean           # remove it

The artifact list comes from `git status --ignored` rather than a
hand-maintained glob list, so tracked files are never at risk -- notably
the handful of *.cpp files that are checked in despite the blanket *.cpp
ignore rule (param_packer.cpp, loader.cpp, *_impl.cpp).

Deletion is then restricted by an allowlist of repository-owned paths, so
an unrecognized ignored directory is preserved rather than removed just
for being ignored. `.pixi` and hand-made virtualenvs are protected at any
depth: they are expensive or developer-local, and never the cause of a
stale-artifact build failure.

An earlier attempt used `git clean -Xdf` with `:(exclude)` pathspecs
instead of a script. That was dropped: on git 2.43 the presence of an
exclude pathspec changes matching for the positive pathspecs too, so the
command silently removed paths outside the directories it was given.

Verified with a dry run on a dirty worktree (212 paths, 698 MiB) and unit
checks of the path-classification rules.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rparolin rparolin added the CI/CD CI/CD infrastructure label Jul 29, 2026
@rparolin
rparolin marked this pull request as draft July 29, 2026 00:34
@copy-pr-bot

copy-pr-bot Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

The bandit workflow (ruff --select S) and pre-commit.ci both failed on
toolshed/clean.py: S607, starting a process with a partial executable
path. Annotate the two `git` argv lists the way the rest of the repo
already does -- see toolshed/check_spdx.py and
toolshed/check_precommit_installed.py, which carry the same noqa for the
same reason.

Verified with the exact command the workflow runs:
`ruff check --select S --ignore S101,S311,S404 .` -> all checks passed,
and `pre-commit run --all-files` is clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rparolin rparolin self-assigned this Jul 30, 2026
@rparolin
rparolin marked this pull request as ready for review July 30, 2026 20:36

Copy link
Copy Markdown
Collaborator

I think we should explore making the build artifacts configuration-aware as the primary fix, rather than making cleanup part of the normal cu12/cu13 workflow.

Specifically:

  • Key the Cython generated-source directory by build identity (at minimum CUDA major).
  • Key the remaining mutable build outputs similarly, or force build_ext when that identity changes, so a cu12 output cannot be considered fresh after a cu13 build.
  • Keep CUDA_PYTHON_COVERAGE separate: it deliberately uses build_dir="." to package generated sources.

This should not conflict with the merged-wheel workflow: the merger consumes completed wheels and copies each cuda/core payload into cu12/ cu13; it never consumes build/cython. It would let us retain editable development without relying on deleting state between target switches.

Could we prototype this approach, with a cu12 → cu13 → cu12 regression test, before adopting a destructive cleaner as the main solution? A narrow cleaner could still be useful recovery tooling, but it should not be the correctness boundary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants