[klueska]

Description

This PR adds GPU support to Ray Sandboxes by leveraging CDI (Container Device Interface) to inject GPU device nodes and their supported user-mode driver files into a gVisor sandbox at runtime.

This PR builds on #66470, which adds the overlayfs rootfs that GPU sandboxes boot from, and on #66469 and #66528 underneath it. Only the last five commits are new here.

The change is implemented as follows.

• Generic CDI framework: We added a vendor-agnostic CDI module that can be used to parse CDI specs and merge devices from them with a standard OCI image spec. The implementation of this module is kept free of any Ray imports so it stays a candidate for extraction into a standalone library at some point in the future. As part of this, theAcceleratorManager class gains two new optional methods, get_cdi_kind()/generate_cdi_spec(). These methods are implemented by an AcceleratorManager to both generate a spec with the full set of CDI devices associated with the accelerator, as well as advertise what kind of CDI devices are associated with it (e.g. nvidia.com/gpu).

• NVIDIA specific implementation : We extended the NvidiaGPUAcceleratorManager to be CDI aware and implement the get_cdi_kind()/generate_cdi_spec() methods mentioned above. At present, we generate CDI devices by shelling out to nvidia-ctk cdi generate, but the preferred long-term path is to build (or leverage) a canonical python library for CDI maintained upstream. Since gVisor only supports NVIDIA GPUs at the moment, the GVisorSandboxBackend gates GPU passthrough to CDI kinds it actually supports (i.e. nvidia.com/gpu), rejecting anything else rather than attempting unverified passthrough.

• num_gpus and gpu_ids: When creating a Sandbox actor, you can now pass a num_gpus field in addition to the existing cpu and memory fields (similar to what is possible when creating a non-sandboxed actor). When doing so, the sandbox auto-inherits the full Ray GPU assignment derived from num_gpus (with no way to limit it further when calling remote() later on, unlike for cpus and memory). When creating a sandbox via SandboxRuntime.create() (i.e. within a pre-existing actor), there is no num_gpus field supported. Instead, we allow passing a gpu_ids field as an explicit parameter to specify which of the GPUs that have been granted to the surrounding actor should be isolated inside the sandbox. This allows a single actor with multiple GPUs to manage several sandboxes, each pinned to a different GPU.

• Overlayfs rootfs for GPU sandboxes: NVIDIA's CDI spec relies on createContainer hooks that create library symlinks and update the ldcache in the sandbox's rootfs, and runsc creates mount points there for the driver files and device nodes. The default EROFS rootfs is read-only and shared by every sandbox of an image, so any sandbox with gpu_ids now runs on [core][sandbox] Boot sandboxes from an overlayfs rootfs if required #66470's overlayfs rootfs. These writes land in the sandbox's own upper layer, and the cached image stays untouched.

• 53 new unit tests and 9 real-GPU CI tests: We added test_cdi.py and test_cdi_lib.py under ray/experimental/sandbox/tests/, and extended the existing suites there and under ray/tests/accelerators/. The real-GPU tests run via two new Buildkite jobs (core: sandbox gpu tests, 1 GPU; core: sandbox multi gpu tests, 4 GPUs).

Validated end-to-end on an NVIDIA L4 GPU and a machine with 4 NVIDIA A100 GPUs.

We also tested on a machine with 4 NVIDIA GB200 GPUs (ARM64/Grace CPU). All of the NVML-based tests passed, but the CUDA tests failed (because gVisor does not yet fully support GB200). Once gVisor supports GB200 we will need to revisit things to ensure that multi-node NVLink is properly supported as well.

AI assistance

Developed with Claude Code (Sonnet 5), including driving the hardware validation above. I reviewed every changed line and take responsibility for the change end-to-end.

Related issues

No tracking issue. Sibling of other active sandbox work: image cache LRU eviction (#65748), per-sandbox netns (#65820), HTTP API (#65633), gRPC facade (#65839), multi-uid netns (#65823); none overlap with this change.

Duplicate check

Searched open/closed issues and PRs for sandbox gpu, CDI, gvisor, nvproxy, nvidia-ctk, and container device interface — nothing addresses GPU or CDI support for Ray Sandboxes.

Testing

• test_gvisor_gpu.py (real nvidia-ctk generation, device injection, --nvproxy boot, nvidia-smi inside the sandbox, and a real CUDA kernel launch via NVIDIA's vectorAdd sample) — passes on a machine with an NVIDIA L4 GPU and a machine with 4 A100 GPUs.

• test_gvisor_multi_gpu.py (all four tests, needing 4 GPUs) — only relies on NVML to test. Passes on both a machine with 4 A100 GPUs and a machine with 4 GB200 GPUs, each sandbox pinned to a distinct GPU.

• python/ray/experimental/sandbox/tests/ (full directory, including the above) — on the L4: 228 passed, 10 skipped. On the GB200 machine, as root: 283 passed, 3 failed, 1 skipped. As the non-root user ray: 269 passed, 3 failed, 15 skipped. The failures are the known CUDA failure (test_sandbox_gpu_cuda_vectoradd_runs_a_real_kernel, see above) and test_netns_egress_and_dns and test_netns_udp_flows_do_not_cross, which need public DNS that this machine's network doesn't provide. The skipped tests need a worker that can mount in the host's user namespace, which ray can't in this Docker test rig, or the CI sandbox job.

• pre-commit run — passed (SKIP=vale; its Go/cgo toolchain doesn't build in this environment).

Testing manually

Building the published ray image needs a wheel via ./build-wheel.sh, which needs Anyscale's internal cr.ray.io/Bazel-remote-cache access to build in reasonable time. To avoid long build times, we instead overlay this branch's changed .py files onto a stock published rayproject/ray:2.58.0 image, allowing us to test without a core rebuild since nothing outside pure Python changed. These tests require a machine with an NVIDIA GPU, Docker, and the nvidia-container-toolkit-base package installed.

Dockerfile: https://gist.github.com/klueska/9f1297a5d7f5a7b7cf9b31b3fb393feb

Save it locally, then build from a checkout of this branch (the repo itself is the build context):

cd ray
curl -fsSL -o Dockerfile.sandbox-gpu-test https://gist.githubusercontent.com/klueska/9f1297a5d7f5a7b7cf9b31b3fb393feb/raw
docker build -t ray-cdi-gpu-test:latest -f Dockerfile.sandbox-gpu-test .

Run it. Each test runs twice, once as root and once as the non-root user ray. --device nvidia.com/gpu=all for GPU access; --privileged is needed for gVisor in this specific nested-Docker test rig. --basetemp keeps pytest's temporary directories out of /tmp, which test_image_utils.py::test_seeding_stays_inside_the_tree requires:

# Pure-logic CDI tests + real-nvidia-ctk accelerator tests, no gVisor/root needed:
docker run --rm --device nvidia.com/gpu=all ray-cdi-gpu-test:latest \
  python3 -m pytest --basetemp=/home/ray/pytest-tmp -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/test_cdi.py \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/test_cdi_lib.py \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/tests/accelerators/test_nvidia_gpu.py

# The real end-to-end GPU sandbox test: exercises real nvidia-ctk generation, device injection,
# and gVisor --nvproxy passthrough. No host setup needed -- nvidia-persistenced can be running
# or not, both are handled.
# The two commands below run these tests as root and as the non-root user ray.
docker run --rm --privileged --user root --device nvidia.com/gpu=all ray-cdi-gpu-test:latest \
  python3 -m pytest --basetemp=/home/ray/pytest-tmp -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/test_gvisor_gpu.py
docker run --rm --privileged --user ray --device nvidia.com/gpu=all ray-cdi-gpu-test:latest \
  python3 -m pytest --basetemp=/home/ray/pytest-tmp -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/test_gvisor_gpu.py

# The four 4-GPU tests: three check that concurrent sandboxes each get a distinct GPU (separate
# Sandbox actors, SandboxRuntime.create, and one actor pinning each of its sandboxes via
# gpu_ids=...), and one checks that a single sandbox given 2 GPUs sees both.
# Needs a machine with 4 GPUs; `--device nvidia.com/gpu=all` must expose at least 4.
# The two commands below run these tests as root and as the non-root user ray.
docker run --rm --privileged --user root --device nvidia.com/gpu=all ray-cdi-gpu-test:latest \
  python3 -m pytest --basetemp=/home/ray/pytest-tmp -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/test_gvisor_multi_gpu.py
docker run --rm --privileged --user ray --device nvidia.com/gpu=all ray-cdi-gpu-test:latest \
  python3 -m pytest --basetemp=/home/ray/pytest-tmp -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/test_gvisor_multi_gpu.py

# Full sandbox suite (mocked-GPU config/scheduling tests + real-gVisor CPU and GPU tests):
# The two commands below run these tests as root and as the non-root user ray.
docker run --rm --privileged --user root --device nvidia.com/gpu=all ray-cdi-gpu-test:latest \
  python3 -m pytest --basetemp=/home/ray/pytest-tmp -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/
docker run --rm --privileged --user ray --device nvidia.com/gpu=all ray-cdi-gpu-test:latest \
  python3 -m pytest --basetemp=/home/ray/pytest-tmp -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/

[gemini-code-assist]

Code Review

This pull request introduces GPU support for Ray Sandboxes using the Container Device Interface (CDI) to safely expose GPUs inside isolated gVisor containers. It adds vendor-agnostic CDI spec parsing and OCI merging utilities, integrates CDI spec generation into the NVIDIA GPU accelerator manager, and updates the sandbox configuration and gVisor backend to validate and pass through scheduled GPUs. The review feedback highlights several robust improvements, including resolving inconsistent default type handling and potential None values in CDI parsing, failing fast in the gVisor backend to prevent unnecessary image pulls, elevating the log level for nvidia-ctk failures to aid troubleshooting, and replacing os.mknod with safer socket and FIFO creation methods to ensure compatibility in unprivileged environments.

[cursor]

Stale Bugbot comment from a previous run.

[dstrodtman]

Docs-side review from the Anyscale docs team (Douglas Strodtman). Claude Code assisted; I read every comment and stand behind each one.

Scope: style, grammar, and docs conventions on doc/source/ray-core/sandboxes.md only, against the Ray documentation style guide. I'm not reviewing the GPU/CDI code or the CI changes; those belong to the Ray core maintainers, and the bot threads already cover them. Nothing here is a merge blocker from the docs side, so this is a COMMENT, not a change request.

The new GPU section is clear and well-organized, and the cross-links between Requirements and the GPU section read well. The suggestions below are all one thing: the style guide asks you to avoid em dashes, semicolons, and parentheticals in prose and restructure into separate sentences instead. The new prose introduced a few of each (Vale independently flags the semicolons and parens; it has no em-dash rule, which is why they need a human eye). Each suggestion preserves your technical claims exactly and only reshapes the punctuation, plus a couple of small word-choice items the guide calls out (actually as filler, since → because for causation).

Take or leave any of them; they're all optional polish. Once you're happy with the prose, the docs side has no objection.

[dstrodtman]

The guide asks to avoid semicolons and parentheticals in prose and split into separate sentences. This also lifts the trailing no other configuration required out of the comma-splice into its own sentence.

Suggested change

	To give sandboxes GPU access (see [GPU access](#gpu-access) below), additionally install `nvidia-container-toolkit-base` (which provides `nvidia-ctk`) on GPU worker nodes; Ray uses it to generate a [CDI](https://github.com/cncf-tags/container-device-interface) spec each time a sandbox actor requests a GPU, no other configuration required.
	To give sandboxes GPU access, also install `nvidia-container-toolkit-base` on GPU worker nodes. This package provides `nvidia-ctk`, which Ray uses to generate a [CDI](https://github.com/cncf-tags/container-device-interface) spec each time a sandbox actor requests a GPU. No other configuration is required. See [GPU access](#gpu-access).

[klueska]

Updated as requested.

[dstrodtman]

Splits the semicolon into two sentences. I also matched the parallel sentence under Schedule a Sandbox actor with custom resources, which opens with Because Sandbox is a standard Ray actor, and swapped like for a plain comparison since the guide reserves like for genuine comparisons.

Suggested change

	Since `Sandbox` is a standard Ray actor, request GPUs with `num_gpus` like any other actor; the sandbox automatically exposes exactly those GPUs inside the isolated container, the same way it inherits `num_cpus` and `memory`. This requires `nvidia-container-toolkit-base` on the node (see [Requirements](#requirements)) and a gVisor build with `--nvproxy` support.
	Because `Sandbox` is a standard Ray actor, request GPUs with `num_gpus` the same way you would for any other actor. The sandbox automatically exposes exactly those GPUs inside the isolated container, the same way it inherits `num_cpus` and `memory`. This requires `nvidia-container-toolkit-base` on the node (see [Requirements](#requirements)) and a gVisor build with `--nvproxy` support.

[klueska]

Reworded beyond what was suggested. PTAL.

[dstrodtman]

This line has two em dashes and a parenthetical, which the guide asks you to restructure into separate sentences. I also dropped actually (the guide lists it as filler) and changed since to because for the causal clause. Same technical content, just reshaped.

Suggested change

GPU sandbox creation requires `ray.get_gpu_ids()` to return at least one GPU for the calling actor or task — Ray populates it when `num_gpus` is requested, which is how the sandbox knows which GPUs are actually its to use. If it's empty (for example, a custom actor that never requested `num_gpus`, or driver code running outside any Ray task or actor), creating a GPU sandbox fails with a clear error rather than guessing. There's no way to opt in without going through Ray's `num_gpus` scheduling — setting `CUDA_VISIBLE_DEVICES` yourself has no effect here, since this checks Ray's own resource assignment, not that environment variable.

GPU sandbox creation requires `ray.get_gpu_ids()` to return at least one GPU for the calling actor or task. Ray populates it when `num_gpus` is requested, which is how the sandbox knows which GPUs are its to use. If it's empty, creating a GPU sandbox fails with a clear error rather than guessing. This happens, for example, for a custom actor that never requested `num_gpus`, or for driver code running outside any Ray task or actor. There's no way to opt in without going through Ray's `num_gpus` scheduling. Setting `CUDA_VISIBLE_DEVICES` yourself has no effect here, because this checks Ray's own resource assignment, not that environment variable.

[klueska]

This paragraph was removed in the last edit.

[dstrodtman]

Em dash → sentence break, per the guide.

Suggested change

	To give several sandboxes managed by one actor each their own GPU, use `SandboxRuntime` directly instead of `Sandbox` — a `Sandbox` actor is always exactly one actor to one sandbox, so it can't do this. Schedule the actor with `num_gpus=N`, then pass an explicit `gpu_ids=[id]` to each `SandboxRuntime.create()` call, one call per GPU the actor was assigned:
	To give several sandboxes managed by one actor each their own GPU, use `SandboxRuntime` directly instead of `Sandbox`. A `Sandbox` actor is always exactly one actor to one sandbox, so it can't do this. Schedule the actor with `num_gpus=N`, then pass an explicit `gpu_ids=[id]` to each `SandboxRuntime.create()` call, one call per GPU the actor was assigned:

[klueska]

Reworded given the broader changes mentioned above. PTAL.

[xyuzh]

Went through the whole thing, including the earlier bot threads. The big ones from those rounds (on-disk spec reuse, wildcard cgroup rule, gpu_ids kwarg collision, writing placeholders into the shared rootfs) all look resolved in the current revision, and the env replace-instead-of-append choice is right given how runsc picks the second-to-last NVIDIA_VISIBLE_DEVICES when the last is void. CI wiring also checks out (coregpubuild shares the Dockerfile, so nvidia-ctk is in the GPU test image).

A handful of things inline. The two I'd really like addressed before merge are the negative caching + 5s timeout combo, and documenting the toolkit / gVisor driver version floors. The rest are questions and nits.

[xyuzh]

Caching None here worries me. Combined with the 5s timeout in nvidia_gpu.generate_cdi_spec, one slow nvidia-ctk run (first NVML init without nvidia-persistenced on a multi-GPU box can easily take several seconds) permanently disables GPU sandboxes for that process. For a Sandbox actor that's a one-off, but for a SandboxRuntime pool actor it's stuck until someone restarts it, and the only signal is a warning log from minutes earlier.

I'd just not cache failures. A failed generation is already the rare path, and retrying it on the next create is cheap relative to the alternative.

[klueska]

Hmm, I'm not even convinced we want to cache anything here at all anymore. This was inspired by the canonical go-based implementation of CDI which never actually does generation at all -- it only reads specs from disk and caches them.

In our case it should be up to the caller to decide if they want the results cached or not, not this library.

[klueska]

I drop the caching of None and moved all caching out of the lib and onto the call-site instead. In the common case a successful CI generation will only happen at at node startup, but if it fails it has another chance to regenerate the next time a spec is requested (i.e. at sandbox creation).

[xyuzh]

5s feels tight. nvidia-ctk cdi generate does an NVML init plus ldcache/library discovery, and on a node where persistence mode is off each GPU has to come up first. I've seen that alone go past 5s on 8-GPU hosts. Something like 30–60s costs nothing on the happy path and avoids a spurious hard failure on a cold node.

[klueska]

I originally had it at 60s but that felt really long. That said, even on the unhappy path (i.e. the one where the tool is not installed at all), we still get a fast failure. It's only if the tool actually starts to run and is taking a really long time for some reason that you ever hit this timeout. So I agree, longer is better and will make the change.

[klueska]

Update back to 60s. PTAL.

[xyuzh]

Two things here:

1. --disable-hook only exists from nvidia-container-toolkit v1.18.0 (multiple --device-name-strategy from v1.15.0). On an older toolkit this just exits non-zero and the user sees a generic "couldn't generate CDI spec". Worth stating the version floor in the docs and maybe hinting at it in the warning.

2. On the comment above about update-ldcache being redundant because the mounted libs "already sit on ld.so's default search path": that's true when host and image are both Debian-style multiarch. A RHEL or Alpine image gets the libs bind-mounted at the host's /usr/lib/x86_64-linux-gnu/..., which isn't on its default search path, so libcuda.so.1 won't resolve. gVisor's GPU docs also say update-ldcache runs fine as a createContainer hook under nvproxy. What did you actually hit when it was enabled? If it really can't run, this should probably be a documented limitation ("image must share the host's library layout").

[klueska]

Regarding (1) -- happy to update the docs with a minimum version of v1.18.0. That version is just around a year old, so shouldn't provide any practical limitations.

Regarding (2) -- you're absolutely right about the libs only sitting in the right path when the host layout matches the container images layout (they don't need to be Debian based, but they do need to match). I didn't think about this when putting together the comment or deciding to omit the update-ldcache hook.

Although the gVisor docs claim they work with update-ldcache, this is only true prior to v1.18.0. In v1.18.0 we changed the way the update-ldcache hook works to ensure that it always runs inside its own isolated mount namespace (rather than directly on the host). As part of this it also isolates the proc filesystem, which gVisor doesn't like and blows up on.

I see 3 options:

1. Live with the limitation that the host (i.e. ray node) fs needs to be of the same "flavor" as the container images being started (I like this option the least)
2. Don't recommend a minimum version of v1.18.0 and instead recommend a version <= v1.18.0. This means that (in general) we may miss out on the many bug fixes that have come in since then (I like this option the second least).
3. Write our own update-ldcache hook that we ship and add to the CDI spec ourselves (and in parallel file a bug with the nvidia-container-toolkit to fix things on their end).

I am leaning towards 3, but happy to hear your thoughts.

[klueska]

I've opened a PR agains the nvidia-container-toolkit to address this at its source.

I will recommend the change be backported to 1.18 and 1.19 and 1.20 once merged:
NVIDIA/nvidia-container-toolkit#2059

[klueska]

While we wait for the above PR to get merged and backported. I put a temporary "hack" in place to make this work in our environment.

The key insight being that nvidia-ctk always sets NVIDIA_CTK_LIBCUDA_DIR in its CDI edits to the set of directories where the driver libs have been placed. With this available, we can prepend these dirs onto whatever a container's existing LD_LIBRARY_PATH is before a sandbox is brought online.

Of course someone could always update / overwrite LD_LIBRARY_PATH after the container has started to bypass this and cause it to break.

That said, it should cover 95% of the use cases out there and is only a temporary measure until NVIDIA/nvidia-container-toolkit#2059 is merged and backported, so it seems to me like a pragmatic solution for now.

[klueska]

Maybe worth adding to the docs as an extra caveat?

[klueska]

NVIDIA/nvidia-container-toolkit#2059 has been merged in the nvidia-container-toolkit. Working with the team to get a backport to v1.18.x, v1.19.x and v1.20.x.

[xyuzh]

"No other configuration is required" is a bit optimistic. Things I think Requirements should mention:

• nvidia-container-toolkit-base >= 1.18 (for --disable-hook).
• runsc only boots with --nvproxy on NVIDIA driver versions it knows about, and there's no override for unknown ones. Pointing people at runsc nvproxy list-supported-drivers will save a lot of confused bug reports.
• MIG isn't supported (gVisor has no nvidia-caps, and Ray's MIG id format wouldn't match the CDI device names anyway).

[klueska]

This is a fair point. I'll reaudit what belongs here once I've made the rest of the changes being discussed here.

[klueska]

I did an audit and updated the docs as suggested. PTAL.

[xyuzh]

Every real-hardware test here and in the multi-GPU file gives each sandbox exactly one GPU. I'd add one num_gpus=2 case asserting nvidia-smi --query-gpu=uuid returns exactly two lines. That's the only path that exercises multiple frontend device nodes in a single spec, and it's the one most likely to interact badly with the NVIDIA_VISIBLE_DEVICES handling.

[klueska]

ACK

[klueska]

Added. PTAL.

[xyuzh]

Minor design point: this re-derives the runsc flags from the process-global spec cache on every runsc call, including exec and delete. If a later process (or the same one after a failed regeneration) can't get a spec, those calls silently run without --nvproxy. It probably works today since runsc talks to the running sentry, but recording the kind on the config at create time would make later calls independent of cache state, and drop the extra get_spec round trips.

[klueska]

Makes sense -- I don't think I want to make it a member of the generic SandboxConfig class (since in theory that is not gVisor specific), but I can find somewhere appropriate to cache it in the GVisorSandboxBackend class during create().

[klueska]

I reworked this so that we only "calculate" whether --nvproxy is needed once at sandbox creation. As part of this, we also only pass these args to the runsc run command now, as the rest of the operations (i.e. exec/kill/delete/state) don't actually need them. PTAL.

[xyuzh]

Two small ones:

• This runs for every core build variant, including CPU ones that never use nvidia-ctk. Gating on BUILD_VARIANT == gpubuild would keep the extra external repo out of the CPU images.
• curl -s -L without -f means a 404 body gets written to sources.list.d and you only find out at the next apt-get update with a confusing error. Same -fsSL as the gpgkey fetch above.

[klueska]

ACK

[klueska]

Updated as suggested.

[xyuzh]

Swallowing the exception and then reporting "returned no GPUs" is misleading when the real problem is that Ray isn't initialized at all. I'd either let the exception propagate or include it in the message.

Bigger-picture (not blocking): doing a ray.get_gpu_ids() call inside a dataclass __post_init__ ties config construction to Ray runtime state. Feels like it belongs in SandboxRuntime.create, with SandboxConfig staying a plain value object.

[klueska]

I'll take a closer look at this and see what makes the most sense. I'll comment back here once I have a better feel for the "right" thing to do.

[klueska]

I just dropped the try so that the error propagates up as suggested.

[xyuzh]

These pull a CUDA base image plus the cuda-sample image and boot gVisor with nvproxy. medium is 300s; I'd go large for both GPU targets so a slow registry doesn't show up as a flaky test.

[klueska]

ACK

[klueska]

Updated as requested.

[andrewsykim]

I don't think we want this in the main core Dockerfile?

[andrewsykim]

@elliot-barn might have some ideas on where to put this. if not, I suggest we just document what users need to include in their Dockerfile

[klueska]

I've added a commit that removes this from the core dockerfile and installs it at test time similar to how runsc gets installed for the standard sandbox tests. PTAL.

[andrewsykim]

this entire comment block is very verbose and difficult to understand. Can we simplify and improve the readability?

[klueska]

ACK. I will do an audit of it.

[klueska]

I have update the text significantly. PTAL.

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[sunnylovestiramisu]

The docs say MIG isn't supported, but nothing rejects it here. If a worker's CUDA_VISIBLE_DEVICES lists MIG devices, ray.get_gpu_ids() returns MIG- ids. nvidia-ctk cdi generate --device-name-strategy=uuid emits devices with those names too, so select_devices finds them and the sandbox only fails later inside gVisor, with an unclear error. Could we reject them up front with a clear error?

[klueska]

The latest changes reject MIG devices early.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[xyuzh]

Moving onto #66470's overlay got rid of the shared-rootfs workarounds, nice. Three things on the current head, inline.

[xyuzh]

Since #67252 (Oct 21), master's python/ rule no longer emits llm, so copying its tags here now adds llm for these three files. That's why premerge is red on this PR: the llm builds run and fail on an unrelated vLLM patch that no longer applies. I'd drop llm here and update the comment above to match.

[klueska]

Done. I dropped llm from that rule and from nvidia_gpu.py's expected tags in test.rules.test.txt, and updated the comment to match.

[xyuzh]

GPU sandboxes now rely on runsc running the CDI createContainer hooks (create-symlinks, update-ldcache). runsc only does that since google/gvisor@ae18a84dfad0, first released in release-20260520.0. Older builds log "CreateContainer hook skipped" and boot anyway. They still have --nvproxy and nvproxy list-supported-drivers, so every check here passes, and the failure only shows up later as nvidia-smi or CUDA not finding the driver libraries. Could this check runsc --version, and sandboxes.md:334 state the minimum?

[klueska]

Added a check. Before giving a sandbox GPUs, the backend now reads runsc --version and fails sandbox creation if it reports a release older than release-20260520.0.

A runsc built from source reports a commit rather than a release, so it can't be checked that way and gets a warning instead, naming the minimum release.

sandboxes.md now states the minimum as well.

[xyuzh]

These tests check that the four sandboxes see four different GPUs, but not that each sees the GPU Ray assigned to it, so a mapping that hands sandbox A sandbox B's GPU still passes. Comparing each sandbox's UUID with the UUID of the GPU in its actor's ray.get_gpu_ids() would catch that. Same for the gpu_ids pool test below.

[klueska]

Done. Each test now checks that the GPU a sandbox sees is the one Ray assigned it, comparing its UUID with the UUID of the GPU in the actor's gpu_ids. The pool test does the same with each sandbox's pinned gpu_id.

[edoakes]

Accelerator change lgtm

[elliot-barn]

nit can you change this to gpu-large-flex. gpu-large-flex supports g6.12xlarge,g5.12xlarge,g6.24xlarge,g5.24xlarge,g6e.12xlarge,g6e.24xlarge vs just g6.12xlarge

[pcmoritz]

Thanks, I updated it!

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 8b054cf. Configure here.}