[dancingactor]

Background & Problem

In short, the issue is essentially the different checking logic between global scheduling (selects a node with "sufficient resources" from all nodes) and local allocation (Raylet allocates specific GPU instances on the selected node).

More specifically, for unit resources like GPU, the accurate resource usage representation needs to fully show the GPU topology. For example, [0.5, 0.5] represents a node with 2 GPUs, each instance having 0.5 availability. So when you want to run a task with 0.8 GPUs, there is no sufficient room to fit even though the aggregate available is larger than 0.8(In Ray, a task request 0.8 GPU means requiring 80% of a single GPU, it cannot use 40% on GPU 0 + 40% on GPU 1). However, at the scheduling level, Ray currently uses scalars to represent resource usage. It falsely assumes the node has sufficient capacity because 1.0 > 0.8, leading to a false positive

The goal of this PR is to make global scheduling also use per-instance vectors to represent unit-instance resources, so that both scheduling and allocation use the same and correct representation

Here is a minimal reproduction script to demonstrate (you could also see #52133 and #54729 for more details):

import ray
from ray.cluster_utils import Cluster

cluster = Cluster()
head = cluster.add_node(num_cpus=4, num_gpus=2)
ray.init(address=cluster.address)
worker = cluster.add_node(num_cpus=4, num_gpus=2)
cluster.wait_for_nodes()
# Two nodes, each with 2 GPUs. Cluster total: 4 GPUs.

@ray.remote(num_gpus=0.6, num_cpus=0)
class Actor:
    def ready(self): return True

# Schedule 2 actors. Ray's hybrid scheduling prefers the local (head) node,
# so both land on head naturally, one per GPU.
# Head GPU state after: [0.4, 0.4], aggregate remaining = 0.8
# Worker still has [1.0, 1.0].
actors = [Actor.remote() for _ in range(2)]
ray.get([a.ready.remote() for a in actors])

# Schedule a 3rd actor (0.6 GPU). Worker can easily handle it, but:
# - Cluster scheduler sees head's aggregate 0.8 >= 0.6, picks head
# - Local allocator: no single GPU on head has >= 0.6, rejects
# - Spill back reruns scheduling, head aggregate still 0.8 >= 0.6, same result
# The actor is stuck pending even though the worker has 2 idle GPUs.
try:
    a3 = Actor.remote()
    ray.get(a3.ready.remote(), timeout=15)
    print("OK: scheduled on worker")
except ray.exceptions.GetTimeoutError:
    print("BUG: stuck pending despite worker having 2 free GPUs")

In this example, the actor gets stuck in an infinite spill back loop because the scheduler keeps picking the fragmented head node.

Actually this is also a big problem in k8s. The different logic between k8s scheduler and allocation logic in kubelet causes scheduling failures, and was an issue at my previous company since these two parts are managed by different teams. The solution is always trying to unify them and ensure no big perf regression.

If we think deeper, the root cause is that the logic in scheduling and allocation is different, and this causes a lot of problems. So this PR is essentially to make sure scheduling and allocation use the same computing logic.

Solution

You might think this is simple! We could either:

• When local allocation fails, exclude the local node and reschedule to another node.
• Change unit resource to also use per-instance vectors, the same as allocation.

Approach 1 is not ideal. When multiple nodes are fragmented, it could cause a ping-pong infinite loop since each node passes the aggregate check but fails local allocation.

So let's focus on changing unit resources to also use per-instance vectors.

Changes

Here is a summary of the key changes:

1. Make scheduling logic in Raylet and GCS use per-instance vector as their cluster resource availability view

• Both use the same class to manage their cluster resource view, so we changed NodeResources.available from scalar to per-instance vector
• Extracted CanAllocate so that scheduling checks and local allocation use the same per instance logic for each resource type

2. Make RaySyncer carry unit resource (GPU) topology in sync messages

3. Fix STRICT_PACK scheduling policy for per instance available

   STRICT_PACK puts all bundles on one node. Previously it aggregated all bundle
   demands into one request. This aggregation is problematic: e.g. node with GPU=[0.7, 0.3], two bundles each need 0.5 GPU.

   Aggregate 1.0 >= 1.0 passes (note: this aggregation happens inside the STRICT_PACK policy itself, not in the general scheduling checking logic), but actual per bundle allocation fails (no single instance has 0.5 free for the second bundle). Causes unnecessary reschedules.

   We already have per instance checking logic from the previous changes. The extra work here is that for this policy, we try allocating each bundle one by one and rollback after, instead of aggregating all bundle demands into one request.

4. Additional updates

   Updated LocalResourceManager to prevent broadcasting zero-availability non-implicit resources. Previously, the old class constructor implicitly pruned 0-availability resources. Because the refactored code uses a new class constructor that preserves them, explicit filtering was added

Performance Testing

Config: 200 PGs, 0.04 GPU/CPU per bundle, 5 nodes, STRICT_SPREAD

GPU per node	GPU (Branch/Master)	CPU (Branch/Master)
8	1.19x	1.05x
16	1.23x	1.12x
32	1.26x	1.14x
64	1.29x	1.09x
128	1.35x	1.10x
200	1.25x	1.12x

[gemini-code-assist]

Code Review

This pull request transitions Ray's resource tracking and scheduling from aggregate scalar values to per-instance resource availability, preventing fractional resource requests (such as GPUs) from being scheduled on nodes with fragmented, insufficient instances. The review feedback highlights critical backward compatibility concerns during rolling upgrades. Specifically, the reviewer recommends retaining the deprecated resources_available field in the protobuf definition, ensuring upgraded raylets continue to populate it, and implementing fallback logic in the GCS and raylet resource managers when the new resources_available_instances field is absent. Additionally, the reviewer suggests a minor cleanup in the tests to use GetAvailableSum directly for better efficiency.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

New APIs missing Doxygen tags

Low Severity

⚠️ Document functions and classes with Doxygen /** ... */ block comments using @ tags (@brief, and @param/@return where applicable).

Additional Locations (2)

• src/ray/common/scheduling/cluster_resource_data.h#L364-L394
• src/ray/raylet/scheduling/cluster_resource_manager.h#L74-L118

 

^{Triggered by project rule: Bugbot Rules}

^{Reviewed by Cursor Bugbot for commit 7543863. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[dancingactor]

unstale

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{Reviewed by Cursor Bugbot for commit 0f80747. Configure here.}

[cursor]

Bundle reserve ignores packing failures

High Severity

SubtractNodeAvailableResources now bin-packs per instance and can fail, but AcquireBundleResources still ignores the result and walks an unordered BundleLocations map. A placement group that just scheduled can fail to reserve in a different order, leaving GCS availability too high so another group is placed on the same GPUs.

 

^{Reviewed by Cursor Bugbot for commit 0f80747. Configure here.}