[Yicheng-Lu-llll]

Description

When an object's reference count drops to zero, the driver publishes two notices to the remote raylets that care about it:

• the raylet pinning the object's primary copy can now free that memory(with Aaron's change, it will send to all nodes that has copy.)
• any raylet still waiting to learn where the object lives should give up, because the object is gone.

Both are published synchronously on the driver's Python thread and contend for the publisher's lock.

On the map-batches benchmark, py-spy native sampling put 3,556 of about 20,000 scheduling-loop samples (17.4%) on that lock, with a further 3.3% on the reference counter's lock as a knock on.

The proposal is to post the two notices asynchronously on the core worker's IO context instead.

Advantage

• Solves the lock contention
• voids any potential deadlock between the python and c++ threads
• If this adds pressure on the C++ core worker, we can focus on splitting the C++ worker into multiple threads without having to worry about the Python thread
• We later even want to move all these kind of operations that run on python thread needs lock, can do async, to past to c core woer thread,

Correctness

I think the argument here is that we already handle "ref counting goes to zero" asynchronously in a lot of places. Like Ed said, our GC now checks for cycles from time to time, and if there is one, it deletes the ref, which can then trigger ref counting going to zero, so that is already async.

And if the last ref to this object is not on the owner but on a remote worker node, then when the subscription notification tells us the remote worker has also deleted its ref, we already put "ref counting goes to zero" on the C core worker thread. So this change actually makes us more uniform.

perf

We could verify that the lock contention disappears and save the 17.4% times.

Note that posting to the IO context turns out to be really cheap, with no serious lock contention there.

[gemini-code-assist]

Code Review

This pull request updates the RemoveLocalReference method in CoreWorker to execute its logic asynchronously on an IO thread, which is intended to reduce lock contention. A critical safety concern was raised regarding the use of a raw this pointer within the asynchronous lambda, as it could lead to a use-after-free error if the CoreWorker instance is destroyed before the task executes. The reviewer suggested capturing shared_from_this() to safely extend the object's lifetime during the operation.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Reviewed by Cursor Bugbot for commit a1c06bf. Configure here.}

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.