Skip to content

[core][__on_object_freed__][2/n] Run the __on_object_freed__ hook through an OnObjectFreed RPC - #67308

Open
karticam wants to merge 1 commit into
karticam/on-object-freed-detectfrom
karticam/on-object-freed-rpc
Open

karticam wants to merge 1 commit into
karticam/on-object-freed-detectfrom
karticam/on-object-freed-rpc

Conversation

@karticam

@karticam karticam commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

This is PR 2 of a series of PRs that add the __on_object_freed__ hook: an actor that defines the hook will be told when an object it created is freed. This PR adds the RPC that runs the hook on the actor's worker. Nothing sends it yet: a later PR makes the raylet send it when the last copy of such an object is deleted.

The complete design doc is here: https://docs.google.com/document/d/1ZJhgRmwVyKfNCR21CWf6xMcc0B6HJwSPQsfebTRvnnE/edit?tab=t.0#heading=h.7wcod1f9afpg

Changes

  1. A new CoreWorkerService.OnObjectFreed RPC. The request carries the freed object's ID and the ID of the worker it is meant for. A worker that reuses the address of a dead one rejects it, like PushTask. The client side (CoreWorkerClient::OnObjectFreed) is added too, for the raylet to use later.
  2. The worker replies before the hook runs. HandleOnObjectFreed schedules the hook and replies right away: the raylet doesn't use the result, and a hook waiting behind a long actor method shouldn't keep the RPC open. If the worker dies before the hook runs, the hook doesn't run, which is the same as for a dead worker.
  3. The hook runs where an actor method in its concurrency group would run. The new TaskReceiver::RunOnActorExecutor picks the group's fiber executor for an asyncio actor, the group's thread pool for a threaded actor, and the task execution thread otherwise, where the hook runs between the actor's methods. The hook's flag and group from [core][__on_object_freed__][1/n] Detect the __on_object_freed__ hook on actor classes #67232, and the actor's executors, are read on the task execution thread, which sets them during actor creation, so the hook never runs before the actor's __init__ has finished (since the hook is also queued on the task execution service).
  4. The Python bridge calls the hook with the object ID as a hex string. On an asyncio actor, both async def and sync hooks run on the group's event loop, as actor methods do. An Exception raised by the hook is logged and the actor keeps running. A test-only CoreWorker._schedule_on_object_freed_hook_for_testing runs the same path as the RPC, so the hook can be tested before the raylet sends the RPC.

@karticam
karticam added this pull request to stack #66768 October 22, 2026 22:51
@karticam karticam added core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests labels Oct 22, 2026
@karticam
karticam marked this pull request as ready for review October 22, 2026 23:02
@karticam
karticam requested a review from a team as a code owner October 22, 2026 23:02

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements the backend and frontend support for running an actor's on_object_freed hook when an object it created is freed. It introduces the OnObjectFreed RPC, adds scheduling logic to run the hook on the actor's executor (respecting concurrency groups and event loops), and includes comprehensive unit tests. A review comment identified an issue in _raylet.pyx where the arguments for PythonFunctionDescriptor are swapped, which would cause incorrect metadata representation.

Comment thread python/ray/_raylet.pyx
Comment on lines +2690 to +2692
hook_descriptor = PythonFunctionDescriptor(
creation_descriptor.module_name, "__on_object_freed__",
creation_descriptor.class_name)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The arguments passed to PythonFunctionDescriptor are swapped. The constructor signature is PythonFunctionDescriptor(module_name, class_name, function_name), but here "__on_object_freed__" (the function name) is passed as the second argument, and creation_descriptor.class_name (the class name) is passed as the third argument. This will cause incorrect metadata representation in dashboard, logging, or tracing.

        hook_descriptor = PythonFunctionDescriptor(creation_descriptor.module_name, creation_descriptor.class_name, "__on_object_freed__")

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/ray/protobuf/core_worker.proto Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5698b6e. Configure here.


virtual void HandleOnObjectFreed(OnObjectFreedRequest request,
OnObjectFreedReply *reply,
SendReplyCallback send_reply_callback) = 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing Doxygen on new functions

Low Severity

⚠️ Document functions and classes with Doxygen /** ... */ block comments using @ tags (@brief, and @param/@return where applicable).

Additional Locations (2)
Fix in Cursor Fix in Web

Triggered by project rule: Bugbot Rules

Reviewed by Cursor Bugbot for commit 5698b6e. Configure here.

@karticam
karticam force-pushed the karticam/on-object-freed-rpc branch from 5698b6e to 3eb9cc8 Compare November 3, 2026 03:07
Signed-off-by: kartica.modi <kartica.modi@anyscale.com>
@karticam
karticam force-pushed the karticam/on-object-freed-rpc branch from 3eb9cc8 to f869a59 Compare November 14, 2026 02:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants