Skip to content

[core][DO NOT MERGE] End-to-end testing changes for removing task events out of GCS - #65163

Closed
karticam wants to merge 69 commits into
ray-project:masterfrom
karticam:karticam/test-task-events-end-to-end
Closed

karticam wants to merge 69 commits into
ray-project:masterfrom
karticam:karticam/test-task-events-end-to-end

Conversation

@karticam

@karticam karticam commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

test pr.

karticam added 30 commits July 28, 2026 11:33
- New TaskEventsHead SubprocessModule exposing POST /api/task_events
- Deserialization isolated in _deserialize_request (wire format TBD)
- In-memory buffer stub; storage/GC is a later task

Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
task/actor task definition events

Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
no thread spawning when task event recorder flags are disabled.

Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
is responsible for making test heavy

Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
karticam added 23 commits August 5, 2026 02:24
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
in py

Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
# Conflicts:
#	src/ray/common/ray_config_def.h
- enable_ray_event: true
- enable_ray_task_event_recorder: true
- enable_task_events_to_dashboard_head: true
- enable_core_worker_task_event_to_gcs: false

Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
- aggregator publish_to_both: pin RAY_enable_ray_event=0 so only the injected
  event reaches the aggregator (len==1 assertion)
- resource_isolation: expect TaskEventsHead only when the flag is on, derived
  from ray._config.enable_task_events_to_dashboard_head()
- gc_jobs: set the head store cap via RAY_task_events_max_num_task_in_gcs env
  var so it reaches the head subprocess

Signed-off-by: Kartica Modi <karticamodi@gmail.com>
The dashboard-head store reads delays/caps from ray._config at import. _system_config
from ray.init may not reach the head subprocess, so also set the values via RAY_<config>
env vars (which reach it through os.environ):
- test_fault_tolerance_job_failed: mark-failed worker-dead / job-done delays
- test_ray_intentional_errors: mark-failed worker-dead delay
- test_task_events_gc_default_policy: max num task events

Signed-off-by: Kartica Modi <karticamodi@gmail.com>
@karticam
karticam force-pushed the karticam/test-task-events-end-to-end branch from c0bc48b to 6f45751 Compare August 5, 2026 12:43
import ray
from ray._common.constants import HEAD_NODE_RESOURCE_NAME, NODE_ID_PREFIX
from ray._common.utils import binary_to_hex, decode, hex_to_binary
from ray._private import ray_constants

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unclear PR title/description

Low Severity

⚠️ This PR needs a clearer title and/or description.

To help reviewers, please ensure your PR includes:

  • Title: A concise summary of the change
  • Description:
    • What problem does this solve?
    • How does this PR solve it?
    • Any relevant context for reviewers such as:
      • Why is the problem important to solve?
      • Why was this approach chosen over others?

See this list of PRs as examples for PRs that have gone above and beyond:

Fix in Cursor Fix in Web

Triggered by project rule: Bugbot Rules

Reviewed by Cursor Bugbot for commit 6f45751. Configure here.

raise DataSourceUnavailable(
"Failed to query task events from the dashboard head. "
f"Response is {resp.status}, reason {resp.reason}"
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unhandled dashboard query failures

Medium Severity

_get_task_events_from_dashboard_head only maps non-2xx HTTP responses to DataSourceUnavailable. Connection failures, timeouts, and other aiohttp errors propagate raw. list_tasks only catches DataSourceUnavailable, so those cases surface as hard failures instead of the existing graceful data-source warning path used for GCS.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 6f45751. Configure here.

monkeypatch.setattr on ray._config.enable_task_events_to_dashboard_head fails at
teardown (read-only Cython attribute). Swap the whole ray._config for a delegating
proxy that overrides only that method instead.

Signed-off-by: Kartica Modi <karticamodi@gmail.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 5 total unresolved issues (including 4 from previous reviews).

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 9812386. Configure here.

get_all_task_info routes on state_manager._READ_TASK_EVENTS_FROM_DASHBOARD_HEAD,
a constant computed once at import from ray._config. Patching ray._config after
import has no effect; patch the module constant instead.

Signed-off-by: Kartica Modi <karticamodi@gmail.com>
Signed-off-by: Kartica Modi <karticamodi@gmail.com>
@karticam karticam closed this Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant