REP: Ray Active-Passive Head Architecture by YoyinZyc · Pull Request #65 · ray-project/enhancements

YoyinZyc · 2026-04-23T19:12:38Z

This REP proposes the transition from a single-point-of-failure GCS model to a highly available Active-Passive setup. The design aims to reduce recovery latency from minutes to seconds.

Scope:

Ray core: GCS native lease-based leader election client
KubeRay: Ability to provision multiple head nodes
Fault-tolerance: Align with existing GCS FT scope, targeting KubeRay based cluster.

Requested reviewers: @andrewsykim @edoakes @MengjinYan

Signed-off-by: Yuchen Zhou <yczhou@google.com>

rueian · 2026-04-30T17:59:16Z

+#### 5. KubeRay Changes
+1. **API change**: specifying the number of desired head replicas.
+2. **Configuration changes**: 
+    - Readiness Probe for passive heads to fail.


IIRC, probes are immutable. So, probably the change should be in GCS?

Yeah, I was originally reaching to k8s APIServer directly to fetch the lease status for readiness check but it could be part of the gcs_health endpoint. In this way, no need to update the readiness probe. Updated.

andrewsykim · 2026-04-23T21:04:41Z

+3. **Fencing Token**: The last barrier to prevent data inconsistency. It is also one of the best practices for implementing lease-based leader election in production. At the end of the day, we must make sure the data stored in redis is pure.
+	- **The Token**: Lease transition, it is updated when a new leader takes over the lease
+	- **Write to Redis**: the leader GCS attaches the token with the query
+	- **Fencing**: The Redis rejects the request if the request_epoch is smaller than its current_epoch


@YoyinZyc how exactly does redis reject requests based on the token? Is it a feature in Redis or something else?

I think this can be implemented with redis lua script, but I am wondering if doing fencing in redis is enough. Shouldn't we also have this fencing in all GCS -> raylet communication?

Yes, it is with lua script, added more details and the example lua script. GCS -> raylet communication involves more design and consideration, I added it as the a phase 2 item which should fall into later iteration.

edoakes · 2026-04-30T21:06:36Z

+The latter two are Ray-specific optimizations for the GCS HA scenario. Techenically, they could provide additional data consistency guarantees. However, it may involve other potential risk and additional implementation work.(e.g. communication between heads/workers could be unreliable) We can leave it to the future iterations. 
+##### Metrics and Monitoring
+- `gcs_is_leader`: Emits a 1 if the current instance is the leader, and a 0 if it is a passive head.
+- `gcs_leader_transitions_total`: Incremented every time a leadership change is detected


where would this be exported?

From the gcs process.

edoakes · 2026-04-30T21:06:43Z

+##### Metrics and Monitoring
+- `gcs_is_leader`: Emits a 1 if the current instance is the leader, and a 0 if it is a passive head.
+- `gcs_leader_transitions_total`: Incremented every time a leadership change is detected
+- `gcs_recovery_latency_ms`: The time it takes for a passive head to recover the state and become the leader.


same for this, where/how is it recorded?

It is recorded from the gcs server. gcs_is_leader is a gauge metrics tracking local leadership status; gcs_leader_transitions_total increaments when the passive becomes the leader;
gcs_recovery_latency_ms tracks the passive head recovery latency from the moment it acquires the leadership to the moment the head is functional.

edoakes · 2026-04-30T21:06:59Z

+The first three mechanisms are production golden standards for leader election implementations. They are generic and must be included in the first phase of this REP. 
+
+The latter two are Ray-specific optimizations for the GCS HA scenario. Techenically, they could provide additional data consistency guarantees. However, it may involve other potential risk and additional implementation work.(e.g. communication between heads/workers could be unreliable) We can leave it to the future iterations. 
+##### Metrics and Monitoring


are there any other metrics around API server latency / health that we should observe?

APIServer has a bunch of metrics but those are pretty generic like apiserver_request_total, apiserver_request_duration_seconds and apiserver_current_inflight_requests

It is probably worth also exporting similar metrics from our lease client

What similar metrics are you referring to? These are k8s APIServer metrics, how could the lease client export them?

I mean our lease client should also export metrics like the latency for its requests to the API server and a failed request count

Ah I see, added two more metrics for lease renew latency and failure count.

edoakes · 2026-04-30T21:08:47Z

+  - **Raylet**
+    Safe to run only if launched with `--num-cpus=0` and `--num-gpus=0`, ensuring the active primary never schedules user tasks onto the standby head node.


Setting these is not strong enough -- actors/tasks with no resource requests can still be scheduled there.

Can we gate the Raylet from joining the cluster until after the GCS is elected the leader? This is similar to the problem w/ the autoscaler that I pointed out above

yes, autoscaler must be suppressed. Raylet may be fine since it anyway only talks to the local gcs but we should restrict its registration which incurs the redis write. Added more detail about components behavior.

Signed-off-by: Yuchen Zhou <yczhou@google.com>

Signed-off-by: Gang Li <ganglica@google.com>

Update autoscaler section in Active-Passive REP

edoakes · 2026-05-07T22:20:25Z

+  - **Raylet**: it is required to be alive to answer the readiness and liveness probes.
+      - **Startup**: Upon boot, the local standby Raylet targets the local GCS and sends a RegisterNode gRPC request. On passive head, the PUT request should not reach to Redis. 
+      - **Passive State**: The Raylet continuously sends heartbeats to the local GCS to keep its node liveness. 
+      - **Promotion**: During promotion, the Raylet should register itself and  physically writes it to Redis.
+  - **Autoscaler**: it is a critical, high-risk background component that must be suppressed on passive head.
+      - **Startup**: Upon boot, the local autoscaler targets the local GCS and periodically sends a GetClusterResourceState gRPC request to check if the cluster resource state in GCS is ready. The cluster resource state should only be ready when the GCS is in active mode.
+      - **Active State**: Autoscaler gets the cluster resource state from GCS and continues with the normal autoscaling logic including updating KubeRay CRDs, reporting the autoscaling state back to GCS.
+      - **Passive State**: Autoscaler gets the cluster resource state from GCS, which indicates the cluster resource state is not ready. The autoscaler skips the rest of the logic.


Will each of these also be monitoring that its GCS is the leader and fast-failing when it loses the lease?

@gangli113 helped adding more details about autoscaler, let us know if there is any other concerns. Thanks!

edoakes

Largely looks good. Primary open feedback:

Please document the full scope of KubeRay changes (especially any user-facing changes).
We will need the same safety guarantees for the autoscaler process. It will either need to fast fail when the GCS fails or also directly check w/ a lease client.

andrewsykim · 2026-05-12T19:51:14Z

+	// +kubebuilder:default:=1
+	// +kubebuilder:validation:Minimum:=1
+	// +kubebuilder:validation:Maximum:=2
+	Replicas *int32 `json:"replicas,omitempty"`


Are we introducing replicas now because we anticipate it can be more than 2 head nodes in the future?

(As opposed to something like mode: SingleReplica|ActivePassive)

I think replicas is confusing because it implies a user can set it to any arbitrary value.

A field in gcsFaultToleranceOption is probably better?

spec: gcsFaultToleranceOptions: mode: ActivePassive .. ...

This is slightly more intuitive because active-passive mode also requires enabling Redis persistence anyways right?

WDYT @rueian ?

I was also thinking about just using a boolean or mode to control the enablement. But using replicas is more kubernetes native for controller to reconcile and can be extended if ray eventually wants to support more replicas?

But using replicas is more kubernetes native for controller to reconcile and can be extended if ray eventually wants to support more replicas?

I think we can just introduce replicas field in the future when we can support > 2 replicas. Putting replicas outside of gcsFaultToleranceOptions implies you can scale to N replicas without enabling GCS persistence, which is not the case. Putting the mode in gcsFaultToleranceOptions makes it clearer that you can only enable active-passive replica mode when GCS persistence is also enabled

Ok changed to using EnableActivePassiveHead under gcsFaultToleranceOptions to gate the enablement of the feature.

Signed-off-by: Yuchen Zhou <yczhou@google.com>

…sive REP Signed-off-by: Gang Li <ganglica@google.com>

Update Autoscaler section for GCS failure and promotion in Active-Passive REP

edoakes · 2026-05-13T21:02:27Z

+      - **Active State**: Autoscaler gets the cluster resource state from GCS and continues with the normal autoscaling logic including updating KubeRay CRDs, reporting the autoscaling state back to GCS.
+      - **Passive State**: Autoscaler gets the cluster resource state from GCS, which indicates the cluster resource state is not ready. The autoscaler skips the rest of the logic.
+  - **Dashboard**: the local Dashboard process is started by KubeRay pointing to the local GCS. Because it queries the passive local GCS, the dashboard UI initially displays a single-node cluster (only itself) with no active jobs or actors. Since the standby pod is Not Ready, its dashboard API is completely unreachable by external clients with the current k8s service setup. 
+  - **Job API**: with right k8s service configuration, the `ray job submit` request should not be able to reach the passive head.


We should not rely on this as the only mechanism. Users may submit job requests in a different way (e.g., exec'ing into the pod and running the job submit command there)

edoakes · 2026-05-13T21:03:31Z

+      - **Passive State**: Autoscaler gets the cluster resource state from GCS, which indicates the cluster resource state is not ready. The autoscaler skips the rest of the logic.
+  - **Dashboard**: the local Dashboard process is started by KubeRay pointing to the local GCS. Because it queries the passive local GCS, the dashboard UI initially displays a single-node cluster (only itself) with no active jobs or actors. Since the standby pod is Not Ready, its dashboard API is completely unreachable by external clients with the current k8s service setup. 
+  - **Job API**: with right k8s service configuration, the `ray job submit` request should not be able to reach the passive head.
+  - **Serve Controller**: The Ray Serve Controller is a standard Ray actor. Since GCS in standby mode has no active workers and no running jobs, no Serve Controller is ever spawned on the standby head.


It doesn't matter if the serve controller isn't run on the standby head, the serve API endpoints in the dashboard on a passive head could still make requests to the active head

Most likely I think we need a shared bit for API server components to check if they are the active or passive and fail requests when passive. This can be used for serve API & jobs API

We could even abstract it into a utility/decorator that performs the check & rejects requests with an appropriate status code

I may need more clarification on this. what do you mean about "serve API endpoints in the dashboard on a passive head could still make requests to the active head"?

And you are suggesting that whenever there is an unexpected request hit Job API or Serve API, it should ask its local gcs to check if it is leader or passive. If it is passive, reject the request. Am I understanding correctly?

also in terms of the decorator, are you saying something on gcs side like a rpc or API for leadership status checking?

you are suggesting that whenever there is an unexpected request hit Job API or Serve API, it should ask its local gcs to check if it is leader or passive. If it is passive, reject the request. Am I understanding correctly?

Yes that's right.

also in terms of the decorator, are you saying something on gcs side like a rpc or API for leadership status checking?

I am saying we could have a shared utility or python decorator in the dashboard logic that automatically performs this check before accepting any requests

MengjinYan · 2026-05-14T07:09:56Z

+  **1. Single-container co-located components**:
+  These components are running in the same container as different subprocesses. The gcs termination caused by lease lost will terminate other processes.
+  - **GCS Server**
+    Safe on Standby. Global tracking managers are instantiated empty; state hydration is deferred until promotion. The rpc server can only answer read requests in passive mode. All mutation requests should not be accepted and the rejection should be handled gracefully on the client side. (With the proper traffic routing, the requests will be sent to the active head only. But it is another layer of protection to avoid any potential split brain issue.)


The rpc server can only answer read requests in passive mode

Nit: It could be implementation detail, but I think it might be better just to identity the endpoints that needs to be called and allow them instead of open all the read endpoints.

At least GetClusterID and healthcheck rpc have to be enabled on passive. I may uncover more during the implementation.

MengjinYan · 2026-05-14T15:47:04Z

+  - **Dashboard**: the local Dashboard process is started by KubeRay pointing to the local GCS. Because it queries the passive local GCS, the dashboard UI initially displays a single-node cluster (only itself) with no active jobs or actors. Since the standby pod is Not Ready, its dashboard API is completely unreachable by external clients with the current k8s service setup. 
+  - **Job API**: with right k8s service configuration, the `ray job submit` request should not be able to reach the passive head.
+  - **Serve Controller**: The Ray Serve Controller is a standard Ray actor. Since GCS in standby mode has no active workers and no running jobs, no Serve Controller is ever spawned on the standby head.
+  - **Dashboard Agent**: Runs locally and reports metrics to the co-located passive GCS. Upon promotion, it continues reporting to the now-promoted local GCS. Ready to initialize runtime environments if new jobs are submitted post-failover.


nit: We might also want to disable event generation in the passive mode. There might not be many events generated but disabling it as a safe guard

I think the event generation is controlled by some envs(like RAY_enable_ray_event or RAY_DASHBOARD_AGGREGATOR_AGENT_PUBLISH_EVENTS_TO_GCS ). Is it possible the re-enable the event generation at run time after promotion? Or are you referring to other approach to disable the event generation?

I wasn't referring to any specific mechanism but more of a high level safe guard.

You are right. The configurations are all specified at the start time. We probably need to add an additional config in the aggregator agent to control whether to emit events at the runtime.

Then the most intuitive approach is to ask the dashboard agent to watch and poll the leadership status periodically (in low frequency, 5s) and emit event post-promotion only.

…assive enablement Signed-off-by: Yuchen Zhou <yczhou@google.com>

Signed-off-by: Yuchen Zhou <yczhou@google.com>

edoakes

Looks good to me as a starting point for this feature; I'm sure there will be some more low level details to iron out as we go.

@andrewsykim

…ion, observability example - Expand the Acceptance Criteria subsection to include the four criteria from @andrewsykim (Ray release tests, Ray Core GCS FT test equivalent, Ray scale tests, Ray Serve chaos testing) and incorporate @edoakes's clarification that the Redis-equivalent test should be a postmerge CI job using a local-file RocksDB, labeled skip-premerge with path-based triggers on src/ray/gcs/**. - Add an "Interaction with Active-Passive Head (REP ray-project#65)" subsection covering the ReadWriteMany topology, two-layer split-brain prevention (lease + RocksDB LOCK file), failover sequence, and RWX latency trade-off. Update Follow-on Work item 2 to mark Active/Passive GCS as in flight and link to REP ray-project#65. - Add observability data (event logs, task/actor history, log indexes) as an explicit example of non-critical, high-volume state that the future cold-paging optimization could keep primarily on disk. Signed-off-by: Santosh Jha <santosh.m.jha@gmail.com>

Add Active-Passive GCS Head architecture REP

47c687c

Signed-off-by: Yuchen Zhou <yczhou@google.com>

YoyinZyc force-pushed the active-passive branch from 5766f6e to 47c687c Compare April 23, 2026 19:17

Add a section about monitoring and alert

ded3bd1

Signed-off-by: Yuchen Zhou <yczhou@google.com>

YoyinZyc force-pushed the active-passive branch from 3caa578 to ded3bd1 Compare April 30, 2026 17:25

rueian reviewed Apr 30, 2026

View reviewed changes

andrewsykim reviewed Apr 30, 2026

View reviewed changes

edoakes reviewed Apr 30, 2026

View reviewed changes

abrarsheikh reviewed Apr 30, 2026

View reviewed changes

Comment thread reps/2026-04-23-active-passive-head.md Resolved

abrarsheikh reviewed Apr 30, 2026

View reviewed changes

Comment thread reps/2026-04-23-active-passive-head.md Resolved

Add more detail about fencing token and readiness probe

e7b5d1a

Signed-off-by: Yuchen Zhou <yczhou@google.com>

YoyinZyc force-pushed the active-passive branch from 18017f5 to e7b5d1a Compare April 30, 2026 23:19

YoyinZyc and others added 5 commits May 4, 2026 21:44

add latency baseline

c3ba3c1

Signed-off-by: Yuchen Zhou <yczhou@google.com>

Add details about ray head components compatibility

a715297

Signed-off-by: Yuchen Zhou <yczhou@google.com>

Add details about metrics collection

300b836

Signed-off-by: Yuchen Zhou <yczhou@google.com>

Update autoscaler section in Active-Passive REP

f08d42b

Signed-off-by: Gang Li <ganglica@google.com>

Merge pull request #1 from gangli113/active-passive

7c61be5

Update autoscaler section in Active-Passive REP

edoakes reviewed May 7, 2026

View reviewed changes

andrewsykim reviewed May 12, 2026

View reviewed changes

Comment thread reps/2026-04-23-active-passive-head.md Resolved

YoyinZyc force-pushed the active-passive branch from 1c5202e to 11bbf1b Compare May 12, 2026 21:27

Add detailed kuberay behavior and the customer facing API change

bfd1bf7

Signed-off-by: Yuchen Zhou <yczhou@google.com>

YoyinZyc force-pushed the active-passive branch from 11bbf1b to bfd1bf7 Compare May 12, 2026 22:04

YoyinZyc and others added 3 commits May 12, 2026 22:36

Address some minor comments

4e39eae

Signed-off-by: Yuchen Zhou <yczhou@google.com>

Update Autoscaler section for GCS failure and promotion in Active-Pas…

54d1da3

…sive REP Signed-off-by: Gang Li <ganglica@google.com>

Merge pull request #2 from gangli113/active-passive

6d01c5c

Update Autoscaler section for GCS failure and promotion in Active-Passive REP

edoakes reviewed May 13, 2026

View reviewed changes

MengjinYan reviewed May 14, 2026

View reviewed changes

Use the option under GcsFaultToleranceOptions to control the active-p…

5890f43

…assive enablement Signed-off-by: Yuchen Zhou <yczhou@google.com>

YoyinZyc force-pushed the active-passive branch from f3f46dd to 5890f43 Compare May 14, 2026 22:54

rephrase the components compatibility section

c69218e

Signed-off-by: Yuchen Zhou <yczhou@google.com>

edoakes approved these changes May 19, 2026

View reviewed changes

edoakes merged commit 564cd49 into ray-project:main May 19, 2026
1 check passed

edoakes mentioned this pull request May 21, 2026

REP: Embedded Storage Backend for GCS Fault Tolerance #64

Merged

YoyinZyc mentioned this pull request May 26, 2026

[Core] Implement Active/passive head replica ray-project/ray#63643

Open

2 of 7 tasks

anwithk mentioned this pull request Jun 23, 2026

NeMo RL Software Architecture Update: PR Tracking NVIDIA-NeMo/RL#2905

Open

5 of 29 tasks

		- Raylet
		Safe to run only if launched with `--num-cpus=0` and `--num-gpus=0`, ensuring the active primary never schedules user tasks onto the standby head node.

Conversation

YoyinZyc commented Apr 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edoakes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewsykim May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewsykim May 12, 2026 •

edited

Loading