Repository navigation
[Core] Implement Active/passive head replica #63643
Description
Description
Context: Implement the enhancement in ray-project/enhancements#65
Implementation plan:
- Standalone C++ Leader Election Client:
- Extends the existing internal Kubernetes client helper
- Implements the platform-agnostic interface
- Implement Kubernetes lease client with best practice from client-go leader election libaray
- GCS and other head components initialization refactoring for Passive Mode
- Refactor GCS managers for standby mode: keep RPC alive for health checks but reject unnecessary queries and mutation to ensure consistency
- Ensure colocated head components compatibility
- GCS Integration with the leader election client
- Implement generic leader election factory
- Defer GCS data loading until it is promoted
- Integrate the background leader election client polling thread inside the main GCS Server process
- Implement promotion and step-down process
- Readiness probe should depend on the leadership status
- Redis fencing token
- Protects Redis persistence against split-brain zombie writes by enforcing monotonically increasing lease transition epochs on all GCS write commands.
- Metrics and monitoring
- KubeRay API and controller support
- E2E failover testing
Use case
GCS fault tolerance
Activity
rpathade commentedon May 27, 2026
Hi, I’d like to help with this.
I went through the issue and the linked REP. Would it make sense for me to start with the standalone C++ leader election client first, and keep the first PR limited to that piece plus tests?
I’ll leave the GCS integration/fencing/KubeRay/E2E parts for follow-up PRs unless you suggest a different split.
nh-atuan commentedon May 29, 2026
Hi, I'd like to work on this issue.
I reviewed the implementation plan and would like to contribute by taking the leader election integration path. My initial focus will be on implementing the generic leader election factory and integrating the background leader election client polling thread into the GCS server process, together with the associated tests.
I'll keep the scope of the first PR limited and coordinate with others if the work gets split across multiple PRs. Please let me know if there is a preferred breakdown of tasks or if any part of the implementation is already being worked on.
Thanks!
YoyinZyc commentedon May 29, 2026
@rpathade @nh-atuan Thank you for your interests. We have already got the draft PRs prepared for the first two phases. Will keep you posted if we need any help in following phases. Thanks!
[Active-passive] Phase1: Introduces a native, lightweight C++ leader …
[core][gcs][Actice-Passive] Phase 2.1 Add Active-Passive leader-elect…
[Active-passive] Phase1: Introduces a native, lightweight C++ leader …
[core][gcs][Actice-Passive] Phase 2.1 Add Active-Passive leader-elect…
[core][gcs][Actice-Passive] Phase 2.2 Add Active-Passive leader-elect…
[core][gcs][Actice-Passive] Phase 2.3 Passive head lifecycle change (#…
[core][gcs][Actice-Passive] Phase 2.1 Add Active-Passive leader-elect…
[core][gcs][Actice-Passive] Phase 2.2 Add Active-Passive leader-elect…
[core][gcs][Actice-Passive] Phase 2.3 Passive head lifecycle change (r…
[Active-Passive] Phase 2.4 Adjust the components on head node to be c…