You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[raft] Improve startup and joining procedure #1802
Following feedback on #1798 and #1780, design a startup and join procedure that is k8s friendly and avoids paths where trivial mistakes lead to a silently failing cluster.
Design proposal according to the feedback
Every node has a unique name within the cluster (TBD - use locality / public endpoint / other ?)
The name is replicated so that the node can still be uniquely identifiable within the cluster
It has 3 Raft IDs (one per service)
The IDs are autogenerated at the start and saved in the WAL's metadata
Starting up and handling membership
Bootstrapping a cluster
Every node starts with its own URL only
Nodes don't find a wal, they generate an ID per service
They wait
User runs dssctl bootstrap --node A (new admin endpoint)
Node A runs raft.StartNode and creates a new cluster with itself as a leader
Add the other nodes one by one to the cluster with dssctl add
Restarting a node
The node finds its old wal and restarts normally
New node joining and existing cluster
Start node then add it with dssctl add like the last step of the bootstrap procedure
Resetting a node that has lost or corrupted its datadir
Delete the datadir
Node starts, generates a new ID and waits
Old ID can be removed and the new one added
(This design removes all state-dependent flags and ensures that, either nodes know what to do at startup or that they will wait for a signal on the admin API from dssctl)
Activity