Skip to content

[raft] Improve startup and joining procedure #1802

Open

Description

@MariemBaccari

Following feedback on #1798 and #1780, design a startup and join procedure that is k8s friendly and avoids paths where trivial mistakes lead to a silently failing cluster.

Design proposal according to the feedback

  • Every node has a unique name within the cluster (TBD - use locality / public endpoint / other ?)
  • The name is replicated so that the node can still be uniquely identifiable within the cluster
  • It has 3 Raft IDs (one per service)
  • The IDs are autogenerated at the start and saved in the WAL's metadata

Starting up and handling membership

Bootstrapping a cluster

  • Every node starts with its own URL only
  • Nodes don't find a wal, they generate an ID per service
  • They wait
  • User runs dssctl bootstrap --node A (new admin endpoint)
  • Node A runs raft.StartNode and creates a new cluster with itself as a leader
  • Add the other nodes one by one to the cluster with dssctl add

Restarting a node

  • The node finds its old wal and restarts normally

New node joining and existing cluster

  • Start node then add it with dssctl add like the last step of the bootstrap procedure

Resetting a node that has lost or corrupted its datadir

  • Delete the datadir
  • Node starts, generates a new ID and waits
  • Old ID can be removed and the new one added

(This design removes all state-dependent flags and ensures that, either nodes know what to do at startup or that they will wait for a signal on the admin API from dssctl)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions