Repository navigation
id: build: TigerBeetle cluster provisioning — RTT mesh gate, 6-replica format, dead-replica runbook + rehearsal #2292
Description
Buildable follow-on from the #2064 topology investigation (doc PR #2291). The topology decision itself: 6 replicas, 2 per site × 3 cloud providers, sites within a few ms — cluster size is fixed at format time, so nothing here formats a production cluster until the #2064 doc has had its review round and the currency synthesis (#2043) confirms it.
Scope
- Site RTT measurement mesh (pre-format gate). Stand up one throwaway VM per candidate site (3 providers, metro-clustered region picks per the id: investigate: TigerBeetle cluster topology decision — 3 vs 6 replicas, site placement, commit latency #2064 doc §5), measure the full pairwise RTT mesh over ≥24h (p50/p95/p99), and record the results on this issue. Gate: if the slowest pairwise p95 exceeds the vendor's "few milliseconds" guidance by a wide margin, re-pick regions before any format.
- Cluster provisioning. Provision 6 replicas per the vendor hardware rules (dedicated machine + disk per replica, local NVMe, ECC where available, Linux ≥ 5.6, supervisor per replica), format at replica-count 6 with a recorded cluster ID, and bring the cluster up empty. No real value, no production data — this proves the shape.
- Dead-replica replacement runbook + rehearsal. Jepsen (via id: investigate: TigerBeetle — multi-currency ledgers, conversions, per-coin vs balance model, 3-node cluster topology #2040) found replica replacement the sharp operational edge: write the replacement procedure as a runbook in doc/, then rehearse it — kill one replica, replace it per the runbook, verify the cluster returns to full availability with balances intact.
Acceptance criteria
- RTT mesh results (p50/p95/p99, ≥24h window) recorded on this issue for the three candidate sites
- 6-replica cluster provisioned and healthy (all replicas up, cluster committing test transfers on a throwaway ledger)
- Replacement runbook in doc/ and one rehearsed replica replacement recorded here
- Production format remains gated on the id: investigate: TigerBeetle cluster topology decision — 3 vs 6 replicas, site placement, commit latency #2064 review round / id: design: ID currencies v1 #2043 synthesis confirmation — this issue builds and rehearses, it does not commit the production decision by itself
Parent epic: #2056. Origin investigation: #2064. Source investigation: #2040.
Metadata
Metadata
Assignees
Labels
Package/app: id (pkgs/ or apps/ in this repo)Large: multi-day or open-ended; expect staged check-insInfrastructure / host-portability workIntake review done: acceptance criteria, links, and normalized labels addedOrdinary priority (the default)Queue or user escalation stateHas a queued task in the Supabase agent queueFiled by the new-issue-creator jobQueue or user escalation stateQueue packet kind: verify, reproduce, and route this itemQueue or user escalation state
Activity
developing-today commentedon Oct 29, 2026
CODE-1149
dezren39 commentedon Oct 29, 2026
Intake review
Related: #2064 (topology investigation, doc PR #2291), #2040 (Jepsen source), #2043 (synthesis gate), epic #2056
Notes: Acceptance criteria were already in the body as a checklist — labels only this pass, no body edit. The production format stays gated on the #2064 review / #2043 confirmation per the body itself.
Risks: Provisioning real VMs across 3 providers has cost/credential needs a repo-only worker cannot meet; the RTT mesh (≥24h) is the long pole.
Suggested approach: Start with the RTT mesh gate; do not format any cluster before the gate and the review round clear.
🤖 Agent note · filed
meta/enrichedapplied. Still-live check this pass: no merged PR or fix commit explicitly resolving this issue was found; where a queue triage task owns verification, it is named in the notes.dezren39 commentedon Nov 3, 2026
🤖 Agent note · blocked (escalated to Drew)
repo-worker-code), OpenChamber scheduled lanec29174f1developing-today commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03