Skip to content

fleet: client-local session forks diverge from the canonical DB (macbook shard 77 ahead; mini fork retired) #1302

Closed
#1407
Closed
fleet: client-local session forks diverge from the canonical DB (macbook shard 77 ahead; mini fork retired)#1302
#1407
Labels
hitlNeeds human review before merge

Description

@aarontrowbridge
aarontrowbridge
opened on Sep 20, 2026

Important

Problem — Fleet clients can silently fork the canonical chat DB. On 2026-09-20 a fleet sweep found: (1) the mini was running a stale local opencode serve on port 4096 from its pre-erlich server role (kept alive by the still-installed co.harmoniqs.amicode-server launchd agent), 3 days idle — remediated same day (fork killed, agent retired to ~/.amico/ops/retired/, its 2 unmerged sessions merged into canonical with zero collisions, local DB archived). (2) The macbook actively writes a local opencode.db via the OpenCode desktop app and a direct TUI session — 77 of its last-21-day sessions are absent from the canonical DB, including 57 substantive ones (a Sep 13–14 amicode dev campaign of issue-cast sessions) and a live bug report from the morning of the sweep ("fleet mode prompts not sending"). The fleet guard (amico-opencode-fleet-guard) covers the extension's binary path but not the desktop app or a bare TUI, both of which write locally by default.
Approach — Two parts. (a) Detection: extend the fleet guard / a nightly divergence check so any client-local opencode process bound to 4096, or any client DB holding sessions absent from canonical, raises a fleet alert before the shard grows. (b) Merge ritual: a documented, scripted shard-merge for the macbook (collision-check → merge position-free → re-home directories → quick_check → archive), runnable once the local TUI/desktop processes are closed.
Approaches considered — Client local-first posture (tracked separately, #1289) changes the default write location but still needs divergence detection while both postures exist. Blocking the desktop app outright was rejected: it is a supported surface, it just must not fork the chat store.
Scope — in: guard coverage for desktop-app + TUI launch paths; nightly client-DB divergence check + alert; scripted macbook shard merge. out: merging while the macbook TUI is live; any change to the canonical server's writer role.
Assumptions — The macbook TUI/desktop sessions are wanted in canonical history (they include real dev-campaign work); the user can close the local TUI at merge time.

Acceptance Criteria

  • A client machine running the desktop app or a bare TUI against a local DB produces a fleet alert (divergence detected) rather than silently forking.
  • The divergence check reports per-client session counts and canonical-overlap, and exits nonzero on divergence (alertable).
  • The macbook shard merges into canonical with zero row collisions on session/message/part/todo, re-homed directories, and a passing PRAGMA quick_check.
  • After merge, the mini-style respawn path is closed: no launchd agent on any client starts a local opencode serve.

Testing Decisions

Extend the existing fleet guard check (--check mode) and the fleet-status job rather than adding a new suite; the merge script gets a dry-run mode exercised against an archived shard copy.

Key Decisions

  • Archive-then-merge: a client DB is never deleted, only renamed .archived-<date> after a verified full merge (the mini's DB is the worked example).
  • The launchd agent retirement is a move-aside (~/.amico/ops/retired/), reversible.
  • Detection is derived from disk/process state only — no hand-maintained device list beyond the SSH aliases the fleet tooling already uses.

Constraints & Invariants

  • One canonical DB, one live writer (the hub server); never two live writers on one SQLite file.
  • Merge follows the chat-database recovery order: collision check → parent tables first → position-free events only → re-home → checkpoint → verify.
  • A merge never runs against a live client writer.

Prior Art

  • The fleet skill's chat-database recovery section (the merge procedure this implements).
  • The 2026-08-07 client-fork incident (IPv6-only forward race) — same failure family, different entry path.
  • The mini remediation performed 2026-09-20 (reference for the scripted version).

Notes

  • Found during the 2026-09-20 session sweep; the macbook half is live right now (its TUI is writing locally as of the sweep).
  • Vault note with the full sweep evidence: personal vault specs/spec-20260920-session-curation.md (§4 shard-divergence watch).

Activity

  1. aarontrowbridge
    added
    hitlNeeds human review before merge
    on Sep 20, 2026
  2. aarontrowbridge
    added a commit that references this issue on Sep 21, 2026

    feat(app): #1302 — unpolluted instruments: per-id paint stamps + the …

    388dbb2

  3. aarontrowbridge commented on Oct 1, 2026

    @aarontrowbridge
    aarontrowbridge
    on Oct 1, 2026
    MemberAuthor

    The macbook shard is merged, verified, and the fork path is closed — 2026-10-01.

    The merge (per the fleet skill's recovery order, scripted at ~/.amico/ops/shard-merge-20261001/shard_merge.py, dry-run first):

    • Collision pass over all 8 row-bearing tables: 85 colliding session ids → 23 byte-identical (skipped), 59 canon-newer (kept canonical — stale macbook copies), 3 shard-newer → 3 session + 3 message + 1 part rows advanced (session.directory never overwritten on collision).
    • Divergent inserts: 80 sessions, 6,949 messages, 26,983 parts, 88 todos, 80 event_sequences, 99,188 position-free events (the divergent positions stayed in the archive, per the recipe), 15 project_directory rows. Re-home: 81 directories mapped per-repo (/Users/aaron/armonia → /home/aaron/armonia, /private/tmp → /tmp) — provenance preserved, panel-listable.
    • Verify: PRAGMA quick_check ok, zero shard sessions missing, zero orphan messages, backup taken pre-merge (opencode.db.backup-20261001-macbook-shard-merge), live /session API 200 with the merged sessions on page 1.

    The respawn path: the macbook's local DB archived (renamed .archived-20261001-shard-merged, never deleted — DB + -shm + -wal). The nightly guard now works end-to-end: the deployed shard-watch censused the macbook for real (165 client / 80 missing, escalated: true to #fleet via the amico-slack convention — the first divergence the watch ever caught was this one, on its first live run), and any future local fork regrows from an empty DB and gets caught within a night.

    AC status: detection alert ✓ (shard-watch, #1306 + #1558 census fix), per-client counts + nonzero-on-divergence ✓, zero-collision merge + re-home + quick_check ✓, no client launchd agent starts a local serve ✓ (mini retired 2026-09-20; the macbook never carried one). One note: the merge ritual script is an ops-dir asset, not yet a repo verb — scripted-merge-as-amico sessions merge remains a good future slice.

  4. aarontrowbridge
    closed this as completedon Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    hitlNeeds human review before merge

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions