Skip to content

Repeatable dictionary migration: prove external setup against the retained working baseline #211

Closed

Description

@holden

Outcome

Move the working dictionary installation to the owner's external 2 TB drive by building and using the repeatable installation, transfer and verification process itself. Deliver both a usable external installation and a documented procedure for a future machine/server. This is one combined implementation, not a manual move followed by a separate bootstrap project.

Combines the practical execution of #209 (storage/worktree/database migration) and #130 (fresh-clone bundle, bootstrap and readiness checks). Those issues remain requirement sources; this issue owns the shared implementation and evidence. Do not close either automatically.

The current installation is the baseline. Preserve it and a verified recovery copy until the destination is accepted. Compare old and new at the same code revision, schema and captured data state. A page that merely loads, matching row counts, or a successful command exit is insufficient.

Entry checkpoint and boundaries

1. Inventory and record an executable plan

  • Inventory source/Git history, worktree registrations, unpushed/detached commits, dirty/untracked and necessary ignored files, source archives, media, models, builds/caches, databases, audit evidence, project-specific agent state and saved paths. Record size, owner, destination, preservation method and verification.
  • Account for runtime benchmark databases and routing recovery artifacts; no database is disposable solely because of its name. Models/runtime assets are already external: reuse verified artifacts rather than download duplicates. Reconcile the runtime's database/cluster authority binding deliberately after restore.
  • Verify external volume identity, filesystem suitability, permissions, available space and measured I/O. A directory at the expected mount path is not proof of the intended mounted drive. Measure current internal headroom and all temporary-space requirements; write bulk transfer artifacts directly externally where safe.
  • Detect shared PostgreSQL resources. Prefer a dedicated external dictionary cluster when moving a shared cluster would affect other projects. Record required PostgreSQL major/tool versions, extensions, encoding, locale/collation, ownership and grants; pin the application toolchain.
  • Produce a path/configuration map covering source, database, models, archives, worktrees, caches, logs and temporary files. Paths must be configurable rather than embedded throughout scripts.
  • Coordinate active agents/processes before changing their paths. Use supported Git and agent-app mechanisms; do not edit live shared agent databases. Record explicit measured exceptions for shared metadata that cannot be relocated independently.

2. Implement the reusable transfer/setup interface

Implement or adapt the bundle, bootstrap and doctor capabilities proposed in #130. Names may follow existing conventions; document the actual supported commands rather than presenting proposed commands as available.

Bundle/export: versioned manifest with code revision, schema/migration state, tool compatibility, file sizes and cryptographic digests; consistent database dump(s); required source/replay/media inputs; and inventory of optional pinned model artifacts. Carry exact pinned inputs that cannot be re-downloaded from rolling URLs. A manifest must bind verification metadata to the actual dump and captured data state.

Bootstrap/restore: explicit destination and mode; verify prerequisites, mount and bundle before mutation; restore into a verified separate empty database; install/build application dependencies on the destination; configure paths and run readiness checks. Reject source aliases and unknown/mismatched/ambiguous target identities. Never silently overwrite an existing installation. Resume interrupted transfers and safely repeat setup without duplicate writes or destructive re-restoration. Existing matching state is verified; conflicting state is reported clearly.

Doctor: read-only machine-readable and human-readable checks for tool versions, destination identity, database/extension/schema compatibility, required files/checksums, configuration, build, endpoints, background-job mode and optional model readiness. Distinguish usable core operation from missing optional provider credentials/model service. No model generation or background imports as a side effect.

Separate credentials from transferable artifacts. Treat a full working database backup as private, regardless of whether .env is excluded. A future public starter bundle needs an explicit data/rights/privacy scope; do not publish the working backup. Operator credentials and grants must be transferred/recreated deliberately through private configuration.

Keep three explicit modes: restore this exact installation; initialize a new development installation from an approved bundle; rebuild from pinned source inputs. Only the first is the migration path. A source rebuild cannot replace restoration of durable identities/editorial state.

3. Establish the comparison baseline

  • Record the consolidated code revision, actual source schema/migration state and relevant feature flags/configuration. Do not apply new migrations to the source merely to simplify relocation.
  • Define the exact comparison contract before moving: table contents keyed by durable identity, references, sequence state, routing decisions/history/paths, curation records/approvals, source manifests and required file hashes. Compare intended role/grant mappings and extensions as well. Exclusions must be narrow, documented and justified; do not hide semantic differences as bookkeeping.
  • Capture representative application behavior on the source: ordinary word page, sourced quotation/artwork page, entity page, search, enabled curated opening, routing lookup where applicable, and relevant authenticated behavior with a controlled test account. Record expected identities, links, attribution, feature availability and errors; screenshots alone are not acceptance.
  • Quiesce writers through their owning services and capture a consistent, immutable baseline. Keep restored jobs, cron, discovery and inference disabled during comparisons. Read-only access to the old installation is allowed only with writers reliably disabled.
  • If an early rehearsal occurs while source writes continue, record its snapshot boundary; before cutover, quiesce and capture/restore a final consistent snapshot. Do not compare a stale snapshot against a changing source and call it exact.

4. Create and verify the external installation

  • Use the documented reusable commands to create the destination; every manual workaround discovered must become an explicit step, configuration input or automated check.
  • Preserve source/worktree history and local work through supported relocation; prove a fresh checkout can also use the setup process without undocumented dependencies on the old checkout. Build artifacts must be produced for the destination platform, not assumed portable to a future Linux server.
  • Use supported PostgreSQL logical backup/restore and isolated endpoints. Never copy individual files out of a running database cluster. Handle cluster-level roles separately without moving unrelated databases.
  • First prove exact restoration at the captured schema. Any migrations needed by the consolidated code are a separate, recorded step after preservation passes; test them on the destination with their expected effects specified.
  • Run exact baseline comparisons, doctor, representative UI/API reads and file/source access. Exercise writes and jobs on an isolated validation database or controlled destination state; never contaminate the baseline comparison or trigger duplicate external work.
  • Run focused integration tests and mix precommit from the external checkout using an external isolated test database. Report capacity-blocked/unrun checks honestly.

5. Prove repeatability and failure behavior

  • Repeat setup from the same code and bundle into a second clean external target, or a documented disposable target, using only the quickstart. No hidden access to the old installation. Verify equivalent preserved state and functionality.
  • Re-run setup on an initialized target: no duplicate data, lost state or silent restore. Demonstrate an interrupted transfer/setup can resume safely.
  • Reject corrupted/missing bundle parts, incompatible tools, incorrect mount, insufficient capacity, source/destination aliasing and an already-populated conflicting target before destructive work.
  • Verify service restart and missing/wrong/unwritable-drive behavior without unplugging a running database. No fallback directories, database, models or bulk caches may be silently created internally.
  • Verify a resumed agent session and a newly created task/worktree use intended paths. Keep unrelated projects operational.
  • Measure representative query/build/test performance and storage growth; document any material regression and acceptance decision. Choose numeric thresholds from the measured baseline before judging results.

6. Cut over, retain rollback, then reclaim space

  • Permit only one intended writable dictionary installation. Update all clients, worktrees, scripts, launchers and jobs; verify their actual endpoints before enabling writes.
  • Preserve the old installation unchanged through destination acceptance. Record a rollback procedure and acceptance window. Once destination writes occur, switching to the old snapshot alone loses those writes: preserve/reconcile them before rollback.
  • After successful verification/cutover, reclaim old internal bulk copies through owning tools and recorded dispositions. Never delete in-use worktrees or database files manually. Retain verified baseline/recovery artifacts externally for later comparisons, with an independent backup destination for drive-failure recovery; keeping all old internal files forever would not meet the disk-relief goal.
  • Report measured reclaimed space, remaining internal exceptions and an operating-headroom target. Verify new routine work continues to write externally.

Deliverables and acceptance

Future server deployment reuses the data manifest, restore, configuration and verification contracts. Server-specific release builds, supervision, TLS/networking and inference hosting remain explicit future work; the external-drive rehearsal is not proof that a Linux production deployment has been tested.

Activity

  1. holden commented on Sep 27, 2026

    @holden
    MemberAuthor

    Readiness audit, 27 September 2026: three live blockers, and the disk picture

    Companion to the routing audit on #194: #194 (comment). Read-only, measured on the machine today; nothing was relocated, deleted, stopped or rolled back.

    Entry checkpoint is stale

    The consolidation checkpoint handoff (697ac8a) holds main c6bd882 + #202 + #210 + #208 at 2d495e8. Since then main moved to 2a851e8 (#215), #208 moved three commits to 7860b9f, and #216 (stacked on #208) did not exist. Once #208 and #216 merge (order and preconditions on #194), the integration branch only needs #202 and #210, and the checkpoint should be re-cut against that main.

    Blockers to the move that the plan does not list

    1. The 4007 development server is SIGSTOPped, not stopped. beam.smp 13180 has been in state T for two and a half days. It holds idle connections to devils_dictionary_v2 (22 idle connections on the database in total, the oldest from that server), and ngrok http --subdomain=wordhoard 4007 is still running against it. It needs fg in its own terminal and a clean stop, and the tunnel closed, before quiescence means anything. The Curated opening section, Phase 1: manual fixture, Bierce-first lead, sourced highlights (#156) #202 preview on 4017 (beam.smp 32882) also holds connections.
    2. devils_dictionary_v2 still carries Curation runtime stage A: one private local model behind one slot (#195) #210's two unmerged migrations (20260927114948, 20260927131023, applied 13:11Z in the incident). The three main migrations above 20260924222346 are now legitimately main's schema (Routing Stage 1: durable pages, path ledger and resolver (#194) #205, Curation persistence slice 1: default configuration, manual compositions and publication authority (#196, #201) #206 merged). Recommendation: roll back only the two, from the consolidation branch, with mix ecto.rollback --to 20260927094256, so the source schema equals main's exactly and no unmerged branch is a dependency of the baseline. The full five-migration rollback the incident comment rehearsed remains the alternative. Owner's decision.
    3. A session is live in the Stage 2: resumable routing backfill, rehearsed on copies of the corpus (#194) #216 worktree (dictionary-issue-194-stage-1-218d99) with uncommitted edits answering CodeRabbit. It must commit and push before the worktree inventory is final.

    The disk is the emergency

    Where Measured
    Internal disk free 5 GiB of 460
    Postgres cluster on 5432 (Postgres.app var-18, internal, shared) 111 GB
    of which dictionary databases about 26 GB: devils_dictionary_v2 14 GB, devils_dictionary_bing135 4.2 GB, devils_dictionary_74_verify 4.2 GB, devils_dictionary_dev 3.3 GB, about twenty devils_dictionary_test* partitions 0.6 GB
    of which other projects about 83 GB: cinegraph_dev 45, volfefe_machine_dev 17, argus_dev 16, eventasaurus_dev 5
    data/ (wiktextract 2.8 GB, not re-downloadable) 2.9 GB
    30 dictionary worktrees about 10 GB
    of which on branches already merged to main, zero commits ahead 6.8 GB across 21 worktrees; 20 have no dirty files, claude/dictionary-word-page-grouping-2c67e4 has two, one (claude/158-build1) is locked
    External /Volumes/LLM Models free 1.5 TiB
    of which dictionary-stage2a scratch cluster on 5433 233 GB, 22 rehearsal databases of 9–13 GB each, plus 2.3 GB of dumps

    Two conclusions for the plan:

    • Moving the dictionary frees roughly 40 GB, not the drive. Three quarters of the internal cluster is other projects, which this issue rightly protects. The internal disk stays tight after the move unless those get their own decision; worth saying so in the acceptance criteria rather than discovering it at the reclaim step.
    • Immediate, reversible headroom exists before any move: git worktree remove on the 20 clean merged worktrees (branches stay), and dropping the finished devils_dictionary_test* partitions. Both need the owner's word under Stage 1's "a merged PR does not by itself make a checkout disposable". The three retained databases (bing135, 74_verify, dev, 12 GB together) are the larger internal win and need an explicit disposition; config/dev.exs still names devils_dictionary_dev as the Gate 0 baseline.
    • The 22 scratch copies on 5433 are external and harmless to the internal disk, but the inventory must give them a disposition too; they are the only copies of the Stage 2 rehearsal states.

    Recommended order

    1. Merge Stage 2 recovery: rehearse on the development corpus, then repair and re-run (#194) #208 and Stage 2: resumable routing backfill, rehearsed on copies of the corpus (#194) #216 as set out on Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194 (two fixes first).
    2. Reclaim the clean merged worktrees and test partitions with the owner's approval.
    3. Unfreeze and stop 4007, close the tunnel, stop 4017; roll back Curation runtime stage A: one private local model behind one slot (#195) #210's two migrations; re-cut the consolidation checkpoint.
    4. Execute this issue. The corpus catch-up, the population, the reviewers and the persistent backfill all wait for the external installation.
  2. holden commented on Sep 27, 2026

    @holden
    MemberAuthor

    The pre-migration work is now specified in #219. Its CP3 is this issue's entry checkpoint: accepted main, at most one named integration branch, the three live blockers cleared or named, and a handoff comment here with the code and schema baseline.

  3. holden commented on Sep 28, 2026

    @holden
    MemberAuthor

    Handoff from #219 (28 September)

    The code: main at 16e35ca. The code is identical to 04f97bc; only docs merged after it. Merged under #219 with merge commits, branches kept, validated after each merge:

    #208 484d21e
    #216 903f82b
    #220 64f12cd
    #202 e1d60b0, on the owner's decision of 28 September
    #225, the On reader 04f97bc
    #226, docs 16e35ca

    #210 had merged before, as d31d212. The final precommit gave 19 doctests, 2,540 tests, 0 failures. The procedure and the full inventories are in docs/operations/consolidation-checkpoint-2026-09-28.md.

    Schema baseline. devils_dictionary_v2 is at 20260927131023 (24 migrations; read-only check on 28 September). main expects exactly two more, applied only after preservation passes:

    1. 20260927200717 (Stage 2: resumable routing backfill, rehearsed on copies of the corpus (#194) #216): routing_backfill_runs and routing_backfill_items.
    2. 20260927220937 (Every database carries the merged local_model_configs_shape, whichever draft it ran (#195, #219) #220): re-states local_model_configs_shape, which the 13:11Z incident left in its early-draft form. Check SELECT count(*) FROM local_model_configs first. It was 0 on 28 September, and the table is append-only.

    A schema-only copy of v2 on the scratch cluster was migrated through these two. The result equals a database migrated from zero (26 migrations), apart from whitespace in function bodies.

    Take a fresh capture. The rehearsal copies come from c1.dump (27 September, 13:29Z). v2 has been written since then, and the 4007 server is still connected to it. The move preserves v2 as it is once quiesced, not c1.

    Blocker before #211 starts: quiescence (D4), the owner's terminals. Still running at 06:10Z on 28 September:

    Then:

    1. Fast-forward the main checkout's local main, which is 66 commits behind: git -C ~/Code/projects-2026/dictionary pull --ff-only. It was left alone because a dev server is running on it against v2.
    2. Do not start that dev server against v2 again before preservation. On the new main, Phoenix's development page for pending migrations would offer to apply the two migrations to v2.
    3. This session stops its own two demo servers (4219, 4220). They use the scratch cluster only.

    The procedure. It is the 27 September one, with these changes:

    Preserved outstanding work:

    Blockers cleared:

    Reclaim. On 27 September nineteen idle sessions were archived; their directories are the owner's to remove. On 28 September 16 more worktrees (3.6 GB) were confirmed removable: no process, nothing untracked, all content on main. The #219 closing comment lists them with the commands. This session removed none, because its permission mode blocks git worktree remove. No database was dropped. Both clusters are inventoried, with proposed dispositions that need separate authorization per database and server.

    After the move: CP4, #224. The owner's decisions (catch-up, population, reviewers), the backfill with real reviews on the working database, and namespace addresses for the everyday reader.

  4. holden commented on Sep 28, 2026

    @holden
    MemberAuthor

    The handoff above was independently checked on 28 September (audit on #219): precommit on 16e35ca reproduced (2,540 tests, 0 failures), the B2 reader table reproduced over HTTP from main in both modes on the retained demo copy, and the migrated-v2 schema equals the fresh reference apart from whitespace. Entry checkpoint: accepted main 16e35ca; the one integration branch is #223 (stacked on #218, leave both unmerged until the move is accepted, since #218 adds a migration). Still before this issue starts, all the owner's: stop the SIGSTOPped 4007 server (PID 13180), the ngrok tunnel (14391) and the 4017 preview (32882); fast-forward the main checkout; remove the 32 clean worktrees. Then take the fresh capture.

  5. holden commented on Sep 28, 2026

    @holden
    MemberAuthor

    D4 done, 28 September 09:12 (+02). The owner stopped the ngrok tunnel (14391) and the 4017 preview (32882); the SIGSTOPped 4007 server (13180) could not take a TERM while stopped by job control, so it was killed with SIGKILL after checking its one open transaction was an empty BEGIN from the moment it resumed. Verified: no process on 4007, zero connections to devils_dictionary_v2, no beam.smp or ngrok left. The source is quiet; a fresh capture may be taken once D5 (fast-forward the main checkout) and D2 (worktree removal) are done.

  6. holden commented on Oct 5, 2026

    @holden
    MemberAuthor

    D2 done, 5 October. 17 dictionary worktrees removed with per-item checks (merged, no process, nothing dirty beyond the data symlink), merged branches deleted, internal disk now 13 GiB free (from 7). #219 is closed. The entry checkpoint holds and nothing on devils_dictionary_v2 has changed since 28 September (schema 20260927131023, zero connections). This issue may begin with a fresh capture.

  7. holden commented on Oct 5, 2026

    @holden
    MemberAuthor

    Entry checkpoint verified live, 5 October 2026 14:30 (+02): ready to start

    Read-only checks on the Mac Studio this afternoon, against this issue's entry conditions and the 28 September handoff.

    Condition Verified
    Accepted main 16e35ca; the main checkout is on it, equal to origin/main, clean. Open PRs: #218 and #223 only, the one integration branch, left unmerged on purpose (#218 adds a migration)
    Source quiet devils_dictionary_v2: 0 connections; no beam.smp, ngrok or ollama serve process; nothing on 4007. Schema head 20260927131023, unchanged since 28 September
    Source characterised 14 GB; cluster system_identifier 7607810074859095446 (Postgres.app 18.2, port 5432, shared with other projects); UTF8; collate/ctype en_US.UTF-8 under the ICU provider; extensions citext 1.8, pg_trgm 1.6; owner postgres; login roles postgres and holden, both superuser
    Destination /Volumes/LLM Models = /dev/disk5s1, APFS, PCI-Express, Device Location: External, SSD, 1,453 GiB free. The rehearsal cluster already on it (dictionary-stage2a/pgdata, port 5433, system_identifier 7690164109148229279, same 18.2 binaries) holds 36 databases, 289 GB
    Tooling pg_dump/pg_restore 18.2 (Postgres.app), Elixir 1.19.0 on OTP 28, rsync, shasum
    Inputs git does not carry .env (931 B); data/english-wordnet-2025-plus-json.zip (11 MB); data/raw-wiktextract-data.jsonl.gz (2.83 GB, not re-downloadable); priv/replay/ (97 MB: six archives plus MANIFEST.json); priv/sources/MANIFEST.json pins
    Retained dumps, external dictionary-stage2a/dumps/v2-source.dump (27 Sep 10:49Z, pre-routing) and baseline-routing.dump, each with its routing sidecar; the Stage 2 rehearsal dumps under dictionary-stage2a/stage2r/dumps/. None is the capture: it must be fresh
    Internal headroom 13 GiB free after D2
    Worktrees 11 registered; none has a database writer; #219 closed

    Two things the implementer decides and presents in stage 1, not assumes: whether the destination is a new dedicated external cluster (this issue's preference) or the existing 5433 rehearsal cluster, with the same encoding, ICU locale, extensions and roles either way; and the comparison contract's treatment of collation version, since the source uses the ICU provider.

    Kickoff prompt

    Work in razrfly/dictionary on issue #211, stages 1 and 2 only: the inventory with an executable plan, and the reusable bundle, bootstrap and doctor interface. Stop for the owner's review before any capture, restore or cutover. Read #211 in full, the 28 September handoff (issuecomment-5864603525), the readiness check above it (5 October), docs/operations/consolidation-checkpoint-2026-09-28.md, docs/routing/recovery.md, and #209 and #130 as requirement sources. The readiness facts are verified; do not re-derive them, but re-check the source is still quiet (pg_stat_activity) before anything else.
    Baseline: main 16e35ca; devils_dictionary_v2 at schema 20260927131023, 14 GB, ICU en_US.UTF-8, citext and pg_trgm; main expects exactly two migrations after preservation (20260927200717, 20260927220937), applied only as a recorded step after the exact restore passes. Destination: /Volumes/LLM Models (/dev/disk5s1, external, 1.4 TiB free).
    Stage 1 deliverables: the inventory table (source, Git worktrees and unpushed commits, databases on both clusters with identity and disposition, archives, models, caches, audit evidence, agent state, saved paths), measured sizes and I/O, the path and configuration map with paths configurable rather than embedded, the destination decision (new dedicated external cluster versus the 5433 rehearsal cluster), the comparison contract (tables keyed by durable identity, references, sequences, routing and curation state, manifests and file hashes, roles and grants, extensions, collation version), the quiescence procedure, the rollback procedure, and the exact commands as they will exist.
    Stage 2 deliverables: bundle, bootstrap/restore and doctor as documented mix tasks or scripts, with the manifest bound to the dump by size and SHA-256, three explicit modes (restore this installation; initialise from a bundle; rebuild from pinned inputs), refusal of source aliases and populated or mismatched targets, resumable transfers, and tests on a private MIX_TEST_PARTITION.
    Constraints: never write to devils_dictionary_v2; every mix ecto.* and mix test carries MIX_ENV=test and a private partition; never use mix dd.snapshot --restore as the transfer and never copy live cluster files; bulk artifacts go straight to the external volume; do not start a dev server against v2; leave #218 and #223 unmerged; mix precommit before any PR; evidence stays private under data/ or the external volume, never in issues with secrets.
    Finish with: the stage 1 record, the stage 2 PR, the decisions the owner must make before the capture, and an honest statement of what was measured versus assumed.

  8. holden commented on Oct 5, 2026

    @holden
    MemberAuthor

    Stages 1 and 2, 5 October: the plan and the interface are ready for review; nothing has been captured

    Stage 2 PR: #227. It adds:

    • mix dd.bundle, which captures, verifies and transfers;
    • mix dd.bootstrap.cluster;
    • mix dd.bootstrap --mode restore|init|rebuild;
    • the read-only mix dd.doctor.

    Runbook: docs/operations/installation.md. mix precommit on a private partition: 19 doctests, 2,572 tests, 0 failures (32 new).

    Stage 1 record: docs/operations/211-stage-1.md. It holds the full inventory with dispositions, the measured I/O, the path and configuration map, the comparison contract, the quiescence and rollback procedures, and the exact commands. The raw inventories are private, under /Volumes/LLM Models/dictionary/private/2026-10-05-211-stage1/.

    The source is still quiet. At 14:29 (+02) no other session was connected to devils_dictionary_v2. Its write counters were identical at 12:32Z and 13:00Z, either side of a read-only doctor run. The doctor reports it usable at 20260927131023, with exactly the two expected pending migrations.

    One finding changes the procedure. Every devils_dictionary* database has TimeZone=Etc/UTC as a database-level setting; ecto_sql's storage_up sets it, and the server's default is Europe/Warsaw. pg_dump without --create does not carry it. A plain createdb and pg_restore would therefore read every timestamp in local time. The contract compares it, and the restore applies it.

    Other findings:

    • 5433 has no holden role.
    • config/test.exs sets no port, so tests stay on 5432 until it does.
    • Two agent processes have a dictionary checkout as their working directory, though neither is connected to the database: a Claude session in stage-2-routing-audit-372bbd, and Codex in the main checkout.

    Decisions owed before the capture

    # Decision Recommendation
    D1 Destination a new dedicated cluster, /Volumes/LLM Models/dictionary/postgres/18/data, port 5434. Not the 5433 rehearsal cluster
    D2 How the new cluster starts after a reboot a Postgres.app server entry, the owner's to add
    D3 wal_sync_method on the external drive fsync_writethrough: 252 commits/s measured, against 7,973/s for open_datasync, which is not flush-safe on macOS
    D4 The new cluster's resources match 5432 for the move
    D5 Authentication trust on localhost for the move, as today
    D6 Which databases move restore v2 and runtime_bench/_b. Archive dev, 74_verify and bing135 as bundles. Keep ex212 with #212. Drop the rest only if authorized
    D7 Bundle contents the inputs and the replay archive: yes. Models: inventory only
    D8 The checkout and agent state a fresh external clone. Fetch the 9 unpushed commits on 7 branches, keep the Codex b0a9 edits as a patch, and copy the Claude memory to the new path's key
    D9 The quiet window the two live agent processes confirm they stay off v2
    D10 Review #227 before it is used on the corpus —
    D11 Where tests and the dev server point after cutover DD_DATABASE_PORT in config/test.exs, and a 5434 default, changed at cutover
    D12 The runtime binding rebind to runtime_bench on the new cluster, deliberately
    D13 One root setting for the runtime's four paths yes, as a small follow-up

    Measured, and assumed

    Measured today:

    • the source's quiet state;
    • both clusters: identities, settings and roles;
    • every database: size, oid, locale, collation version, settings and schema head;
    • worktrees and unpushed commits;
    • the destination's identity and I/O;
    • the Wiktionary input's digest.

    Exercised, on disposable databases only: the whole stage 2 interface. That includes restore onto a throwaway cluster made by dd.bootstrap.cluster. An independent review of its destructive paths found no way to touch the source or a finished bundle. The ten resume and ordering defects it found are fixed in #227.

    Assumed:

    • the capture, restore and verify times at corpus scale. These come from Stage 2A, not from this interface;
    • that a dump keeping owners and privileges restores an equal schema at corpus scale;
    • that Postgres.app accepts this data directory;
    • that the drive honours F_FULLFSYNC.

    Stopping here for the owner's review. #218 and #223 remain unmerged.

  9. holden commented on Oct 5, 2026

    @holden
    MemberAuthor

    Stages 1–2 reviewed (A−, on #227); recommendations on D1–D13

    The owner decides. These are the reviewer's recommendations, each with its reason.

    # Recommendation Why
    D1 New dedicated cluster on 5434 at /Volumes/LLM Models/dictionary/postgres/18/data a distinct system_identifier keeps the guards and the runtime authority unambiguous; rehearsal drops never happen beside the working database; 5433 lacks holden
    D2 A Postgres.app server entry, added by the owner after stage 3 proves the data directory a cluster started by pg_ctl from a terminal is what left 5433 with no launcher
    D3 fsync_writethrough 252 commits/s is far above this application's write rate; open_datasync does not reach the drive's stable storage on macOS, and a pulled cable is the realistic failure for an external disk
    D4 Match 5432's resources for the move compare like with like at stage 4; tune afterwards under #211's performance step
    D5 trust on localhost, as today the move changes no security boundary; a later change is its own decision
    D6 Restore v2 and both runtime_bench databases; bundle dev, 74_verify and bing135 as archives; keep ex212 until #212 lands; drop ex181, ex181b2 and the test partitions only on an explicit list config/dev.exs still names dev as the Gate 0 baseline, so its bundle must verify before anything is dropped
    D7 Inputs and the replay archive in the bundle; models inventoried only the Wiktionary dump cannot be re-downloaded; the weights are already external
    D8 A fresh external clone; fetch the 9 unpushed commits (7 branches) into it; keep the Codex b0a9 edits as a patch; copy the Claude project state to the new path's key the memory directory is keyed by checkout path, so without the copy every session starts blind at the new path
    D9 Confirmed for this session: it connects to v2 only through read-only psql and the doctor, and will not during the capture the other process is Codex in the main checkout; it needs the same confirmation
    D10 Done: #227 reviewed A−; fix its three open threads before the first corpus run the cluster-left-running and list-valued-settings findings are operational
    D11 DD_DATABASE_PORT honoured in config/test.exs, and the dev default switched to 5434 at cutover, not before tests and the dev server must not reach the new cluster while it is still under comparison
    D12 Rebind the model service to runtime_bench on the new cluster, after the exact restore passes the authority file names cluster 7607810074859095446 today
    D13 Yes, as the small follow-up it is four absolute paths in two places

    Then stage 3. The first corpus-scale rehearsal must measure the four assumptions the record declares: capture, restore and verify times; schema equality with owners and privileges at 14 GB; Postgres.app accepting the data directory; F_FULLFSYNC honoured by the drive.

  10. holden commented on Oct 5, 2026

    @holden
    MemberAuthor

    How far from done, and the prompt for the next session (5 October)

    Stage (#211) State Remaining effort
    1 Inventory and plan done, reviewed A− re-cut the worktree table in the stage 3 record
    2 Bundle, bootstrap, doctor done, reviewed A− three small fixes on #227, then merge
    3 Baseline and capture not started; the tooling exists about half a day: capture, a disposable restore to measure the four assumptions
    4 External installation not started; needs D1–D13 about a day: cluster, restore, verify, migrations as a recorded step, external clone, precommit from there
    5 Repeatability and failure drills not started about a day
    6 Cutover, rollback window, reclaim not started; the owner's go a day plus the acceptance window

    Roughly a third of the issue is done, with the hard thinking behind it. Three to four working days of agent time remain, interleaved with the owner's decisions and the acceptance window. Nothing has moved yet: the source is unchanged, no bundle has been taken, no new cluster exists.

    Prompt for the next session

    Work in razrfly/dictionary on issue #211, picking up after the stage 1–2 review (PR #227 reviewed A− on 5 October; D1–D13 recommendations posted the same day). Read #211's 5 October comments in order (readiness check, stage 1 record, the #227 review, the D1–D13 recommendations), docs/operations/211-stage-1.md and docs/operations/installation.md on #227's branch, and the 28 September handoff. Before anything else, re-check that the source is quiet (pg_stat_activity on devils_dictionary_v2 empty) and that main is still 16e35ca.

    Phase A, no owner decision needed.

    1. Bring #211 stages 1–2: the inventory and plan, and the bundle, bootstrap and doctor interface #227 to A: resolve its three open CodeRabbit threads (stop the cluster when role setup fails after pg_ctl start; re-apply list-valued settings entry by entry; the rename-then-comment order in installation.md), each with a regression test; re-run the installation tests and mix precommit on a private partition; merge #211 stages 1–2: the inventory and plan, and the bundle, bootstrap and doctor interface #227 with a merge commit, branch kept.
    2. Take the fresh capture (stage 3): mix dd.bundle of devils_dictionary_v2 into /Volumes/LLM Models/dictionary/bundles/<date>-v2 with --volume "/Volumes/LLM Models" --volume-uuid F7FDE75A-3FE9-43D9-AC1E-71FDDEDBAF31, inputs and replay archive included, models inventoried, quiet required. Post the manifest digest on Repeatable dictionary migration: prove external setup against the retained working baseline #211 as the baseline candidate. Also bundle runtime_bench, runtime_bench_b, dev, 74_verify and bing135 as archives (the D6 recommendation; nothing is dropped).
    3. Measure the four declared assumptions on a disposable cluster: mix dd.bootstrap.cluster on a scratch data directory on the external volume (not the recommended final path), mix dd.bootstrap --mode restore of the v2 bundle into it with --expect-manifest-sha256, the contract comparison, then mix dd.routing.verify against the live source as the second check; time every step at corpus scale; confirm Postgres.app opens that data directory; confirm F_FULLFSYNC with pg_test_fsync there. Then stop and remove the throwaway cluster, recording its system_identifier so it can never be confused with the real one.
    4. Post the stage 3 record on Repeatable dictionary migration: prove external setup against the retained working baseline #211: timings, the exact comparison result, the re-cut worktree table, and every measurement that differs from the stage 1 assumptions.

    Phase B, only after the owner has ruled on D1–D13 on #211. Use the posted recommendations only where the owner says so.
    5. Stage 4: create the dedicated cluster at the decided path and port with the decided settings (D1–D5); restore v2 and the runtime_bench pair (D6) from the approved bundle and verify exactly; apply the two pending migrations as a separate recorded step and verify their expected effect; make the fresh external clone (D8), fetching the 9 unpushed commits, keeping the Codex b0a9 edits as a patch and copying the Claude project state to the new path's key; deps, build and mix precommit from the external checkout on an external test database; mix dd.doctor --expect-cluster … --volume …; representative reads and writes on an isolated validation database only.
    6. Stage 5: repeat setup into a second clean external target from the bundle alone; re-run on the initialised target (no duplicate, no silent restore); interrupt a transfer and a restore and resume both; reject a corrupted bundle part, a wrong mount, insufficient capacity and a populated target; measure query, build and test performance against the stage 1 figures.
    7. Stop before stage 6. Post the cutover run sheet with the rollback window for the owner's go.

    Constraints: never write to devils_dictionary_v2 beyond the read-only capture; every mix ecto.* and mix test carries MIX_ENV=test and a private partition; no dev server against v2; nothing deleted on 5432 or 5433 without the owner naming the exact database; #218 and #223 stay unmerged; the 9 unpushed commits on 7 local branches are preserved by fetching them into the new clone and are not merged to main as part of the move; bulk artifacts only on the external volume; evidence private under /Volumes/LLM Models/dictionary/private/; secrets never in issues; the corpus catch-up, the population, the reviewers and CP4 (#224) remain after the move.

    Finish each phase with what was measured versus assumed, what changed on disk, and what the owner must decide next.

  11. added 2 commits that reference this issue on Oct 5, 2026
  12. holden commented on Oct 5, 2026

    @holden
    MemberAuthor

    Stage 3, 5 October: the capture is taken, and the four assumptions are measured at corpus scale

    This is Phase A of the next-session prompt. Phase B waits for your ruling on D1–D13.

    Kept unchanged:

    Quiet throughout:

    • At the start, at 16:41:57 (+02): devils_dictionary_v2 had 0 sessions, and main was 16e35ca.
    • At the end, at 21:42 (+02): 0 sessions. The write counters were unchanged all day, at 37,301,248 / 61,342,633 / 8,670,285, and the schema stays 20260927131023 (24 migrations).

    The private evidence is under /Volumes/LLM Models/dictionary/private/2026-10-05-211-stage3/. session-readings.txt there records the readings that exist only in the session's output, and every exact command.

    #227: its three threads closed, then merged

    The three CodeRabbit threads are fixed in 78ed132, each with a regression test, and resolved:

    • List settings come back element by element, as pg_dump does.
    • Any failure after pg_ctl start stops the new server.
    • The runbook says rename, then comment, and a test holds it to the code.

    An independent multi-lens review of those fixes, with two skeptics per finding, raised four findings:

    • Upheld: one, a contradictory ICU message. Fixed.
    • Refuted: three, covering two distinct issues: resuming after a stopped setup (raised twice), and raw FROM CURRENT list settings. The stop message now says what to do. The FROM CURRENT limit is documented in the runbook.

    Two interface additions (ada0948), so that stages 3–4 run on the tools:

    • mix dd.bundle --root reads the main checkout's data/ and replay archive. It records that checkout's revision as code, beside the tool's own as tool.
    • mix dd.bootstrap.cluster --setting key=value writes each setting with ALTER SYSTEM, restarts when one needs it, and reads every value back from pg_file_settings.

    These two additions had no independent review before the merge. The adversarial review that has run on them since upheld five defects in --setting:

    • lists written as one name;
    • pg_file_settings.applied trusted as "running";
    • mixed-case names crashing;
    • a failed restart misreported;
    • duplicates found only after initdb.

    None is triggered by the D3/D4 values used below, so stage 3's measurements stand. All five are fixed, each with a test, in #228 (precommit 19 doctests, 2,580 tests, 0 failures), open for review. Phase B should use --setting only after #228 is merged.

    Checks:

    • MIX_TEST_PARTITION=_b211 mix precommit on the final tree: 19 doctests, 2,578 tests, 0 failures. No source file changed between that run and the commits.
    • The installation suite, re-run on the merged b69bc3e: 38 tests, 0 failures. b69bc3e's tree is identical to ada0948's.

    Merged as b69bc3e, with a merge commit and the branch kept. CodeRabbit's status was still pending at the merge; its confirmations on the three threads came after.

    The baseline candidate

    Approve one digest or more. Every later step pins one with --expect-manifest-sha256.

    All six bundles have kind installation. "Archive" below is only their D6 disposition. Each holds a dump and its state; only v2 carries the inputs.

    Bundle MANIFEST.json SHA-256 Contents Capture time
    v2 5d36203469089f05bee510eb35c6ef126fd5e8ca2adf7384b3674a5bc98e63ff 11 files, 4,197,797,670 bytes 76 s, 83 s with the inputs
    runtime_bench (restore, D6) 2ac42dcdf6d4b5061a4190c312ab3545b15b4de3339f9283f73764ba6e77a5b2 2 files, 768,144 bytes 1.3 s
    runtime_bench_b (restore, D6) 69b36af8722b482f37e6bc54873ea7fafe12f2c63ad56a50752dca6a7ef03014 2 files, 745,847 bytes 1.1 s
    dev (archive) 3993f1cc6ce08b0e271efa8323d481334d4f47f13436ffd766e3e34c65180fb2 2 files, 232,646,982 bytes 18.6 s
    74_verify (archive) f0d6fee0c1e96d93bc97d94d01b82e7ad7a0182472ab2fe16c99468760fafdc5 2 files, 299,355,094 bytes 24.3 s
    bing135 (archive) 8e24850a7b34271973eb92e0c87430eaed2d5157b06142aede718b9237230ea1 2 files, 299,573,873 bytes 26.7 s

    Where they are: /Volumes/LLM Models/dictionary/bundles/2026-10-05-<db>/, 5,030,919,127 bytes (5.03 GB) in all.

    What holds for every one:

    • it verifies by size and SHA-256;
    • its capture window was quiet: no other session at the start or the end, and its write counters, sequences and user catalog unchanged;
    • it records the volume pin: UUID F7FDE75A-3FE9-43D9-AC1E-71FDDEDBAF31 on /dev/disk5s1.

    The v2 bundle:

    • The dump: 1,257,958,079 bytes, under snapshot 00000037-0000F3CB-1.
    • Revisions: code is 16e35ca (the main checkout); tool is b69bc3e.
    • The source: cluster 7607810074859095446, oid 31334772, 14,742,779,583 bytes.
    • Its record: setting TimeZone=Etc/UTC; cluster roles holden and postgres.
    • Inputs: 2,939,504,238 bytes of archived inputs and the replay archive, each checked against its pin before it was copied.
    • Models: an inventory of 8 files, 9,984,448,444 bytes. Not copied, and their digests are taken from the blobs' file names.

    The five archives were taken without --root, so their code is the tool's, b69bc3e. bing135 carries no TimeZone setting, as stage 1 found.

    The bundles directory is now 0700. On this noowners volume that keeps nobody out on a multi-user machine. The v2 bundle holds users and users_tokens, so it must never leave private storage.

    The four assumptions, measured on a disposable cluster

    The disposable cluster:

    • made by mix dd.bootstrap.cluster in 1.9 s;
    • data directory /Volumes/LLM Models/dictionary/scratch/211-stage3/data, port 5444;
    • system_identifier 7693266767055315945;
    • settings: wal_sync_method=fsync_writethrough, shared_buffers=4GB, max_wal_size=8GB, maintenance_work_mem=2GB, checkpoint_timeout=15min. These are 5433's resources, for comparison with the earlier figures.

    It is now stopped and removed (46 GB, measured with du). Its identifier is recorded in the evidence as never the dictionary's.

    1. Times at corpus scale. The earlier figures come from Stage 2A (docs/routing/recovery.md, Limits) and the stage 1 record.

    Step Earlier records Measured
    Capture: dump and state, under one snapshot ~80 s snapshot, then ~70 s manifest (Stage 2A) 76 s together. 83 s for the whole 4.2 GB bundle, 2.9 GB of it inputs
    mix dd.bundle --verify, every file hashed — 2.6 s
    dd.bootstrap restore or init, end to end, 4 runs — 119 s, 175 s, 139 s and 159 s
    of which verify and pre-checks — 2–3 s each
    of which pg_restore --jobs 8 76–110 s (stage 1) 57 s and 75 s under fsync_writethrough; 111 s and 96 s under open_datasync
    of which the contract comparison 73–88 s (stage 1) 59–61 s
    mix dd.routing.verify (two manifests at once, as Stage 2A) 73–88 s 62.9 s
    mix dd.doctor --deep, contract comparison included — 57 s
    Restored size 9.98 GB (stage 1) 9,977,403,071 bytes right after the restore; 9,977,861,823 at teardown

    The four runs alternated fsync_writethrough, open_datasync, fsync_writethrough, open_datasync. Run 1 started on an empty cluster. Runs 2–4 each started while the previous run's spread checkpoint was still writing. The server log warned "consider increasing max_wal_size" during the restores, even at 8 GB.

    2. An equal schema, with owners and privileges, at 14 GB. Three independent checks agree:

    • The contract comparison inside dd.bootstrap: differences: [] in every section, before the rename. That covers all 69 tables, Oban's two included, the schema, 54 sequences, routing, database properties and settings, extensions, and the referenced roles.
    • mix dd.routing.verify against the live source: "matches … exactly". That covers every table except Oban's two queue tables, which it leaves out by design (67 of 69), the 2,253 schema rows, owners and grants included, and the 54 sequences.
    • pg_dump --schema-only, source against copy: 7,801 lines each, with the 129 OWNER TO lines equal. The 37 lines that differ are all the documented array-cast re-parse: (ARRAY[…])::text[] comes back as ARRAY[(…)::text, …], 36 times under = ANY and once under <> ALL. After the canonical form Recovery.canonical/1 applies, 0 lines differ.

    The owner and privilege part passed against an easy case. The corpus has one owner (postgres) and no non-default or default privileges, so it does not test differing owners or grants; the suite's throwaway-cluster tests do. Likewise, dd.routing.verify's "every path and page resolves the same" compares empty sets, because the corpus holds 0 pages and 0 paths.

    3. Postgres.app and the data directory: partly shown.

    • Shown: I stopped the cluster and started it with Postgres.app's own launch line, copied from the 5432 server's process: Postgres.app's binary with shared_preload_libraries=auth_permission_dialog and its dialog executable. It started on the external volume. Read back after that launch:

      • wal_sync_method=fsync_writethrough and shared_buffers=4GB;
      • the corpus: 3,741,755 objects, TimeZone=Etc/UTC.

      The doctor reported "usable; optional parts degraded": inputs, replay, .env and the model-service binding, all optional.

    • Not shown: Postgres.app itself opening the directory.

      • That was this session imitating its launch.
      • Postgres.app 2.9.3 keeps its servers in its own preferences (Servers: binPath, port, varPath, startOnLogin). Registering one is your step (D2), and I did not edit them.
      • How the GUI behaves with the drive absent is untested.

    4. F_FULLFSYNC. I ran pg_test_fsync -s 5 with its file in the cluster's own directory:

    Method ops/s
    fsync_writethrough 250 (4.0 ms each)
    fsync 18,111 (55 µs)
    open_datasync 7,779

    A drive that ignored the full-flush request would show no gap; here it is 72×. That is consistent with the drive honouring it. But stage 1 measured almost the same rate on the internal disk (247 ops/s), so the ~4 ms may be largely the macOS F_FULLFSYNC path itself. This is not a power-loss test, which would need pulling the cable during a write.

    Measurements that differ from stage 1

    • D3: fsync_writethrough was not slower at restore; on these runs it was faster. pg_restore took 57 and 75 s under it, against 111 and 96 s under open_datasync. Stage 1's per-commit figures (252 against 7,973 per second) suggested the opposite, and stage 1 left import time to be measured. A likely explanation: open_datasync syncs every WAL write, which a bulk load issues by the thousand. This is two runs per method, not isolated, and only run 1 started on an empty cluster.
    • D4 needs the WAL settings, not only the memory. The restores hit checkpoint pressure at 8 GB max_wal_size. "Match 5432" would mean 1 GB and checkpoint_timeout 5 min, which makes a restore slower. Whatever you decide for D4, the WAL settings belong in it.
    • The capture took one pass where Stage 2A's figures added two: 76 s, because the dump and the state read concurrently.
    • Comparisons are faster: 57–63 s for the three checks, against 73–88 s.
    • The end-to-end restore has a measured range: 119–175 s over four runs.
    • Worktrees:
      • stage-2-routing-audit-372bbd moved from a89c2ff to f08b638;
      • this session's worktree moved from its branch at 16e35ca to b69bc3e, detached;
      • origin/main is b69bc3e while the main checkout's main stays at 16e35ca;
      • sizes grew slightly, the main checkout from 7.35 to 7.42 GB.

    Worktrees, re-cut at 21:42 (+02)

    There are 13: the main checkout, 10 under .claude/worktrees, and 2 under Codex.

    • Untracked files: each .claude worktree's only untracked file is its data symlink, except this session's, which has none.
    • Unpushed commits: 9 commits on 7 local branches are still on no remote.
    • "In origin/main" means its head is in b69bc3e.
    • Processes are those whose working directory is exactly that worktree.
    Worktree Head Branch In origin/main Processes
    main checkout 16e35ca main yes Codex 30776, 30779, 30794; a Claude Code CLI session 22877 (since 10 September); 35 node helpers; 8 shells. None is connected to a database
    ~/.codex/worktrees/b0a9 67be21a detached yes none. README.md modified, docs/encyclopedia/ untracked: keep as a patch (D8)
    ~/.codex/worktrees/routing-policy 0cd1114 detached yes none
    143-spotify-music-6bb8a8 9076b78 claude/143-spotify-music-6bb8a8 no none. 1 unpushed commit
    144-kit-de9e9d febfe30 claude/144-kit-phase3 no none. 5 unpushed commits
    172-final-sweep 8e695ad claude/172-final-sweep yes none
    212-exemplar-items-b549d7 a89c2ff claude/212-exemplar-provenance (#223) no none
    curation-persistence-196 697ac8a codex/consolidation-2026-09-27 (superseded, pushed) no none
    dictionary-consolidation-stages-1-2-38c750 b69bc3e detached at origin/main yes this session (Claude 67553)
    dictionary-curated-opening-phase1-f69daa 7711e31 #202's yes none
    source-listing-ui-ef5442 0576800 claude/219-checkpoint yes none
    stage-2-routing-audit-372bbd f08b638 detached at #227's stage 1 commit, from before the thread fixes yes Claude 61805, the reviewer's session
    word-level-tier-shelves-720f8b 07c8a17 claude/word-level-tier-shelves-720f8b yes none

    Measured, and assumed

    Measured today, with evidence:

    • the capture and its quiet window;
    • the six bundles and their verification;
    • four corpus restores, timed;
    • three exact comparisons;
    • Postgres.app's launch line on the external data directory;
    • the fsync figures;
    • the worktrees, unpushed commits and the source's state;
    • that nothing was dropped.

    Still assumed:

    • the drive's durability under power loss;
    • Postgres.app's GUI handling the entry, and an absent drive;
    • timings with D4's resources, since these used 5433's;
    • owner and grant equality for a database that, unlike this one, has differing owners or grants. Only the suite tests that;
    • that a later capture, if one is needed, matches this one. A capture describes only its own quiet window.

    Changed on disk

    Added:

    • /Volumes/LLM Models/dictionary/bundles/: 6 bundles, 5.03 GB, mode 0700;
    • …/private/2026-10-05-211-stage3/: evidence and logs;
    • this session's test partition devils_dictionary_test_b211 on 5432.

    Removed: the throwaway cluster, 46 GB.

    Unchanged:

    What the owner decides next

    1. The baseline:

      • approve the v2 digest 5d36203469089f05bee510eb35c6ef126fd5e8ca2adf7384b3674a5bc98e63ff, or ask for a fresh capture;
      • if D6's restore of the runtime_bench pair is adopted, approve 2ac42dcd… and 69b36af8… too.
    2. Rule on D1–D13. The recommendations stand, with two measured additions:

      • D3: cost nothing at restore here;
      • D4: should name max_wal_size and checkpoint_timeout, not only memory.

      Phase B starts only on your ruling, and uses a recommendation only where you adopt it.

    3. D2: accept the partial Postgres.app proof, or say if the GUI step must come first. You add the server entry yourself, once the real cluster exists.

    4. D9: confirm that Codex (30776) and the Claude Code session 22877 in the main checkout stay off v2 until the cutover.

    5. Review and merge #228 before Phase B. It holds the --setting fixes above, and two cosmetic ones:

      • the debug-level query log in the capture, the restore and dd.routing.verify;
      • thousands separators in printed identifiers.
  13. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    Independent audit of Phase A (7 October): A−; the baseline is sound and Phase B can start on your ruling

    Read-only. Nothing captured, restored, dropped or written to devils_dictionary_v2.

    Executed here

    Check Result
    mix dd.bundle --verify on the v2 bundle, with main b69bc3e (the tool that made it) 11 files verified; manifest SHA-256 5d36203469089f05bee510eb35c6ef126fd5e8ca2adf7384b3674a5bc98e63ff, 2.7 s
    The six manifests read directly every digest equals the record's; each window quiet: true; the v2 manifest's write counters, sequence digest and catalog digest are identical before and after; snapshot 00000037-0000F3CB-1; source cluster 7607810074859095446, oid 31334772, 14,742,779,583 bytes; TimeZone=Etc/UTC recorded (none on bing135, as stage 1 found); roles holden and postgres; volume F7FDE75A… on /dev/disk5s1; code 16e35ca, tool b69bc3e
    The source today tuple counters unchanged since 5 October (inserted 47,617,920 / updated 61,366,997 / deleted 8,675,172 at the database level); schema head 20260927131023; 0 connections; the same nine non-test databases on 5432
    The throwaway cluster no process, no scratch/ directory, 5434 and 5444 silent; its identifier 7693266767055315945 is recorded as never the dictionary's
    The evidence behind the four measurements restore-v2-scratch.json: outcome: restored, differences: [], elapsed_ms 118,498, the two expected pending migrations listed, not applied; routing-verify-scratch.log: "matches … exactly"; schema diff 74 raw lines, all the documented array-cast re-parse, 0 after the canonical form; pg_test_fsync: fsync_writethrough 250 ops/s against open_datasync 7,779 and fsync 18,111
    #227 at merge its three threads resolved by 78ed132; merged b69bc3e with a merge commit
    #228 at bd067b9 installation suite 40 tests, 0 failures, twice (seeds 0 and 7), no throwaway cluster left; mix precommit 19 doctests, 2,580 tests, 0 failures, formatter clean

    Verdict

    Phase A did what the prompt asked, in order, and recorded it honestly: the three #227 findings fixed with regressions and the PR merged; six quiet, verified bundles; the four assumptions measured on a disposable cluster that is gone; a stage 3 record whose every number I could check matched its evidence. The two findings that changed stage 1's plan are real and useful: fsync_writethrough cost nothing at restore, and D4 must name the WAL settings because the restores hit checkpoint pressure even at 8 GB.

    A−, not A, for two process slips, both self-reported and both remedied:

    What is now true

    Stages 1–3 of six are done. Roughly half of #211 remains, and all of it is now mechanical given the tooling: the real cluster and restore (stage 4), repeatability and failure drills (stage 5), and cutover with the acceptance window (stage 6). The source is untouched and the baseline candidate is verified.

    Decisions, consolidated

    1. Approve the baseline: v2 5d362034…63ff, and with D6 the runtime_bench pair 2ac42dcd… and 69b36af8…. A new capture is only needed if v2 is written to before stage 4; the bundle's quiet window proves it has not been so far.
    2. Merge #211: --setting proves what runs, lists element by element, refusals before initdb #228 before Phase B. Its one open thread is Debian-specific (dropping unix_socket_directories relies on a default socket directory that Debian moves to /var/run/postgresql): irrelevant to the Mac move, relevant to the future Linux server. Resolve it now or file it against the server deployment; either way, record which.
    3. Rule on D1–D13, with the two measured amendments: D3 confirmed by the restore timings; D4 should state max_wal_size and checkpoint_timeout for the restore window (8 GB and 15 min were used) and separately for steady state.
    4. D9: confirm Codex 30776 and the Claude CLI session 22877 in the main checkout stay off v2. This audit session connects only read-only and will not during stage 4.
  14. 14 remaining items

  15. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    Close-out block 1, 7 October: C4 done on the validation copy, the LaunchAgent proven, pre-authorisations 4–6 applied

    Gate (13:17 +02): every condition held.

    • devils_dictionary_v2 on 5432 has 0 sessions, and its counters are 47,617,920 / 61,366,997 / 8,675,172.
    • 5434 is cluster 7693849764459364596.
    • The old checkout's main is 16e35ca.

    The macOS permissions. At 13:17 both processes were still waiting in open() on /Volumes. I restarted each wait so the dialogs appeared fresh, and the owner allowed both at 13:22:

    • beam.smp: the Claude preview's dev server;
    • pg_ctl: the LaunchAgent.

    Evidence: /Volumes/LLM Models/dictionary/private/2026-10-07-211-closeout/.

    C4: the external application, on an isolated copy

    Setup.

    • The dev server: mix phx.server from /Volumes/LLM Models/dictionary/src/dictionary at d890bbf.
      • Environment: DD_DATABASE=devils_dictionary_validation211, DD_DATABASE_PORT=5434, Oban off.
      • Every database connection went to 127.0.0.1:5434.
    • The source during the check: v2 on 5432 had 0 sessions and its counters did not move.
    • One fix to the launch line. Its first start failed on a Hex archive built for an older OTP: the launch line had skipped mise, so it lacked mise's MIX_HOME and MIX_ARCHIVES. Adding the two variables fixed it.
    Kind Link Identity
    Ordinary word (On page) /on/love "On love": 4 dictionaries, 43 senses, 4 subjects; the definitions and the culture shelves
    Entity, with sourced quotations /entities/1/ambrose-bierce person, b. 1842-06-24; #entity-works; #entity-quotations with quotation-3741339, -3741837, -3741946 … and their evidence links
    Sourced quotation /evidence/content/108485 "The fly that sips treacle is lost in the sweets.": Wikiquote, public domain, revision 108485, with its source URL
    Artwork /entities/3736457/x → 302 /entities/3736457/lotto-carpet "'Lotto' Carpet": 16th century, the Met, Rogers Fund 1908, #artwork-metadata
    Search /?q=oyst /on/oyster, plus /words/9420/oyster, /388598, /216976, /1335342
    Curated opening fixture /on/love?opening=fixture #opening: "Development fixture · AI-selected, not reviewed by a person · not published"; Bierce's LOVE, 1911
    On page, a second word /on/mars 3 dictionaries, 14 senses, 1 subject: /entities/1831413/mars
    Subject address, internal mode /people/ambrose-bierce 404, #subject-unresolved data-outcome=missing, "Nothing at this address". This is right for a corpus with 0 pages, 0 paths and 0 decisions (#194 after the move)
    Reviewer test account /users/log-in, magic link through the local mailbox, then /reconciliation signed in as issue84-browser@local.test (reviewer=true); "reviewer queue · Resolve split attachments". Unauthenticated, /reconciliation returns 302 to the login page

    Oban on: one bounded run of an existing job type (03-oban-bounded.exs, against the validation copy only).

    • Guards first. A direct connection proved the database was devils_dictionary_validation211 on cluster 7693849764459364596. Only the discovery queue ran (limit 1), with no plugins: no cron and no pruner.
    • Every provider stubbed. All three request-option keys that a discovery job's HTTP can reach pointed to a plug that counts the request and answers 503 on the machine:
      • discovery_req_options (Discovery.Transport, Unsplash);
      • req_options (the Absorb client, used for creator lookups);
      • verification_req_options.
    • Result. Fifteen Discovery.RunWorker jobs were waiting, admitted earlier by the connected page views while Oban was off. In the 90-second bound:
      • one completed: run 401 (Bing News), failed with provider_unavailable after its 3 transport attempts;
      • one was mid-backoff: run 402 (Guardian).
      • 5 provider requests, every one answered by the stub (www.bing.com, content.guardianapis.com), matching the 5 new discovery_request_attempts. None left the machine, so there was no duplicate external work. The run that failed is recorded with its retry time, as any outage would be.

    Restart. I stopped the application and started it again. /on/love answered 200 one second after the start, and /entities/1/ambrose-bierce (200) and /people/ambrose-bierce (404) answered the same as before. Its 11 database connections are all on 5434.

    Dropped by name. DROP DATABASE devils_dictionary_validation211 on 5434 (cluster 7693849764459364596): 0 sessions, 9.5 GB. 5434 now holds v2, runtime_bench, runtime_bench_b and the test partition devils_dictionary_test_b211x.

    C6: the LaunchAgent starts the cluster

    com.razrfly.dictionary.postgres-5434 uses Postgres.app's pg_ctl, with RunAtLoad and StartOnMount.

    • The proof. I stopped the cluster by hand, then started it only with launchctl kickstart gui/501/com.razrfly.dictionary.postgres-5434. It was up 0.6 s later; the agent's run exited 0, and the postmaster (pid 86973) is parented by launchd.
    • What came back: identity 7693849764459364596, data directory on /Volumes/LLM Models, fsync_writethrough, max_wal_size 2 GB, checkpoint_timeout 5 min, nothing pending.
    • The run before it (13:22, the owner's grant) found the server already running and exited 1, as the plist's comment says.

    Pre-authorisations 4, 5 and 6

    • 4, stop 5433. pg_ctl stop on cluster 7690164109148229279. It had been up since 27 September with 0 client sessions and shared_buffers 4 GB. Its 36 databases (310.5 GB) and its data directory are untouched: nothing was dropped.

    • 5, shared_buffers=8GB on 5434. ALTER SYSTEM, then applied by the LaunchAgent restart above: 16 GB went to 8 GB, recorded. Revisit it after the reclaim.

    • 6, synchronous_commit=off for test databases only.

      • The change. It is a connection parameter of the suite's own Repo connections in config/test.exs. No database's settings change.
      • The test. A new SynchronousCommitTest proves:
        • the suite's sessions run with it off;
        • no database or role on the suite's server carries it, and that server is 5434, where v2 lives;
        • no configuration but test.exs mentions it;
        • the development configuration's Repo has no connection parameters.
      • Merge and measurement: #233, merged as 06df00a after an independent review. The review found no path that gives v2 the setting, and its four suggestions to tighten the test are in. The whole suite now takes 311 s on 5434, against 614 s before (the old cluster took 249 s): 2,591 tests, 0 failures.

      One slip, fixed. Tests commit without waiting for the disk; the corpus still does (#211, pre-authorisation 6) #233's last commit also tracked .claude/worktrees/cutover-231, a worktree gitlink: a pointer only, no file contents. A fresh clone doesn't carry the old checkout's .git/info/exclude, which ignored agent worktrees. #234 (d493889) untracks the entry and puts /.claude/worktrees/ in .gitignore. The external clone now carries the old local excludes, plus its coderabbit.basebranch and extensions.worktreeconfig settings (a D8 gap, now closed).

    Still standing

    v2 on 5432 is untouched (0 sessions, the same counters). #218 and #223 are unmerged, #229 and #232 filed, and the 9 commits kept on their branches.

  16. added a commit that references this issue on Oct 7, 2026
  17. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    Close-out block 2: the cutover, 7 October 13:52 +02. The window closes Friday 9 October 2026, 13:52 +02 (11:52 UTC)

    The dictionary now runs from the external drive.

    • Server: cluster 7693849764459364596, port 5434, data directory /Volumes/LLM Models/dictionary/postgres/18/data.
    • Checkout: /Volumes/LLM Models/dictionary/src/dictionary.
    • Evidence: /Volumes/LLM Models/dictionary/private/2026-10-07-211-closeout/, files 10–18.

    The run sheet, step by step

    Step Result
    1. The new cluster answers as itself The doctor passes --expect-cluster 7693849764459364596: devils_dictionary_v2 (oid 16385, 9.3 GiB) on /Volumes/LLM Models, head 20260927220937, none pending
    2. Nothing writes to the old v2 0 sessions; counters 47,617,920 / 61,366,997 / 8,675,172
    3. The defaults switch (D11) First a precommit of #231 merged with main, run with nothing set: 19 doctests, 2,592 tests, 0 failures, 317 s on 5434. Then #231 merged with a merge commit as 7b731f5 (11:50:53Z), and the external checkout's main moved to it. The old checkout stays at 16e35ca, defaulting to 5432
    4. The model service rebinds (D12) DD_DATABASE=devils_dictionary_runtime_bench mix dd.runtime service start --rebind, with no port set. run/authority.json went from runtime_bench on 7607810074859095446 to runtime_bench on 7693849764459364596. ready --model-config qwen3.5-9b-v1 passes every check: slot, volume, authority, runtime 0.34.4, served digest, manifest. The old slot held no live attempt (34 completed and 1 ended_by_restart, identical on both clusters). The service is now stopped, as it was before the move, and its binding is kept
    5. Every endpoint, before any write, with nothing set (no DD_DATABASE*, no PGPORT) Dev: devils_dictionary_v2 on 5434, cluster 7693849764459364596, head 20260927220937. Tests: devils_dictionary_test_b211x on 5434, synchronous_commit off. Doctor: every core check passes with --expect-cluster 7693849764459364596 --volume "/Volumes/LLM Models" --volume-uuid F7FDE75A…; it warns, as expected, that Oban will run and that the runtime is bound to runtime_bench by design. Runtime binding: the new cluster
    The first writes mix phx.server from the external checkout, Oban on, nothing set. Its 11 database connections are all on 5434. The first writes landed on 5434 (13:52:17): this node replaced the stale oban_peers row, and Oban's pruner removed the 978 job rows finished in September. On 5432: 0 sessions on any dictionary database; v2's counters unchanged. Its xact_commit moved only through autovacuum's routine visits

    After the checks I stopped the dev server and the model service, so the owner starts their own from the new path.

    Rollback, checked against the real clusters (nothing executed)

    The procedure is docs/operations/211-cutover.md.

    • The old installation is intact.
      • The old checkout is at 16e35ca and its dev default is still "5432".
      • The old v2 on 7607810074859095446 is at 20260927131023, unwritten since 5 October.
      • runtime_bench is still on 5432.
      • The LaunchAgent is loaded and the data directory is present.
    • Both copies of the v2 bundle verify (5d362034…63ff): on the external drive, and in ~/Backups.
    • Writes made on 5434 since 13:52 are not in the source. Rolling back means capturing them first with dd.bundle --source ecto://…:5434/devils_dictionary_v2, then pointing the clients back.

    The window

    48 hours of normal use: until Friday 9 October 2026, 13:52 +02. Through it:

    At the close, the pre-authorised reclaim runs (block 4): each bundle re-verified that day, then only the listed databases on 5432 by exact name, then the listed worktrees, then the old checkout.

    For the owner, now: open the project in Claude Code and Codex from /Volumes/LLM Models/dictionary/src/dictionary, and use it normally. Sessions opened from the old path keep pointing at the old checkout and at 5432.

  18. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    Independent audit of the cutover (7 October, 14:05 +02): A. The dictionary runs from the external drive; the issue stays open for the window, the reclaim and the docs

    Read-only against both clusters, the checkouts and the runtime state. Nothing written or dropped.

    Verified live

    Claim Measured here
    The corpus is served from the external drive 5434 is cluster 7693849764459364596 at /Volumes/LLM Models/dictionary/postgres/18/data; devils_dictionary_v2 9.5 GB, runtime_bench and _b present; devils_dictionary_validation211 dropped; settings shared_buffers 8 GB, max_wal_size 2 GB, fsync_writethrough; TimeZone=Etc/UTC on every database
    Defaults switched (D11) #231 merged 7b731f5 with a merge commit; the external checkout is on main 7b731f5, clean; config/dev.exs and config/test.exs default to 5434
    Model service rebound (D12) run/authority.json: runtime_bench on 7693849764459364596
    The first writes landed on 5434 its v2 shows 1,002 updates and 1,092 deletes since the restore (Oban's peer row and its September job prune), with the dev server since stopped
    The old installation untouched 5432 v2: tuple counters 47,617,920 / 61,366,997 / 8,675,172, head 20260927131023, 0 connections; all nine non-test databases and 115 test partitions still there; the old checkout at 16e35ca with 13 worktrees (7.1 GB)
    D2 start at login LaunchAgent com.razrfly.dictionary.postgres-5434 loaded, 2 runs, last exit 0; the cluster process (pid 86973, started 13:28:37) has launchd as its parent. The log shows the first start failing while the removable-volume permission was pending and the second succeeding
    Pre-authorisations 4–6 5433 stopped (no response); shared_buffers 8 GB; tests commit without waiting for the disk (#233)
    Machine swap down from 12.5 GB to 9.0 GB used; 65 % memory free; internal disk 8 GiB free (the reclaim has not happened yet)
    Bundles both copies of 5d362034…63ff verified earlier today; unchanged

    Verdict

    A for blocks 1 and 2. The run sheet was followed step by step, every endpoint was checked with nothing set before the first write, the servers were stopped afterwards so the owner starts their own from the new path, the rollback was checked against the real clusters, and the window's closing time is posted. The one page that returns 404, a subject address, is correct for a corpus with no routing pages yet; that is #224's work.

    What remains before closing

    Criterion State
    C7 reclaim after the window: Friday 9 October 2026, 13:52 +02; then the pre-authorised list by exact name, measured space, headroom target
    C1 README quickstart, C9 updates to #130, #209, #194 block 5, after the reclaim
    C8 final review block 6; this audit session will do it
    C6 last item the owner opens the project from /Volumes/LLM Models/dictionary/src/dictionary; sessions from the old path still use the old checkout

    Two ways to finish. Wait for Friday 13:52, when the finishing session resumes on its own; or, since every exactness proof already holds and both bundle copies verify, the owner may close the window early with a comment here, and blocks 4–6 run today.

  19. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    Pre-authorisation amended (owner, 7 October): #218 and #223 no longer wait for the 48-hour hold. Merging them to main leaves the old checkout at 16e35ca and the old database untouched, so the rollback path is unchanged. The hold still gates only the reclaim (block 4). The owner may also close the hold early here; the evidence does not require the full 48 hours.

  20. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    Record amended, 7 October 16:05 (+02), from the #212 final review. The old checkout ~/Code/projects-2026/dictionary is no longer at 16e35ca: its main was fast-forwarded to 7b731f5 at 14:16 (+02). It is still a rollback asset, but a server started from it now defaults to 5434 like the new one, so the rollback step is DD_DATABASE_PORT=5432 set explicitly. Its 14 registered worktrees are unchanged. The old devils_dictionary_v2 on 5432 still has 0 sessions and the gate's counters.

    Two facts for the owner's decision on the hold:

  21. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    Hold closed by the owner, 7 October 16:40 (+02): the reclaim runs now. Pre-wipe check attached

    The owner closed the 48-hour hold in chat: check that it is working, then wipe. The check below was read-only, and everything passed, so blocks 4–6 of the close-out prompt run today, in their order, by the finishing session. The window therefore ends at the owner's word, not at Friday 13:52.

    The new installation, checked 16:10–16:30

    Check Result
    mix dd.doctor --expect-cluster 7693849764459364596 --volume "/Volumes/LLM Models" from the external checkout every core line green: cluster as pinned, data directory on the volume (F7FDE75A-…, 1,436 GiB free), devils_dictionary_v2 9.4 GiB, ICU 153.128, TimeZone=Etc/UTC, head 20260927220937, 27 migrations, none pending, inputs and replay present, build compiled. The warnings are the expected ones: 4007 held by the running server, the Oban note, optional env keys, the runtime bound to runtime_bench as D12 intends
    mix precommit on main 3fe1cd1 from the external checkout 19 doctests, 2,658 tests, 0 failures (335 s)
    The server pid 8730, cwd the external checkout: 11 connections to 5434, none to 5432. /, /on/love, /on/coward, /on/logomachy, /artworks, /sources/wordnet, /users/log-in 200; /health → /ops/health; /connect → log-in; https://wordhoard.eu.ngrok.io/on/love 200. Oban: 39 jobs, 37 completed, 2 cancelled, the maintenance workers on schedule (last run 14:07Z)
    The LaunchAgent RunAtLoad and StartOnMount; pg_ctl -w -t 120 start on the external data directory; runs 2, last exit 0; the postmaster has run under launchd since 13:25
    The 5434 log the only ERROR lines today are at 16:04, from the review's own test partition (constraint tests); that partition is dropped

    Nothing the reclaim destroys is unpreserved

    Asset on the internal disk Where it already is
    devils_dictionary_v2 on 5432, 14 GB its newest row is 2026-09-27 20:29 (discovery_runs); the bundle was captured 2026-10-05 19:20Z, so it holds all of it. Both copies verified by SHA-256 just now: 11 files, MANIFEST.json 5d36203469089f05bee510eb35c6ef126fd5e8ca2adf7384b3674a5bc98e63ff, at /Volumes/LLM Models/dictionary/bundles/2026-10-05-v2 and ~/Backups/dictionary/bundles/2026-10-05-v2, 3.9 GB each
    5434 against 5432, every table 69 shared tables, 48 equal. The 21 that differ are all explained: schema_migrations 24 → 27 and routing_backfill_* only on 5434 (the two recorded migrations and #212's); +104 source_records, +185 revisions, +14 entities, +26 objects, +13 assertions, +29 revisions, +12 content items, +38 identifiers, +13 verification runs, +26 materialized outputs, +8 person and +6 work details, +1 mapping, +1 reconciliation case, +16 evidence rows: each count equals the rows inserted after 11:50Z today, so they are the day's use of the new installation; discovery_runs, discovery_results, discovery_request_attempts and oban_jobs are fewer on 5434 because retention and Oban's pruner ran there (all 24 runs on 5434 are today's). The sequences differ for the same reasons
    dev, 74_verify, bing135, the runtime_bench pair bundled 2026-10-05 on the external drive; the runtime pair is also live on 5434
    14 unpushed commits on 7 branches (claude/143-spotify-music-6bb8a8, the claude/144-kit-* stack, integ, merge-154) all 7 branches exist in the external clone at the same SHAs (D8 done)
    Codex b0a9's edits (README.md, docs/encyclopedia/) private/2026-10-07-211-stage4/codex-b0a9/ (b0a9-all.patch, tracked.patch, untracked.tar, SHA256SUMS)
    The other dirty worktree, dictionary-consolidation-stages-1-2-38c750 only .claude/launch.json modified: a local launch setting, nothing to keep
    The reviewing session's own worktree, stage-2-routing-audit-372bbd clean, detached at d890bbf, nothing to keep; but it is that session's working directory and the app refuses to move it, so it goes last, after the C8 review is posted (below)

    For the finishing session: the order, unchanged from the pre-authorisation

    1. 5432, by exact name, devils_dictionary_v2 last: the runtime_bench pair, dev, 74_verify, bing135, ex181, ex181b2, every devils_dictionary_test* (123 databases, 27 GB in all); keep ex212; nothing of any other project. Record pg_database_size per name before, and df before and after.
    2. The old checkout, 7.2 GB, all but the reviewer's worktree: git worktree remove every registered worktree except stage-2-routing-audit-372bbd (the two Codex ones are Codex's to prune; their 17 MB and 294 MB directories go when the checkout does), delete data/ (3.5 GB), _build and deps, and leave the .git directory and that one worktree in place. About 6.8 of the 7.2 GB comes back here. The internal ~/Backups/dictionary second copy stays, by the retention plan.
    3. The README quickstart (Kickstart on a new machine: one command from a fresh clone to the full corpus (bundle, doctor, README quickstart) #130 §1), the Kickstart on a new machine: one command from a fresh clone to the full corpus (bundle, doctor, README quickstart) #130, Move dictionary storage, PostgreSQL and agent workspaces to the external 2 TB drive — execution in #211 #209 and Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194 comments, the measured before/after, then the final review here (C8) by the reviewing session.
    4. After C8 is posted: the owner archives the reviewing session; then remove stage-2-routing-audit-372bbd and delete the checkout's remaining .git and root (about 0.4 GB), record the final df, and close.

    Internal disk at 16:30: 6.0 GiB free. About 34 GB comes back.

  22. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    Close-out block 4, 7 October 17:31–17:40 (+02): the reclaim. +32.87 GiB on the internal disk (5.12 → 37.99 GiB free)

    Run by the finishing session after the owner closed the hold (16:40). Everything followed the pre-authorised list and the order in the hold-closed comment.

    Re-checked first (17:31–17:33)

    Check Result
    5432's identity system_identifier 7607810074859095446
    devils_dictionary_v2 on 5432 0 sessions; tup_inserted / tup_updated / tup_deleted = 47,617,920 / 61,366,997 / 8,675,172, the gate's counters, so nothing was written after the capture
    Every bundle, re-verified today with mix dd.bundle --verify v2 in both places (/Volumes/LLM Models/dictionary/bundles/2026-10-05-v2 and ~/Backups/dictionary/bundles/2026-10-05-v2): 11 files each, MANIFEST.json 5d36203469089f05bee510eb35c6ef126fd5e8ca2adf7384b3674a5bc98e63ff. runtime_bench 2ac42dcd…a5b2 and runtime_bench_b 69b36af8…3014, the approved digests. dev 3993f1cc…80fb2, 74_verify f0d6fee0…fdc5 and bing135 8e24850a…30ea1: every file verified
    The replacements on 5434 devils_dictionary_v2 9.4 GiB; runtime_bench 14 MB and runtime_bench_b 13 MB, live
    5432, everything 156 databases. 123 on the list: 27.38 GiB, none with a session. ex212 kept. 32 of other projects, not touched

    1. 5432: 123 databases dropped by exact name, 17:33:23–17:33:47, v2 last

    Each was a plain DROP DATABASE "<name>": no FORCE, the server identity re-read before each drop, and the loop set to stop on the first error. It did not stop.

    Database pg_database_size before (bytes) GiB
    devils_dictionary_runtime_bench 14,685,887 0.01
    devils_dictionary_runtime_bench_b 13,522,623 0.01
    devils_dictionary_dev 3,423,786,687 3.19
    devils_dictionary_74_verify 4,457,354,943 4.15
    devils_dictionary_bing135 4,458,903,231 4.15
    devils_dictionary_ex181 16,676,543 0.02
    devils_dictionary_ex181b2 16,864,959 0.02
    every devils_dictionary_test* (115, devils_dictionary_testspotify included) 2,259,032,013 2.10
    devils_dictionary_v2, last 14,742,779,583 13.73
    total, 123 databases 29,403,606,469 27.38

    After the drops, 5432 holds 33 databases: devils_dictionary_ex212 (14.9 MB, kept) and the same 32 of other projects, by name.

    2. The old checkout ~/Code/projects-2026/dictionary, 17:35–17:36

    Removed, 6 worktrees. Before removal, each was checked: clean, no process, and its HEAD present in the external clone.

    Worktree HEAD Where it is kept
    143-spotify-music-6bb8a8 9076b78 local branch claude/143-spotify-music-6bb8a8 in the external clone (1 unpushed commit, D8)
    144-kit-de9e9d febfe30 local branch claude/144-kit-phase3 in the external clone (5 unpushed, D8)
    172-final-sweep 8e695ad contained in the claude/211-* branches (merged)
    curation-persistence-196 697ac8a origin/codex/consolidation-2026-09-27
    dictionary-curated-opening-phase1-f69daa 7711e31 merged
    word-level-tier-shelves-720f8b 07c8a17 merged

    Deleted: data/ (3,524,736 KiB), _build (165,384 KiB) and deps (101,848 KiB). All three were real directories, git-ignored, and had no file open in any process. The external checkout's own data/ is untouched.

    Not removed:

    • Four worktrees each hold a running Claude Code session. Pre-authorisation 3 removes only worktrees that "hold no process", the reason the reviewer's is deferred:

      Worktree Session (claude pid, started 7 Oct) KiB
      212-exemplar-items-b549d7 90764, 13:29 338,220
      dictionary-consolidation-stages-1-2-38c750 25424, 09:14 276,644
      source-listing-ui-ef5442 90405, 13:29 363,516
      dictionary-issue-212-673812 65943, 14:16 (this finishing session) 148,868

      They go with the last step, once the owner archives those sessions.

    • The reviewer's stage-2-routing-audit-372bbd (364,720 KiB), as ordered. It also holds the running ngrok tunnel's working directory (pid 42700), so remove it only after ngrok is restarted elsewhere or no longer wanted.

    • The two ~/.codex worktrees, which are Codex's to prune.

    The old checkout is now 1.81 GiB (1,902,688 KiB): .git 146 MiB, the five worktrees above 1.42 GiB, and the tracked files.

    Measured

    Free on the internal disk (df -k /System/Volumes/Data)
    Before, 17:31 5,363,772 KiB (5.12 GiB)
    After the 5432 drops, 17:33 34,149,688 KiB (32.57 GiB)
    After the worktrees, 17:35 36,037,272 KiB (34.37 GiB)
    After data/, _build, deps, 17:36 39,832,936 KiB (37.99 GiB)
    Reclaimed 34,469,164 KiB = 32.87 GiB

    The owner's estimate was about 34 GB, of which about 0.4 GB was to follow the review. What remains is now 1.81 GiB, because four worktrees hold live sessions.

    After the reclaim (C6):

    • mix dd.doctor --expect-cluster 7693849764459364596 --volume "/Volumes/LLM Models" from the external checkout: every core line green; 27 migrations, none pending.
    • The dev server (pid 8730): 11 connections to 5434, 0 to 5432; /on/love 200.

    Operating headroom on the internal disk (the target C7 asks for)

    • Floor: 30 GiB free. Below that, macOS swap and APFS snapshots on a 460 GiB disk start to crowd out work; this disk was at 5–6 GiB.
    • Watch line: 40 GiB.
    • Today: 37.99 GiB, and about 39.8 GiB once the last 1.81 GiB goes. The dictionary itself keeps on the internal disk only:
      • the 3.9 GB second bundle copy in ~/Backups/dictionary, as the retention plan intends;
      • devils_dictionary_ex212 on 5432 (15 MB).
    • Where pressure will come from: the other 32 databases on 5432 (83.6 GiB) are other projects', and the next room to free is theirs.
  23. added a commit that references this issue on Oct 7, 2026
  24. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    Close-out block 5 done, and the request for the final review (C8), 7 October

    Block 5: C1 and C9

    Where
    README quickstart (#130 §1) and docs reconciled PR #236, merged as 9eb0ee4. It contains: Quickstart on a new machine (PostgreSQL 18 pinned, and why; restore from an approved bundle; .env kept private; the 5434 default) and a table of the working installation. The rebuild script is retitled as the slow path. 211-cutover.md is brought to the reclaimed state, with rollback now a restore from the bundle. The dated checkpoints are marked moved, AGENTS.md names where the installation lives, and the stale "5432 keeps the pre-move copy" comments are gone. Precommit on its head: 2,658 tests, 0 failures
    #130 comment: §1, §2, §4 and §5 are done, and §3 used option 1 plus the D14 copy. Open: a second Mac following the README alone, and object storage if wanted
    #209 comment: every stage is mapped to its record here, with the internal exceptions accepted and measured, and the headroom target
    #194 comment: Stage 2 resumes on the external installation, and CP4 (#224) is next

    One slip, recovered. I fast-forwarded the external checkout to 9eb0ee4 under the running dev server. config/dev.exs had changed (a comment only), so Phoenix's code reloader refused to recompile, and /on/love returned 500 for a few minutes. The server was restarted from the external checkout in a Terminal tab (mise exec -- mix phx.server, now pid 5039):

    • /, /on/love, /on/coward, /artworks, /sources/wordnet and /users/log-in return 200;
    • it holds 11 connections, all to 5434;
    • https://wordhoard.eu.ngrok.io/on/love returns 200.

    The doctor is green on every core check.

    Measured, before and after (block 4)

    Internal disk free 5.12 → 37.99 GiB (5,363,772 → 39,832,936 KiB), +32.87 GiB
    5432 123 dictionary databases dropped by exact name: 27.38 GiB by pg_database_size, v2 last. ex212 (15 MB) and 32 databases of other projects remain
    The old checkout 6 idle worktrees removed. data/ 3.36 GiB, _build 162 MiB and deps 99 MiB deleted. 1.81 GiB left (.git, the tracked files, five worktrees of open sessions)
    Headroom target a floor of 30 GiB free, with a watch line at 40 GiB. Met today

    Measured versus assumed.

    • Measured today:
      • every bundle verified file by file;
      • the gate counters;
      • each database's size before its drop;
      • df at each step;
      • each worktree's state, processes and size, and the external clone holding every HEAD;
      • the doctor;
      • mix dd.manifest, 5 / 5;
      • precommit.
    • Assumed: that the five worktrees held by running sessions (four Claude sessions plus the reviewer's, whose directory also hosts the ngrok tunnel) hold nothing to keep. Their branches are pushed or contained; their removal waits for the sessions to be archived. That a second Mac follows the README: not exercised.

    For the reviewing session: C8

    Please review main at 9eb0ee4 and the evidence in this issue against C1–C9:

    • the close-out plan;
    • blocks 1, 2 and 4;
    • this record.

    Then post the verdict here.

    After C8 (the last step, not done here):

    1. The owner archives the reviewing session and the four Claude sessions whose worktrees remain:
      • 212-exemplar-items-b549d7;
      • dictionary-consolidation-stages-1-2-38c750;
      • source-listing-ui-ef5442;
      • this finishing session's dictionary-issue-212-673812.
    2. ngrok is restarted from elsewhere, if it is still wanted.
    3. Then remove those worktrees and the old checkout's .git and root (1.81 GiB), record the final df, and close Repeatable dictionary migration: prove external setup against the retained working baseline #211.

    The finishing session stops here.

  25. holden commented on Oct 7, 2026

    @holden
    MemberAuthor

    C8, the final review, 7 October 18:55 (+02): accepted, grade A. #211 closes after the last 1.81 GiB, which waits on five sessions being archived

    Read-only against main 9eb0ee4, both clusters, the old checkout, the running server and the evidence; then reproduced from the external checkout. Nothing written, nothing dropped.

    Reproduced now

    Check Result
    mix precommit on main 9eb0ee4, external checkout, MIX_ENV=test, partition _c8 on 5434 exit 0 · 19 doctests, 2,658 tests, 0 failures (318 s) · the formatter changed nothing; the partition dropped afterwards
    mix dd.doctor --expect-cluster 7693849764459364596 --volume "/Volumes/LLM Models" every core line green: cluster 7693849764459364596 as pinned, data directory on volume F7FDE75A-… (1,436 GiB free), v2 9.4 GiB, ICU 153.128, TimeZone=Etc/UTC, head 20260927220937, 27 migrations, none pending, inputs and replay present, build compiled; the warnings are the expected four
    5432 devils_dictionary_ex212 (14 MB) is the only devils_dictionary* database left; 31 databases of other projects, plus postgres, as before. The cluster directory is 84 GB, down from 111 GB
    5434 devils_dictionary_v2 9.6 GB, the runtime_bench pair, test_212shots and test_b211x. The postmaster has run under launchd since 13:25
    The old checkout 1.8 GB: .git, the tracked files, and five worktrees. data/, _build and deps are gone. git worktree list shows exactly the five plus the two Codex ones; each of the five has a live claude process with it as its working directory (pids 25424, 26232, 65943, 90405, 90764), so none could be removed yet
    The server pid 5039 from the external checkout, in the finishing session's Terminal tab; 11 connections to 5434 and none to 5432; /on/love 200 locally and through wordhoard.eu.ngrok.io (ngrok pid 42700, in this reviewing session's tab)
    Internal disk 38 GiB free (df: 392 of 460 GiB used), against 6.0 GiB at 16:30 and 5.12 GiB at 17:31
    Both bundle copies verified by SHA-256 at 16:20 (11 files, MANIFEST.json 5d36203469089f05bee510eb35c6ef126fd5e8ca2adf7384b3674a5bc98e63ff) and again by the finishing session at 17:31

    The criteria

    Verdict On what
    C1 quickstart and runbook met The README's Quickstart on a new machine goes from brew install mise to mix phx.server through dd.bundle --verify, dd.bootstrap --mode init --expect-manifest-sha256, the doctor and ecto.migrate, with the PostgreSQL 18 pin explained, the .env rule, the 5434 default and the DD_DATABASE_PORT=5432 case for a machine with only the default server. The working installation table names the checkout, the cluster, the LaunchAgent, the doctor line and where recovery comes from. 211-cutover.md's rollback section now says there is no internal copy and recovery is a restore from the bundle, and keeps the window-era procedure under a heading that says it is over. installation.md, the two dated checkpoints and AGENTS.md no longer describe the installation as internal
    C2 the tools used for the move met at stage 4 unchanged
    C3 private evidence, every difference explained met, one nit stages 1–6 and blocks 1–2 have their files under private/; block 4's record (each database's bytes before its drop, df in KiB at each step, each worktree's HEAD and where it is kept) exists only as the comment here. The pre-wipe check at 16:30 explained every row-count difference between the two copies before anything was dropped
    C4 the external application usable met in block 1 pages, the bounded Oban run, the restart, precommit; files 01–04, 08, 10
    C5 second clean target, repeat and interruption met at stage 5 unchanged
    C6 no hidden dependency; future work external met #231 merged; the LaunchAgent proven (file 06) and still the cluster's parent; the #212 finishing session and #236 worked from the external checkout; the doctor passes with --expect-cluster and --volume; AGENTS.md now tells every session where to work
    C7 cutover, rollback, retention, space reclaimed met, bar 1.81 GiB cutover verified at 14:05; the window observed until the owner closed it at 16:40 on evidence; 123 databases dropped by exact name with v2 last, 27.38 GiB; six idle worktrees and the ignored data removed; +32.87 GiB measured; headroom target recorded (30 GiB floor, 40 GiB watch). The remaining 1.81 GiB is five worktrees whose sessions are running, which pre-authorisation 3 rightly forbids removing
    C8 this review accepted
    C9 #130, #209, #194 updated; stale text reconciled met the three comments at 16:24; PR #236

    Noted, none blocking

    1. Block 4 left no evidence file. The comment is exact enough to stand as the record; a copy of it under private/2026-10-07-211-closeout/ would make the folder complete.
    2. The fast-forward under the running server (a 500 for a few minutes, restarted) was the session's own report; the restart left the server in that session's Terminal tab, so archiving the session stops it. The last step below restarts it from a durable place.
    3. Outside this issue's list: the stopped 27 September rehearsal cluster at /Volumes/LLM Models/dictionary-stage2a holds 309 GB on the external drive (pgdata 297 GB, stage2r 9.7 GB, dumps 2.3 GB). The pre-authorisation said stop it, not delete it, and the drive has 1.4 TB free, so nothing is wrong. It is the next deliberate deletion to decide, in its own comment or issue.

    The last step, after this review

    1. The owner archives the five sessions whose worktrees remain: this reviewing session (stage-2-routing-audit-372bbd), the finishing session (dictionary-issue-212-673812), 212-exemplar-items-b549d7, dictionary-consolidation-stages-1-2-38c750, source-listing-ui-ef5442. That stops the dev server (in the finishing session's tab) and the ngrok tunnel (in the reviewing session's tab).
    2. A new session opened from /Volumes/LLM Models/dictionary/src/dictionary restarts both from there, confirms no process has a working directory under ~/Code/projects-2026/dictionary, removes the five worktrees and the checkout (1.81 GiB; the two ~/.codex worktree directories, 311 MB, lose their registration with it and are Codex's to prune), records the final df here, and closes Repeatable dictionary migration: prove external setup against the retained working baseline #211.

    Verdict: accepted, grade A. The move was proved before it was made, made with the record intact, and the internal copies were removed only after every one of them was shown to exist elsewhere.

  26. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    The last step, 8 October 10:30–10:50 (+02), and closing

    Done by the reviewing session on the owner's word. Nothing on 5432 or 5434 was touched.

    Services no longer depend on a session. Dev server: LaunchAgent com.razrfly.dictionary.phx-4007 (~/Library/Application Support/dictionary/phx-4007.sh waits for the external checkout and Postgres on 5434, then runs mix phx.server from /Volumes/LLM Models/dictionary/src/dictionary; RunAtLoad, KeepAlive, StartOnMount). Running: pid 20255, 11 connections to 5434 and none elsewhere, /on/love 200. Tunnel: LaunchAgent com.razrfly.dictionary.ngrok (ngrok http --url=wordhoard.eu.ngrok.io 4007), pid 20468, https://wordhoard.eu.ngrok.io/on/love 200. Logs in ~/Library/Logs/dictionary-*.launchd.log; launchd-spawned processes may not open files on the removable volume, the same reason the Postgres agent logs there. The session-held server and tunnel were stopped first; the tunnel was down about four minutes.

    Sessions. "Dictionary issue #212 builds 1-3", "Issue #219 routing and aggregate reading" and "Curation persistence slice 1" archived and their worktrees removed (each clean, no process in it, HEAD on origin or in the external clone; git worktree prune run). "Dictionary consolidation stages 1–2" refused to archive ("live work") even with Remote Control off; the owner archives it from the sidebar. The reviewing session archives itself after this.

    Left on the internal disk before the final removal. ~/Code/projects-2026/dictionary 1,051,648 KiB (.git 149,556; the tracked files; two worktrees 276,644 and 364,720); ~/.codex/worktrees/{b0a9,routing-policy}/dictionary 319,136 KiB (b0a9's edits are saved in private/2026-10-07-211-stage4/codex-b0a9/; routing-policy is clean and on origin). Kept by design: devils_dictionary_ex212 14 MB on 5432 (pre-authorisation) and ~/Backups/dictionary 3.9 GB (the second bundle copy, by the retention plan). df -k /System/Volumes/Data: 38,910,952 KiB free (37.1 GiB) against 5,363,772 before block 4. On the owner's word the old checkout and the two Codex folders are removed by the reviewing session right after this comment; the final df follows below.

    Closing. C1–C9 are met (C8 above). Follow-ups recorded outside this issue: the stopped 27 September rehearsal cluster at /Volumes/LLM Models/dictionary-stage2a (309 GB on the external drive, a deletion to decide), and #130's second-machine run. #209 closes with this.

  27. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    The final removal is the owner's, 8 October 10:55 (+02). The owner ordered the residue removed, but Claude Code's built-in safety check refuses to let a session delete its own workspace root (~/Code/projects-2026/dictionary is this reviewing session's), and only a person may approve that. So all three paths still present. Once the "Dictionary consolidation stages 1–2" session and the reviewing session are archived, one command from any terminal finishes it:

    rm -rf ~/Code/projects-2026/dictionary ~/.codex/worktrees/b0a9/dictionary ~/.codex/worktrees/routing-policy/dictionary

    df -k /System/Volumes/Data now: 38893380 KiB free (37.1 GiB); about 1.3 GiB more follows that command. The issue stays closed: the dictionary needs nothing from those folders.

  28. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    Final state, 8 October 11:05 +02, verified by the reviewing session after the owner ran the last command. The old checkout and both Codex worktree folders are gone. Internal disk: 46 GiB free, 90 percent used, against 5.12 GiB before the reclaim. On 5432 only devils_dictionary_ex212 remains of the dictionary's databases. The 5434 cluster is 7693849764459364596, 27 migrations, devils_dictionary_v2 9.6 GB. The three LaunchAgents are loaded; the dev server, pid 20255, and the tunnel, pid 20468, run; /, /on/love, /on/coward, /artworks and /users/log-in answer 200 locally, and /on/love through wordhoard.eu.ngrok.io; 11 connections to 5434, none to 5432. The doctor passes every core check from the external checkout, which is on main and clean. Kept by design: ~/Backups/dictionary, the 3.9 GB second bundle copy, and ex212, 14 MB. Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions