Skip to content

CP4: enable reviewed namespace addresses on the working installation (after #211) #224

Closed

Description

@holden

Checkpoint CP4 of #219. #219 delivered accepted code and an On → subject reader, demonstrated on a disposable copy of the corpus. #211 moves the storage. This issue comes after both. It enables the reviewed population on the external working installation, so that the owner's everyday reader opens subjects at their namespace addresses.

Success, in one sentence (from #219): on the accepted application revision and the designated working dataset, opening On Mars and selecting a subject reliably opens that exact subject's stored namespace address, while lexical distinctions, content and actual editorial approvals remain intact.

Entry conditions

Work

  1. Re-derive the population. Export the working database with the committed SQL, then run mix dd.routing.audit and candidates.py. The backfill refuses a population derived from any other export.
  2. Run without reviews (mix dd.routing.backfill --snapshot … --population … --manifest …). This writes decisions, draft pages and dispositions, and no address. Reviewers review its manifest.
  3. Record real reviews. Named reviewers write a review file bound to the population. Each confirmation names the fingerprint it reviewed. Rehearsal reviews ("rehearsal": true) are refused outside rehearsal copies, and none from Pre-#211 delivery: accept the routing backfill, make On the main reading entry, and consolidate main for the storage move #219 may be reused.
  4. Run with the reviewers' file, twice. The second run must write nothing. Keep the candidate launch manifest.
  5. Decide how the owner reads drafts. Internal mode comes from :internal_reading in trusted configuration, or from an authenticated internal contributor or reviewer. It never comes from a request parameter, and it never publishes or approves anything. Close any public tunnel before a server with internal reading runs.
  6. Verify on the working dataset: search → On mars → each Mars subject's card → its address. Then back, reload, and the other states. The corpus holds one Mars entity (the planet). Pre-#211 delivery: accept the routing backfill, make On the main reading entry, and consolidate main for the storage move #219's deity and album were fixtures on a copy (on_demo.exs) and must not be recreated here. A deity page exists only if a real identity for it is created through the normal process.
  7. Snapshot the result and verify the restore (mix dd.snapshot, mix dd.routing.verify).

Not in this issue

  • Publication, indexing, sitemaps and SEO (Stage 5).
  • The page–composition binding for curated On pages (Stage 4).
  • New move, merge, split and retire administration.

References

Activity

  1. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    Brought up to date, 8 October 2026: entry conditions met, the owner's decisions, and the closing criteria

    Checked against main 9eb0ee4 and the working installation after #211 and #209 closed.

    Entry conditions, as they stand

    Condition in the body State today
    #211 complete, the working database preserved and verified at its new location Done and closed (7–8 October). devils_dictionary_v2 is on the dictionary's own cluster, system_identifier 7693849764459364596, port 5434, at /Volumes/LLM Models/dictionary/postgres/18/data. The checkout is /Volumes/LLM Models/dictionary/src/dictionary. The internal copies are gone; the rollback is the verified bundle 2026-10-05-v2 (MANIFEST.json 5d362034…98e63ff, in two places)
    The two migrations reach it, in order Done. 27 migrations, head 20260927220937, none pending; #212's 20260927204902 was applied on 7 October as a recorded step. SELECT count(*) FROM local_model_configs is 0, as expected
    Processes quiesced as #211 records Changed. The dev server and the ngrok tunnel are user LaunchAgents since 8 October (com.razrfly.dictionary.phx-4007, com.razrfly.dictionary.ngrok), not Terminal tabs. Quiescing is launchctl bootout gui/$(id -u)/com.razrfly.dictionary.phx-4007; resuming is launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.razrfly.dictionary.phx-4007.plist. The Postgres agent stays up. Every mix task against the corpus runs with DD_NO_OBAN=1
    The owner has decided Decided 8 October, in chat, taking the recommendations (change any line here before the run starts): (1) the corpus catch-up is option A of corpus-catch-up.md, with the six Wikipedia artwork images as they are today (no code change to hold them back); (2) the population is the committed selection rule, re-derived from a fresh export as the body requires, so the count may move from 170; (3) the reviewer is the owner's account, the one that already carries reviewer: true (there is exactly one such account among the two). The owner reviews each record from the manifest; no review file is written for anyone
    Step 5, close any public tunnel while internal reading runs Waived by the owner for the development deployment. The tunnel stays up and internal_reading stays on behind it, as it has since 7 October. Internal mode still comes from configuration, never from a request parameter, and never publishes or approves anything

    Facts the run should know

    • The routing ledger on the working corpus is empty: 0 pages, 0 public_paths, 0 classification_decisions, 0 backfill runs. The bundle's state.json recorded pages: 0 at the 5 October capture. The "12 earlier pages, 14 paths, 12 decisions and 42 ledger rows" that backfill.md describes were on the 27 September rehearsal copies, not on the corpus. CP4 starts from a clean ledger.
    • The corpus holds one active entity labelled Mars. Pre-#211 delivery: accept the routing backfill, make On the main reading entry, and consolidate main for the storage move #219's deity and album were fixtures on a copy and are not recreated.
    • Every reference in the body exists on main: docs/routing/stage-2/{backfill.md, corpus-catch-up.md, candidates.json, candidates.py, rehearsal/on_demo.exs}, ADR 0004, and the tasks dd.routing.audit, dd.routing.backfill, dd.routing.verify, dd.routing.guard, dd.snapshot, dd.bundle, dd.bootstrap, dd.doctor. dd.routing.verify's default baseline is 5434 since Document the installation as it is after #211's move and reclaim (C1, C9) #236. The backfill still refuses rehearsal reviews on any devils_dictionary_v2.
    • The corpus has moved on since the proposal was measured (13:29Z, 27 September): the catch-up's measured delta must be matched less what the corpus gained since, which is explained row by row before going on.

    Closing criteria, so one execution closes this issue

    • C1 Catch-up applied. Window proven write-free; a verified pre-catch-up bundle taken as the rollback point; the re-projection rehearsed on a scratch copy and its delta explained against corpus-catch-up.md; mix dd.materialize … --all --resolve run on the corpus for the six providers in the documented order with DD_NO_OBAN=1; mix dd.routing.verify --projected agrees between corpus and copy on everything but bookkeeping; the caught-up corpus bundled as the new accepted baseline, with its internal second copy, both verified; corpus-catch-up.md's status updated.
    • C2 Population re-derived from a fresh export of the caught-up corpus with the committed SQL, mix dd.routing.audit and candidates.py; bound to the export's and policy's digests; the differences from the committed candidates.json recorded; the new population committed.
    • C3 Run without reviews, twice; the second writes nothing (pg_stat_user_tables); the manifest kept; a review worksheet prepared for the owner from it.
    • C4 Reviews recorded by the owner, one per record the owner decided, each naming the fingerprint reviewed, under the owner's reviewer account; the reviewed run, twice; the second writes nothing; the candidate launch manifest kept.
    • C5 The journey verified on the working dataset, server restarted: search → On mars → the planet's card → its stored namespace address; back, reload, and the other states; a people subject and a works subject likewise; /define/* still 404; exact-ID behaviour as ADR 0004 §6.
    • C6 Snapshot and restore verified (mix dd.snapshot, restore into a scratch database on 5434, mix dd.routing.verify, scratch dropped); mix precommit on main; the doctor green; the dev server and tunnel back under their LaunchAgents.
    • C7 Record. Evidence under /Volumes/LLM Models/dictionary/private/<date>-224/; the handoff on Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194 updated (CP4 done; Stages 4 and 5 remain, and Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194 stays open for them); the docs' status lines updated in a small PR; this issue closed.

    Not in this issue, unchanged: publication, indexing, sitemaps and SEO (Stage 5); the page–composition binding (Stage 4); move, merge, split and retire administration.

  2. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    CP4 block 1 (C1): the corpus catch-up is applied. Stopped for independent audit before block 2

    8 October 2026, on main 9eb0ee4, on devils_dictionary_v2 (cluster 7693849764459364596, port 5434). Option A of corpus-catch-up.md, as the owner decided, with the six artwork images as they are. Nothing on 5432 was touched. Every mix task ran with DD_NO_OBAN=1. The dev server has been stopped since 09:34Z and stays stopped through block 4; the tunnel is up.

    Result. The catch-up delta matches the proposal exactly, less what the corpus gained after 13:29Z on 27 September, explained row by row: 20 creator identities minted after the capture. The caught-up corpus is the new accepted baseline, 2026-10-08-v2. Its MANIFEST.json is a8d61b11ea50532ba7d5967f639a52e0d149463e4f21d891aca03531cd0c4d22, held in two places, both verified. One finding needs the auditor's eye. mix dd.routing.verify --projected between the corpus and the copy exits 1, on 8 sections. Every differing row is a row the projection inserted, and differs only in inserted_at and its sequence-drawn id. Step 4 of option A assumed bookkeeping would be the only difference (see The projected verification below).

    Gate

    Check Result
    main 9eb0ee4 = origin/main, clean
    mix dd.doctor --expect-cluster 7693849764459364596 --volume "/Volumes/LLM Models" every core line green once the dev server was stopped. Before that, http_port was the only warning, because 4007 was held by the server itself
    Migrations 27, head 20260927220937, none pending
    Ledger on v2 pages, public_paths, classification_decisions, routing_backfill_runs (and _items) all 0
    Reviewer exactly one users row with reviewer = true (users.id 2). The owner confirmed in chat that it is theirs

    Quiesce, and the rollback point

    • launchctl bootout gui/501/com.razrfly.dictionary.phx-4007 at 09:34:11Z. Afterwards no client session but mine remained on the cluster.

    • Write-free over 78 s:

      Counter 09:34:22Z 09:35:40Z
      pg_stat_user_tables ins / upd / del 28,628,884 / 6,402 / 4,118 the same
      pg_stat_database tup ins / upd / del 28,959,534 / 8,040 / 4,327 the same
      other sessions on v2 0 0

      Only xact_commit moved, by my own read-only queries. The counters stayed equal from then until the catch-up began at 11:33Z.

    • Rollback point: the bundle 2026-10-08-v2-pre-catchup.

      • MANIFEST.json 08c31cd32fa6d572f45dc46b5814db3ccf9940a92f7ef39e0205c6a3edc0baef, dump 3c642924….
      • The window was quiet, and --verify checked all 11 files.
      • It was restored twice with mix dd.bootstrap --mode init: each restore equalled the bundle's captured state in every section. mix dd.routing.verify then matched one of the restores exactly against live v2.

    The rehearsal on a copy, and its delta

    The copy. devils_dictionary_cp4_proj was restored from the bundle and re-projected as the rehearsal did:

    1. Wikipedia and Wikidata, each replayed from an archive exported from the copy itself;
    2. materialize --all --resolve for wikipedia, wikidata, johnson, bierce, wiktionary and wordnet.

    Measuring the delta. It was measured against a frozen restore of the same bundle (cp4_pre), with the September method (delta.py, attribute.py). Two more frozen restores made it row by row:

    • the 27 September 13:29Z capture (v2-source.dump, a9384f44…), which gives exactly what the corpus gained since;
    • the proposal's own endpoints, B0 and C1 (8090988c…, b78aa6c8…). Their delta, recomputed with every key kept, reproduces the proposal's table count for count.
    Table Proposal: before → after Today: before → after Added: proposal / today Changed: proposal / today
    entities 100,795 → 100,795 100,919 → 100,919 0 / 0 8,973 / 8,993
    person_details 213 → 6,878 227 → 6,892 6,665 / 6,665 0 / 0
    work_details 7,690 → 9,771 7,805 → 9,886 2,081 / 2,081 0 / 0
    external_identifiers 104,576 → 104,589 104,787 → 104,803 13 / 16 8,770 / 8,770
    assertions 3,901,280 → 3,902,230 3,901,314 → 3,902,284 950 / 970 0 / 0
    assertion_revisions 4,373,560 → 4,376,966 4,373,877 → 4,377,303 3,406 / 3,426 2,456 / 2,456
    source_assertion_outputs 3,823,923 → 3,824,873 3,823,923 → 3,824,893 950 / 970 0 / 0
    source_materialized_outputs 2,106,046 → 2,115,016 2,106,206 → 2,115,195 8,970 / 8,989 0 / 0
    pending_relations 163,537 → 163,537 163,537 → 163,537 0 / 0 388 / 388
    actors 18 → 19 14 → 15 1 / 1 0 / 0

    Nothing was removed from any table in either measurement. Every other section is exact: objects, lexemes, senses, content, all routing tables, all curation tables, and every path and page resolution.

    Identical to the proposal:

    • the kinds: 2,081 concept → work and 2 concept → event;
    • 6,665 person_details and 2,081 work_details;
    • 8,770 identifiers re-pinned from null, with no value, object or status changed;
    • 2,456 Wiktionary edges each gaining one revision, with the same metadata changes (2,455 labels, and down → downlike);
    • 388 pending labels;
    • the provider-relationships actor;
    • the 23 artwork images, including the six that change picture (Mona Lisa to the C2RMF retouched scan, and so on);
    • 0 outputs retired, and every assertion with exactly one current revision.

    Why the "before" columns differ.

    Row by row, every difference is the 20 creators.

    • Every row of the proposal's delta is in today's. They are matched by primary key, or by natural key for rows the projection mints. No row of the proposal's delta is missing today.
    • Every row changed in both measurements is identical, bookkeeping aside, before (B0 against the bundle) and after (C1 against the copy). That is 8,973 entities, the 23 artworks among them; 8,770 identifiers; 2,456 revisions; and 388 pending edges.
    • Today has 102 extra rows: 20 + 20 + 20 + 20 + 19 + 3. Each belongs to one of 20 creator identities minted from Wikidata after the capture: 12 later on 27 September and 8 on 7 October. They are 19 people and one organization, Chantilly porcelain (Q3398119). No materializer had read them yet, which is the proposal's cause 3. Materializing them:
      • fills their display metadata (_projection_origin on all 20; wikipedia_title 20, commons_category 17, image 16, aliases 12, WordNet ids 2);
      • asserts each one's Wikidata instance_of: 20 assertions, 20 first revisions and 20 ownership rows;
      • gives the 19 people source-identity evidence and a source_identity:wikidata:… output;
      • adds Artsy artist slugs for three of them: Barye, Tait and Baudrillard.
    • The organization, like DK in the proposal, keeps its mint output and gains no source-identity output. That is why 20 creators give 19 outputs.
    • 0 rows are unexplained.
    • Note: two of the creators' object ids (3742806, 3742808) coincide with ids the fixture minted on B0. Both continued the same sequence from 13:29Z. On the corpus, they are the creators.

    No materializer, resolver or source-identity code changed between the proposal's fb9392f and 9eb0ee4.

    The catch-up on the corpus

    mix dd.materialize --source S --all --resolve on devils_dictionary_v2, in the documented order, with DD_NO_OBAN=1. There was no replay.

    Provider Records Exit M2 Corpus Copy 27 Sep (scratch)
    wikipedia 85,044 1 changed: entities 285 s 719 s 342 s
    wikidata 73,384 1 changed: 8 tables; 356 verifier_cache, 6 label_missing 336 s 759 s 393 s
    johnson 42,726 0 identical 261 s 776 s 324 s
    bierce 997 0 identical 243 s 249 s 307 s
    wiktionary 1,485,718 1 changed: assertion_revisions, pending_relations; 2,475,210 edges resolved 801 s 3,284 s 799 s
    wordnet 120,564 0 identical 311 s 312 s 374 s
    • On the corpus, the catch-up took 37 minutes, from 11:33Z to 12:10Z.
    • An exit of 1 means the run completed and found derived state changed, which is the catch-up. The corpus's run statistics equal the copy's line for line, for every provider.
    • The copy was slower because of concurrent restores, and because a fresh restore has no planner statistics. Each Wiktionary lookup of assertions by origin_key was a parallel sequential scan of 3.9 million rows, until an ANALYZE on the copy at 11:17Z. That changes only statistics. The corpus had its statistics.

    The projected verification between corpus and copy

    DD_DATABASE=devils_dictionary_v2 mix dd.routing.verify --baseline devils_dictionary_cp4_proj --projected exits 1:

    • 61 sections are exact, entities and pending_relations among them, and every path and page resolves the same;
    • 8 sections differ, each with the same row count on both sides.

    The verification leaves out only updated_at, materialized_at, last_seen_run_id and import_runs. Rows that a projection inserts carry their wall-clock inserted_at. The copy's Wikidata replay also minted the never-materialized records' rows before its full pass, so their sequence ids follow another order. The September stability runs could not show this: they re-projected an already caught-up state, which inserts nothing.

    compare.py matched every row of the ten delta tables between corpus and copy by natural key. Minted foreign keys were replaced by the natural key of the row they name. It compared every other column:

    Table Only on one side Content differs Same content, other inserted_at …and other own id
    actors 0 0 1 0
    external_identifiers 0 0 16 16
    person_details 0 0 6,665 —
    work_details 0 0 2,081 —
    source_materialized_outputs 0 0 8,989 —
    assertions 0 0 970 970
    source_assertion_outputs 0 0 970 (its assertion_id)
    assertion_revisions 0 0 3,426 970
    entities, pending_relations 0 0 0 0
    • The rows that differ are exactly the 23,118 the catch-up inserted, equal to the delta's "added" counts.
    • Their inserted_at falls inside each side's run: corpus 11:39–12:03Z, copy 09:43–11:18Z.
    • No id differs without inserted_at also differing.

    For the audit to decide:

    • whether this is accepted as "agreement on everything but bookkeeping";
    • or whether --projected needs an explicit, separately reviewed rule for rows a projection inserts.

    I changed no exclusion.

    The new accepted baseline

    Bundle MANIFEST.json SHA-256 Dump SHA-256 Where Verified
    2026-10-08-v2, caught up a8d61b11ea50532ba7d5967f639a52e0d149463e4f21d891aca03531cd0c4d22 5f31266b…512913e /Volumes/LLM Models/dictionary/bundles/ --verify, 11 files, and an independent shasum
    its second copy the same the same ~/Backups/dictionary/bundles/ (--transfer --internal --expect-manifest-sha256) --verify, 11 files, and an independent shasum
    2026-10-08-v2-pre-catchup, the rollback 08c31cd32fa6d572f45dc46b5814db3ccf9940a92f7ef39e0205c6a3edc0baef 3c642924… external volume --verify, 11 files
    • The baseline was captured with the window quiet: pg_stat_user_tables, sequences, catalog and sessions all unchanged.
    • Autovacuum was reclaiming the catch-up's 15.6 million updates during the capture. It moved only pg_stat_database's catalog counts.
    • The doctor is green on every core line afterwards. v2 is 13.5 GiB until autovacuum finishes.
    • The 2026-10-05-v2 baseline and its second copy are unchanged.

    Evidence, and what stays for the audit

    • Evidence is in /Volumes/LLM Models/dictionary/private/2026-10-08-224/, with SHA256SUMS: steps.log, source-counters.log, every command's output, cp4.sh, delta.py, attribute.py, reconcile.py, compare.py and their JSON.
    • Scratch databases on 5434, kept for the audit and dropped by exact name afterwards:
      • devils_dictionary_cp4_proj, the projected copy;
      • _cp4_pre, the bundle, frozen;
      • _cp4_s27, the 13:29Z capture, frozen;
      • _cp4_b0 and _cp4_c1, the proposal's endpoints, frozen.
    • corpus-catch-up.md's status line is updated in block 6's PR, as C1 asks.

    Stopped here. Block 2 (the population) starts after the independent audit.

  3. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    Independent audit of block 1 (C1), 8 October 15:20 (+02): passes, grade A. The projected-verify question is ruled: agreement. Continue with block 2

    Read-only against devils_dictionary_v2 on 5434, the five cp4_* copies, both new bundles and the evidence folder. Nothing written.

    Reproduced here

    Check Result
    Evidence folder private/2026-10-08-224/ 59 files; SHA256SUMS verifies, 0 mismatches
    Both bundles, by SHA-256 with mix dd.bundle --verify 2026-10-08-v2-pre-catchup: 11 files, MANIFEST.json 08c31cd3…baef. 2026-10-08-v2: 11 files, a8d61b11…cd22, both copies (external volume and ~/Backups/dictionary/bundles/), the two manifests byte-identical
    The write-free window source-counters.log: user-table n_tup_ins/upd/del unchanged from 09:34Z (t0) through the pre-catch-up bundle and the reference checks to 11:33Z (before-catchup); 0 client sessions other than the measuring one at every reading. The pre-catch-up copy cp4_pre matched live v2 exactly, 73 sections including sequences (verify-exact-pre-vs-v2)
    What the catch-up inserted into v2, by inserted_at ≥ 11:30Z assertions 970, assertion_revisions 3,426, person_details 6,665, work_details 2,081, source_materialized_outputs 8,989, source_assertion_outputs 970, external_identifiers 16, actors 1 (wikidata (provider relationships), 11:39:56Z); entities, objects, source_records, pending_relations 0. Sum 23,118, the session's number
    Against the proposal's table every "Added" count is the proposal's plus the 20 creators minted after the 27 September capture (19 people, 1 organization): 950 + 20, 3,406 + 20, 8,970 + 19, 13 + 3; person_details 6,665 and work_details 2,081 exactly; reconcile.out attributes each of the 102 extra rows to "entity added since" or "owning source record added or revised since" and reports no UNEXPLAINED row. The "Changed" counts match: entities 8,973 (+20 since), external_identifiers 8,770 re-pinned from null with no value, object or status change, assertion_revisions 2,456 superseded and still active, pending_relations 388 labelled. content-check-both: the changed rows' before-state equals the proposal's and their after-state equals today's copy's, 0 differences in all four tables
    Invariants on v2 now every assertion has exactly one current revision; kinds moved concept → work 2,081 and concept → event 2, nothing else; 0 materialized outputs retired; 6,665 new person rows are all persons; routing and curation tables untouched (0 pages, paths, decisions, runs, compositions); 27 migrations, head unchanged
    Per-provider stats, copy vs corpus identical line for line on the two I read in full (Wikidata: 73,384 records, 51,959 concept relations, 73,026 concepts; Wiktionary: 1,485,718 records, 1,470,174 lexemes, 1,776,553 senses, 2,475,210 edges). Each task exits 1 with "semantic replay changed derived state", which is the tool saying the projection changed something, the catch-up's purpose, not a failure
    State dev server stopped (no listener on 4007, the phx-4007 agent booted out), tunnel up and answering 502 as expected, Postgres agent up; v2 14 GB; five scratch copies 48 GB on the external volume

    The ruling: --projected DIFFERS on 8 tables is agreement on everything but bookkeeping

    mix dd.routing.verify --projected leaves out updated_at, materialized_at and last_seen_run_id and compares everything else by exact id. The eight tables that differ (actors, assertions, assertion_revisions, external_identifiers, person_details, work_details, source_assertion_outputs, source_materialized_outputs) are exactly the tables the catch-up inserted into, and every other table, entities and pending_relations included, is exact. compare.py then matches every row of those eight tables by natural key, replacing projection-minted ids with the natural key of the row they name (an assertion by its source and origin key, a revision by its assertion and number, an actor by kind and label), and reports for all eight: 0 rows only on one side, 0 rows whose content differs; the only differences are inserted_at on 23,118 rows, and the id on 1,956 of them, with other_id_without_other_inserted_at: 0, so no row differs in id without also being one the projection inserted.

    A row a projection inserts necessarily carries the time it was written and the next sequence value; two databases projected separately from one snapshot cannot agree on either, and neither is content. The September stability runs could not meet this because they re-projected a caught-up state and inserted nothing. So this is agreement on everything but bookkeeping, with bookkeeping now understood to include inserted_at and minted ids of rows the projection inserted. No exclusion was widened: the rule was applied to a case the rule's authors had not met, and the proof is row-level, not a count. Option A step 4 should say so (block 6's PR), and dd.routing.verify --projected should learn the same natural-key matching so the next recovery that inserts can verify without a side script: a small follow-up issue, not a blocker.

    Noted, none blocking

    1. ANALYZE after a restore. Wiktionary took 3,284 s on the fresh copy against 801 s on v2 until the copy was analysed. dd.bootstrap should run ANALYZE at the end of a restore, and the recovery runbook should say so: a follow-up with the one above.
    2. v2 grew from 9.6 GB to 14 GB. The catch-up updated bookkeeping on nearly every source_records and pending_relations row, so the growth is dead tuples. Autovacuum will reclaim it; a VACUUM (ANALYZE) in the quiet window is optional and does not change what any verify compares.
    3. pg_stat_database moved by +15/+56/+10 during the baseline bundle while the user-table counters stayed flat: catalog maintenance (autoanalyse writing pg_statistic after the mass update), not user data. The bundle's own quiet-window proof is on user tables and passed.
    4. Copies. The pre-catch-up bundle exists on the external volume only; it is a transient rollback point, superseded by 2026-10-08-v2 once C6 passes. ~/Backups/dictionary now holds two second copies (5 and 8 October, about 4 GB each); the owner decides after C6 whether the 5 October copy stays.
    5. The five cp4_* scratch databases may be dropped now, by exact name.

    Verdict: block 1 accepted. Proceed to block 2. The new accepted baseline is 2026-10-08-v2, a8d61b11ea50532ba7d5967f639a52e0d149463e4f21d891aca03531cd0c4d22.

  4. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    CP4 blocks 2 and 3 (C2, C3): the population re-derived, the run without reviews done twice. Stopped: the owner reviews

    8 October 2026, on main 9eb0ee4, on devils_dictionary_v2 (cluster 7693849764459364596, port 5434). DD_NO_OBAN=1 on every mix task. Nothing on 5432 was touched. The dev server has stayed stopped since 09:34Z; the tunnel is up. The only client sessions on v2 were this run's.

    Gate, re-checked before block 2

    Check Result
    main 9eb0ee4 = origin/main, clean
    mix dd.doctor --expect-cluster 7693849764459364596 --volume "/Volumes/LLM Models" every core line green; http_port 4007 free
    Migrations 27, head 20260927220937, none pending; local_model_configs 0
    Ledger on v2 before block 3 pages, public_paths, classification_decisions, routing_backfill_runs, routing_backfill_items all 0
    Reviewer exactly one users row with reviewer = true (users.id 2), the one the owner confirmed as theirs in block 1
    Write counters at 13:08Z equal to the post-baseline reading of 12:18Z (source-counters.log): nothing wrote to v2 between the baseline bundle and this block

    Block 2 (C2): the population

    Step Result
    Export, the committed policy-export.sql against v2 173,949 lines, SHA-256 a33f1bc8c62b630be701d84b48f994fcfb6c127a8d4012bf492911cdb1b55fd0; read_only: on; 2 s; pg_stat_user_tables and pg_stat_database write counters identical before and after
    mix dd.routing.audit 100,919 entities, 73,028 class-evidence records; policy 1.0.0, policy_sha256 54695faa… (unchanged); input_sha256 equal to the export's; 5 s
    candidates.py --audit … --input … 170 records, file SHA-256 c2cbd2ef7703b7a01acbd9e3f63051097779ce75ebed874d83ce408ded553d51, records digest c493ba0e…; selection_missing_from_audit: [], global_proposal_conflicts: 0; its 10 tests pass

    Against the committed candidates.json (27 September), record by record (population-diff.py):

    • the same 170 object ids, none added or removed; the same 11 collision groups with the same members;
    • the same disposition, status, family, candidate families, candidate path, description and selection reason on every record;
    • 14 records differ, in stored_kind only: concept → work on 12 (Butterfly 1874255, 1880172, 1885840, 1886567, 1887631, 1888104, 1888558, 1890422; Human 1874309, 1887405; Crocodile Tears 1878370; Wicked: For Good 1882172) and concept → event on 2 (Apollo 8 1878949, Apollo 15 1896394). That is what the stage-2 README said the catch-up would change, and the policy does not classify by stored kind;
    • deferred_outside_population 100,614 → 100,749: the 135 entities the corpus gained after the 27 September capture;
    • summary.inputs, source and sha256_of_records name the new export and audit. The records digest c493ba0e… is the one the 28 September re-run derived on a caught-up copy.

    The new population is committed in #240 (candidates.json, and the README's population bullet), byte-identical to the file the backfill read.

    Block 3 (C3): the run without reviews, twice

    DD_NO_OBAN=1 mix dd.routing.backfill --snapshot routing-input.jsonl --population candidates.json --manifest … on v2. Run key 7008c6f3ddbf434a1538b724cf043d6de250fa2444ed8e76b274c7e32f4305e5.

    Run Records Result Elapsed Wrote
    1 170 129 awaiting_review (108 draft pages), 41 not_addressed 2,363 ms 170 classification_decisions, 108 pages, 170 routing_backfill_items, 1 routing_backfill_runs (+1 update, finished_at); 0 public_paths
    2, the same inputs 170 the same 916 ms nothing: every pg_stat_user_tables n_tup_ins/upd/del of pages, page_revisions, page_memberships, public_paths, classification_decisions, routing_backfill_runs, routing_backfill_items, entities and objects equal before and after (ledger-counters.log); the manifest byte-identical to run 1's
    • Manifest SHA-256 77fa0849a3018d148def340f6c26f2f4dd8d9077698b0c96bf32aa756ca9392b (manifest-run1.json = manifest-run2.json). Its inputs name the export, policy and population digests above; reviews_sha256: null; publication_approved: 0.
    • No record was deferred as evidence_changed or input_changed, and none was refused. All 170 decisions are the evaluator's (origin evaluator, reviewer_actor_id null); every page is draft, active, 106 subject and 2 edition (the two editions of The Devil's Dictionary, as the rehearsal found); public_paths is 0.
    • Counts, dispositions, page roles and the 41 not-addressed records (38 no family evidence, 2 source pages, 1 identity-lifecycle review) are the 28 September rehearsal's exactly. Only fingerprints and ids are the corpus's own.

    The owner's review worksheet

    review-worksheet.md, .csv and .json in the evidence folder: 129 rows, one per record awaiting review, in object-id order, each with object id, label, stored kind, proposed family, candidate families, proposed or blocked path, collision group, evidence fingerprint, page id, the internal link /entities/:id/:slug (made by Routing.Links.entity_path/2) and the description.

    Review Rows What a confirmation needs
    Allocation candidates 50 the family (the proposed path is uncontested)
    Classification review: a sole candidate family 14 the family and a path, or defer
    Collision review, qualifier proposed 55 the path, named
    Collision review blocked on classification 3 the family and the path (/places/vik ×2, /works/crocodile-tears ×1)
    Duplicate-identity review 7 defer if one subject twice (a registry merge is outside the backfill), else the family and a distinct path (/works/love ×3, /works/mona-lisa ×2, /works/the-devils-dictionary ×2)

    The 11 groups are the README's: /works/butterfly 19, /works/human 15, /places/vik 7, /places/mount-tom 6, /people/billy-lee 5, /works/love 3, and 2 each for /places/daman, /organizations/the-creatures, /works/crocodile-tears, /works/mona-lisa, /works/the-devils-dictionary. The Markdown worksheet keeps each group together.

    One thing to know when deciding: 126 distinct fingerprints cover the 129 rows. The two editions (objects 3 and 6) share one, and the three Love works share one, because their evidence is identical. A review names the object id and the fingerprint, so this is not ambiguous, but it is also why the Love trio cannot be told apart from evidence alone.

    Evidence

    /Volumes/LLM Models/dictionary/private/2026-10-08-224/, SHA256SUMS regenerated: cp4-b2.sh (the steps), routing-input.jsonl, routing-audit/, candidates.json, candidates.out, population-diff.{py,json,out}, test-candidates.out, backfill-run{1,2}.out, manifest-run{1,2}.json, ledger-counters.log, source-counters.log, db-after-run2.out, entity-links.json, worksheet.py, review-worksheet.{md,csv,json}, steps.log. One export file taken at 13:08:54Z was discarded because the wrapper's timing stamp landed as its first line; the export was re-run at 13:09:30Z with the step fixed, and that is the one everything is bound to (steps.log records both).

    Stopped here. Block 4 writes only what the owner decides, per record, into a review file bound to population c2cbd2ef…, each confirmation naming the fingerprint above, under the owner's reviewer account. The five cp4_* scratch databases from block 1 are still on 5434 (48 GB), not dropped.

  5. added a commit that references this issue on Oct 8, 2026
  6. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    Independent audit of blocks 2 and 3 (C2, C3), 8 October 17:40 (+02): both pass, grade A. #240 merged with its one-line fix. Block 4 is the owner's reviews

    Read-only against devils_dictionary_v2 on 5434, the evidence folder and PR #240; nothing written to any database.

    Reproduced here

    Check Result
    Evidence folder 89 files; SHA256SUMS verifies, 0 mismatches
    The export routing-input.jsonl: 173,949 lines, SHA-256 a33f1bc8…, the digest the audit, the population and the manifest all name
    The population candidates.json in the evidence is byte-identical to the file on #240's head (c2cbd2ef…); 170 records, 11 groups, deferred_outside_population 100,749 = 100,919 − 170; inputs name the export (a33f1bc8…), the audit (ee31ddd0…) and policy 54695faa… 1.0.0; selection_missing_from_audit: [], global_proposal_conflicts: 0; records digest c493ba0e…, the same the 28 September re-run derived on a caught-up copy
    Against the committed population population-diff.out: same 170 ids, same 11 groups, 14 records differ in stored_kind only (12 concept → work, 2 concept → event), the catch-up's own change; the summary differs only in inputs, source, the records digest and the deferred count
    The runs ledger-counters.log: before run 1 every routing table 0; after run 1 classification_decisions 170, pages 108, routing_backfill_items 170, routing_backfill_runs 1 (+1 update, finished_at), public_paths, page_revisions, page_memberships 0, entities and objects untouched; before and after run 2 identical on every counter. manifest-run1.json and manifest-run2.json are byte-identical (77fa0849…, 2,396 lines): 170 items, 129 awaiting_review, 41 not_addressed, 108 with a page, 0 with an address, inputs binding export, policy and population, reviews_sha256: null
    The ledger on v2 now pages 108 (106 subject, 2 edition), all draft and active, none with a canonical; decisions 170, all current, none overridden: 112 mapped, 55 needs_review, 2 excluded_source_page, 1 identity_review; public_paths 0, route_changes 0; 1 run (7008c6f3…), 170 checkpoint rows, 129/41
    The corpus beside it entities 100,919; assertions 3,902,284; revisions 4,377,303; objects 3,741,781; identifiers 104,803; composition items 0: the catch-up audit's numbers, unchanged
    The worksheet review-worksheet.csv 129 rows (+ header), .md 203 lines: object id, label, stored kind, proposed family, candidate families, proposed or blocked path, collision group, fingerprint, page id, internal link, description; grouped as the report says, the 11 collision groups kept together
    State reviewer rows 1 (users.id 2); no client session on v2 but the auditor's; dev server booted out (no phx-4007 agent, no listener on 4007), tunnel up answering 502, Postgres up; the five cp4_* scratch copies still present, 48 GB

    #240

    The PR changes two files, +23/−23: the population file and the stage-2 README's population bullet. CodeRabbit's one finding was right: the README still said 100,614 entities stay deferred, the 27 September remainder; on the caught-up corpus it is 100,749. I made that one-line fix on the branch and merged #240 with a merge commit, branch kept. Nothing else in the PR changes code.

    Read, for the owner's benefit

    • The 14 classification-review rows each have a sole candidate family, and none of the proposals looks wrong on its face (the Polish language → concepts, the Polish breed → nature, Mars and Mercury the planets → nature, Zeus the god → subjects, Zeus the fish genus → nature, the United States → places, Warsaw → places, the Wikimedia Foundation → organizations). They still need the owner's confirmation, one by one, because the policy deliberately holds a sole candidate for a human.
    • 126 fingerprints cover 129 rows: the two editions share one, and the three Love works share one, because their evidence is identical. A review names the object id as well, so the file is unambiguous; the worksheet's labels, descriptions and links are how to tell the trio apart.
    • The 41 not-addressed records need no decision: 38 have no family evidence, 2 are source pages, 1 is the "Issue 84 split artwork" fixture awaiting identity review.

    Noted, none blocking

    1. The run key and the manifest bind the export, policy and population digests; the reviews file will add the fourth. Keep routing-input.jsonl and candidates.json exactly as they are until block 4 is done: a byte's change makes a new run.
    2. The scratch copies from block 1 may go now, by exact name, before block 5 restarts the server.

    Verdict: C2 and C3 accepted. Block 4 waits on the owner's decisions, transcribed by the session under users.id 2 and nothing else.

  7. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    The owner's decision for block 4, recorded 8 October 2026 at 18:30 +02 from the owner in chat with the steering session: My decisions are review-decisions-owner-draft.csv in the evidence folder, approved as written. The file sits in private/2026-10-08-224/ beside the worksheet, with its Markdown twin review-decisions-owner-draft.md: 124 confirm and 5 defer, each row naming the object id, the family and the path, nine of them with a path the owner names in place of the proposal. The executing session transcribes exactly that under users.id 2, each row with its fingerprint from the worksheet, regenerates SHA256SUMS to cover the two new files, and runs block 4 twice as C4 requires. The owner has also said that per-record review must not recur: a standing, owner-signed review rule is being added to the publication issue as a requirement before any second manifest.

  8. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    CP4 blocks 4, 5 and 6 (C4, C5, C6): the owner's reviews run, the journey verified, the snapshot restored exactly. C7 is this record, #241 and the close

    8 October 2026, on main 6d57a0c (#240 merged), on devils_dictionary_v2 (cluster 7693849764459364596, port 5434). DD_NO_OBAN=1 on every mix task. Nothing on 5432 was touched. Blocks 4 and 6 ran with the dev server stopped; block 5 ran with it up. The tunnel stayed up throughout, by the owner's waiver.

    Gate, re-checked before block 4

    Check Result
    main 6d57a0c = origin/main, clean
    mix dd.doctor --expect-cluster 7693849764459364596 --volume "/Volumes/LLM Models" every core line green; http_port 4007 free (doctor-before-block4.out)
    Write counters at 15:49Z equal to the after-run-2 reading of 13:11Z on every user table (source-counters.log, ledger-counters.log before-block4): nothing wrote to v2 between block 3 and block 4
    Reviewer one users row with reviewer = true: users.id 2
    The evidence folder's SHA256SUMS verified before anything was added; only the three running logs had moved, by the gate's own appends

    Block 4 (C4): the owner's reviews, transcribed and run twice

    The review file, reviews-owner.json (SHA-256 57e26405cce64fd8b05a1d79e3d811f85e91b0cd428a1862946cb07f7a987792), made by transcribe-reviews.py from review-decisions-owner-draft.csv (9a65149f…), the file the owner approved as written (comment):

    • 129 reviews, one per worksheet row: 124 confirm, 5 defer. Each row's action, family, path and reason are the CSV's bytes; the reason is the CSV's note. Each names evidence_fingerprint from review-worksheet.csv for its object id (126 distinct, as the worksheet), and the reviewer issue84-browser@local.test, users.id 2. The file binds population_sha256 c2cbd2ef… and records the run key and manifest it reviewed (7008c6f3…, 77fa0849…).
    • The script refuses anything else: an object not on the worksheet, a row decided twice, a confirmation without family and path, a path outside its family, a path not NFC or not lower-case, two confirmations of one path, a family outside the record's candidate families, a path that is another record's proposed qualifier. Nothing is decided in it.
    • Six confirmations take a path the owner named in place of the proposal (3, 6, 1886465, 1885953, 1893450, 1846395), and two take a duplicate-identity group's bare path (2, 1894610), as the CSV says.
    • Backfill.load was run read-only first (preflight-load-reviews.out): the file loads, 129 reviews, run key 9639fc28e445a579f656e426374569f4deeb734b55cc5229dacae00f4f1512b8.

    The runs, DD_NO_OBAN=1 mix dd.routing.backfill --snapshot routing-input.jsonl --population candidates.json --reviews reviews-owner.json --manifest … (cp4-b2.sh backfill):

    Run Records Result Elapsed Wrote
    rev1 170 124 allocated, 5 deferred_by_review, 41 not_addressed; 0 refused 2,453 ms 124 classification_decisions (+124 updates: the superseded evaluator rows), 16 pages (+124 updates: the canonical), 124 public_paths, 248 route_changes, 170 routing_backfill_items, 1 routing_backfill_runs (+1, finished_at); entities, objects, page_revisions, page_memberships untouched
    rev2, the same inputs 170 the same 921 ms nothing: every pg_stat_user_tables counter equal before and after (ledger-counters.log), the manifest byte-identical
    • The candidate launch manifest is manifest-rev1.json = manifest-rev2.json, SHA-256 12384745ff3b34bc2ef793e2a3f5e5e33a28cdd12dc564be0767873faee3fb6e. Its inputs bind the export a33f1bc8…, policy 1.0.0 54695faa…, population c2cbd2ef… and reviews 57e26405…; publication_approved: 0.
    • manifest-check.py: every record agrees with the owner's CSV, 124 allocated at exactly the CSV's path and family, 5 deferred with no address, 41 not addressed; the evidence fingerprint unchanged on all 170 against the run without reviews, so no review was stale and nothing was deferred as evidence_changed or input_changed.
    • db-after-rev2.out: 2 runs; the reviewed run's 170 items 124/5/41; 0 refused in any run; 124 public_paths, all canonical; 124 pages, all draft and active, each with its canonical (122 subject, 2 edition); decisions 294: 124 superseded evaluator rows, 46 current evaluator rows (the 41 not addressed and the 5 deferred), 124 current overrides, every one by the actor of users.id 2; every allocated path's page is the item's own object; the five deferred have no path and no override; no other client session on v2.
    • The 16 pages are the records that had none after block 3: the 14 classification reviews and the two duplicate-identity confirmations. The three blocked collision rows and the 14 classification rows now carry the owner's family (manifest-check.out: family and status differ from the run without reviews on 17 records).

    Block 5 (C5): the journey on the working dataset

    The dev server was started under its LaunchAgent at 15:52Z (config/dev.exs, internal reading on; the tunnel up by the owner's waiver). From the desktop app's built-in browser against localhost:4007, recorded in c5-journey.md, c5-http-statuses.out and c5-screenshots/:

    Step Seen
    search → Enter on mars /on/mars: "1 subject · 1 at their own address"; the one card Nature · Draft · Mars, /nature/mars
    the card /nature/mars: Nature · Draft · Mars, "On mars" back, Q111; About this address: /nature/mars, "subject page, a draft: not public", allocated 8 October 2026 by routing backfill (#194 Stage 2), "approved by reviewer #2 — classification review: sole candidate", "Nature, confirmed by Account #2 · policy 1.0.0", evidence 21f2dbd22a89e815…
    back, reload /on/mars again; a direct GET of /nature/mars is the same page, 200
    a people subject, a works subject, an edition /people/ambrose-bierce (People · Draft, Q191050); /works/the-devils-dictionary (Works · Draft, Q1197843, editions: Project Gutenberg #972); /works/the-devils-dictionary-gutenberg-972 (Works · Draft, 997 contents, "an edition of The Devil's Dictionary")
    the other states /on/love: love Concepts · Draft /concepts/love, the three Love works "No public address yet"; /on/cat: two addressed, two "Awaiting classification review", five "Unclassified"; /on/human: 16 addressed, the disambiguation page "A source page, not a subject", seven awaiting review, "9 more with this name not shown"
    /define/* 404 (/define/mars, /define/)
    exact ID (ADR 0004 §6) /entities/1831413/mars 200, the exact-identity page; /entities/1831413/wrong → /entities/1831413/mars; /words/999999999/x 404
    statuses /nature/Mars and /nature/mars/ 301 → /nature/mars; /places/vik-sømna 301 → its percent-encoded form, 200; /subjects/plan%C4%93 200; /nature/marz 404; /works/love 404 (deferred, so nothing holds the bare path)

    Not demonstrable here: the public-mode "not yet public" card, because the development configuration turns internal reading on for every reader; #219 B6's tests cover it. /entities/999999999/x answers 200 with the exact-identity LiveView's own "no such thing" page, as before #219; the ADR's 404 for a missing id is /words/:id/:slug's, which it gives.

    Block 6 (C6): snapshot, restore, verify; precommit; the doctor; the server back

    Server stopped at 15:57Z (no listener on 4007; no client session on v2 but the measuring one).

    Step Result
    mix dd.snapshot --out scratch/dumps/v2-c6.dump 1.2 GB in 82 s, SHA-256 fd34a0bc…; the .routing.json sidecar records cluster 7693849764459364596, oid 16385, and the routing digests: 294 decisions, 124 pages, 124 paths, 248 route changes, 0 revisions and memberships
    mix dd.snapshot --restore … --database devils_dictionary_cp4_c6 79 s, into a new scratch database on 5434
    DD_DATABASE=devils_dictionary_cp4_c6 mix dd.routing.verify --baseline devils_dictionary_v2 exact: 71 sections, every path and page resolves the same, 93 s (verify-c6-vs-v2.{out,summary})
    Write-free window user-table ins/upd/del 31,130,738 / 15,618,722 / 2,480,244 before the snapshot, after it and after the verify (source-counters.log); only pg_stat_database's tup counters moved, by catalog maintenance, as in block 1
    The scratch devils_dictionary_cp4_c6 dropped by name after the verify
    mix precommit (compile with warnings as errors, unused deps, format, the suite) 19 doctests, 2,658 tests, 0 failures in 342 s (precommit.out), on the code of 6d57a0c (the working tree was then on claude/224-close, which adds only docs); the tree clean afterwards
    mix dd.doctor every core line green (doctor-after-c6.out); the two optional warnings (.env refresh seconds; the runtime service bound to devils_dictionary_runtime_bench) are the ones every earlier doctor in this issue reported
    Server and tunnel the phx-4007 agent bootstrapped at 16:02Z; /nature/mars 200 on 4007 and the tunnel answering 200; the ngrok agent running throughout

    The ledger counters were read again before the snapshot: unchanged since rev2 (294/124/124/340/2), so the server's block-5 window wrote nothing to the routing tables.

    C7: the record

    • Evidence: /Volumes/LLM Models/dictionary/private/2026-10-08-224/, SHA256SUMS regenerated over every file: block 4's transcribe-reviews.{py,out}, reviews-owner.json, preflight-load-reviews.out, backfill-rev{1,2}.out, manifest-rev{1,2}.json, manifest-check.{py,out}, db-after-rev2.{sql,out}; block 5's c5-journey.md, c5-http-statuses.out, c5-screenshots/; block 6's cp4-b4.sh, snapshot-c6.out, scratch/dumps/v2-c6.dump (+ sidecar), restore-devils_dictionary_cp4_c6.out, verify-c6-vs-v2.{out,summary}, precommit.out, doctor-{before-block4,after-c6}.out, and the running logs.
    • Docs: #241, four status lines (backfill.md, corpus-catch-up.md with step 4's finding, the stage-2 README, the README).
    • The handoff on Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194 follows; this issue closes after the independent audit.

    Left as they were, for the owner: the five cp4_* scratch databases from block 1 (48 GB on 5434), the pre-catch-up bundle, and the v2-c6.dump (1.2 GB) in the evidence folder. Publication, the standing review rule and the reviewer's tools are #237's.

  9. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    Independent audit of blocks 4, 5 and 6 (C4, C5, C6) and the C7 record, 8 October 18:18 (+02): all four pass, grade A−. One thing the report leaves out: the server writes to v2 whenever it is up. #224 may close; #241 may merge

    Read-only against devils_dictionary_v2 on 5434, the evidence folder, PR #241, the dev server on 4007 (GET only) and the tunnel. Nothing written to any database; nothing posted; git untouched.

    Reproduced here

    Check Result
    Evidence folder private/2026-10-08-224/ 125 files (124 in SHA256SUMS + the sums file); shasum -c: 123 OK, 1 FAILED: steps.log. Its recorded digest f1e975b2… is the file minus its last line, the 16:08:14Z "SHA256SUMS regenerated" entry appended after the regeneration. That is all. The posted block 4–6 comment is byte-equal to block4-6-comment.md
    The review file reviews-owner.json (57e26405…) 129 reviews, 124 confirm / 5 defer; every action, family, path and reason byte-equal to review-decisions-owner-draft.csv (9a65149f…, reason = the CSV's note): 0 mismatches; every evidence_fingerprint the worksheet's for that object id, 126 distinct; the 129 ids are exactly the CSV's and exactly the worksheet's; every review names issue84-browser@local.test = users.id 2, the one reviewer = true row of the two users; population_sha256 = sha256 of candidates.json = c2cbd2ef…; no rehearsal key at the top or on any review; it also records reviewed_run_key 7008c6f3…, reviewed_manifest_sha256 77fa0849… and transcribed_from the CSV with its digest and the approval comment. Every confirmed path is NFC, lower-case, inside its family, distinct (124), and the family is within the record's candidate families
    The owner's named paths Eight, not nine: 3 renamed (3, 6, 1886465), 3 blocked rows given a path (1885953, 1893450, 1846395), 2 duplicate-identity bare paths (2, 1894610). The draft's "four renamed" counted 1880893 But, which keeps its proposed path. All eight are in the ledger at exactly the CSV's path and family
    routing_backfill_runs 2 rows. Run 2: key 9639fc28…, input_sha256 a33f1bc8…, policy 54695faa… 1.0.0, population c2cbd2ef…, reviews_sha256 57e26405… = sha256 of reviews-owner.json, 170 records, 15:51:25–27Z. Run 1: 7008c6f3…, reviews null
    routing_backfill_items run 2: 124 allocated / 5 deferred_by_review / 41 not_addressed; run 1: 129 awaiting_review / 41; 0 refused in any run. The five deferred carry the CSV's reason verbatim, no path, no page
    ledger-counters.log before-rev1: decisions 170, pages 108, paths 0, items 170, runs 1. after-rev1 = before-rev2 = after-rev2 on every counter: decisions 294 (+124 upd), pages 124 (+124 upd), public_paths 124, items 340, runs 2 (+2 upd); entities 100,919 and objects 3,741,781 unchanged through all four readings. source-counters.log: after-rev1 = before-rev2 = after-rev2, 31,129,266 / 15,618,464 / 2,480,243
    The manifests manifest-rev1.json and manifest-rev2.json byte-identical, 12384745…; inputs = export a33f1bc8…, policy 54695faa… 1.0.0, population c2cbd2ef…, reviews 57e26405…; publication_approved: 0; 170 records 124/5/41; every allocated record's address and family equal the CSV's, every deferred has no address, none of the 41 is in the CSV; decided_by override 124 / evaluator 46; 122 subject + 2 edition, all draft
    The ledger on v2 now public_paths 124, all canonical; pages 124, all draft + active, each with its canonical (122 subject, 2 edition); classification_decisions 294 = 124 superseded evaluator + 46 current evaluator + 124 current overrides, every override's reviewer_actor_id → actor 5 → users.id 2, and each supersedes its own evaluator row (124); route_changes 248, all allocate, 124 operations, 15:51:26–27Z; for the 124 allocated items the path is canonical, the page's target is the item's object and the page's canonical is that path (124/124/124). The five deferred (3740879, 3734402, 3737584, 3739174, 3742759): 0 paths, 0 overrides, 0 pages, their current decision the evaluator's mapped
    The sidecar's digests on live v2 recomputed the way Routing.Recovery does (sha256 over md5(to_jsonb(row)) + newline, by id): decisions 178c774a…, pages 57746744…, paths 2386404a…, route_changes 8d31df15…, revisions and memberships e3b0c442… (empty): all six equal v2-c6.dump.routing.json. The routing ledger is byte-for-byte what C6 snapshotted
    C5 statuses, re-run on 4007 all 19 lines of c5-http-statuses.out identical: /nature/mars 200, /nature/Mars 301 → /nature/mars, /nature/mars/ 301, /nature/marz 404, /define/mars 404, /define/ 404, /on/mars 200, /entities/1831413/mars 200, /entities/1831413/wrong 302 → …/mars, /entities/999999999/x 200, /words/999999999/x 404, /works/love 404, /people/ambrose-bierce 200, /works/the-devils-dictionary 200, …-gutenberg-972 200, /places/vik-s%C3%B8mna 200, /subjects/plan%C4%93 200, /concepts/love 200, /works/mona-lisa 200
    C5 journey the seven screenshots show what c5-journey.md says: home → /on/mars "1 subject · 1 addressed", the one card Nature · Draft · Mars /nature/mars; the page with "On mars", Q111; About this address: /nature/mars, "a draft: not public", allocated 8 October 2026 by routing backfill, "approved by reviewer #2 — classification review: sole candidate", "Nature, confirmed by Account #2 · policy 1.0.0", evidence 21f2dbd22a89e815… = the worksheet's fingerprint for 1831413; /works/the-devils-dictionary Works · Draft, Q1197843, editions: Project Gutenberg #972; /on/love one addressed Concepts · Draft and three Works "No public address yet"; /on/cat two addressed, two "Awaiting classification review", five "Unclassified"
    C6 snapshot v2-c6.dump 1,254,411,129 bytes, SHA-256 fd34a0bc… = the sidecar's; sidecar names cluster 7693849764459364596, oid 16385, port 5434, counts 294/124/124/248/0/0; 82 s
    C6 restore and verify restore into devils_dictionary_cp4_c6 79 s, exit 0; verify-c6-vs-v2.summary: 71 sections exact (every table, schema, sequences) + "routing: every path and page resolves the same" + "matches … exactly"
    C6 write-free window source-counters.log: before-snapshot-c6 = after-snapshot-c6 = after-verify-c6 on user tables, 31,130,738 / 15,618,722 / 2,480,244; the pg_stat_database counters moved as in block 1
    The scratch pg_database lists 14: v2, the five cp4_{b0,c1,pre,proj,s27}, the test and bench databases, postgres, the templates. No cp4_c6
    mix precommit precommit.out: 19 doctests, 2658 tests, 0 failures, 342.4 s, exit 0 at 16:07:56Z
    mix dd.doctor doctor-after-c6.out: every core line green; the two optional warnings (.env refresh seconds; runtime bound to devils_dictionary_runtime_bench) are the ones every earlier doctor reported
    Server and tunnel phx-4007 agent running, pid 67812, beam.smp listening on 127.0.0.1:4007; ngrok agent running, pid 20468; https://wordhoard.eu.ngrok.io/ 200 and /nature/mars 200 through it
    PR #241 4 files, +11/−11, all Markdown (README.md, docs/routing/stage-2/{README,backfill,corpus-catch-up}.md); head 1ad54a7 is one commit on top of main 6d57a0c; mergeable; CodeRabbit and GitGuardian green. Every number in its lines is one I reproduced (124/5/41, 122 + 2, c2cbd2ef…, 9639fc28…, 100,919 at the export, 100,749 deferred by the population)
    handoff-194-comment.md every fact checks: baseline a8d61b11…, population c2cbd2ef…, runs 7008c6f3… and 9639fc28…, manifest 12384745…, 124 pages (122 + 2), five deferred named right, 71 sections, the evidence path, #241, the two follow-ups
    State main = origin/main = 6d57a0c; the tree on claude/224-close at 1ad54a7 = its origin, git status clean; 11 client sessions on v2, all the server's pool (16:02:44Z, LISTEN "public.oban_insert"); the five cp4_* copies present, 9.5 + 9.5 + 9.6 + 13 + 9.5 GB

    What the report does not say: block 5 wrote to v2, and the server still does

    The dev server is mix phx.server under the LaunchAgent, without DD_NO_OBAN, so Oban's discovery queue runs. Browsing the journey admitted 39 discovery runs (DevilsDictionary.Discovery, "visit-driven cultural discovery … approved adapters may resolve an eligible result to a durable registry identity"; actor 6, Wordhoard automatic discovery). In the block-5 window, 15:52–15:58Z, v2 gained 1,472 inserts, 258 updates, 1 delete on user tables (source-counters.log, after-rev2 → before-snapshot-c6): entities 74, objects 96, source_records 232, source_record_revisions 239, external_identifiers 133, work_details 68, source_materialized_outputs 100, discovery_results 245, discovery_mappings 36, discovery_runs 39, content_items 22, content_revisions 22, assertions 16, assertion_revisions 16, person_details 6, reconciliation_cases 1 (Mars imagery, Buzz Aldrin, The Cat in the Hat, …). The session's own ledger-counters.log records it (after-c6: entities 100,993, objects 3,741,877) and the verify summary carries the same counts; the comment says only that the window "wrote nothing to the routing tables", which is true, and nothing about the corpus.

    Since the restart at 16:02Z the queued jobs resumed and wrote again (entities +12, objects +24, source_records +88, external_identifiers +36, …): at my last reading the counters were 31,131,196 / 15,618,870 / 2,480,377, and two discovery jobs (2416, 2426) sit executing since 15:56–57Z, orphaned by the boot-out. Today v2 holds 101,005 entities and 3,741,901 objects, not the 100,919 / 3,741,781 the blocks before block 5 were measured on.

    What this does and does not change:

    • C4 is untouched. Nothing of it happened while the runs ran; the routing ledger is byte-identical to the snapshot (the six digests above).
    • C6 stands. The snapshot is of v2 as it was at 15:57:36Z, 74 entities past the baseline, the restore matched it exactly and nothing wrote between the snapshot and the verify. But v2-c6.dump is already behind v2, and the accepted baseline 2026-10-08-v2 (a8d61b11…) is the caught-up corpus before today's discoveries, not today's v2. Neither document claims otherwise.
    • The population's evidence is untouched: 0 new identifiers and 0 entity updates on any of the 170 objects; one new assertion revision (15:57:23Z) names Ambrose Bierce (object 1) as its object. A second run would need a fresh export in any case, as the docs say, and object 1's fingerprint may move then.
    • Ruling: not blocking. C5's text asks for the journey on the working dataset with the server up, and the working dataset is live by design: it moves whenever the server is up and someone reads. "Write-free" in this issue means with the server booted out, which is what every mix task in CP4 had. The record should say so plainly; this comment does, and Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194's handoff should carry one line: the corpus is live under discovery, so a recovery baseline is only ever the last snapshot taken with the server down.

    The two C5 caveats, ruled

    1. Public mode not demonstrable. config/dev.exs turns internal reading on for every reader, so the "not yet public" card cannot be shown on this server without a configuration change; Pre-#211 delivery: accept the routing backfill, make On the main reading entry, and consolidate main for the storage move #219 B6's tests cover it. Accepted. Note that the public tunnel renders the drafts (I fetched /nature/mars through wordhoard.eu.ngrok.io: Draft, "a draft: not public"). That is the owner's recorded waiver, not a C5 failure; but Say CP4 ran: the reviewed population is backfilled on the working corpus as drafts (#224 C7) #241's README.md keeps the line "A development server with drafts must not sit behind a public tunnel", which states the rule the waiver suspends. Fine as a rule; the owner knows the state.
    2. /entities/999999999/x 200. ADR 0004 §6's table says Unknown exact path/ID: 404, without name-based substitution or a label-only success page. A 200 "no such thing" page for an unknown id is not that; the report's reading that the prose's 404 is /words/:id/:slug's is only half the text. It predates Pre-#211 delivery: accept the routing backfill, make On the main reading entry, and consolidate main for the storage move #219, CP4 changes no code, and the behaviour for an existing id is as §6 says (200; a wrong slug 302 to the right one; /words/999999999/x 404). C5 is met for what CP4 can show; the missing-id 200 is a small follow-up issue on the exact-identity route (ReadingStatus should answer 404 before the LiveView), not a blocker.

    Noted, none blocking

    1. "Nine" owner-named paths are eight (above); the file and the ledger are right, the draft's arithmetic was not.
    2. The report's db-after-rev2 bullet says "no other client session on v2"; the file's last line says 11, because it ran at 15:53:36Z with the server already up (15:52Z). Harmless.
    3. Two Oban discovery jobs left executing since 15:56–57Z, orphaned by the boot-out; Oban's rescue will retry them or they can be left. Nothing of routing depends on them.
    4. steps.log gained one line after the regeneration; the next SHA256SUMS regeneration covers it.
    5. Left for the owner, as the report says: the five cp4_* copies (about 50 GB on 5434), the pre-catch-up bundle, v2-c6.dump (1.2 GB). The snapshot is the last state taken with the server down; the owner decides whether it or 2026-10-08-v2 is the rollback point from here.
    6. Follow-ups already named: natural-key matching in dd.routing.verify --projected; ANALYZE after a restore; and now the /entities/:id 404.

    Verdict: C4, C5, C6 and C7 accepted; grade A− (the execution reproduces exactly; the record left out the one thing its own counters show). #224 may close with this comment on it. #241 may merge as it stands.

  10. added a commit that references this issue on Oct 8, 2026
  11. holden commented on Oct 8, 2026

    @holden
    MemberAuthor

    Closed, 8 October 2026: C1–C7 met, each block independently audited

    The audit's one finding is on the record: the working corpus is live under the dev server's discovery queue and moves whenever the server is up (101,005 entities at the close, against the 100,919 the blocks ran on); the routing ledger and the population's evidence do not move that way. Left for the owner: the five cp4_* scratch copies (about 50 GB on 5434), the pre-catch-up bundle, v2-c6.dump, and the choice of rollback point. Publication, the standing review rule and the reviewer's tools: #237. Follow-ups named on #194: natural-key matching in dd.routing.verify --projected, ANALYZE after a restore, a 404 for an unknown id on /entities/:id/:slug.

  12. holden commented on Oct 9, 2026

    @holden
    MemberAuthor

    Steering session's re-verification of the close, 9 October 2026: the close stands. #224 as a whole: A−

    Read-only against devils_dictionary_v2 on 5434, the evidence folder, the server on 4007 (GET only), the tunnel and GitHub, after the second auditing session's block 4–6 audit (A−) and the close. Nothing written.

    Reproduced

    Check Result
    State #224 closed 16:21Z; #241 merged f9c653f (README and three stage-2 docs, +11/−11); the handoff on #194 at 16:21Z; main = origin/main = f9c653f, clean
    Evidence 126 entries in SHA256SUMS, 0 mismatches; v2-c6.dump 1.25 GB under scratch/dumps/; 7 journey captures under c5-screenshots/ plus the HTTP status table
    The owner's decisions reviews-owner.json: 129 reviews, 124 confirm / 5 defer, bound to population c2cbd2ef…, no rehearsal flag, reviewer users.id 2 (actor 5); compared row by row with review-decisions-owner-draft.csv: 0 mismatches, no row missing, every confirmation carries its fingerprint
    The reviewed run run 2 (9639fc28…, reviews_sha256 set, finished): 124 allocated, 5 deferred_by_review, 41 not_addressed; manifest manifest-rev1.json binds export, policy, population and reviews, publication_approved 0
    The ledger on v2 124 canonical paths, 248 allocate ledger rows (a path row and a page row each), no other operation; 124 draft pages (122 subject, 2 edition), every one with its canonical; decisions 170 current = 124 mapped overrides by actor 5 + 5 mapped evaluator (the deferred) + 38 needs-review + 2 excluded + 1 identity review, with 124 superseded evaluator decisions kept as history; every owner-named path present (/works/the-devils-dictionary-gutenberg-972, /works/johnsons-dictionary-1755-leme-transcription, /organizations/martins-famous-pastry-shoppe, /places/vik-sømna, /places/vik-grimstad, /works/crocodile-tears-novel, /works/the-devils-dictionary, /works/mona-lisa)
    Over HTTP, internal mode /nature/mars, /people/ambrose-bierce, /works/the-devils-dictionary, /works/mona-lisa, /on/mars 200; /places/vik-sømna 301 to its canonical encoding; /works/love (deferred) 404; /define/love 404; through the tunnel /nature/mars 200, by the owner's waiver
    Services phx-4007 (pid 67812), ngrok, Postgres all under their LaunchAgents

    Confirmed findings

    1. The corpus moves under the dev server. 101,005 entities now, +86 since 12:30Z on 8 October; 39 discovery runs since; the ledger did not move. This is the product working as designed (discovery on visit), not a defect, but it means "the accepted baseline" is a point in time and the next bundle should be taken before Routing: everything left to close #194 — Stage 5 publication behind a feature flag, then the reviewer tool (after #224) #237's publication run.
    2. /entities/<unknown id>/<slug> answers 200, where ADR 0004 §6 says 404 with no name-based substitution. A small fix; it is on Routing before launch: On overviews, standards-informed namespaces, and stable subject URLs #194's follow-up list and should land with Routing: everything left to close #194 — Stage 5 publication behind a feature flag, then the reviewer tool (after #224) #237's first PR.
    3. A devils_dictionary_test database was created on 5434 at 18:02 (+02), block 6's precommit: a run without MIX_TEST_PARTITION. Harmless on the dictionary's own cluster, but the rule is a private partition on every run; drop it with the scratch copies.
    4. The second auditor's note stands: the draft said nine owner-named paths; the ledger holds eight (But kept its proposal). The file and the ledger are right.

    Left on the installation, the owner's to decide

    Five cp4_* scratch copies (about 51 GB on 5434) and the stray test database: drop by exact name. The pre-catch-up bundle (external only) and v2-c6.dump: keep one as the rollback point for #237's publication run, or take a fresh bundle then, which is the better choice given finding 1.

    Verdict: the close stands. #224 as a whole is A−: every block proved before it was trusted and reproduced by two auditors; the minus is the 200 on an unknown entity id, the unpartitioned test run, and that the launch population's review had to be read row by row by a human once, which #237's standing rule exists to end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions