Skip to content

Build and verify the encyclopedia identity and relationship model - #76

Merged
holden merged 19 commits into
mainfrom
74-encyclopedia-model
Sep 10, 2026
Merged

holden merged 19 commits into
mainfrom
74-encyclopedia-model

Conversation

@holden

@holden holden commented Sep 9, 2026

Copy link
Copy Markdown
Member

Replaces the flat lexical model with an encyclopedia foundation: stable word identities, source-specific senses, typed entities, published content, and attributed, revisioned assertions with evidence and context. Ambrose Bierce is one person across biography and authorship; words, definitions, works, editions and examples remain distinct while connecting through typed relationships.

Ports the six existing source adapters and the connected read layer. Authenticated users can propose connections with revision-specific evidence; reviewers record accountable decisions. Public reads apply rejection/withdrawal policy from either endpoint and on historical connection URLs. Database constraints enforce registry/subtype consistency, endpoint compatibility, revision ownership and current-revision integrity.

The rebuild path retains lexical observations, selects conflicting publication/display projections deterministically, and preserves archived observation dates. Durability and extension tests cover source evolution, identity reconciliation and new relationship contexts. Independent-database comparison checks exact semantic multisets; replay checks derived state and revision histories rather than counts alone.

Validation results, source limitations and reproduction commands: docs/rebuild/completion-2026-09-09.md and its adjacent evidence directory. Verification uses clean disposable databases and archived inputs; the original development database is preserved, with no legacy-row migration.

ConceptNet production ingestion, Artsy integration and public-deployment hardening remain downstream work.

Review snapshot for #74; do not merge or close the issue until the remaining acceptance checks are complete.

Validation: mix precommit passes 721 tests; animals 44/44, emotions 43/43, culture 42/42 scorecard rows pass with no pending rows; all six source semantic replays pass; independent rebuilt corpora compare exactly. Desktop/mobile read-page checks are recorded. Warm-cache animals p95: page composition 44.191 ms, traversal 0.418 ms, search 69 ms.

Outstanding: authenticated contribution/review browser walk, final CLI/browser health agreement, disposable verification database cleanup, and investigation of 117,363 null claim-output run stamps in the latest diagnostic. These are explicitly retained in the completion report; the passing scorecard alone is not a full closure claim.

holden and others added 15 commits September 8, 2026 17:36
The first checkpoint of #74. No schema yet — this is the evidence the schema
decision has to rest on, and two of its findings changed the design.

**The audit is in the repo.** It was untracked, and #74 supersedes MVP-0's
"only design remains" reading on the strength of it, so it needs to be
referenceable. README §Status withdraws the claim that nothing left to build
changes the thirteen tables; the dated historical results stay as they were.

**Inputs are pinned.** `priv/sources/MANIFEST.json` + `mix dd.manifest` digest
all five archived inputs; Johnson's computed digest matches the sha256 that
`Catalog` has pinned since S5, which is how the mechanism was checked rather
than asserted. The one that matters is `raw-wiktextract-data.jsonl.gz`: 2.6 GB
from a **rolling** URL, and only 22,534 of its records ever reach
`source_records` because the 1.5M-row index is written straight to `lexemes`.
It is a required, unreproducible recovery input and the manifest says so.

`mix dd.export.replay` writes all 344,635 source records to a checksummed
gzipped archive (101 MB, 15.7 s). Wikipedia's 85,044 and Wikidata's 72,770
exist only inside the database — roughly three hours of batched API calls — so
this is those two sources' equivalent of a pinned dump. Without it P5 re-fetches
and gets whatever Wikipedia says that day, which is a different corpus.

**Gate 0 ran on the full corpus**, not a sample: 1,541,669 lexemes, 250,389
senses, 1,163,659 relations loaded into a scratch database as 1.79M objects and
2.03M assertion revisions, with revision depth so revision-selection cost is
actually exercised. Eleven integrity rejections, and three decisions settled by
measurement:

  * **The current revision is a partial `is_current` index, not a pointer.**
    0.256 ms p95 against 2.576 ms on the highest-degree node, because the
    partial index touches 41 buffers where the pointer touches 7,957. And
    `DISTINCT ON` beat the pointer too — the opposite of what the audit
    predicted, so neither claim gets repeated. Correctness is completed by
    deferred constraint triggers for exactly-one and `FOR UPDATE` as the write
    path's first statement; without it an adversarial ordering loses a write.
  * **Lexical relations stay in the assertion model.** 1,118 MB against
    `lexical_relations`' 730 MB — 1.53× for a full revision history. The typed
    fallback table planned as an escape hatch is not needed.
  * **A sense is identified by its content, never its position.** Proved on the
    real Wiktionary `bank/noun/1`, captured from the pinned dump: delete a
    meaning from the middle and "Money; profit." moves from position 5 to 4,
    keeps its identity, and the attachment made before the edit still means what
    it meant — while position 5 resolves to a different sense. Ambiguity has two
    routes and both are exercised, including a 0.897 score with a 0.000 gap.

Two things the gate found that reading a diagram would not. Guarding "exactly
one subtype" with an AFTER INSERT trigger left object 101 typed `entity` with no
entity row after its subtype was deleted, while `authored_by` still pointed at
it — so subtypes get AFTER DELETE guards, and an identity is retired, never
deleted out from under its claims. And declaring only lexeme→lexeme and
sense→sense endpoints silently dropped 36,434 real relations that are a meaning
pointing at a word without naming which meaning, which is exactly what §C says
to preserve without inventing a sense id.

ADR 0001 records the decisions and what was rejected. `docs/rebuild/` maps every
file to port/rewrite/retire and every one of the 37 scorecard rows to
retained/revised/retired — including E1, whose `migrations == 2` equality would
fail this rebuild on day one, and E3, which grades a rolled-back sketch by
`File.exists?`.

504 tests, 0 failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VDPp7Bwosav4Fs8fKtRug7
…eir proofs (#74)

**This commit does not compile.** It is a labelled checkpoint on a feature
branch, not a shippable state: the new model is in and green on its own tests,
and the port of the existing read/write/health/web stack onto it is half done.
`main` is untouched and MVP-0 is reachable at tag `mvp-0`. What remains is
listed at the bottom.

## What is done and verified

**The migration.** One `create_encyclopedia_schema` replaces the MVP-0 baseline
— #74 decides that a rebuild path over thirteen tables we are deleting is waste
when nothing is deployed and the data is disposable. 40 tables in
`devils_dictionary_v2`, which is a NEW database: `devils_dictionary_dev` is
untouched and stays as the matched baseline for benchmarks and P5's semantic
comparison.

The three rules no single constraint can express are all built and all tested:
at-most-one-subtype-of-the-right-kind by composite FK against a generated
column; at-least-one at COMMIT by deferred constraint trigger; kind immutable
by trigger. Plus the direction the Gate 0 spike found the hard way — deleting a
subtype row orphans its object while assertions still point at it — closed by
AFTER DELETE guards on all four subtypes. Predicate endpoint compatibility is a
real foreign key behind a BEFORE trigger, so it holds for insert_all and COPY,
which matters because the linker is entirely raw SQL.

**Schemas and contexts.** ~28 modules; `Registry` (identities, names, external
ids, revisions, retire/merge/split), `Claims` (assert, revise, withdraw,
evidence, review contexts, reviews, votes), `Corpus` (versioned payloads).
`source_records` keeps identity and `source_record_revisions` holds payloads
keyed by content hash — still taken before `trim/1` — so a cited revision is
never overwritten. `SourceRecord.raw` survives as a **virtual** field filled
from the current revision, which is what lets all six parsers and their 159
offline tests stay untouched by the payload moving tables.

**Predicates are data.** `priv/predicates/core.json`, read like `priv/scopes/`.
Thirteen predicates with enumerated endpoint pairs. A new relationship type is
a file entry, which is what E1 measures now that it has stopped counting
migrations.

**Bierce is one identity, with the right QID.** `people` is deleted; a person is
an entity plus `person_details`. The catalog seeds Bierce, his work and the
edition, and asserts `authored_by` and `edition_of` between them — verified
idempotent: seeding three times gives one Bierce, six entities, four assertions.
`catalog.ex` now carries **Q191050**, not the Q310190 it has held since S3,
which is Tobin Bell.

**28 rejection tests, green**, every one going through `Repo.query!` or a
multi-row INSERT rather than a changeset — because the linker, resolver and
scope builder are raw SQL and a rule they can walk around is not a rule. They
found two real bugs on the way in: `create_object` raised instead of returning
a changeset on a unique violation, and the subtype trigger rejected the
legitimate create-then-drop cascade.

Auth is `phx.gen.auth` with Swoosh (Local/Test adapters only; no API client, so
no second HTTP client — AGENTS.md says Req).

## What remains, and why it is not here

The port of 36 files onto the new model: `Lexicon` (started), `Browse`,
`WordPage`, `Materializer`, the six adapters' output maps, `Resolver`,
`Linker`, `ScopeBuilder`, `Health` ×3, two mix tasks, and 18 test files
including `WordFixtures`, which writes raw structs and is the highest-leverage
file in the suite.

Stopping here rather than pushing on: a regex sweep across `lexicon.ex`
produced `language_taguage_tag`, which is the kind of damage haste does to a
1,700-line SQL port. The rest wants the same care the schema got, not the same
speed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VDPp7Bwosav4Fs8fKtRug7
…aps (#74)

`mix precommit` green: **645 tests, 0 failures**, from a branch that did not
compile. The 2026-09-09 audit's continuation order, steps 1 and 2.

## The seven correctness gaps

**`Claims.revise/2` carried seven of thirteen context columns.** `valid_from`,
`valid_to`, `jurisdiction_entity_id`, `context_object_id` and `metadata` were
silently dropped, so a date-bounded claim became unbounded on a typo fix. All
thirteen carry now; naming a field with `nil` still clears it, because "absent"
and "present and nil" are different instructions.

**Revision switching rewrote history.** The outgoing revision was stamped
`superseded`, which is currentness written into the column that records what the
revision *asserted*. Only `is_current` moves now. `lifecycle_state` carries
forward too — without it, fixing a typo in a withdrawn claim's rationale put the
claim silently back into force. `reinstate/2` is the deliberate way back.

**Public reads ignored editorial decisions.** `incoming/2` and `outgoing/2` are
public readers and hide a claim whose standing review is `rejected` or
`withdrawn`, filtered before counts and before pagination, from both directions;
`visibility: :internal` is the review queue's reader.

**Pagination had no stable order.** Keyset on `id` with `next_cursor/1`, filters
first, so no row is skipped or repeated under a concurrent write.

**Review-context items were not tied to their endpoint.** A `BEFORE INSERT OR
UPDATE` constraint trigger resolves the cited revision to its owning object and
compares it to the endpoint the row names — so it holds for `insert_all` and
COPY, not only for changesets.

**`merge/3` and `split/3` were an audit event and nothing else.** Merge moves
names and external identifiers (an id already verified in that namespace lands
as a `candidate` rather than breaking the unique index), retires inputs as
`merged`, and leaves assertions alone — a revision is immutable and re-pointing
one would forge history. `resolve/1` follows the event instead. Split moves
nothing: every current assertion touching the input opens a
`reconciliation_cases` row naming the candidate outputs, and a person decides.

**`Registry.Object` declared no associations**, so four preloads in the rewritten
`encyclopedia.ex` would have raised at run time rather than at compile time.

## The port

Read layer, write path, health and web, onto `objects` + typed subtypes +
assertions. What survives unchanged is the reasoning: the seven-query word page,
the per-sense placement rule, `lookup/2`'s three steps, the `%` trigram operator
(43 ms against `similarity()`'s 420 ms), the linker's six rungs and four
corroboration steps, the resolver's preference order, the scope builder's three
rules, the demo seam, and every cap and order on the page.

**`Materializer` rewritten.** Revision insertion, not in-place upsert — and a
revision only when the text *actually differs*, which is what makes identical
input produce zero new semantic revisions. Output ownership by `last_seen_run_id`
in `source_materialized_outputs` / `source_assertion_outputs`, and `reconcile/2`
retires what a run stopped emitting: audit finding #1, that refresh was purely
additive. Retired, never deleted, and only ever this source's own output.

**`Absorb.SenseIdentity`** implements the content-matched policy Gate 0 proved:
part of speech, etymology number as a hard filter, and a `pg_trgm`-faithful
trigram score. Matched, new, or a `reconciliation_cases` row — never a reused
id. `similarity/2` is a pure function and `sense_identity_test.exs` checks it
against Postgres's own on eight real glosses, so "faithful" is checked rather
than intended.

**`Health.Parity` rewritten (M1).** Natural-key *presence* was the wrong
question: replacing every gloss in the database with `CORRUPTED` returned zero
gaps. It now compares content against the current revision, endpoints, presence,
and outputs a run should have retired — and `health_test.exs` runs the
corruption probe and asserts it fails.

**`Linker` rewritten.** Rungs 1–3 read a source's own sense mapping and write
`refers_to`; rungs 4–5 match a spelling and write `lexeme_entity_candidate`,
never sense equivalence (audit finding #7). A rerun may not resurrect a rejected
link — reviews are append-only rows and the rungs respect them — and the
automatic `confirmed` is gone: two methods agreeing is evidence, not a review
nobody performed. The reads stay set-based SQL; the write goes through
`Materializer.write_assertions/3`, so the ladder and the importer share one
revision policy, one idempotency key and one ownership rule.

**`pending_relations`** is where an edge waits whose other end does not exist.
Both endpoints of an assertion are real objects and NOT NULL, and inventing a
lexeme for a lemma no source listed is the identity-by-string mistake #74 exists
to end. #69 §4 keeps `to_lemma` forever; this is where. R2 reports it.

**Predicates as data.** `priv/predicates/lexical.json` and `taxonomy.json` add
the twelve source-native lexical types and Wikidata's P171/P279/P31/P13176, with
the three endpoint pairs Gate 0 found necessary — including `sense → lexeme`,
without which 36,434 of 1,163,659 relations were silently dropped. `entity/*`
expands at seed time. Scorecard **E1** stops counting `schema_migrations` and
measures what a source costs instead.

**`explicit_lemmas`**, the fourth scope rule, and `priv/scopes/culture.json`:
nepotism, bank, egomaniac. A lemma the index does not hold is reported, so a
typo reads as a typo rather than as a small scope.

## Defects the port found

- `Absorb.Batch` and two more sites selected the now-virtual `raw` column;
  every batched materialize would have failed at run time.
- `Sources.upsert_record/2` returned a record with no payload, so **every
  on-demand enrich materialized an empty map**.
- `wordnet.ex` still ran `UPDATE lexical_relations`. WordNet's sense keys are
  deterministic, so the edge now names its target meaning and the pass is gone.
- `corroborate_taxon`'s `EXISTS` had `OR` binding looser than `AND`, so every
  title match read as corroborated by a taxon name.
- A base rung and corroboration flip-flopped a confidence for ever; MVP-0's
  in-place UPDATE hid it, a revision history would not have.
- `Registry.create_sense/1` dropped `metadata`, which is where the QIDs the
  linker reads live.

`raw_sql_test.exs` is the guard for the class: no file may name a retired table,
and every table raw SQL does name must exist. `mix compile` cannot see any of
this — the three modules that are raw SQL alias nothing that was deleted — and
two of the defects above reached a green build that way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
…#74)

`mix precommit`: **659 tests, 0 failures.**

## D1–D4 — the audit's defects, reproduced through the application

`durability_test.exs`. Not spike SQL: every one of these makes an attachment in
the new schema through the real paths, then makes the source change underneath
it. That is #74's milestone 4 — "re-import and later source changes cannot
silently change what an existing attachment means".

**D1** a source withdraws one of its two meanings. The withdrawn sense is
retired, not deleted; the curator's `illustrates` claim still points at the
meaning it was made about; the other source's support for the same word is
untouched; and a scoped run that visited one record does not retire the rest.

**D2** the audit's own reproduction, in the application. `"Money; profit."`
moves from key `#1` to `#0` when the sense above it is deleted, **keeps its
identity**, and keeps the attachment made before the edit. A genuinely new
meaning gets a new identity. Byte-identical input writes **no new revision at
all**. Two indistinguishable candidates open a `reconciliation_cases` row rather
than being decided by a row id.

**D3** a rejected link stays rejected when its rung reruns, and is invisible
from *both* endpoints and from `count_incoming/2` and `count_outgoing/2` — not
just from the first page. A review queue still sees it.

**D4** the endpoint's text changes after a review. The pinned manifest still
names the revision that was displayed, the vote does not transfer to the new
revision, and the old review keeps its decision. A review context that cites a
revision of some other object is refused by the trigger, on raw SQL.

## E3 — the extension exercise replaces the sketch

`extension_test.exs`: a **translated poem passage**. A work, its poet, a
translator who is not its author, an edition, a passage that is an `excerpt_of`
the work, and a quotation drawn from it that a curator attaches to a meaning
with the passage's own revision as evidence.

It cost **zero migrations and zero new tables**: `excerpt_of` and
`translated_by` were already in `priv/predicates/core.json`, and `work`,
`edition`, `passage` and `quotation` were already kinds. The database refuses a
passage that is an excerpt of a *person*, as an endpoint rule rather than a
validation.

`docs/sketches/community_layer_migration.exs` is retired, and with it the four
places that referenced it. The layer it sketched is shipped schema now —
`users`, `actors`, `assertion_reviews`, `assertion_votes` — so `File.exists?` on
it was measuring a file. Its dated result is preserved in the sketches README;
what is withdrawn is only its use as a current measurement.

## The scorecard

Six new rows, D1–D6, each because the audit reproduced a defect none of the
thirty-seven could catch. Five are test-proved and say so, the way M3 already
is — "an attachment does not move when a source reorders" is a claim about a
*sequence*, and no query over the end state can see it.

**D6 checks presence and byte count, not digests.** Hashing 2.6 GB on every
scorecard run made `Score.rows/1` take a minute and time other tests out; the
cheap check still catches a missing or truncated archive, `mix dd.manifest
--verify` does the digests, and the row says which of the two it did.

## One more false constraint removed

`senses (source_id, external_key)` was **unique**, which contradicts the schema's
own docstring: the key is provenance, and a position-based key is reusable by
construction. A meaning that keeps its identity through a reorder takes a key
another sense still holds until its retirement lands — so the index refused
exactly the case #74 exists to make safe. It is an ordinary index now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
`mix precommit`: **672 tests, 0 failures.** #74's milestone 3, driven end to end
rather than described.

## The pages

**`/entities/:id/:slug`** — the page the whole rebuild was for. Bierce's
biography, the works he wrote and the definitions he wrote are three sections,
and **all three are the same object id asked a different question**. In MVP-0
that page could not exist: `people` held authors, `concepts` held encyclopedia
subjects, and nothing joined them.

The sections are predicates, not columns — `about`, `authored_by`, `edition_of`,
`published_in` — and beneath them is a connections list of *everything else*,
both directions. A page that renders only the predicates it anticipated is a
page that silently hides a claim, and #73 says a claim must appear from either
endpoint.

**`/connections/:id`** — one claim, from either side, with its rationale, its
evidence and counterevidence, its derived review state, its relevance score, and
a history that reads any revision. Nothing arranges "the same claim appears from
both endpoints": there is one directional row and the inverse is derived, so the
two ends cannot disagree — including about visibility, since every surface reads
through `Claims.incoming/2` and `outgoing/2`.

**`/connect`** — the composer, authenticated. Names help discovery and the
**selected id** is what the claim points at. The relation list is read off
`predicate_endpoint_rules`, so an artifact is offered `illustrates` and a word is
not, rather than from a list somebody keeps in step with the rules. A claim with
no rationale is refused: an assertion nobody explained is one nobody can review.
It is attributed twice, to the account and to the claimant, because #74 §B is
explicit that an account is never automatically the person it claims to be.

**`/words/:id/:slug`** — canonical, addressed by identity (ADR decision 10).
`/define/:slug` stays as the resolver, and when a slug names more than one
distinct lemma it now **offers the choice** instead of picking: `C++`, `C+` and
`c` are three identities that share a slug, and MVP-0 headed that page `-c-`. A
wrong slug on either canonical route redirects to the right one.

Scorecard **U1** grows from six routes to ten.

## The walk

`connected_flow_test.exs` is the acceptance proof, and every navigation is a real
click on a rendered link, so a section that renders but does not connect fails:

    /define/nepotism → Bierce's definition
      → /entities/<id>/ambrose-bierce → his biography *and* his definitions
        → The Devil's Dictionary → the Gutenberg edition → its contents
          → back to /words/<id>/nepotism

Plus: the artifact→sense illustration read from the claim page, the artifact's
page and the meaning's side with the same status; a rejected claim vanishing from
all three *and* from the counts; the sense-backed `refers_to` kept distinct from
a spelling-level candidate; and the composer refusing a pair the endpoint rules
disallow — by not offering it.

## One defect the walk found

The canonical route delegated to the resolver with `"id"` still in params, so it
matched its own clause for ever. And it rendered the *slug's* lookup rather than
the word the id named — `/words/<C++>/c` came back headed `c`, which is the
defect the canonical address exists to fix, reintroduced one layer up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
`mix precommit`: **682 tests, 0 failures.** The tooling #74's milestone 5 needs —
`mix dd.reset`, `mix dd.rebuild`, `mix dd.replay`, `mix dd.compare`, and the
page/traversal budgets — plus four defects that only a full re-import could
surface. Each is fixed, and each now has a test that fails without the fix.

## Sense identity collapsed two synsets into one

WordNet's *sequoia* the tree and *sequoia* the wood differ by the words "wood
of" — 0.79 trigram similarity, well past the 0.62 threshold — so the second
materialize pass matched them onto a single identity. **194 synsets** lost their
meanings that way, and every survivor rewrote its gloss one way on the first pass
and back on the second: four revisions in a single run, for ever. A permanent M2
failure that no row count would show, because no row is missing.

ADR decision 5 is **amended on the evidence** rather than quietly reinterpreted.
`Absorb.Source.sense_key_stability/0`: `:positional` is the default and the
assumption that costs nothing when wrong, and Wiktionary keeps it — a position
key is the whole reason #74 exists. WordNet declares `:stable`, because
`oewn-84481488-n#sequoia` is a synset id, which is exactly what WordNet promises
not to move. Under `:stable` the key is the identity and nothing is scored.

## The import path never called `reconcile/2`

`durability_test.exs` proved the function; nothing proved the import used it. The
first full run wrote **240,056 outputs with a null `last_seen_run_id`**, because
no caller on the path had ever opened a run — and an unstamped output is worse
than an unreconciled one, since reconciliation then cannot tell "no longer
published" from "never seen".

`Batch.run/3` now owns this: a run id per stage, every output stamped with it,
and `reconcile/2` after each batch, scoped to the records that batch visited. It
is at this level and not in each source deliberately — forgetting a keyword
should not be able to make refresh additive again.

## The index pass minted objects outside a transaction

`objects` and its subtype are two statements and the registry's constraint
trigger is deferred to **COMMIT**, so in autocommit the object's own commit asks
for a lexeme row the next statement has not written yet:

    ERROR 23000 object 374480 (kind lexeme) has no lexeme row

The Wiktionary index pass is the one writer outside `Materializer.run_batch/3`,
and it died on its first flush. One transaction per chunk fixes it.

The test needed a new capability: **the SQL sandbox hides this entire class**,
because a deferred constraint is checked at the outermost COMMIT and the sandbox
is one. `@moduletag :unboxed` in `DataCase` runs for real and truncates after —
and with it, the test reproduces the production error exactly.

## `mix dd.rebuild` passed a source where a scope belongs

`absorb/2`'s first argument is the scope; a source module looks up its own
source. Wiktionary and Wikipedia are the two that pattern-match it, and both
stages were lost to a `FunctionClauseError`. Also `opts[:live] or …` raised on
`nil` rather than reading as false.

## The rest

- **`mix dd.compare`** — P5's semantic comparison, eight dimensions reduced to
  the same normalised key on both sides (lemma, gloss, body digest, edge, QID,
  taxonomy, link, scope membership). The baseline's keys are streamed out and
  `COPY`d into a temp table beside the current query, so the set operations
  happen in Postgres and neither side is held in memory. `relations` unions
  `pending_relations` into the current side, because the baseline kept an
  unresolved edge as a row and the new model keeps it as a pending one.
- **`mix dd.replay`** now re-materializes while the open edges are still
  falling, capped at five passes. A single pass closes the graph only as far as
  the records happened to be ordered, and Wikidata's own absorb loops for that
  reason — a replay that did not would hand back a truncated taxonomy.
- **`Health.Budgets`** — rows P1 and P2, measured on the *highest-degree* nodes,
  warm, with the population stated beside the number.
- **`pending_relations(source_id, origin_key)`** — the second pass deletes by
  origin key the moment an edge becomes an assertion, and was seq-scanning half
  a million rows per batch.
- `DD_DATABASE` in `config/dev.exs`, for the third database P5's rebuildability
  proof creates and drops. `mix dd.reset` still demands the name of what it is
  about to drop.
- Docs: ADR decision 5 amended, the map's tables updated to the encyclopedia
  model, README status rows for P0–P3, and a dead sketch link fixed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
…em (#74)

`mix precommit`: **683 tests, 0 failures.** Both of these were dead code paths
until a real re-import walked them, and both now have a test that fails without
the fix.

## The scoped Wiktionary pass queried a column that no longer exists

`source_records.raw` became **virtual** at P1 — the payload moved to
`source_record_revisions` so a cited revision can never be overwritten — and the
phase filter that orders the scoped pass by scope reason still read it:

    ERROR 42703 (undefined_column) column s0.raw does not exist

It compiles, because `r.raw` is a real schema field. It just is not a column.
`mix dd.fixtures.capture` had the same defect in two more queries. Both now join
the current revision the way `Sources.with_raw/1` does, on
`revision_key == content_hash` — the two are the same value by construction, so
there is no pointer to drift.

The test runs the scoped pass end to end on a three-word dump, because that is
the only thing that would have caught it. Reverting the fix reproduces the
error verbatim.

## The raw-SQL guard was exempting every lowercase statement

`raw_sql_test.exs` scans for `FROM`/`JOIN`/`INTO`/`UPDATE` case-**sensitively**,
and `mix dd.compare` was written in lowercase — so the file most likely to name
a retired table was the one file the guard never read.

Making the scan case-insensitive is not the fix: `from the`, `into a` and
`update it` appear in every other sentence of prose, and it reported a hundred
English words. So the uppercase rule stays, the reason it exists is written down
where the next person will look, and `dd.compare` is rewritten in the house
style — where the guard promptly found it naming six retired tables.

That is correct, and `dd.compare` is the one legitimate exception: it reads the
**baseline** database, whose schema is the one #74 replaced. It is exempted by
name rather than by a pattern, so a second such file has to be argued for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
`mix precommit`: **687 tests, 0 failures.** The rebuild reached the stages that
only run once a whole corpus exists, and each of them had never been executed at
this scale before. Two failed loudly; the third succeeded and did nothing.

## `Mix.Task.run/2` runs a task once per session

A rebuild replays twice — Wikidata, then Wikipedia. `Mix.Task.run/2` returns
`:noop` after the first call, so the second stage printed

    ── wikipedia · Wikipedia: articles and candidates
      replayed                 priv/replay/wikipedia.jsonl.gz
      elapsed                  0 ms

and **85,044 records were not in the database**. Nothing failed; the rebuild
reported success for a stage that had not run. `Mix.Task.rerun/2` is the fix,
and the reason this is worth a test is exactly that a silent stage is worse
than a crashing one.

## The resolver sized its write for the scan

`@window 200_000` is the id range a scan covers. It is not a number of rows a
write can carry: an `assertion_revisions` row is twelve bind parameters, so a
window holding 30,767 matches asks Postgres for 369,208 of them against a limit
of 65,535. That is a protocol error, not a slow query, and the Wiktionary
backlog of 105,744 pending edges hit it immediately. The window still sizes the
scan; the write is now chunked for the wire.

## `write_assertions/3` needed a transaction of its own

Minting an assertion and writing its first revision are two statements, and
`assertion_has_one_current_revision` is deferred to **COMMIT**. Inside the
materializer's `Ecto.Multi` that is fine. `Absorb.Linker` has no transaction of
its own — it is raw SQL by design — so the ladder's first rung committed an
assertion with no revision:

    assertions 1236827 has 0 current revisions, expected exactly 1

This is the same defect as the Wiktionary index pass, in a second writer, which
is the argument for fixing it *inside* `write_assertions/3` rather than at each
call site. Within a `Multi` the nested transaction joins the enclosing one and
costs nothing.

## The tests

`deferred_constraints_test.exs` is unboxed and holds the family: the sandbox
wraps a test in one transaction, so a constraint deferred to COMMIT never fires
inside it, and every defect of this shape is invisible to an ordinary test.
Reverting either fix reproduces the production error verbatim — `assertions 5
has 0 current revisions` and `72,000 parameters, the maximum is 65535`.

The `Mix.Task.rerun` and scope-argument guards assert on the task's source,
because both are properties of the Mix runtime and of a callback's argument
order rather than of anything a test can call: proving them otherwise means
running two real replays and a 2.6 GB dump.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
…74)

`mix precommit`: **689 tests, 0 failures.** The Wikipedia replay found the first
place where #74's own fix creates a new obligation.

## Six records, one article

Giving a Wikipedia article a canonical publication identity is the point of the
change — it is what stops one article rendering six times, which the 7 September
audit reproduced. The consequence is that six probe records now resolve to the
same content key inside one batch, and Postgres refuses a statement that hits
the same conflict key twice. The replay died in its first batch of 85,044.

So the item is written once. But `own_outputs/5` still sees **every** row,
because each of those records genuinely does attest the article, and an
ownership table that recorded only one of them would be recording a fiction.

## Which makes "retire what a run stopped emitting" wrong as written

`reconcile/2` retired the *object* whenever an output went stale. With one owner
that is right. With several it is not: one probe ceasing to redirect to an
article is not the source withdrawing the article, and the old code withdrew it
anyway — while the other record's ownership row sat there, unretired, still
pointing at it.

Outputs are now marked retired first, and only then are the objects nothing
still attests withdrawn. The docstring already promised that a withdrawn
Wiktionary sense "cannot remove WordNet's support for the same word"; this is
the first configuration in which the code had to actually do it.

Both are tested through `Batch.run/3` — two records naming one article, then one
of them going quiet — and both tests fail without their fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
`mix precommit`: **689 tests, 0 failures.**

`mint/4` computed `fresh = wanted_keys -- Map.keys(existing_keys)`. `--` removes
**one occurrence per element**, so a key listed twice survived into `fresh`
twice, minted two objects, and `Map.new(Enum.zip(fresh, minted))` kept only the
second — leaving the first an object of its kind with no subtype row:

    object 1902950 (kind content) has no content row

This was invisible until P5d, because until Wikipedia had a canonical
publication identity no caller ever passed a duplicate key. It is fixed in
`mint/4` rather than at the call site: every identity in the registry is minted
through that function, and "the caller deduped" is not a property the function
should have to assume.

## And the reason the test did not catch it

`batch_test.exs` was sandboxed. Its two-records-one-article test wrote the
orphan quite happily, because the enclosing transaction never commits and a
deferred constraint is checked at COMMIT. It is unboxed now, which is what the
file was always testing for — the import path, where half of what can go wrong
is exactly this.

That is the third defect of this shape in one session (the index pass, the
linker's first rung, and now `mint/4`), and each was hidden by the same
mechanism. Any test that exercises a bulk writer belongs outside the sandbox.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
`mix precommit`: **691 tests, 0 failures.** The semantic comparison against
`devils_dictionary_dev` found two shortfalls, and neither was a bug in what the
new model *represents* — both were stages the pipeline never ran.

## The scope grows after the stages that read it

`wikidata_taxon` walks a taxonomy that does not exist until Wikidata has been
absorbed, so on Animals it adds **8,724 lemmas after Wiktionary has finished**.
*pica*, *loris* and *calyptra* sat in the scope with no senses — 3,333 of the
3,371 senses the comparison found missing. MVP-0 hit this too and closed it by
hand ("Wiktionary re-absorbed on the grown scope"); it is a stage now, so a
rebuild does not depend on anybody remembering.

## The taxonomy grows after the stage that writes it

Wikidata records `Q24089999 instance_of Q34038` — *Slippery Falls* is a
*Waterfall* — but P31 and P279 targets are **recorded, not chased**, so the edge
waits for its target to exist. Wikipedia introduces 3,350 of those targets after
Wikidata has finished, and nothing re-read Wikidata afterwards. That is the whole
of the taxonomy gap: 51,513 baseline edges against 48,163.

So the tail of a rebuild is now `wikidata-taxon → scope → wiktionary → resolve →
link`. None of it is a retry: each stage can only do its work once a later one
has run, and a straight line has to come back round.

## Which the resolver could not survive

Re-materializing a record re-creates the pending row for an edge already
resolved — 21,398 times on the second Wiktionary sweep — and the resolver
inserted a second assertion with the same origin key:

    duplicate key value violates unique constraint "assertions_origin_index"

It had its own answers to "is this claim already here?", "did anything change?"
and "who owns it?", which is three copies of a policy ADR 0001 says is one. It
goes through `Materializer.write_assertions/3` now, and sixty lines of duplicated
write path are gone with it.

## And a real bug that change exposed

`write_assertions/3` keyed everything on `{source_id, origin_key}` — including
claims with **no** origin key. But `assertions.origin_key` is nullable and its
unique index is partial: it does not cover NULLs. `{source_id, nil}` is not an
identity, it is the absence of one, and treating it as a key collapsed every
unkeyed claim in a batch into a single assertion. The resolver's own fixtures
caught it the moment it started using the shared writer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
`mix precommit`: **692 tests, 0 failures.** The corpus came out of the second
sweep with **110,331 Wiktionary claims withdrawn** — every one of them re-emitted
by the source and re-resolved by the very next stage. Two faults compounded, and
each is a rule the model needs stated properly.

## Ownership passes between stages of one import

The materializer emits an edge. If the target word is not in the batch the edge
goes to `pending_relations` and the **resolver** closes it later — so the
assertion is owned by the resolver's run, not the materializer's. Re-materialize
that record under a new run and `reconcile/2` sees an assertion output the
current run did not stamp, reads it as "the source stopped publishing this", and
withdraws it.

But the source did not stop publishing it. It published the same edge, and the
edge is what ownership is about. So an edge deferred to `pending_relations` now
**re-stamps** the assertion output for its origin key: this record still supports
this claim, whichever stage happens to close it.

## A withdrawn revision is never "unchanged"

The second half is why it stayed broken. `resolve-taxon` re-resolved all 110,331,
but `write_revisions/5` writes a revision only when a compared field moved — and
nothing about the wording had moved. The claim was re-asserted and remained
withdrawn, with a live ownership row pointing at it.

Currentness and lifecycle are separate columns by design, which is right; what
was missing is that they are not independent *here*. Every revision this function
writes is written active, so a current revision that is withdrawn always differs
from what is being asserted. It is compared now, and a re-asserted claim comes
back.

Both tested through the real path — resolve, re-materialize, reconcile — and the
test reports `withdrawn` without the fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
…h query (#74)

`mix precommit`: **693 tests, 0 failures.** The scorecard finally ran on the full
corpus, and found two things.

## M1: 99 gaps, all one shape

Wiktionary keys a record by etymology, so `bear/noun/2`, `/3` and `/4` all say
*bear* is a kind of *mammal*. That is **one claim and three attestations** — and
`pending_relations` was keyed by the edge alone, so two of those three records
lost their pending row before the resolver ever saw them. The claim was written
correctly; it just ended up with one owner where it should have had three, and
`Health.Parity` was right to call the other two missing.

`source_record_id` is part of that key now. And `write_assertions/3` deduplicates
for the **identity** while keeping every claim for the **ownership** — the same
rule content items got at P5d, when six Wikipedia probes resolved to one article.
This is the third time the model has said the same thing: an object is written
once, and everything that attests it says so.

## A health query that took 154 seconds

`union_covered/1` — A6's "attested by Wiktionary **or** linked to something" —
was written as `A OR EXISTS (…)`. The `OR` stops Postgres using a semi-join for
the EXISTS, so it re-ran the subquery once per scope member: 25,383 times, 154
seconds, and a pool timeout that killed the whole scorecard. The same set as a
union of two id sets is **176 ms**. Both halves still read the link rule from
`linked_to_word/1`, so there is still one definition of what a link is.

Also `mix dd.score` could not print the rows P2/P4 added: `heading/1` had no case
for `D` or `P` and crashed on the first durability row with a `CaseClauseError`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
`mix precommit`: **696 tests, 0 failures.** `mix dd.materialize --all` is what
grades M2, and it had never been run against the new model — its `@tables` list
still named `Lexicon.Entry` and five other retired schemas. A module named as a
**bare atom in a list** is the one place the compiler does not check that it
exists, so the task raised `Ecto.Queryable not implemented for Atom` on its first
line. `raw_sql_test.exs` guards that class now, beside the stale-table-name one.

With the task running, M2 reported WordNet, Wikidata, Bierce and Johnson
identical, and two sources not. Both were real.

## One article, two identities

`upsert_content/6` looked up an existing content item among **the records in this
batch**. But a content key is a canonical publication identity —
`article:Q2703156` is *San Jose scale* whoever asked for it — so as soon as the
two records that name it fell in different batches, the second minted a second
object. Both then owned the same `about` claim, and each re-materialize rewrote
its subject to whichever record ran last: eight revisions of one assertion, all
identical apart from the subject flipping back and forth.

The lookup is scoped to the source now. That is the third form of the same rule
— an object is written once and everything that attests it says so — after
content items at P5d and edges at P5h.

## An ambiguous sense minted a new identity every run

`SenseIdentity` returning `:ambiguous` means no *existing* meaning is claimed and
a person decides. It does not mean a fresh identity per import — but no id was
assigned, so `mint/4` made a new object for that key, every time. Sixteen new
senses on a byte-identical re-import, growing without limit.

The row this source published under this key last time is **ours**, not a
candidate. Reusing it is not matching on the key; it is declining to open a
second unresolved case for the same one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
@coderabbitai

coderabbitai Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

On-demand reviews are free for the next 10 days. After that, they cost $0.25 per reviewed file.

View limit details

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Essentials

Run ID: 704c592a-2ceb-42d6-8ea7-f7bf313f3af3

📥 Commits

Reviewing files that changed from the base of the PR and between a2ef867 and 293a8af.

⛔ Files ignored due to path filters (9)
  • docs/rebuild/evidence-2026-09-09/bierce-1280.png is excluded by !**/*.png
  • docs/rebuild/evidence-2026-09-09/bierce-375.png is excluded by !**/*.png
  • docs/rebuild/evidence-2026-09-09/edition-1280.png is excluded by !**/*.png
  • docs/rebuild/evidence-2026-09-09/edition-375.png is excluded by !**/*.png
  • docs/rebuild/evidence-2026-09-09/nepotism-1280.png is excluded by !**/*.png
  • docs/rebuild/evidence-2026-09-09/nepotism-375.png is excluded by !**/*.png
  • docs/rebuild/evidence-2026-09-09/work-1280.png is excluded by !**/*.png
  • docs/rebuild/evidence-2026-09-09/work-375.png is excluded by !**/*.png
  • mix.lock is excluded by !**/*.lock
📒 Files selected for processing (246)
  • .gitignore
  • README.md
  • config/config.exs
  • config/dev.exs
  • config/test.exs
  • docs/adr/0001-encyclopedia-model.md
  • docs/audits/2026-09-07-project-audit.md
  • docs/audits/2026-09-09-issue73/README.md
  • docs/audits/2026-09-09-issue73/precommit.txt
  • docs/audits/2026-09-09-issue73/reproductions.exs
  • docs/audits/2026-09-09-issue73/reproductions.txt
  • docs/audits/evidence-2026-09-07/animals-scorecard.txt
  • docs/audits/evidence-2026-09-07/emotions-scorecard.txt
  • docs/audits/evidence-2026-09-07/refresh-probes-output.txt
  • docs/audits/evidence-2026-09-07/refresh-probes.exs
  • docs/map/README.md
  • docs/rebuild/completion-2026-09-09.md
  • docs/rebuild/evidence-2026-09-09/browser-walk.md
  • docs/rebuild/evidence-2026-09-09/clean-animals.txt
  • docs/rebuild/evidence-2026-09-09/clean-culture.txt
  • docs/rebuild/evidence-2026-09-09/clean-emotions.txt
  • docs/rebuild/evidence-2026-09-09/clean-reset.txt
  • docs/rebuild/evidence-2026-09-09/corpus-metrics.exs
  • docs/rebuild/evidence-2026-09-09/corpus-metrics.json
  • docs/rebuild/evidence-2026-09-09/entity-projection-failure.json
  • docs/rebuild/evidence-2026-09-09/entity-projection-regression.txt
  • docs/rebuild/evidence-2026-09-09/final-comparison.json
  • docs/rebuild/evidence-2026-09-09/health.txt
  • docs/rebuild/evidence-2026-09-09/initial-comparison-failures.json
  • docs/rebuild/evidence-2026-09-09/initial-replay.txt
  • docs/rebuild/evidence-2026-09-09/johnson-resolver-refresh.txt
  • docs/rebuild/evidence-2026-09-09/page-profile-before.txt
  • docs/rebuild/evidence-2026-09-09/precommit.txt
  • docs/rebuild/evidence-2026-09-09/resolve-after-replay.txt
  • docs/rebuild/evidence-2026-09-09/resolver-tie-regression.txt
  • docs/rebuild/evidence-2026-09-09/score-animals.txt
  • docs/rebuild/evidence-2026-09-09/score-culture.txt
  • docs/rebuild/evidence-2026-09-09/score-emotions.txt
  • docs/rebuild/evidence-2026-09-09/semantic-replay.txt
  • docs/rebuild/evidence-2026-09-09/single-claim-comparison-failure.json
  • docs/rebuild/evidence-2026-09-09/wiktionary-index-regression.txt
  • docs/rebuild/port-checklist.md
  • docs/rebuild/score-rows-p5.md
  • docs/rebuild/score-rows.md
  • docs/sketches/README.md
  • docs/sketches/community_layer_migration.exs
  • docs/spikes/2026-09-gate0/01-schema.sql
  • docs/spikes/2026-09-gate0/02-integrity-output.txt
  • docs/spikes/2026-09-gate0/02-integrity.sql
  • docs/spikes/2026-09-gate0/03-load.sql
  • docs/spikes/2026-09-gate0/04-unsensed.sql
  • docs/spikes/2026-09-gate0/05-indexes.sql
  • docs/spikes/2026-09-gate0/06-plans.txt
  • docs/spikes/2026-09-gate0/07-switching.sql
  • docs/spikes/2026-09-gate0/08-exactly-one-output.txt
  • docs/spikes/2026-09-gate0/08-exactly-one.sql
  • docs/spikes/2026-09-gate0/09-concurrency.sql
  • docs/spikes/2026-09-gate0/10-reconcile.sql
  • docs/spikes/2026-09-gate0/11-drift.sql
  • docs/spikes/2026-09-gate0/12-drift-run3.sql
  • docs/spikes/2026-09-gate0/13-ambiguity.sql
  • docs/spikes/2026-09-gate0/README.md
  • lib/devils_dictionary/absorb/batch.ex
  • lib/devils_dictionary/absorb/clients/wikidata.ex
  • lib/devils_dictionary/absorb/linker.ex
  • lib/devils_dictionary/absorb/materializer.ex
  • lib/devils_dictionary/absorb/resolver.ex
  • lib/devils_dictionary/absorb/scope_builder.ex
  • lib/devils_dictionary/absorb/sense_identity.ex
  • lib/devils_dictionary/absorb/source.ex
  • lib/devils_dictionary/absorb/sources/bierce.ex
  • lib/devils_dictionary/absorb/sources/johnson.ex
  • lib/devils_dictionary/absorb/sources/wikidata.ex
  • lib/devils_dictionary/absorb/sources/wikipedia.ex
  • lib/devils_dictionary/absorb/sources/wiktionary.ex
  • lib/devils_dictionary/absorb/sources/wordnet.ex
  • lib/devils_dictionary/accounts.ex
  • lib/devils_dictionary/accounts/scope.ex
  • lib/devils_dictionary/accounts/user.ex
  • lib/devils_dictionary/accounts/user_notifier.ex
  • lib/devils_dictionary/accounts/user_token.ex
  • lib/devils_dictionary/claims.ex
  • lib/devils_dictionary/claims/assertion.ex
  • lib/devils_dictionary/claims/assertion_evidence.ex
  • lib/devils_dictionary/claims/assertion_review.ex
  • lib/devils_dictionary/claims/assertion_revision.ex
  • lib/devils_dictionary/claims/assertion_vote.ex
  • lib/devils_dictionary/claims/catalog.ex
  • lib/devils_dictionary/claims/connection.ex
  • lib/devils_dictionary/claims/contributions.ex
  • lib/devils_dictionary/claims/pending_relation.ex
  • lib/devils_dictionary/claims/predicate.ex
  • lib/devils_dictionary/claims/predicate_endpoint_rule.ex
  • lib/devils_dictionary/claims/review_context.ex
  • lib/devils_dictionary/claims/review_context_item.ex
  • lib/devils_dictionary/corpus/source_record_revision.ex
  • lib/devils_dictionary/encyclopedia.ex
  • lib/devils_dictionary/encyclopedia/concept.ex
  • lib/devils_dictionary/encyclopedia/concept_link.ex
  • lib/devils_dictionary/encyclopedia/concept_relation.ex
  • lib/devils_dictionary/encyclopedia/entity_page.ex
  • lib/devils_dictionary/health.ex
  • lib/devils_dictionary/health/budgets.ex
  • lib/devils_dictionary/health/coverage.ex
  • lib/devils_dictionary/health/parity.ex
  • lib/devils_dictionary/health/rebuild_snapshot.ex
  • lib/devils_dictionary/health/score.ex
  • lib/devils_dictionary/health/state_fingerprint.ex
  • lib/devils_dictionary/lexicon.ex
  • lib/devils_dictionary/lexicon/browse.ex
  • lib/devils_dictionary/lexicon/entry.ex
  • lib/devils_dictionary/lexicon/lexeme.ex
  • lib/devils_dictionary/lexicon/lexical_relation.ex
  • lib/devils_dictionary/lexicon/scope.ex
  • lib/devils_dictionary/lexicon/scope_lexeme.ex
  • lib/devils_dictionary/lexicon/scope_member.ex
  • lib/devils_dictionary/lexicon/sense.ex
  • lib/devils_dictionary/lexicon/word_page.ex
  • lib/devils_dictionary/mailer.ex
  • lib/devils_dictionary/registry.ex
  • lib/devils_dictionary/registry/content_item.ex
  • lib/devils_dictionary/registry/content_revision.ex
  • lib/devils_dictionary/registry/edition_details.ex
  • lib/devils_dictionary/registry/entity.ex
  • lib/devils_dictionary/registry/external_identifier.ex
  • lib/devils_dictionary/registry/identity_event.ex
  • lib/devils_dictionary/registry/identity_event_member.ex
  • lib/devils_dictionary/registry/lexeme.ex
  • lib/devils_dictionary/registry/lexeme_form.ex
  • lib/devils_dictionary/registry/object.ex
  • lib/devils_dictionary/registry/object_name.ex
  • lib/devils_dictionary/registry/person_details.ex
  • lib/devils_dictionary/registry/sense.ex
  • lib/devils_dictionary/registry/sense_revision.ex
  • lib/devils_dictionary/registry/work_details.ex
  • lib/devils_dictionary/sources.ex
  • lib/devils_dictionary/sources/actor.ex
  • lib/devils_dictionary/sources/assertion_output.ex
  • lib/devils_dictionary/sources/catalog.ex
  • lib/devils_dictionary/sources/manifest.ex
  • lib/devils_dictionary/sources/materialized_output.ex
  • lib/devils_dictionary/sources/person.ex
  • lib/devils_dictionary/sources/reconciliation_case.ex
  • lib/devils_dictionary/sources/source_input.ex
  • lib/devils_dictionary/sources/source_record.ex
  • lib/devils_dictionary_web/components/kit.ex
  • lib/devils_dictionary_web/components/layouts/root.html.heex
  • lib/devils_dictionary_web/components/word.ex
  • lib/devils_dictionary_web/controllers/user_session_controller.ex
  • lib/devils_dictionary_web/live/connection_live.ex
  • lib/devils_dictionary_web/live/entity_live.ex
  • lib/devils_dictionary_web/live/scope_live.ex
  • lib/devils_dictionary_web/live/user_live/confirmation.ex
  • lib/devils_dictionary_web/live/user_live/login.ex
  • lib/devils_dictionary_web/live/user_live/registration.ex
  • lib/devils_dictionary_web/live/user_live/settings.ex
  • lib/devils_dictionary_web/live/word_live.ex
  • lib/devils_dictionary_web/router.ex
  • lib/devils_dictionary_web/user_auth.ex
  • lib/mix/tasks/dd.absorb.ex
  • lib/mix/tasks/dd.compare.ex
  • lib/mix/tasks/dd.export.replay.ex
  • lib/mix/tasks/dd.fixtures.capture.ex
  • lib/mix/tasks/dd.health.ex
  • lib/mix/tasks/dd.manifest.ex
  • lib/mix/tasks/dd.materialize.ex
  • lib/mix/tasks/dd.rebuild.ex
  • lib/mix/tasks/dd.replay.ex
  • lib/mix/tasks/dd.reset.ex
  • lib/mix/tasks/dd.resolve.ex
  • lib/mix/tasks/dd.score.ex
  • lib/mix/tasks/dd.verify.rebuild.ex
  • mix.exs
  • priv/predicates/core.json
  • priv/predicates/lexical.json
  • priv/predicates/taxonomy.json
  • priv/replay/MANIFEST.json
  • priv/repo/migrations/20260905094219_create_lexicon_schema.exs
  • priv/repo/migrations/20260908153900_create_users_auth_tables.exs
  • priv/repo/migrations/20260908153949_create_encyclopedia_schema.exs
  • priv/repo/migrations/20260909115311_add_reviewer_role_to_users.exs
  • priv/repo/migrations/20260909121050_add_deterministic_source_projection_merge.exs
  • priv/repo/migrations/20260909122440_normalize_pronunciation_observations.exs
  • priv/repo/migrations/20260909134659_add_current_claim_page_indexes.exs
  • priv/scopes/culture.json
  • priv/sources/MANIFEST.json
  • test/devils_dictionary/absorb/batch_test.exs
  • test/devils_dictionary/absorb/concepts_test.exs
  • test/devils_dictionary/absorb/deferred_constraints_test.exs
  • test/devils_dictionary/absorb/linker_test.exs
  • test/devils_dictionary/absorb/materializer_test.exs
  • test/devils_dictionary/absorb/resolver_test.exs
  • test/devils_dictionary/absorb/scope_builder_test.exs
  • test/devils_dictionary/absorb/sense_identity_test.exs
  • test/devils_dictionary/absorb/sources/wikidata_absorb_test.exs
  • test/devils_dictionary/absorb/sources/wikidata_test.exs
  • test/devils_dictionary/absorb/sources/wikipedia_absorb_test.exs
  • test/devils_dictionary/absorb/sources/wikipedia_test.exs
  • test/devils_dictionary/absorb/sources/wiktionary_index_test.exs
  • test/devils_dictionary/absorb/sources/wiktionary_test.exs
  • test/devils_dictionary/accounts_test.exs
  • test/devils_dictionary/catalog_identity_test.exs
  • test/devils_dictionary/durability_test.exs
  • test/devils_dictionary/encyclopedia_test.exs
  • test/devils_dictionary/extension_test.exs
  • test/devils_dictionary/health_concepts_test.exs
  • test/devils_dictionary/health_coverage_test.exs
  • test/devils_dictionary/health_pages_test.exs
  • test/devils_dictionary/health_records_test.exs
  • test/devils_dictionary/health_score_test.exs
  • test/devils_dictionary/health_test.exs
  • test/devils_dictionary/lexicon/word_page_test.exs
  • test/devils_dictionary/lexicon_browse_test.exs
  • test/devils_dictionary/lexicon_test.exs
  • test/devils_dictionary/raw_sql_test.exs
  • test/devils_dictionary/rebuild_safety_test.exs
  • test/devils_dictionary/rebuild_snapshot_test.exs
  • test/devils_dictionary/replay_export_test.exs
  • test/devils_dictionary/schema_test.exs
  • test/devils_dictionary/source_projection_test.exs
  • test/devils_dictionary/sources/manifest_test.exs
  • test/devils_dictionary/state_fingerprint_test.exs
  • test/devils_dictionary/tasks/dd_rebuild_test.exs
  • test/devils_dictionary/workers_test.exs
  • test/devils_dictionary_web/controllers/user_session_controller_test.exs
  • test/devils_dictionary_web/live/connected_flow_test.exs
  • test/devils_dictionary_web/live/health_live_test.exs
  • test/devils_dictionary_web/live/home_live_test.exs
  • test/devils_dictionary_web/live/imports_live_test.exs
  • test/devils_dictionary_web/live/scope_live_test.exs
  • test/devils_dictionary_web/live/source_live_test.exs
  • test/devils_dictionary_web/live/user_live/confirmation_test.exs
  • test/devils_dictionary_web/live/user_live/login_test.exs
  • test/devils_dictionary_web/live/user_live/registration_test.exs
  • test/devils_dictionary_web/live/user_live/settings_test.exs
  • test/devils_dictionary_web/live/word_live_test.exs
  • test/devils_dictionary_web/user_auth_test.exs
  • test/support/conn_case.ex
  • test/support/data_case.ex
  • test/support/fake_source.ex
  • test/support/fixtures/MANIFEST.json
  • test/support/fixtures/accounts_fixtures.ex
  • test/support/fixtures/wiktionary/bank.json
  • test/support/fixtures/wiktionary/egomaniac.json
  • test/support/fixtures/wiktionary/nepotism.json
  • test/support/word_fixtures.ex

Comment @coderabbitai help to get the list of available commands.

holden and others added 3 commits September 9, 2026 20:05
`mix precommit`: **723 tests, 0 failures.** Closes the provenance-stamp
discrepancy the completion pass reported: **117,363 claim outputs with a null
`last_seen_run_id`**.

Wiktionary, Johnson and Bierce name a relation's target by *lemma*, not by a
sense key, so every claim those three sources make is written by the resolver
rather than by the materializer's second pass. Both of the resolver's callers —
`mix dd.resolve` and `dd.rebuild`'s resolve stage — opened an `import_runs` row
and then called `Resolver.run/1` without it. `own_assertions/3` duly wrote the
run it was given, which was `nil`.

That is the state `reconcile/2` cannot read: `is_nil(last_seen_run_id) or
last_seen_run_id != run_id` is its test for "the source stopped publishing
this", so an unstamped output is not merely unreconciled, it is indistinguishable
from a withdrawn one. It is the same defect as P5g in the one write path P5g did
not cover.

The split is exactly along that line, which is what confirms the diagnosis:
WordNet's edges carry a deterministic sense key and are closed by the
materializer under its own run, so none of its 1.05 M claims were affected.

## And the verification gap that hid it

My own integrity query checked `source_materialized_outputs.last_seen_run_id`
and reported zero. It never looked at `source_assertion_outputs` — a second
table, the same column, the same invariant. The zero was true and it was not the
whole answer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
`mix precommit`: **724 tests, 0 failures.** Scorecards re-run on the repaired
corpus: animals **44 / 44**, emotions **43 / 43**, culture **42 / 42**, zero
pending.

O4's actual: *"`mix dd.health` and /health: coverage, resolution, the link
histogram, candidates, conflicts, parity"*. Its grade:

    opts[:skip_health_check] != true

which is `true` unless something passes the flag. The row asserted the agreement
and measured nothing — the same shape of evidence the 7 September audit rejected
for E3, where `File.exists?` on a rolled-back file stood in for a measurement.
It is also the one row the completion pass could not confirm, and it could not be
confirmed, because there was nothing there to confirm.

The agreement is real and structural: both surfaces read `Health.coverage/2`,
which is exactly what `mix dd.health` prints. So it is a property that can be
checked rather than claimed, and `health_live_test.exs` now settles the async
assigns and asserts the rendered coverage rows carry the values that function
returns. O4 reads "proven by test", like D1–D5.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
The provenance-stamp diagnostic and O4 are both resolved; the report listed them
as open. It also framed the 117,363 figure as contradicting the earlier zero —
it did not: the two counted different tables, and only one of them had ever been
checked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X4We2gAaBbGDxFZggWRM5F
@holden

holden commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
⚠️ Action not completed

Review skipped: 240 files exceed the limit of 150.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant