Skip to content

Implement the encyclopedia model from a clean database: schema, connected identities, validation and re-import #74

Closed

Description

@holden

Completion audit — 2026-09-09: implementation is not complete

Audited local commit: 2bc6810, branch 74-encyclopedia-model (“WIP P1”). At inspection, this issue remained open, its posted checkpoint was P0, and remote main remained at a2ef867. This is an assessment of that checkout, not any uninspected work elsewhere.

Grade: B for the architectural foundation; D for completion against this issue. The design/spike evidence is substantial; the application integration and completion gate are unfinished. Continue this issue through implementation and validation rather than opening another architecture issue.

What was verified

  • ADR, source inventory, score-row mapping, manifests/replay tooling and detailed Gate 0 artifacts exist.
  • Registry/domain tables and Claims APIs are substantially implemented.
  • Authentication scaffolding exists, but the connected contribution experience does not.
  • Router still exposes the old /define/:slug flow without the planned identity-based word/entity/connection routes.
  • Materializer and other integration paths still reference the retired models.
  • Executed mix test --max-failures 5: compilation failed before tests could execute. Failures reference removed DevilsDictionary.Lexicon.Lexeme and DevilsDictionary.Encyclopedia.Concept structs in:
    • lib/devils_dictionary/health/coverage.ex:443
    • lib/devils_dictionary/lexicon/word_page.ex:327
    • lib/devils_dictionary/absorb/resolver.ex:203
    • test/support/word_fixtures.ex:28
  • No completed full re-import, semantic comparison or integrated page benchmark evidence was found in the inspected code/issue history.

Gate 0 timings were reviewed as recorded evidence, not independently rerun. The code findings below require regression reproductions once compilation is restored; they are not claimed as executed runtime tests.

Concrete correctness gaps to close

  • Preserve claim context on revision. Claims.revise/2 carries a limited field set, omitting validity dates, jurisdiction, context object and metadata. Changing only the rationale must retain all unspecified semantic context. Explicit clearing remains possible. Add tests for every context field.
  • Preserve historical lifecycle. Revision switching currently changes the old revision's lifecycle to superseded. Keep currentness separate from what the historical revision asserted; test revising/withdrawing a previously withdrawn or rejected claim without rewriting its historical meaning.
  • Enforce editorial visibility in public reads. Claims.incoming/2 and outgoing/2 filter currentness/lifecycle but do not apply separate review decisions. Define explicit public versus review/admin query policy; verify a rejected/restricted claim cannot appear from either endpoint, including counts and pagination. A raw internal reader may remain available if clearly separated.
  • Implement stable bounded pagination. The relationship readers currently impose a limit without stable ordering or cursor pagination. Verify no skipped/duplicated results across pages and apply visibility filters before pagination.
  • Validate review-context endpoint ownership. The migration checks that a context item references a content or sense revision, but no ownership enforcement was found tying that revision to the selected claim endpoint. Reject an unrelated passage/sense revision on direct SQL/bulk writes as well as through the API.
  • Complete identity merge/split behavior. Registry.merge/3 retires inputs and logs an event; split/3 logs an event. Complete resolution of old identities/aliases and explicit review of ambiguous attachments, or narrow these APIs until those behaviors are implemented. Prove old links remain meaningful and no attachment is guessed onto a split output.

Evidence locations at the audited commit: claims.ex:141–161 (revision carry-forward), claims.ex:213+ (relationship reads), priv/repo/migrations/20260908153949_create_encyclopedia_schema.exs:784+ (review context), registry.ex:365+ (merge/split), and devils_dictionary_web/router.ex (routes).

Required continuation order

  1. Finish the schema transition sufficiently to restore compilation and a green test suite; port fixtures and schema-coupled references together.
  2. Close the correctness gaps above with meaningful regression/direct-write tests.
  3. Complete source adapters/materializer, resolver/linker, scopes, read contexts and the actual scorecard implementation. A mapping document is not the implementation.
  4. Demonstrate definition → author → biography/work navigation, plus one submitted/reviewed example visible correctly from both endpoints.
  5. Execute lifecycle/reconciliation and extension tests through the real application paths, not only spike SQL.
  6. Complete checksummed replay/full re-import into the new database, semantic comparison, rebuild proof and matched page/query measurements.

Closure evidence required: exact final commit/PR, successful mix precommit results, source import counts/errors and verified manifests, current scorecard results, lifecycle/extension test results, integrated query/page budgets, and an explicit list of any remaining scope exceptions. Keep the existing database and archived inputs under the agreed separate-database policy. No old-row migration or compatibility layer is requested.

This audit does not require repeating already evidenced Gate 0 work unless implementation changes invalidate it. It does require proving those guarantees in the finished application. Current status: architecture tested in a spike; application integration underway.


Decision: rebuild from sources; do not migrate existing data

The owner has explicitly stated that current application data is disposable. Optimize for the correct models and schema, then recreate the database and re-import. No backfills, compatibility reads, dual writes, preservation of existing row IDs, or data-migration rollback project.

This supersedes the existing-data migration requirements in #73, especially its identity backfill and compatibility stages. Keep #73's semantic contract, wireframes and use cases, and #72's source correctness requirements.

This is an implementation issue, not another open-ended architecture brainstorm. Keep Phoenix, Ecto, Postgres and the useful ingestion framework.

Distinguish:

  • Now: throw away current derived application data and rebuild.
  • After launch: new identities and attachments must survive source refresh, corrections and withdrawals. Disposable current data does not remove those future lifecycle requirements.
  • Ecto schema migrations are still the reproducible way to create/change schema. We are eliminating migration of existing records, not database version control.
  • Reset commands must explicitly target the intended development/test database. Do not delete original source archives, unrelated databases or credentials.

Goal

Build and demonstrate a clean encyclopedia model in which:

  1. A word and each source-specific meaning have explicit identities.
  2. A person has one entity identity across author, biography subject and example roles.
  3. Definitions, works, editions and passages retain their distinct semantics and provenance.
  4. Typed, attributed relationships connect valid endpoints and appear consistently from both directions.
  5. Local objects do not require Wikidata IDs.
  6. Re-import and later source changes cannot silently change what an existing attachment means.

Anchor example: Ambrose Bierce → authored definition → nepotism; Bierce → encyclopedia biography; one cultural artifact → attributed example of a selected sense/concept. Contested person classifications remain attributable interpretations.

Concrete proposed schema and model wireframes

This section supplies the first-version design that the planning session must validate and finalize. Recommended starting point: a small identity registry plus typed domain tables. It is a proposal to implement and test, not a claim that these tables already exist. Change this design only with a documented reason and equivalent acceptance coverage.

Before → after

Current model Proposed model Practical result
Authors in people; encyclopedia subjects in concepts One entity identity, with a person subtype and authorship relationships Bierce's biography and definitions lead to the same person
concepts.qid required Internal entity ID; external IDs in a namespace-aware table Local artwork, event or concept can exist before Wikidata matching
entries carries mixed prose and one author field Typed content items, revisions and role-aware relationship assertions Definition, biography and quotation remain distinguishable; multiple credits supported
Source senses have gloss and source identity Keep source-specific senses; separate stable identity from revisions Changing source text does not silently replace the meaning being referenced
Example sketch requires lexeme and one example-concept field Assertion has explicit subject and object IDs Person→sense, artwork→concept, content→author and event→concept all have valid endpoints
Vote target is unchecked type/id Vote/review references a real assertion revision No orphan vote; changing a claim does not inherit approval for its old wording
Slugs and source IDs influence identity Internal object IDs plus aliases and external mappings Names/URLs can change without redefining an object

A. Identity and typed records

All IDs below are internal bigint IDs unless noted. Object-backed domain tables use object_id as both primary key and foreign key to objects.id; timestamps omitted for readability. Relationships never point at an unchecked pair of table name and row ID.

Table Principal columns Key rules
objects id, kind, lifecycle_state Kinds initially lexeme, sense, entity, content; stable identity independent of labels
lexemes object_id, language_tag, lemma, part_of_speech, lexical_key Source-independent word identity policy; URLs use ID; never merge by slug alone
lexeme_forms id, lexeme_id, written_form, form_kind, language_tag, source_record_revision_id Explicit variants/forms and their evidence; accents/case/punctuation retained
senses object_id, lexeme_id, source_id, external_key, identity_state Unique source/external key where trustworthy; source sense belongs to exactly one lexeme
sense_revisions id, sense_id, revision_number, gloss, group_key, source_record_revision_id, lifecycle_state Unique sense/revision; source position is ordering, not semantic identity
entities object_id, entity_kind, preferred_label, description Person, organization, concept, event, work, edition, etc.; no required QID
person_details entity_id, birth_date, death_date One-to-one person subtype; authors are not a separate identity population
work_details entity_id, work_kind, original_language Artwork, literary work, recording, etc.; descriptive facts retain evidence
edition_details entity_id, work_id, edition_label, publication_year, language_tag Edition belongs to a work; do not duplicate the work for every hosted record
content_items object_id, content_kind, original_language Definition, article, quotation, passage, image, etc.; stable object apart from mutable body
content_revisions id, content_id, revision_number, body, body_format, canonical_url, source_record_revision_id, rights_metadata Immutable cited versions; explicit permitted display; unique content/revision
object_names id, object_id, name, language_tag, name_kind, source_record_revision_id Names/pseudonyms/aliases do not create another entity
external_identifiers id, object_id, namespace, external_id, source_record_revision_id One active verified mapping per namespace/external ID; uncertain candidates kept separately
identity_events id, operation, actor_id, reason Audited retire/merge/split operation
identity_event_members event_id, object_id, role Explicit input/output identities; split can have multiple outputs

The registry is only identity and lifecycle. Domain-specific fields remain in typed tables. Person, work and edition detail rows are typed extensions of entities, not new competing identities.

Retain source-specific auxiliary fields such as lexical tags, topics, taxon data and import metadata where appropriate; this table list defines ownership and core relationships, not permission to drop imported semantics. Define their final columns in the ADR.

Enforcement: require exactly one matching domain row per object at transaction commit; enforce entity subtype/kind consistency. Implement and test deferred constraints/triggers where ordinary foreign keys cannot express these rules. Restrict kind changes after insertion. Retire identities rather than cascading away referenced content. A registry without these protections fails this design.

B. Source and relationship records

Keep the existing source registry, import runs and archived inputs. Extend stored source records so a cited revision is not overwritten.

Table Principal columns Key rules
source_records id, source_id, external_key Stable source-native record identity
source_record_revisions id, source_record_id, revision_key, payload, checksum, observed_at, import_run_id Immutable/version-addressable evidence; exact replay inputs or archive locator
actors id, actor_kind, user_id, bot_id, entity_id Exactly one typed principal for account, bot or external claimant; account≠public person by default
predicates id, key, forward_label, reverse_label, is_symmetric, is_transitive Registered meanings; default no transitivity
predicate_endpoint_rules predicate_id, subject_kind, subject_subkind, object_kind, object_subkind Enumerated allowed combinations; not arbitrary strings
assertions id, origin_key, origin_actor_id, submitted_by_actor_id, current_revision_id Stable claim identity; source-import origin keys enable idempotency
assertion_revisions id, assertion_id, revision_number, subject_object_id, predicate_id, object_object_id, rationale, valid_from, valid_to, language_tag, jurisdiction_entity_id, context_object_id, method, confidence, lifecycle_state Explicit endpoints and contextual meaning; changing these creates a revision
assertion_evidence id, assertion_revision_id, source_record_revision_id, content_revision_id, sense_revision_id, evidence_role, locator, attribution_text At least one evidence target; if multiple are given validate compatibility; support/contradiction retained
assertion_reviews id, assertion_revision_id, reviewer_actor_id, decision, reason Append-only review events; effective state derived deterministically
assertion_votes assertion_revision_id, actor_id, value Unique actor/revision; FK target; relevance only
source_assertion_outputs source_record_id, assertion_id, last_seen_run_id, output_key Tracks ownership/current outputs for safe reconciliation
source_materialized_outputs source_record_id, output_object_id, output_role, output_key, last_seen_run_id Tracks source-produced senses/content; never delete another source's support

Actors may be unknown for a historical claim: retain explicit unknown attribution rather than inventing a person. Submission/import actors must be accountable. A curator account claiming to be a known author is not automatically linked to that author.

Keep source support and editorial state separate. Reviews of a claim revision do not mutate source content. Withdrawn source output is retired; repeated input does not create a new semantic revision just because an import timestamp changed.

For source adapters, derive origin keys from stable source-native assertion identifiers where available. Where unavailable, specify and test an adapter-specific matching policy. Ambiguous sense identity changes create a reconciliation case; they never reuse an old ID for a different meaning. This cannot be solved by a uniqueness constraint alone.

Do not collapse claims by subject/predicate/object alone. Two claimants, contexts or source assertions can differ. Multiple citations can support one assertion revision without erasing their origins. Grouping similar claims for display is separate from identity.

C. Initial predicate contract

Predicate Allowed endpoints Example
defines definition content → lexeme or source sense Bierce definition → nepotism lexeme
authored_by content or work entity → person/organization entity Definition → Bierce
translated_by content or work entity → person/organization entity Translated poem → translator
published_in content → edition entity Definition → dictionary edition
about article content → entity Biography → Bierce
refers_to source sense → entity Selected meaning → concept
lexeme_entity_candidate lexeme → entity Title/disambiguation candidate with method, confidence and review state; not sense equivalence
Source-native lexical predicates lexeme → lexeme or sense → sense, as declared per mapping Preserve unsensed lexical relations without inventing sense IDs
illustrates entity or content → source sense or concept entity Artwork → concept
satirizes work entity or content → entity Meme → practice/person
depicts work entity or image content → entity Portrait → person
participates_in person/organization entity → event entity Person → appointment event
excerpt_of passage/quotation content → content or work entity Quoted passage → poem

Keep source-native lexical and semantic predicates, including WordNet and ConceptNet distinctions, through a reviewed mapping registry. These initial product predicates do not replace those source relations with generic illustrates or related_to.

Foreign keys enforce existence. Database validation must additionally reject incompatible endpoint kinds for the predicate, including on bulk writes. Check timestamps and confidence ranges where applicable; source weight is not a calibrated probability. Declare cycle rules per predicate rather than banning all graph cycles.

D. Schema relationship diagram

erDiagram
  OBJECTS ||--o| LEXEMES : typed_as
  OBJECTS ||--o| SENSES : typed_as
  OBJECTS ||--o| ENTITIES : typed_as
  OBJECTS ||--o| CONTENT_ITEMS : typed_as
  LEXEMES ||--o{ SENSES : has_source_meanings
  SENSES ||--o{ SENSE_REVISIONS : versioned_as
  ENTITIES ||--o| PERSON_DETAILS : person_subtype
  ENTITIES ||--o| WORK_DETAILS : work_subtype
  ENTITIES ||--o| EDITION_DETAILS : edition_subtype
  CONTENT_ITEMS ||--o{ CONTENT_REVISIONS : versioned_as
  OBJECTS ||--o{ EXTERNAL_IDENTIFIERS : identified_by
  ASSERTIONS ||--|{ ASSERTION_REVISIONS : revised_as
  OBJECTS ||--o{ ASSERTION_REVISIONS : subject
  OBJECTS ||--o{ ASSERTION_REVISIONS : object
  PREDICATES ||--o{ ASSERTION_REVISIONS : types
  ASSERTION_REVISIONS ||--o{ ASSERTION_EVIDENCE : supported_or_challenged_by
  SOURCE_RECORDS ||--|{ SOURCE_RECORD_REVISIONS : versioned_as
  SOURCE_RECORD_REVISIONS o|--o{ ASSERTION_EVIDENCE : cited_by
  CONTENT_REVISIONS o|--o{ ASSERTION_EVIDENCE : cited_by
  ASSERTION_REVISIONS ||--o{ ASSERTION_REVIEWS : reviewed_as
  ASSERTION_REVISIONS ||--o{ ASSERTION_VOTES : relevance_votes
Loading

The four registry subtype links are individually optional but collectively exactly one. The diagram omits auxiliary tables and actor relationships for readability; the column contract above still applies.

E. Stored example: Bierce, not duplicated

IDs are illustrative local IDs, not production data.

OBJECT 101  entity/person   Ambrose Bierce
OBJECT 102  entity/work     The Devil's Dictionary
OBJECT 103  entity/edition  Selected imported edition
OBJECT 104  lexeme          nepotism · en · noun
OBJECT 105  content         definition
OBJECT 106  content         biographical article
OBJECT 107  entity/concept  nepotism as a practice
OBJECT 108  content         original illustrative meme
OBJECT 109  sense           WordNet's nepotism meaning

Typed records:
  person_details.entity_id = 101
  edition_details: entity_id = 103, work_id = 102
  senses: object_id = 109, lexeme_id = 104, source = WordNet

Assertion revisions:
  102 — authored_by  → 101
  105 — authored_by  → 101
  105 — published_in → 103
  105 — defines      → 104
  106 — about        → 101
  109 — refers_to    → 107   [mapping evidence/review]
  108 — illustrates  → 107   [curator attribution/review]

The historical definition targets the lexeme unless a defensible source-sense target exists. The meme's claim is separate from its content. All links to Bierce point to object 101. No special “Bierce the author” duplicate is necessary.

F. Reader-facing wireframes tied to those records

WORD PAGE /words/104/nepotism
─────────────────────────────────────────────────
nepotism · English · noun
[Definitions] [Meanings] [Examples]

DEFINITIONS
The Devil's Dictionary · selected edition
  Definition text…                         [105]
  By Ambrose Bierce →                      [101]
  [Edition 103] [Source / revision]

MEANINGS
WordNet meaning →                          [109]
  Refers to: nepotism, the practice →       [107]

EXAMPLES OF THIS CONCEPT
Illustrative meme →                        [108]
  “Illustrates nepotism”
  Proposed by curator A · review state
  [Evidence] [Open claim]
─────────────────────────────────────────────────
PERSON PAGE /entities/101/ambrose-bierce
─────────────────────────────────────────────────
Ambrose Bierce · Person
[Biography] [Works] [Definitions] [Connections]

BIOGRAPHY
Article content…                           [106]
  [Publication / revision]

WORKS AUTHORED
The Devil's Dictionary →                    [102]

DEFINITIONS AUTHORED
nepotism →                                 [105]
  Published in selected edition →          [103]
─────────────────────────────────────────────────
Same identity 101; sections query different predicates.
WORK PAGE /entities/102/the-devils-dictionary
─────────────────────────────────────────────────
The Devil's Dictionary · Work
Author: Ambrose Bierce →                    [101]

EDITIONS
Selected imported edition →                [103]
  Contents: nepotism definition →          [105]

ABOUT / INTERPRETATIONS
Attributed connections, when available
─────────────────────────────────────────────────
Work, edition and definition are separate records.
CONNECTION DETAIL
─────────────────────────────────────────────────
Illustrative meme [108] — illustrates → concept [107]
Claim #… · revision 2
Original claimant: curator A
Submitted by: account B
Context: …                  Review: Disputed

Rationale: …
Evidence: [source record/revision] [passage locator]
Counterevidence: …
[Open meme] [Open concept] [History] [Challenge]
─────────────────────────────────────────────────
The same claim revision appears from either endpoint.
ADD CONNECTION
─────────────────────────────────────────────────
Subject:  [Find existing object / Create local object]
Relation: [Only predicates valid for subject type ▾]
Object:   [Search valid target kinds               ]

For “illustrates”:
  [Specific source meaning / Concept ▾]
  Context: [event, period, passage…]
  Rationale: […]
  Evidence: [+ citation and locator]

Preview: “A proposes X as an illustration of Y.”
[Submit for review]
─────────────────────────────────────────────────
Names help discovery; selected IDs determine endpoints.

G. Query and testing consequences

  • Word definitions: indexed incoming defines assertions for lexeme/sense, with source content and credits preloaded.
  • Person definitions: incoming authored_by, filtered to definition content. Biography: incoming about, filtered to article content.
  • Examples: incoming illustrates for the exact selected target. Concept-path discovery explicitly follows a reviewed mapping and labels the indirect path.
  • Add indexes on assertion revision endpoints/predicate, latest revision lookup, content/sense revision lookup, external namespace/ID and source ownership keys. Use the explicit current-revision selection contract below; never return historical revisions as duplicate active claims.
  • Page displays must apply review/visibility rules before counts and pagination, in both directions.
  • Prove the exactly-one subtype rule, predicate endpoint checks, external-ID collision behavior, deterministic current revision selection and vote revision isolation with direct-write tests.
  • Start with these concrete records and flows, then exercise source change and the held-back extension case from the milestones below.

Reviewed audit corrections and mandatory pre-freeze gate

The supplied audit identified real omissions. This revision accepts the missing lexeme-level bridge, scope model, scorecard/read-layer inventory, input verification, revision context and early measurement requirements. It does not adopt its grades, unprovided benchmark results or line counts as verified facts. The number of agents is not evidence.

The change remains the backbone-correctness milestone from #72 executed through a clean rebuild instead of record migration. It is not a rewrite of all parsing or a mandate to replace the stack. The owner's disposable-data decision is sufficient authorization for that strategy; verify environment state before an actual reset rather than relying on unverified claims about deployments/users. Preserve original inputs and export any raw source records/import metadata that exist only inside the database before dropping derived tables. A reset without independently recoverable source inputs fails the rebuild contract.

Gate 0 — executable proof before schema freeze or broad adapter rewrites

Use an explicitly named separate scratch database for the candidate schema and benchmark. A second permanent Phoenix Repo, dual application runtime or compatibility layer is not required. Keep existing checks runnable during the spike. Use transactional candidate writes and real constraints; a fixed number of tables is not a deliverable.

Commit reproducible scripts, fixtures, SQL, query plans and environment details:

  • Load Bierce content and WordNet senses for nepotism/bank; include a small Wiktionary bank fixture for its position-key failure. WordNet alone cannot prove Wiktionary reconciliation.
  • Demonstrate deferred rejection of an object without its matching subtype and rejection of invalid predicate endpoints.
  • Identical input creates zero new semantic revisions.
  • Inserting/reordering/removing Wiktionary senses leaves attachments on the original meaning or flags ambiguity; it never silently reassigns them.
  • Load a representative real WordNet edge population including high-degree nodes, and enough historical revisions to exercise revision-selection cost.
  • Compare endpoint reads using DISTINCT ON/latest-row view, partial-current indexing and an explicit current-revision pointer. Measure both directions, complete page-query composition, write/update cost and concurrent writers.
  • Record p50/p95, query count, EXPLAIN (ANALYZE, BUFFERS), rows/index bytes, load/index wall time, revision depth, cache conditions, hardware and Postgres version.
  • Decide the current-revision implementation and freeze the adapter write contract only after this evidence passes agreed budgets.

The audit's claimed 72–91 ms versus sub-millisecond reads are a hypothesis to reproduce, not a portable timing guarantee. DISTINCT ON can create poor plans for some query shapes; it is not proof that every current-revision view must scan all history.

Current-revision correctness contract

Recommended baseline: assertions.current_revision_id, with a deferred composite FK (id, current_revision_id) → assertion_revisions(assertion_id, id) and supporting uniqueness. At committed state every persisted assertion has one designated current revision belonging to that assertion. Lock/serialize updates of that assertion when adding a revision and changing its pointer; test concurrent writers and rollback.

Current selection is distinct from lifecycle: the designated current revision can say withdrawn, while history stays immutable. Do not encode both “currentness” and “withdrawal” into one overloaded lifecycle value. Use endpoint indexes on revisions and pointer equality in the read; benchmark the actual query plan, since a pointer alone does not guarantee efficient reads.

If the spike selects a separate is_current flag instead, require a partial unique index on assertion ID plus a commit-time guarantee for exactly one selected revision and atomic switching. A partial unique index alone proves at most one, not exactly one. Do not maintain both pointer and flags without enforcing agreement.

Missing schema coverage

Table/addition Contract
scopes(id, slug, name, rules, stats) Retain data-driven Animals/Emotions definitions, including score bars
scope_lexemes(scope_id, lexeme_id, reasons) Composite key; FK to new lexeme object ID; retain membership reasons
import_runs.scope_id Retain scope FK and replay context
assertion_revisions.method, confidence Method records derivation; nullable confidence constrained to [0,1], explicitly heuristic; source weight stays a separate source-native quantity
assertion_evidence.sense_revision_id Direct FK for evidence citing a source meaning; compatible with other cited revision targets
review_contexts(id, assertion_revision_id) Immutable manifest of the endpoint content/sense revisions actually displayed
review_context_items(context_id, endpoint_role, content_revision_id, sense_revision_id) Exactly one revision target per row; validate it belongs to the designated endpoint; an entity may have no textual revision
assertion_reviews.review_context_id, assertion_votes.review_context_id Must match the reviewed assertion revision; preserve what the reviewer saw even if endpoint text later changes

Endpoint revision changes can mark a prior review stale/needs-review according to declared materiality rules; they must not rewrite the old review context. Do not require a revision ID for immutable identity-only endpoints or fabricate an entity version.

Port the six linking methods and their corroboration stage explicitly. Sense-backed mappings remain refers_to; title/disambiguation heuristics use the lexeme-level candidate relation. Candidates can appear as possible subjects but do not automatically confer sense equivalence or propagate examples. Preserve method, evidence, confidence and editorial decisions; do not blindly preserve the existing automatic “confirmed” policy.

Add priv/scopes/culture.json with a documented explicit-lemma rule for the bounded pilot (nepotism, bank, egomaniac); implement that rule if needed, and document the actual scoped import/link commands. Do not assume those words are in Animals or Emotions.

Correct the Bierce catalog identity before the anchor import: current catalog.ex assigns Q310190. Wikidata Q191050 is Ambrose Bierce; Q310190 is Tobin Bell. Assert the verified fixture identity in a regression test; never solve this by merging the two QIDs.

Use is_symmetric/is_transitive column names. PostgreSQL lists SYMMETRIC as reserved. Quoted identifiers can work, so this was a naming hazard rather than a fatal schema defect.

Code and acceptance inventory — required at the first checkpoint

Subsystem Treatment
Pure source parsers, committed fixtures, source catalog Reuse where semantically correct; fix evidenced source bugs
Materializer, batch writes, resolver, linker Audit/rewrite schema-coupled SQL and output ownership; retain useful algorithms
Scope builder, scope tasks, scoped import entry points Port to new lexical IDs; preserve rules/reasons
Lexicon/encyclopedia read contexts and WordPage builder Port queries and preloads; test semantics and query counts
All six existing pages and search/navigation Inventory routes and required behavior; preserve or explicitly replace each
Health/coverage/parity/pages/score and their tests Map every existing score row to retained, revised or retired, with reason and replacement
Unshipped community sketch/demo assumptions Replace with new contract; distinguish demo fixtures from real acceptance
README and docs/map Update at the first schema checkpoint; remove forward-looking “no changes to thirteen tables” claims while preserving dated historical results

Do not use approximate line counts as effort estimates. Produce a file-level port/rewrite/retire checklist. Fix @baseline_migrations 2 in the same checkpoint as schema changes, replacing table-count/migration-count acceptance with tested extensibility. No permanently red suite and no disabling checks merely to keep a score green. New acceptance measurements can be explicitly pending during the spike; completion requires their tests to pass.

Carry existing X2 search p95 <150 ms and O2 aggregate dump processing ≤2 hours, API elapsed separately reported forward with their actual populations and measurement definitions. The audit conflates search and page budgets; establish an additional full page/query budget from fresh matched baseline runs.

At Gate 0 record current corpus counts, maximum node degree, database/index sizes and archived input sizes. The supplied audit reports 1,541,669 lexemes, 1,179,377 edges, 344,635 source records and 3,057 MB database size; these are reported reference values requiring remeasurement, not newly verified results. Agree numeric storage/headroom and candidate page-query budgets before the benchmark. Measure costs with multiple revisions, not just a history-free corpus.

Reproducible source inputs

Before reset/re-import, create a manifest per input: source/edition, immutable archive locator, acquisition URL/date, byte count, SHA-256, parser version and license. Verify checksum before import and fail clearly on mismatch. A rolling latest URL and File.exists? are insufficient. A locally computed digest pins what was actually used; it does not independently authenticate a publisher. Test altered bytes and missing archives. Export DB-only API records/revisions to a checksummed replay archive before deleting the database.

Session checkpoints

Keep this issue as the implementation tracker rather than creating another overlapping issue solely for session bookkeeping. Each checkpoint posts commands, commit, test results, measured numbers and remaining blockers:

  1. P0: inventory, score-row mapping, input manifest, scratch correctness/read-path spike, final schema decision.
  2. P1: clean schema plus minimal Bierce/WordNet/Wiktionary fixture adapters, early lifecycle tests and updated README/checks.
  3. P2: remaining source writes, scope/linker/read ports with green targeted tests and score coverage.
  4. P3: connected UI and reviewed contribution slice; endpoint/revision context tests.
  5. P4: extension exercise, full verified re-import, matched scale measurements and completion report.

One planning session can define this sequence and contracts. It cannot honestly claim performance validation until the executable spike runs; do not promise a fixed one/two-day implementation based on the audit.

Related downstream work: #65 quotations, #66 design, #67 evidence wall. Their features consume the new model; do not add disposable user submissions on the old schema to discover a sense-drift defect already reproduced. #72's operational hardening remains required before public deployment.

Can this be planned in one session?

Yes: one bounded planning session should settle an implementable first version. It need not predict all future media, and it should not promise the entire implementation or full import finishes in that session.

The session must produce a committed ADR, concrete schema/ER diagram, importer mapping and ordered implementation checklist. End with decisions, not another list of equally weighted alternatives.

Required decisions

  • Choose an addressable-object registry with typed domain tables, or explicit typed relationship tables. Document the choice against actual foreign-key integrity, query complexity and extension cost.
  • Specify exact first-version tables, columns, indexes, checks, uniqueness and deletion rules; include how kind/subtype existence and predicate endpoint compatibility are enforced.
  • Unify person/author identity; distinguish public entities from submitting user/bot accounts.
  • Preserve source-specific senses. Explicitly model optional sense→concept mapping without inventing a universal merged sense.
  • Define local entity identity plus namespace-aware external identifiers.
  • Define the minimum work/edition/content/credit model needed for Bierce and an artwork; support additional roles without a full bibliographic ontology.
  • Specify assertion identity, evidence, context, authorship, revision and review state; separate relevance votes from factual/source support.
  • Define source-native assertion keys and refresh behavior: add, revise, retire, ambiguous identity change and replay.
  • Specify commands and inputs for clean setup, bounded fixture imports and full source re-import.
  • Set measurable query budgets and seed sizes for ordinary word/person pages before measuring them.

If one storage question remains uncertain, resolve it with a small executable schema/query experiment within this milestone. Do not make a speculative universal graph the prerequisite for importing a definition.

Implementation milestones

Execution order: pre-freeze gate → milestone 1 → bounded subset of milestone 2 with milestone 4 lifecycle tests → remaining adapters in milestone 2 → milestone 3 → remaining milestone 4 checks → milestone 5. Lifecycle correctness and read-path measurements are prerequisites to adapting all six sources, not end-of-project checks. Run mix precommit at every implementation checkpoint; update affected checks with the corresponding schema/read changes.

1. New schema and domain API

Create the new baseline through normal Ecto tooling; choose and document whether to replace the pre-release baseline migration history or add a rebuild path. Do not implement a transformation of old rows.

  • Build chosen schema, domain structs/changesets and database constraints.
  • Add narrow context APIs to create/read identities, source content, credits, assertions and evidence.
  • Test direct SQL/bulk-write rejection of invalid targets, incompatible predicates and missing required records.
  • Define canonical URLs using stable internal identity, with readable labels; spelling/slug collisions must not assert equivalence.
  • Create local person/work/artifact/concept identities without external IDs; adding an external ID leaves identity unchanged.
  • Remove or replace the unshipped community migration sketch so it cannot act as a competing contract.

Achievable result: an empty database can be created reproducibly; small fixtures represent the model and invalid data is rejected.

2. Import a small, representative source slice

Use archived/licensed source inputs already available where practical. Fixtures must retain source-native identifiers and relevant revisions.

  • Adapt all six existing source contracts: WordNet, Wiktionary/Kaikki, Wikidata, Wikipedia, Bierce, Johnson.
  • First execute bounded imports around Bierce, nepotism, an ambiguous word such as bank, and an entity with multiple source records.
  • Import the author and biography subject as one entity using evidenced identity matching.
  • Keep definitions as source content with author/work/edition links; do not relabel them as examples.
  • Canonicalize Wikipedia publication identity independently of lookup probes.
  • Honor supported Wikidata statement rank/qualifier/reference policy and retain original evidence.
  • Preserve source-specific relation types and direction.
  • Document unsupported source structures and unresolved mappings; do not silently flatten them.

Achievable result: real imported data follows the new model; one person can be navigated as both author and encyclopedia subject.

ConceptNet's production import remains the bounded pilot in #72. Validate that the new relationship contract can represent its native endpoint types, relation and dataset lineage before freezing the schema; do not require a full ConceptNet ingest here. Artsy integration is likewise not required: use a clearly identified local artifact fixture to prove the model.

3. Prove one connected product flow

Use #73's wireframes as acceptance guidance, scaled to the minimum useful slice.

  • Render a word with separately attributed source definitions.
  • Navigate from a definition's author to the person's page and back to their authored definitions.
  • Show person biography and authored works in their appropriate sections.
  • Create one attributed artifact→sense/concept relationship with exact endpoints, context, rationale and evidence.
  • Show that same relationship from both endpoints with the same status and provenance.
  • Support a minimal authenticated submit/review flow; do not build an entire social platform.
  • Preserve rejected/disputed state across reads and importer reruns.
  • Label indirect concept-path discovery separately from a direct example.

Achievable result: a working, inspectable end-to-end example, not just tables or a diagram.

4. Establish durability before broad import

Current data is disposable, but these tests use attachments created in the new schema.

  • Re-import identical input: no semantic duplicates and equivalent results.
  • Reorder/add/remove source meanings: attachments never silently move to a different meaning.
  • Withdraw one supporting record: history and other support remain correct.
  • Correct authorship: content identity remains stable and attribution history is inspectable.
  • Merge identities with evidence; split an incorrectly combined identity: ambiguous attachments enter explicit review.
  • Change claim endpoints/meaning: revisions and review policy prevent old votes/reviews silently endorsing a different claim.
  • Generate operation sequences (import, attach, update, withdraw, replay) and check integrity/lifecycle invariants after each operation.
  • Verify restricted/rejected content cannot leak through reverse views.
  • Compare normalized semantic content, not just row counts, including unexpected stale records.

Achievable result: evidence that ordinary source evolution does not corrupt meaning or relationships.

5. Test extension, rebuild and scale

Achievable result: the model works beyond hand-picked fixtures and has a repeatable rebuild process.

Scope boundaries

Do not spend time on:

Do retain:

  • Actual database integrity, not changeset-only guarantees.
  • Explicit source provenance and rights/display metadata.
  • Future refresh/reconciliation correctness.
  • Bounded queries, tested rebuildability and semantic validation.

Completion gate

Ready for broader feature development when:

  • Bierce has one identity across authorship and biography.
  • Words, source senses, concepts and published content remain distinct.
  • A reviewed relationship can connect new local objects and existing encyclopedia identities.
  • The relationship survives source changes without silently changing meaning.
  • An additional use case fits through a controlled extension.
  • Clean setup, re-import, tests and representative queries are reproducible.

The first planning session's expected output is a final first-version schema and executable work order. The implementation's expected output is a demonstrably correct vertical slice followed by full re-import. Do not equate a completed planning session or green CRUD tests with completion of this gate.

Activity

  1. holden commented on Sep 8, 2026

    @holden
    MemberAuthor

    P0 — inventory, manifest, spike, decision

    Commit 10097c4 on branch 74-encyclopedia-model. mix precommit: 504 tests, 0 failures.

    Two decisions on the working arrangement, taken with the owner:

    • The new schema is built in a new database, devils_dictionary_v2. devils_dictionary_dev is left intact — nothing is destroyed, the old corpus stays available as the matched baseline for the benchmarks and the P5 semantic comparison, and "recreate a separate empty database from documented inputs" becomes what we are doing anyway rather than an exercise at the end.
    • Branch 74-encyclopedia-model, one commit per checkpoint, one PR at P5. main stays green and MVP-0 stays reachable.

    Baseline, remeasured

    The audit's reported reference values are confirmed on this machine (PostgreSQL 18.2, Apple M4 Max / 64 GB, Elixir 1.19.0-otp-28): 3,270 MB database · 1,541,669 lexemes · 1,179,377 lexical relations (1,085,482 sense-scoped, 15,718 unresolved) · 344,635 source records · max lexeme degree 1,768 out / 1,586 in.

    Inputs pinned

    priv/sources/MANIFEST.json + mix dd.manifest digest all five archived inputs. 5/5 verified. Johnson's computed digest matches the sha256 Sources.Catalog has pinned since S5 — that agreement is how the mechanism was checked rather than asserted. Altered bytes, a flipped single byte, a missing archive and an unpinned entry all fail, with tests.

    Two findings:

    • data/raw-wiktextract-data.jsonl.gz cannot be re-fetched. 2.6 GB from a rolling URL (kaikki.org/dictionary/raw-wiktextract-data.jsonl.gz), previously pinned only by dump_bytes. Only 22,534 of its records reach source_records; the 1.5M-row index is written straight to lexemes by the index pass. It is a required, unreproducible recovery input, and the manifest records url_is_rolling: true so nothing later implies otherwise.
    • mix dd.export.replay writes all 344,635 records with payloads to a checksummed gzipped archive — 101 MB, 15.7 s. Wikipedia's 85,044 and Wikidata's 72,770 exist only in the database (~3 h of batched API calls), so this is those two sources' equivalent of a pinned dump. Verified to round-trip including raw["_probe"], which materialize/1 reads.

    Gate 0

    Full evidence: docs/spikes/2026-09-gate0/. Run on a named scratch database against the whole resolved corpus, not a sample — 1,792,058 objects, 1,163,659 assertions, 2,029,069 assertion revisions, with every fifth assertion carrying five revisions so revision-selection cost is actually exercised.

    Integrity — eleven rejections, and the Bierce anchor commits. Object with no subtype rejected at COMMIT (and legal mid-transaction, as it must be); wrong-kind subtype, two subtypes, kind mutation, person_details on a concept, defines → a person, refers_to from a lexeme, confidence 1.4, a stolen revision pointer — all rejected. Including on a bulk write: one good row and one bad row in a single multi-row INSERT rejects the whole statement, which is what makes this hold for insert_all, COPY and the linker's raw SQL.

    Current revision — the shootout. 3,000 transactions each, warm cache, pgbench -l percentiles:

    Strategy outgoing p95 incoming p95 buffers (outgoing)
    explicit pointer 2.576 ms 4.535 ms 7,957
    DISTINCT ON 1.670 ms 2.976 ms 5,989
    partial is_current index 0.256 ms 0.489 ms 41

    A 194× difference in pages touched. Two of the audit's expectations do not survive contact: it predicted the pointer would be fast and DISTINCT ON slow, and DISTINCT ON beat the pointer on every figure. Neither claim should be repeated as fact.

    So: strategy C, and the pointer is dropped entirely — one source of truth, so §"do not maintain both pointer and flags without enforcing agreement" cannot be violated. Correctness is completed by a partial unique index (at most one), deferred constraint triggers (at least one, at commit), and SELECT … FOR UPDATE as the write path's first statement. Atomic switching was measured rather than assumed, since a partial unique index cannot be DEFERRABLE: single-statement flag moves work in both directions on PG 18. Concurrent writers serialise; without the explicit lock, an adversarial statement ordering loses a write to a duplicate-key error. Currentness and lifecycle stay separate columns — the designated current revision of assertion 11 reads withdrawn.

    Write path sustains 13,756 tps at 8 clients, p95 0.904 ms.

    Lexical relations stay in the assertion model. 1,118 MB against lexical_relations' 730 MB — 1.53× for a full revision history and a 10× faster endpoint read. The typed-fallback table named as an escape hatch is not needed.

    Sense identity — proofs 4 and 5 on the real Wiktionary bank/noun/1 record, captured from the pinned dump (one full 2.6 GB pass, 47 s; 7 records across 4 etymologies). Identity is matched on content — pos, etymology number, normalised gloss scored with pg_trgm — never on position.

    • Byte-identical input: all ten senses match themselves at 1.000. Zero new identities, zero new revisions.
    • The audit's reproduction: delete "A branch office of such an institution." from position 1. Everything after shifts down. "Money; profit." moves from position 5 to 4 and keeps sense 20000005; incoming position 5 resolves to 20000006, a different identity; the curator attachment made before the edit still reads "Money; profit."; the deleted sense is retired, not deleted; the genuinely new sense scores 0.165 and gets a new identity.
    • Ambiguity has two routes and both work: nothing above the strong threshold (0.430), and two candidates too close to choose (0.897 with a 0.000 gap — picking the higher would have been decided by object_id). Both open a reconciliation_cases row and set needs_review. No revision written, no attachment moved, no identity reused.

    Two things the gate found that reading would not

    1. The subtype-delete gap. Guarding "exactly one subtype" with an AFTER INSERT trigger on objects is half a guard. Deleting Bierce's entities row left object 101 typed entity with no entity row while authored_by still pointed at it. AFTER DELETE constraint triggers on all four subtypes close it, and the rule that survives is the one §A asks for: an identity is retired via lifecycle_state, never deleted out from under its claims.
    2. sense → lexeme is a real endpoint pair. Declaring only lexeme → lexeme and sense → sense silently dropped 36,434 of 1,163,659 relations — a meaning pointing at a word without naming which of the word's meanings. §C requires exactly that, "without inventing sense IDs". After declaring it: 1,047,918 sense→sense, 79,307 lexeme→lexeme, 36,434 sense→lexeme, 0 assertions without a current revision.

    Also, two naming hazards beyond the known SYMMETRIC: POSITION is reserved in a RETURNS TABLE column list, and one trigger function shared across two tables whose key columns differ by name fails at run time, not at creation.

    Budgets, agreed from these measurements

    Budget Value Basis
    Bounded incoming/outgoing assertion read p95 < 5 ms measured 0.256 / 0.489 ms; ~10× headroom
    Single-assertion write path p95 < 10 ms measured 0.904 ms at 8 clients
    Search p95 < 150 ms X2 carried forward unchanged, same 20 probes
    Storage, assertion half ≤ 2× lexical_relations today measured 1.53×
    Dump processing ≤ 2 h, APIs reported separately O2 carried forward

    Full word-page composition p95 is pending, not guessed — it needs the ported WordPage builder to measure honestly, and is set at P2 against a matched baseline run on devils_dictionary_dev.

    Inventory

    docs/adr/0001-encyclopedia-model.md — the ten decisions, what was rejected and why.
    docs/rebuild/port-checklist.md — every file to reuse / port / rewrite / retire. No line counts.
    docs/rebuild/score-rows.md — all 37 scorecard rows mapped, plus 8 new ones. E1 stops counting schema_migrations (its == 2 equality would fail this rebuild on day one) and becomes a test that actually adds a source, a predicate and an entity kind. E3 stops grading a rolled-back sketch by File.exists?. M1 must fail on the corruption probe it currently passes.

    README §Status withdraws the "none of which changes the thirteen tables" claim; the dated historical results are preserved.

    Remaining blockers

    None for P1. Two things carried forward: full page-composition budget is pending until P2, and disk is the tightest constraint — the volume is at 92%, 35 GB free, against a ~3.3 GB second corpus, so P5's rebuildability proof creates its third database, verifies and drops it rather than keeping one.

    Next: P1 — the migration, the domain API, and the direct-write rejection tests.

  2. added 20 commits that reference this issue on Sep 10, 2026
  3. holden commented on Sep 17, 2026

    @holden
    MemberAuthor

    Closing: the issue's own record stops at P0, but the README status table carries P1 (6a6ae8f), P2/P4 (e60ab6e) and P3 (0dbb752) as landed 2026-09-08/09, with the independent completion verification in docs/rebuild/completion-2026-09-09.md (independent clean rebuild devils_dictionary_74_verify, semantic M2, mix dd.verify.rebuild) and the P5 score-row mapping in docs/rebuild/score-rows-p5.md. The reader milestone that consumed the model shipped as PR #83 (#82) and the connection-reliability fixes as PR #87 (#84).

    Remaining correctness work is owned by #78/#84's successors, not by this issue. Reopen if the P5 close-out list has residue those do not cover.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions