Repository navigation
Description
Completion audit — 2026-09-09: implementation is not complete
Audited local commit: 2bc6810, branch 74-encyclopedia-model (“WIP P1”). At inspection, this issue remained open, its posted checkpoint was P0, and remote main remained at a2ef867. This is an assessment of that checkout, not any uninspected work elsewhere.
Grade: B for the architectural foundation; D for completion against this issue. The design/spike evidence is substantial; the application integration and completion gate are unfinished. Continue this issue through implementation and validation rather than opening another architecture issue.
What was verified
- ADR, source inventory, score-row mapping, manifests/replay tooling and detailed Gate 0 artifacts exist.
- Registry/domain tables and Claims APIs are substantially implemented.
- Authentication scaffolding exists, but the connected contribution experience does not.
- Router still exposes the old
/define/:slugflow without the planned identity-based word/entity/connection routes. - Materializer and other integration paths still reference the retired models.
- Executed
mix test --max-failures 5: compilation failed before tests could execute. Failures reference removedDevilsDictionary.Lexicon.LexemeandDevilsDictionary.Encyclopedia.Conceptstructs in:lib/devils_dictionary/health/coverage.ex:443lib/devils_dictionary/lexicon/word_page.ex:327lib/devils_dictionary/absorb/resolver.ex:203test/support/word_fixtures.ex:28
- No completed full re-import, semantic comparison or integrated page benchmark evidence was found in the inspected code/issue history.
Gate 0 timings were reviewed as recorded evidence, not independently rerun. The code findings below require regression reproductions once compilation is restored; they are not claimed as executed runtime tests.
Concrete correctness gaps to close
- Preserve claim context on revision.
Claims.revise/2carries a limited field set, omitting validity dates, jurisdiction, context object and metadata. Changing only the rationale must retain all unspecified semantic context. Explicit clearing remains possible. Add tests for every context field. - Preserve historical lifecycle. Revision switching currently changes the old revision's lifecycle to
superseded. Keep currentness separate from what the historical revision asserted; test revising/withdrawing a previously withdrawn or rejected claim without rewriting its historical meaning. - Enforce editorial visibility in public reads.
Claims.incoming/2andoutgoing/2filter currentness/lifecycle but do not apply separate review decisions. Define explicit public versus review/admin query policy; verify a rejected/restricted claim cannot appear from either endpoint, including counts and pagination. A raw internal reader may remain available if clearly separated. - Implement stable bounded pagination. The relationship readers currently impose a limit without stable ordering or cursor pagination. Verify no skipped/duplicated results across pages and apply visibility filters before pagination.
- Validate review-context endpoint ownership. The migration checks that a context item references a content or sense revision, but no ownership enforcement was found tying that revision to the selected claim endpoint. Reject an unrelated passage/sense revision on direct SQL/bulk writes as well as through the API.
- Complete identity merge/split behavior.
Registry.merge/3retires inputs and logs an event;split/3logs an event. Complete resolution of old identities/aliases and explicit review of ambiguous attachments, or narrow these APIs until those behaviors are implemented. Prove old links remain meaningful and no attachment is guessed onto a split output.
Evidence locations at the audited commit: claims.ex:141–161 (revision carry-forward), claims.ex:213+ (relationship reads), priv/repo/migrations/20260908153949_create_encyclopedia_schema.exs:784+ (review context), registry.ex:365+ (merge/split), and devils_dictionary_web/router.ex (routes).
Required continuation order
- Finish the schema transition sufficiently to restore compilation and a green test suite; port fixtures and schema-coupled references together.
- Close the correctness gaps above with meaningful regression/direct-write tests.
- Complete source adapters/materializer, resolver/linker, scopes, read contexts and the actual scorecard implementation. A mapping document is not the implementation.
- Demonstrate definition → author → biography/work navigation, plus one submitted/reviewed example visible correctly from both endpoints.
- Execute lifecycle/reconciliation and extension tests through the real application paths, not only spike SQL.
- Complete checksummed replay/full re-import into the new database, semantic comparison, rebuild proof and matched page/query measurements.
Closure evidence required: exact final commit/PR, successful mix precommit results, source import counts/errors and verified manifests, current scorecard results, lifecycle/extension test results, integrated query/page budgets, and an explicit list of any remaining scope exceptions. Keep the existing database and archived inputs under the agreed separate-database policy. No old-row migration or compatibility layer is requested.
This audit does not require repeating already evidenced Gate 0 work unless implementation changes invalidate it. It does require proving those guarantees in the finished application. Current status: architecture tested in a spike; application integration underway.
Decision: rebuild from sources; do not migrate existing data
The owner has explicitly stated that current application data is disposable. Optimize for the correct models and schema, then recreate the database and re-import. No backfills, compatibility reads, dual writes, preservation of existing row IDs, or data-migration rollback project.
This supersedes the existing-data migration requirements in #73, especially its identity backfill and compatibility stages. Keep #73's semantic contract, wireframes and use cases, and #72's source correctness requirements.
This is an implementation issue, not another open-ended architecture brainstorm. Keep Phoenix, Ecto, Postgres and the useful ingestion framework.
Distinguish:
- Now: throw away current derived application data and rebuild.
- After launch: new identities and attachments must survive source refresh, corrections and withdrawals. Disposable current data does not remove those future lifecycle requirements.
- Ecto schema migrations are still the reproducible way to create/change schema. We are eliminating migration of existing records, not database version control.
- Reset commands must explicitly target the intended development/test database. Do not delete original source archives, unrelated databases or credentials.
Goal
Build and demonstrate a clean encyclopedia model in which:
- A word and each source-specific meaning have explicit identities.
- A person has one entity identity across author, biography subject and example roles.
- Definitions, works, editions and passages retain their distinct semantics and provenance.
- Typed, attributed relationships connect valid endpoints and appear consistently from both directions.
- Local objects do not require Wikidata IDs.
- Re-import and later source changes cannot silently change what an existing attachment means.
Anchor example: Ambrose Bierce → authored definition → nepotism; Bierce → encyclopedia biography; one cultural artifact → attributed example of a selected sense/concept. Contested person classifications remain attributable interpretations.
Concrete proposed schema and model wireframes
This section supplies the first-version design that the planning session must validate and finalize. Recommended starting point: a small identity registry plus typed domain tables. It is a proposal to implement and test, not a claim that these tables already exist. Change this design only with a documented reason and equivalent acceptance coverage.
Before → after
| Current model | Proposed model | Practical result |
|---|---|---|
Authors in people; encyclopedia subjects in concepts |
One entity identity, with a person subtype and authorship relationships | Bierce's biography and definitions lead to the same person |
concepts.qid required |
Internal entity ID; external IDs in a namespace-aware table | Local artwork, event or concept can exist before Wikidata matching |
entries carries mixed prose and one author field |
Typed content items, revisions and role-aware relationship assertions | Definition, biography and quotation remain distinguishable; multiple credits supported |
| Source senses have gloss and source identity | Keep source-specific senses; separate stable identity from revisions | Changing source text does not silently replace the meaning being referenced |
| Example sketch requires lexeme and one example-concept field | Assertion has explicit subject and object IDs | Person→sense, artwork→concept, content→author and event→concept all have valid endpoints |
Vote target is unchecked type/id |
Vote/review references a real assertion revision | No orphan vote; changing a claim does not inherit approval for its old wording |
| Slugs and source IDs influence identity | Internal object IDs plus aliases and external mappings | Names/URLs can change without redefining an object |
A. Identity and typed records
All IDs below are internal bigint IDs unless noted. Object-backed domain tables use object_id as both primary key and foreign key to objects.id; timestamps omitted for readability. Relationships never point at an unchecked pair of table name and row ID.
| Table | Principal columns | Key rules |
|---|---|---|
objects |
id, kind, lifecycle_state |
Kinds initially lexeme, sense, entity, content; stable identity independent of labels |
lexemes |
object_id, language_tag, lemma, part_of_speech, lexical_key |
Source-independent word identity policy; URLs use ID; never merge by slug alone |
lexeme_forms |
id, lexeme_id, written_form, form_kind, language_tag, source_record_revision_id |
Explicit variants/forms and their evidence; accents/case/punctuation retained |
senses |
object_id, lexeme_id, source_id, external_key, identity_state |
Unique source/external key where trustworthy; source sense belongs to exactly one lexeme |
sense_revisions |
id, sense_id, revision_number, gloss, group_key, source_record_revision_id, lifecycle_state |
Unique sense/revision; source position is ordering, not semantic identity |
entities |
object_id, entity_kind, preferred_label, description |
Person, organization, concept, event, work, edition, etc.; no required QID |
person_details |
entity_id, birth_date, death_date |
One-to-one person subtype; authors are not a separate identity population |
work_details |
entity_id, work_kind, original_language |
Artwork, literary work, recording, etc.; descriptive facts retain evidence |
edition_details |
entity_id, work_id, edition_label, publication_year, language_tag |
Edition belongs to a work; do not duplicate the work for every hosted record |
content_items |
object_id, content_kind, original_language |
Definition, article, quotation, passage, image, etc.; stable object apart from mutable body |
content_revisions |
id, content_id, revision_number, body, body_format, canonical_url, source_record_revision_id, rights_metadata |
Immutable cited versions; explicit permitted display; unique content/revision |
object_names |
id, object_id, name, language_tag, name_kind, source_record_revision_id |
Names/pseudonyms/aliases do not create another entity |
external_identifiers |
id, object_id, namespace, external_id, source_record_revision_id |
One active verified mapping per namespace/external ID; uncertain candidates kept separately |
identity_events |
id, operation, actor_id, reason |
Audited retire/merge/split operation |
identity_event_members |
event_id, object_id, role |
Explicit input/output identities; split can have multiple outputs |
The registry is only identity and lifecycle. Domain-specific fields remain in typed tables. Person, work and edition detail rows are typed extensions of entities, not new competing identities.
Retain source-specific auxiliary fields such as lexical tags, topics, taxon data and import metadata where appropriate; this table list defines ownership and core relationships, not permission to drop imported semantics. Define their final columns in the ADR.
Enforcement: require exactly one matching domain row per object at transaction commit; enforce entity subtype/kind consistency. Implement and test deferred constraints/triggers where ordinary foreign keys cannot express these rules. Restrict kind changes after insertion. Retire identities rather than cascading away referenced content. A registry without these protections fails this design.
B. Source and relationship records
Keep the existing source registry, import runs and archived inputs. Extend stored source records so a cited revision is not overwritten.
| Table | Principal columns | Key rules |
|---|---|---|
source_records |
id, source_id, external_key |
Stable source-native record identity |
source_record_revisions |
id, source_record_id, revision_key, payload, checksum, observed_at, import_run_id |
Immutable/version-addressable evidence; exact replay inputs or archive locator |
actors |
id, actor_kind, user_id, bot_id, entity_id |
Exactly one typed principal for account, bot or external claimant; account≠public person by default |
predicates |
id, key, forward_label, reverse_label, is_symmetric, is_transitive |
Registered meanings; default no transitivity |
predicate_endpoint_rules |
predicate_id, subject_kind, subject_subkind, object_kind, object_subkind |
Enumerated allowed combinations; not arbitrary strings |
assertions |
id, origin_key, origin_actor_id, submitted_by_actor_id, current_revision_id |
Stable claim identity; source-import origin keys enable idempotency |
assertion_revisions |
id, assertion_id, revision_number, subject_object_id, predicate_id, object_object_id, rationale, valid_from, valid_to, language_tag, jurisdiction_entity_id, context_object_id, method, confidence, lifecycle_state |
Explicit endpoints and contextual meaning; changing these creates a revision |
assertion_evidence |
id, assertion_revision_id, source_record_revision_id, content_revision_id, sense_revision_id, evidence_role, locator, attribution_text |
At least one evidence target; if multiple are given validate compatibility; support/contradiction retained |
assertion_reviews |
id, assertion_revision_id, reviewer_actor_id, decision, reason |
Append-only review events; effective state derived deterministically |
assertion_votes |
assertion_revision_id, actor_id, value |
Unique actor/revision; FK target; relevance only |
source_assertion_outputs |
source_record_id, assertion_id, last_seen_run_id, output_key |
Tracks ownership/current outputs for safe reconciliation |
source_materialized_outputs |
source_record_id, output_object_id, output_role, output_key, last_seen_run_id |
Tracks source-produced senses/content; never delete another source's support |
Actors may be unknown for a historical claim: retain explicit unknown attribution rather than inventing a person. Submission/import actors must be accountable. A curator account claiming to be a known author is not automatically linked to that author.
Keep source support and editorial state separate. Reviews of a claim revision do not mutate source content. Withdrawn source output is retired; repeated input does not create a new semantic revision just because an import timestamp changed.
For source adapters, derive origin keys from stable source-native assertion identifiers where available. Where unavailable, specify and test an adapter-specific matching policy. Ambiguous sense identity changes create a reconciliation case; they never reuse an old ID for a different meaning. This cannot be solved by a uniqueness constraint alone.
Do not collapse claims by subject/predicate/object alone. Two claimants, contexts or source assertions can differ. Multiple citations can support one assertion revision without erasing their origins. Grouping similar claims for display is separate from identity.
C. Initial predicate contract
| Predicate | Allowed endpoints | Example |
|---|---|---|
defines |
definition content → lexeme or source sense | Bierce definition → nepotism lexeme |
authored_by |
content or work entity → person/organization entity | Definition → Bierce |
translated_by |
content or work entity → person/organization entity | Translated poem → translator |
published_in |
content → edition entity | Definition → dictionary edition |
about |
article content → entity | Biography → Bierce |
refers_to |
source sense → entity | Selected meaning → concept |
lexeme_entity_candidate |
lexeme → entity | Title/disambiguation candidate with method, confidence and review state; not sense equivalence |
| Source-native lexical predicates | lexeme → lexeme or sense → sense, as declared per mapping | Preserve unsensed lexical relations without inventing sense IDs |
illustrates |
entity or content → source sense or concept entity | Artwork → concept |
satirizes |
work entity or content → entity | Meme → practice/person |
depicts |
work entity or image content → entity | Portrait → person |
participates_in |
person/organization entity → event entity | Person → appointment event |
excerpt_of |
passage/quotation content → content or work entity | Quoted passage → poem |
Keep source-native lexical and semantic predicates, including WordNet and ConceptNet distinctions, through a reviewed mapping registry. These initial product predicates do not replace those source relations with generic illustrates or related_to.
Foreign keys enforce existence. Database validation must additionally reject incompatible endpoint kinds for the predicate, including on bulk writes. Check timestamps and confidence ranges where applicable; source weight is not a calibrated probability. Declare cycle rules per predicate rather than banning all graph cycles.
D. Schema relationship diagram
erDiagram
OBJECTS ||--o| LEXEMES : typed_as
OBJECTS ||--o| SENSES : typed_as
OBJECTS ||--o| ENTITIES : typed_as
OBJECTS ||--o| CONTENT_ITEMS : typed_as
LEXEMES ||--o{ SENSES : has_source_meanings
SENSES ||--o{ SENSE_REVISIONS : versioned_as
ENTITIES ||--o| PERSON_DETAILS : person_subtype
ENTITIES ||--o| WORK_DETAILS : work_subtype
ENTITIES ||--o| EDITION_DETAILS : edition_subtype
CONTENT_ITEMS ||--o{ CONTENT_REVISIONS : versioned_as
OBJECTS ||--o{ EXTERNAL_IDENTIFIERS : identified_by
ASSERTIONS ||--|{ ASSERTION_REVISIONS : revised_as
OBJECTS ||--o{ ASSERTION_REVISIONS : subject
OBJECTS ||--o{ ASSERTION_REVISIONS : object
PREDICATES ||--o{ ASSERTION_REVISIONS : types
ASSERTION_REVISIONS ||--o{ ASSERTION_EVIDENCE : supported_or_challenged_by
SOURCE_RECORDS ||--|{ SOURCE_RECORD_REVISIONS : versioned_as
SOURCE_RECORD_REVISIONS o|--o{ ASSERTION_EVIDENCE : cited_by
CONTENT_REVISIONS o|--o{ ASSERTION_EVIDENCE : cited_by
ASSERTION_REVISIONS ||--o{ ASSERTION_REVIEWS : reviewed_as
ASSERTION_REVISIONS ||--o{ ASSERTION_VOTES : relevance_votes
The four registry subtype links are individually optional but collectively exactly one. The diagram omits auxiliary tables and actor relationships for readability; the column contract above still applies.
E. Stored example: Bierce, not duplicated
IDs are illustrative local IDs, not production data.
OBJECT 101 entity/person Ambrose Bierce
OBJECT 102 entity/work The Devil's Dictionary
OBJECT 103 entity/edition Selected imported edition
OBJECT 104 lexeme nepotism · en · noun
OBJECT 105 content definition
OBJECT 106 content biographical article
OBJECT 107 entity/concept nepotism as a practice
OBJECT 108 content original illustrative meme
OBJECT 109 sense WordNet's nepotism meaning
Typed records:
person_details.entity_id = 101
edition_details: entity_id = 103, work_id = 102
senses: object_id = 109, lexeme_id = 104, source = WordNet
Assertion revisions:
102 — authored_by → 101
105 — authored_by → 101
105 — published_in → 103
105 — defines → 104
106 — about → 101
109 — refers_to → 107 [mapping evidence/review]
108 — illustrates → 107 [curator attribution/review]
The historical definition targets the lexeme unless a defensible source-sense target exists. The meme's claim is separate from its content. All links to Bierce point to object 101. No special “Bierce the author” duplicate is necessary.
F. Reader-facing wireframes tied to those records
WORD PAGE /words/104/nepotism
─────────────────────────────────────────────────
nepotism · English · noun
[Definitions] [Meanings] [Examples]
DEFINITIONS
The Devil's Dictionary · selected edition
Definition text… [105]
By Ambrose Bierce → [101]
[Edition 103] [Source / revision]
MEANINGS
WordNet meaning → [109]
Refers to: nepotism, the practice → [107]
EXAMPLES OF THIS CONCEPT
Illustrative meme → [108]
“Illustrates nepotism”
Proposed by curator A · review state
[Evidence] [Open claim]
─────────────────────────────────────────────────
PERSON PAGE /entities/101/ambrose-bierce
─────────────────────────────────────────────────
Ambrose Bierce · Person
[Biography] [Works] [Definitions] [Connections]
BIOGRAPHY
Article content… [106]
[Publication / revision]
WORKS AUTHORED
The Devil's Dictionary → [102]
DEFINITIONS AUTHORED
nepotism → [105]
Published in selected edition → [103]
─────────────────────────────────────────────────
Same identity 101; sections query different predicates.
WORK PAGE /entities/102/the-devils-dictionary
─────────────────────────────────────────────────
The Devil's Dictionary · Work
Author: Ambrose Bierce → [101]
EDITIONS
Selected imported edition → [103]
Contents: nepotism definition → [105]
ABOUT / INTERPRETATIONS
Attributed connections, when available
─────────────────────────────────────────────────
Work, edition and definition are separate records.
CONNECTION DETAIL
─────────────────────────────────────────────────
Illustrative meme [108] — illustrates → concept [107]
Claim #… · revision 2
Original claimant: curator A
Submitted by: account B
Context: … Review: Disputed
Rationale: …
Evidence: [source record/revision] [passage locator]
Counterevidence: …
[Open meme] [Open concept] [History] [Challenge]
─────────────────────────────────────────────────
The same claim revision appears from either endpoint.
ADD CONNECTION
─────────────────────────────────────────────────
Subject: [Find existing object / Create local object]
Relation: [Only predicates valid for subject type ▾]
Object: [Search valid target kinds ]
For “illustrates”:
[Specific source meaning / Concept ▾]
Context: [event, period, passage…]
Rationale: […]
Evidence: [+ citation and locator]
Preview: “A proposes X as an illustration of Y.”
[Submit for review]
─────────────────────────────────────────────────
Names help discovery; selected IDs determine endpoints.
G. Query and testing consequences
- Word definitions: indexed incoming
definesassertions for lexeme/sense, with source content and credits preloaded. - Person definitions: incoming
authored_by, filtered to definition content. Biography: incomingabout, filtered to article content. - Examples: incoming
illustratesfor the exact selected target. Concept-path discovery explicitly follows a reviewed mapping and labels the indirect path. - Add indexes on assertion revision endpoints/predicate, latest revision lookup, content/sense revision lookup, external namespace/ID and source ownership keys. Use the explicit current-revision selection contract below; never return historical revisions as duplicate active claims.
- Page displays must apply review/visibility rules before counts and pagination, in both directions.
- Prove the exactly-one subtype rule, predicate endpoint checks, external-ID collision behavior, deterministic current revision selection and vote revision isolation with direct-write tests.
- Start with these concrete records and flows, then exercise source change and the held-back extension case from the milestones below.
Reviewed audit corrections and mandatory pre-freeze gate
The supplied audit identified real omissions. This revision accepts the missing lexeme-level bridge, scope model, scorecard/read-layer inventory, input verification, revision context and early measurement requirements. It does not adopt its grades, unprovided benchmark results or line counts as verified facts. The number of agents is not evidence.
The change remains the backbone-correctness milestone from #72 executed through a clean rebuild instead of record migration. It is not a rewrite of all parsing or a mandate to replace the stack. The owner's disposable-data decision is sufficient authorization for that strategy; verify environment state before an actual reset rather than relying on unverified claims about deployments/users. Preserve original inputs and export any raw source records/import metadata that exist only inside the database before dropping derived tables. A reset without independently recoverable source inputs fails the rebuild contract.
Gate 0 — executable proof before schema freeze or broad adapter rewrites
Use an explicitly named separate scratch database for the candidate schema and benchmark. A second permanent Phoenix Repo, dual application runtime or compatibility layer is not required. Keep existing checks runnable during the spike. Use transactional candidate writes and real constraints; a fixed number of tables is not a deliverable.
Commit reproducible scripts, fixtures, SQL, query plans and environment details:
- Load Bierce content and WordNet senses for nepotism/bank; include a small Wiktionary bank fixture for its position-key failure. WordNet alone cannot prove Wiktionary reconciliation.
- Demonstrate deferred rejection of an object without its matching subtype and rejection of invalid predicate endpoints.
- Identical input creates zero new semantic revisions.
- Inserting/reordering/removing Wiktionary senses leaves attachments on the original meaning or flags ambiguity; it never silently reassigns them.
- Load a representative real WordNet edge population including high-degree nodes, and enough historical revisions to exercise revision-selection cost.
- Compare endpoint reads using DISTINCT ON/latest-row view, partial-current indexing and an explicit current-revision pointer. Measure both directions, complete page-query composition, write/update cost and concurrent writers.
- Record p50/p95, query count, EXPLAIN (ANALYZE, BUFFERS), rows/index bytes, load/index wall time, revision depth, cache conditions, hardware and Postgres version.
- Decide the current-revision implementation and freeze the adapter write contract only after this evidence passes agreed budgets.
The audit's claimed 72–91 ms versus sub-millisecond reads are a hypothesis to reproduce, not a portable timing guarantee. DISTINCT ON can create poor plans for some query shapes; it is not proof that every current-revision view must scan all history.
Current-revision correctness contract
Recommended baseline: assertions.current_revision_id, with a deferred composite FK (id, current_revision_id) → assertion_revisions(assertion_id, id) and supporting uniqueness. At committed state every persisted assertion has one designated current revision belonging to that assertion. Lock/serialize updates of that assertion when adding a revision and changing its pointer; test concurrent writers and rollback.
Current selection is distinct from lifecycle: the designated current revision can say withdrawn, while history stays immutable. Do not encode both “currentness” and “withdrawal” into one overloaded lifecycle value. Use endpoint indexes on revisions and pointer equality in the read; benchmark the actual query plan, since a pointer alone does not guarantee efficient reads.
If the spike selects a separate is_current flag instead, require a partial unique index on assertion ID plus a commit-time guarantee for exactly one selected revision and atomic switching. A partial unique index alone proves at most one, not exactly one. Do not maintain both pointer and flags without enforcing agreement.
Missing schema coverage
| Table/addition | Contract |
|---|---|
scopes(id, slug, name, rules, stats) |
Retain data-driven Animals/Emotions definitions, including score bars |
scope_lexemes(scope_id, lexeme_id, reasons) |
Composite key; FK to new lexeme object ID; retain membership reasons |
import_runs.scope_id |
Retain scope FK and replay context |
assertion_revisions.method, confidence |
Method records derivation; nullable confidence constrained to [0,1], explicitly heuristic; source weight stays a separate source-native quantity |
assertion_evidence.sense_revision_id |
Direct FK for evidence citing a source meaning; compatible with other cited revision targets |
review_contexts(id, assertion_revision_id) |
Immutable manifest of the endpoint content/sense revisions actually displayed |
review_context_items(context_id, endpoint_role, content_revision_id, sense_revision_id) |
Exactly one revision target per row; validate it belongs to the designated endpoint; an entity may have no textual revision |
assertion_reviews.review_context_id, assertion_votes.review_context_id |
Must match the reviewed assertion revision; preserve what the reviewer saw even if endpoint text later changes |
Endpoint revision changes can mark a prior review stale/needs-review according to declared materiality rules; they must not rewrite the old review context. Do not require a revision ID for immutable identity-only endpoints or fabricate an entity version.
Port the six linking methods and their corroboration stage explicitly. Sense-backed mappings remain refers_to; title/disambiguation heuristics use the lexeme-level candidate relation. Candidates can appear as possible subjects but do not automatically confer sense equivalence or propagate examples. Preserve method, evidence, confidence and editorial decisions; do not blindly preserve the existing automatic “confirmed” policy.
Add priv/scopes/culture.json with a documented explicit-lemma rule for the bounded pilot (nepotism, bank, egomaniac); implement that rule if needed, and document the actual scoped import/link commands. Do not assume those words are in Animals or Emotions.
Correct the Bierce catalog identity before the anchor import: current catalog.ex assigns Q310190. Wikidata Q191050 is Ambrose Bierce; Q310190 is Tobin Bell. Assert the verified fixture identity in a regression test; never solve this by merging the two QIDs.
Use is_symmetric/is_transitive column names. PostgreSQL lists SYMMETRIC as reserved. Quoted identifiers can work, so this was a naming hazard rather than a fatal schema defect.
Code and acceptance inventory — required at the first checkpoint
| Subsystem | Treatment |
|---|---|
| Pure source parsers, committed fixtures, source catalog | Reuse where semantically correct; fix evidenced source bugs |
| Materializer, batch writes, resolver, linker | Audit/rewrite schema-coupled SQL and output ownership; retain useful algorithms |
| Scope builder, scope tasks, scoped import entry points | Port to new lexical IDs; preserve rules/reasons |
| Lexicon/encyclopedia read contexts and WordPage builder | Port queries and preloads; test semantics and query counts |
| All six existing pages and search/navigation | Inventory routes and required behavior; preserve or explicitly replace each |
| Health/coverage/parity/pages/score and their tests | Map every existing score row to retained, revised or retired, with reason and replacement |
| Unshipped community sketch/demo assumptions | Replace with new contract; distinguish demo fixtures from real acceptance |
| README and docs/map | Update at the first schema checkpoint; remove forward-looking “no changes to thirteen tables” claims while preserving dated historical results |
Do not use approximate line counts as effort estimates. Produce a file-level port/rewrite/retire checklist. Fix @baseline_migrations 2 in the same checkpoint as schema changes, replacing table-count/migration-count acceptance with tested extensibility. No permanently red suite and no disabling checks merely to keep a score green. New acceptance measurements can be explicitly pending during the spike; completion requires their tests to pass.
Carry existing X2 search p95 <150 ms and O2 aggregate dump processing ≤2 hours, API elapsed separately reported forward with their actual populations and measurement definitions. The audit conflates search and page budgets; establish an additional full page/query budget from fresh matched baseline runs.
At Gate 0 record current corpus counts, maximum node degree, database/index sizes and archived input sizes. The supplied audit reports 1,541,669 lexemes, 1,179,377 edges, 344,635 source records and 3,057 MB database size; these are reported reference values requiring remeasurement, not newly verified results. Agree numeric storage/headroom and candidate page-query budgets before the benchmark. Measure costs with multiple revisions, not just a history-free corpus.
Reproducible source inputs
Before reset/re-import, create a manifest per input: source/edition, immutable archive locator, acquisition URL/date, byte count, SHA-256, parser version and license. Verify checksum before import and fail clearly on mismatch. A rolling latest URL and File.exists? are insufficient. A locally computed digest pins what was actually used; it does not independently authenticate a publisher. Test altered bytes and missing archives. Export DB-only API records/revisions to a checksummed replay archive before deleting the database.
Session checkpoints
Keep this issue as the implementation tracker rather than creating another overlapping issue solely for session bookkeeping. Each checkpoint posts commands, commit, test results, measured numbers and remaining blockers:
- P0: inventory, score-row mapping, input manifest, scratch correctness/read-path spike, final schema decision.
- P1: clean schema plus minimal Bierce/WordNet/Wiktionary fixture adapters, early lifecycle tests and updated README/checks.
- P2: remaining source writes, scope/linker/read ports with green targeted tests and score coverage.
- P3: connected UI and reviewed contribution slice; endpoint/revision context tests.
- P4: extension exercise, full verified re-import, matched scale measurements and completion report.
One planning session can define this sequence and contracts. It cannot honestly claim performance validation until the executable spike runs; do not promise a fixed one/two-day implementation based on the audit.
Related downstream work: #65 quotations, #66 design, #67 evidence wall. Their features consume the new model; do not add disposable user submissions on the old schema to discover a sense-drift defect already reproduced. #72's operational hardening remains required before public deployment.
Can this be planned in one session?
Yes: one bounded planning session should settle an implementable first version. It need not predict all future media, and it should not promise the entire implementation or full import finishes in that session.
The session must produce a committed ADR, concrete schema/ER diagram, importer mapping and ordered implementation checklist. End with decisions, not another list of equally weighted alternatives.
Required decisions
- Choose an addressable-object registry with typed domain tables, or explicit typed relationship tables. Document the choice against actual foreign-key integrity, query complexity and extension cost.
- Specify exact first-version tables, columns, indexes, checks, uniqueness and deletion rules; include how kind/subtype existence and predicate endpoint compatibility are enforced.
- Unify person/author identity; distinguish public entities from submitting user/bot accounts.
- Preserve source-specific senses. Explicitly model optional sense→concept mapping without inventing a universal merged sense.
- Define local entity identity plus namespace-aware external identifiers.
- Define the minimum work/edition/content/credit model needed for Bierce and an artwork; support additional roles without a full bibliographic ontology.
- Specify assertion identity, evidence, context, authorship, revision and review state; separate relevance votes from factual/source support.
- Define source-native assertion keys and refresh behavior: add, revise, retire, ambiguous identity change and replay.
- Specify commands and inputs for clean setup, bounded fixture imports and full source re-import.
- Set measurable query budgets and seed sizes for ordinary word/person pages before measuring them.
If one storage question remains uncertain, resolve it with a small executable schema/query experiment within this milestone. Do not make a speculative universal graph the prerequisite for importing a definition.
Implementation milestones
Execution order: pre-freeze gate → milestone 1 → bounded subset of milestone 2 with milestone 4 lifecycle tests → remaining adapters in milestone 2 → milestone 3 → remaining milestone 4 checks → milestone 5. Lifecycle correctness and read-path measurements are prerequisites to adapting all six sources, not end-of-project checks. Run mix precommit at every implementation checkpoint; update affected checks with the corresponding schema/read changes.
1. New schema and domain API
Create the new baseline through normal Ecto tooling; choose and document whether to replace the pre-release baseline migration history or add a rebuild path. Do not implement a transformation of old rows.
- Build chosen schema, domain structs/changesets and database constraints.
- Add narrow context APIs to create/read identities, source content, credits, assertions and evidence.
- Test direct SQL/bulk-write rejection of invalid targets, incompatible predicates and missing required records.
- Define canonical URLs using stable internal identity, with readable labels; spelling/slug collisions must not assert equivalence.
- Create local person/work/artifact/concept identities without external IDs; adding an external ID leaves identity unchanged.
- Remove or replace the unshipped community migration sketch so it cannot act as a competing contract.
Achievable result: an empty database can be created reproducibly; small fixtures represent the model and invalid data is rejected.
2. Import a small, representative source slice
Use archived/licensed source inputs already available where practical. Fixtures must retain source-native identifiers and relevant revisions.
- Adapt all six existing source contracts: WordNet, Wiktionary/Kaikki, Wikidata, Wikipedia, Bierce, Johnson.
- First execute bounded imports around Bierce, nepotism, an ambiguous word such as bank, and an entity with multiple source records.
- Import the author and biography subject as one entity using evidenced identity matching.
- Keep definitions as source content with author/work/edition links; do not relabel them as examples.
- Canonicalize Wikipedia publication identity independently of lookup probes.
- Honor supported Wikidata statement rank/qualifier/reference policy and retain original evidence.
- Preserve source-specific relation types and direction.
- Document unsupported source structures and unresolved mappings; do not silently flatten them.
Achievable result: real imported data follows the new model; one person can be navigated as both author and encyclopedia subject.
ConceptNet's production import remains the bounded pilot in #72. Validate that the new relationship contract can represent its native endpoint types, relation and dataset lineage before freezing the schema; do not require a full ConceptNet ingest here. Artsy integration is likewise not required: use a clearly identified local artifact fixture to prove the model.
3. Prove one connected product flow
Use #73's wireframes as acceptance guidance, scaled to the minimum useful slice.
- Render a word with separately attributed source definitions.
- Navigate from a definition's author to the person's page and back to their authored definitions.
- Show person biography and authored works in their appropriate sections.
- Create one attributed artifact→sense/concept relationship with exact endpoints, context, rationale and evidence.
- Show that same relationship from both endpoints with the same status and provenance.
- Support a minimal authenticated submit/review flow; do not build an entire social platform.
- Preserve rejected/disputed state across reads and importer reruns.
- Label indirect concept-path discovery separately from a direct example.
Achievable result: a working, inspectable end-to-end example, not just tables or a diagram.
4. Establish durability before broad import
Current data is disposable, but these tests use attachments created in the new schema.
- Re-import identical input: no semantic duplicates and equivalent results.
- Reorder/add/remove source meanings: attachments never silently move to a different meaning.
- Withdraw one supporting record: history and other support remain correct.
- Correct authorship: content identity remains stable and attribution history is inspectable.
- Merge identities with evidence; split an incorrectly combined identity: ambiguous attachments enter explicit review.
- Change claim endpoints/meaning: revisions and review policy prevent old votes/reviews silently endorsing a different claim.
- Generate operation sequences (import, attach, update, withdraw, replay) and check integrity/lifecycle invariants after each operation.
- Verify restricted/rejected content cannot leak through reverse views.
- Compare normalized semantic content, not just row counts, including unexpected stale records.
Achievable result: evidence that ordinary source evolution does not corrupt meaning or relationships.
5. Test extension, rebuild and scale
- After the core design works, add one challenging type/context: translated poem passage, fictional character in an adaptation, or an event with participants.
- Record the extension's schema/code changes. A small additive migration is acceptable; rewriting existing identities or hiding essential fields in JSON is a failed extension exercise.
- Recreate a separate empty database from documented source inputs and reproduce equivalent semantics.
- Run full re-import of the six existing sources with resumability and explicit unresolved/error reporting.
- Benchmark representative high-degree word/person pages and bounded incoming/outgoing queries against the budgets agreed in planning.
- Run
mix precommit, semantic import checks and end-to-end tests; publish results and remaining source coverage limitations. - Update Harden the encyclopedia backbone: durable word identities, correct source relationships, safe refresh, and ConceptNet integration #72/Unify encyclopedia identities and typed connections: people, authors, definitions, works and contextual examples #73 and model documentation to distinguish demonstrated behavior from deferred work.
Achievable result: the model works beyond hand-picked fixtures and has a repeatable rebuild process.
Scope boundaries
Do not spend time on:
- Migrating or preserving current application rows.
- Old/new schema compatibility or dual-write infrastructure.
- Full imports before the bounded correctness tests pass.
- Implementing every example in Unify encyclopedia identities and typed connections: people, authors, definitions, works and contextual examples #73 as a polished feature.
- A full media ingestion platform, recommendation engine or comprehensive voting system.
- Building a universal ontology or changing database technology.
Do retain:
- Actual database integrity, not changeset-only guarantees.
- Explicit source provenance and rights/display metadata.
- Future refresh/reconciliation correctness.
- Bounded queries, tested rebuildability and semantic validation.
Completion gate
Ready for broader feature development when:
- Bierce has one identity across authorship and biography.
- Words, source senses, concepts and published content remain distinct.
- A reviewed relationship can connect new local objects and existing encyclopedia identities.
- The relationship survives source changes without silently changing meaning.
- An additional use case fits through a controlled extension.
- Clean setup, re-import, tests and representative queries are reproducible.
The first planning session's expected output is a final first-version schema and executable work order. The implementation's expected output is a demonstrably correct vertical slice followed by full re-import. Do not equate a completed planning session or green CRUD tests with completion of this gate.
Activity
holden commentedon Sep 8, 2026
P0 — inventory, manifest, spike, decision
Commit
10097c4on branch74-encyclopedia-model.mix precommit: 504 tests, 0 failures.Two decisions on the working arrangement, taken with the owner:
devils_dictionary_v2.devils_dictionary_devis left intact — nothing is destroyed, the old corpus stays available as the matched baseline for the benchmarks and the P5 semantic comparison, and "recreate a separate empty database from documented inputs" becomes what we are doing anyway rather than an exercise at the end.74-encyclopedia-model, one commit per checkpoint, one PR at P5.mainstays green and MVP-0 stays reachable.Baseline, remeasured
The audit's reported reference values are confirmed on this machine (PostgreSQL 18.2, Apple M4 Max / 64 GB, Elixir 1.19.0-otp-28): 3,270 MB database · 1,541,669 lexemes · 1,179,377 lexical relations (1,085,482 sense-scoped, 15,718 unresolved) · 344,635 source records · max lexeme degree 1,768 out / 1,586 in.
Inputs pinned
priv/sources/MANIFEST.json+mix dd.manifestdigest all five archived inputs. 5/5 verified. Johnson's computed digest matches thesha256Sources.Cataloghas pinned since S5 — that agreement is how the mechanism was checked rather than asserted. Altered bytes, a flipped single byte, a missing archive and an unpinned entry all fail, with tests.Two findings:
data/raw-wiktextract-data.jsonl.gzcannot be re-fetched. 2.6 GB from a rolling URL (kaikki.org/dictionary/raw-wiktextract-data.jsonl.gz), previously pinned only bydump_bytes. Only 22,534 of its records reachsource_records; the 1.5M-row index is written straight tolexemesby the index pass. It is a required, unreproducible recovery input, and the manifest recordsurl_is_rolling: trueso nothing later implies otherwise.mix dd.export.replaywrites all 344,635 records with payloads to a checksummed gzipped archive — 101 MB, 15.7 s. Wikipedia's 85,044 and Wikidata's 72,770 exist only in the database (~3 h of batched API calls), so this is those two sources' equivalent of a pinned dump. Verified to round-trip includingraw["_probe"], whichmaterialize/1reads.Gate 0
Full evidence:
docs/spikes/2026-09-gate0/. Run on a named scratch database against the whole resolved corpus, not a sample — 1,792,058 objects, 1,163,659 assertions, 2,029,069 assertion revisions, with every fifth assertion carrying five revisions so revision-selection cost is actually exercised.Integrity — eleven rejections, and the Bierce anchor commits. Object with no subtype rejected at COMMIT (and legal mid-transaction, as it must be); wrong-kind subtype, two subtypes,
kindmutation,person_detailson a concept,defines→ a person,refers_tofrom a lexeme, confidence 1.4, a stolen revision pointer — all rejected. Including on a bulk write: one good row and one bad row in a single multi-rowINSERTrejects the whole statement, which is what makes this hold forinsert_all,COPYand the linker's raw SQL.Current revision — the shootout. 3,000 transactions each, warm cache,
pgbench -lpercentiles:DISTINCT ONis_currentindexA 194× difference in pages touched. Two of the audit's expectations do not survive contact: it predicted the pointer would be fast and
DISTINCT ONslow, andDISTINCT ONbeat the pointer on every figure. Neither claim should be repeated as fact.So: strategy C, and the pointer is dropped entirely — one source of truth, so §"do not maintain both pointer and flags without enforcing agreement" cannot be violated. Correctness is completed by a partial unique index (at most one), deferred constraint triggers (at least one, at commit), and
SELECT … FOR UPDATEas the write path's first statement. Atomic switching was measured rather than assumed, since a partial unique index cannot beDEFERRABLE: single-statement flag moves work in both directions on PG 18. Concurrent writers serialise; without the explicit lock, an adversarial statement ordering loses a write to a duplicate-key error. Currentness and lifecycle stay separate columns — the designated current revision of assertion 11 readswithdrawn.Write path sustains 13,756 tps at 8 clients, p95 0.904 ms.
Lexical relations stay in the assertion model. 1,118 MB against
lexical_relations' 730 MB — 1.53× for a full revision history and a 10× faster endpoint read. The typed-fallback table named as an escape hatch is not needed.Sense identity — proofs 4 and 5 on the real Wiktionary
bank/noun/1record, captured from the pinned dump (one full 2.6 GB pass, 47 s; 7 records across 4 etymologies). Identity is matched on content — pos, etymology number, normalised gloss scored withpg_trgm— never on position.object_id). Both open areconciliation_casesrow and setneeds_review. No revision written, no attachment moved, no identity reused.Two things the gate found that reading would not
AFTER INSERTtrigger onobjectsis half a guard. Deleting Bierce'sentitiesrow left object 101 typedentitywith no entity row whileauthored_bystill pointed at it.AFTER DELETEconstraint triggers on all four subtypes close it, and the rule that survives is the one §A asks for: an identity is retired vialifecycle_state, never deleted out from under its claims.sense → lexemeis a real endpoint pair. Declaring onlylexeme → lexemeandsense → sensesilently dropped 36,434 of 1,163,659 relations — a meaning pointing at a word without naming which of the word's meanings. §C requires exactly that, "without inventing sense IDs". After declaring it: 1,047,918 sense→sense, 79,307 lexeme→lexeme, 36,434 sense→lexeme, 0 assertions without a current revision.Also, two naming hazards beyond the known
SYMMETRIC:POSITIONis reserved in aRETURNS TABLEcolumn list, and one trigger function shared across two tables whose key columns differ by name fails at run time, not at creation.Budgets, agreed from these measurements
lexical_relationstodayFull word-page composition p95 is pending, not guessed — it needs the ported
WordPagebuilder to measure honestly, and is set at P2 against a matched baseline run ondevils_dictionary_dev.Inventory
docs/adr/0001-encyclopedia-model.md— the ten decisions, what was rejected and why.docs/rebuild/port-checklist.md— every file to reuse / port / rewrite / retire. No line counts.docs/rebuild/score-rows.md— all 37 scorecard rows mapped, plus 8 new ones. E1 stops countingschema_migrations(its== 2equality would fail this rebuild on day one) and becomes a test that actually adds a source, a predicate and an entity kind. E3 stops grading a rolled-back sketch byFile.exists?. M1 must fail on the corruption probe it currently passes.README §Status withdraws the "none of which changes the thirteen tables" claim; the dated historical results are preserved.
Remaining blockers
None for P1. Two things carried forward: full page-composition budget is pending until P2, and disk is the tightest constraint — the volume is at 92%, 35 GB free, against a ~3.3 GB second corpus, so P5's rebuildability proof creates its third database, verifies and drops it rather than keeping one.
Next: P1 — the migration, the domain API, and the direct-write rejection tests.
P5j: the resolver's claims were owned by nobody (#74)
P5k: O4 claimed the page and the CLI agree, and checked nothing (#74)
Record the two closed acceptance items in the completion report (#74)
P0: the audit committed, the inputs pinned, and Gate 0 run (#74)
WIP P1: the encyclopedia schema, the registry and claims APIs, and th…
P1: the port onto the encyclopedia model, and the seven correctness g…
P2/P4: durability, the extension exercise, and six new scorecard rows (…
P3: the connected flow — one identity, three pages, one claim (#74)
P5a: the rebuild path, and the four defects a real re-import found (#74)
P5b: two more defects the rebuild found, and the guard that missed th…
P5c: three defects between the last source and the first claim (#74)
P5d: one article, many records — and what "retired" has to mean then (#…
P5e: `--` removes one occurrence, and the sandbox hid it (#74)
P5f: the second sweep, and one writer for claims (#74)
P5g: an edge deferred is not an edge withdrawn (#74)
P5h: three records, one claim, three attestations — and a 154 s healt…
P5i: M2 caught two things a re-import repeats for ever (#74)
P5j: the resolver's claims were owned by nobody (#74)
P5k: O4 claimed the page and the CLI agree, and checked nothing (#74)
Record the two closed acceptance items in the completion report (#74)
holden commentedon Sep 17, 2026
Closing: the issue's own record stops at P0, but the README status table carries P1 (
6a6ae8f), P2/P4 (e60ab6e) and P3 (0dbb752) as landed 2026-09-08/09, with the independent completion verification indocs/rebuild/completion-2026-09-09.md(independent clean rebuilddevils_dictionary_74_verify, semantic M2,mix dd.verify.rebuild) and the P5 score-row mapping indocs/rebuild/score-rows-p5.md. The reader milestone that consumed the model shipped as PR #83 (#82) and the connection-reliability fixes as PR #87 (#84).Remaining correctness work is owned by #78/#84's successors, not by this issue. Reopen if the P5 close-out list has residue those do not cover.