Repository navigation
Description
Superseded by #84. This issue is retained as historical context. Completed reader work has been removed from the active plan; remaining requirements, examples, wireframes and acceptance gates now live in #84. Do not execute the stale baseline/checklists below as a new work order.
Objective
Finish the encyclopedia foundation and connected product experience left incomplete by #73/#74. This is the consolidated implementation work order, including the subject-independent cleanup in #77. Keep the useful schema and source pipeline; fix integration defects and complete the missing workflows rather than starting another architecture rewrite.
Target: an evidence-backed A+ against the defined contract below, not a promise of universal future compatibility. Green existing tests, one hand-seeded person, or passing topic scorecards are not sufficient. A reviewer must be able to reproduce the connected workflows and the negative cases.
Baseline: commit b0abb07, branch 74-encyclopedia-model, PR #76. Audit assessment: C+ overall; architectural direction B+; narrower #74 completion B−. Existing suite passed 722 tests. Seven isolated audit checks yielded six failures and one pass. The full local report and executable reproductions are currently at docs/audits/2026-09-09-issue73/ and were not committed at issue creation. P0 below must publish them as part of the implementation evidence. The concrete reproductions are also described here so this issue does not depend on access to that local directory.
Scope and ownership
- This issue owns the remaining Unify encyclopedia identities and typed connections: people, authors, definitions, works and contextual examples #73/Implement the encyclopedia model from a clean database: schema, connected identities, validation and re-import #74 correctness and minimum useful product workflows, and executes Remove Animals prototype coupling: make the encyclopedia subject-independent #77's removal of Animals-specific product/default assumptions. Remove Animals prototype coupling: make the encyclopedia subject-independent #77 remains the detailed supporting specification; do not run a second competing implementation of the same work.
- Coordinate source-adapter changes with Harden the encyclopedia backbone: durable word identities, correct source relationships, safe refresh, and ConceptNet integration #72. Full ConceptNet production ingestion, Artsy production integration, a general media ingestion platform, curator bots (The Dead Editors’ Society: historical personas curate examples and marginalia #75), a recommendation system and comprehensive social/voting features remain downstream. Test representability and lineage using explicit offline fixtures where integrations are not live.
- Implement the encyclopedia model from a clean database: schema, connected identities, validation and re-import #74 superseded old-data migration, compatibility reads and dual writes. Preserve that decision: no legacy-row migration project. If disposable derived data must change, re-import. Keep original source archives, credentials and unrelated databases intact; reset only explicitly designated disposable databases.
- Do not erase legitimate animal entries or taxonomic relationships. Animals should be ordinary content and an optional test population, never a privileged product category or hidden import default.
- Do not close Unify encyclopedia identities and typed connections: people, authors, definitions, works and contextual examples #73/Implement the encyclopedia model from a clean database: schema, connected identities, validation and re-import #74/Remove Animals prototype coupling: make the encyclopedia subject-independent #77 automatically on the strength of this issue or a green suite. The closure report must map their requirements to completed evidence and explicitly accepted downstream boundaries.
P0 — Freeze the acceptance contract and baseline
- Inspect current branch/PR state and coordinate ownership before editing shared files or running corpus mutations. Use one frozen implementation revision for final measurements; record code/input/schema/database fingerprints.
- Bring the audit reproductions into the permanent regression suite, verifying each fails for the intended reason before its fix. Record baseline test results and the current imported population.
- Create a requirement matrix for Unify encyclopedia identities and typed connections: people, authors, definitions, works and contextual examples #73, Implement the encyclopedia model from a clean database: schema, connected identities, validation and re-import #74 and Remove Animals prototype coupling: make the encyclopedia subject-independent #77: implemented, incomplete, superseded by the no-migration decision, or explicitly deferred. No unchecked requirement silently disappears.
- Preserve existing valid evidence. Re-run expensive imports/measurements only when changes invalidate them; enumerate what was reused and why. A changed fixture assertion is not a substitute for a changed implementation.
Already demonstrated; preserve it
Typed object registry/subtype integrity; source-specific senses and revisions; local IDs without required QIDs; constrained predicates; immutable claim/evidence history; independent review/vote/source-support dimensions; source reorder/withdrawal regressions; deterministic replay; exact independent rebuild comparison; indexed bounded API reads; Bierce definition→person→biography/work navigation.
The Surprise me scope restriction was fixed in b0abb07: enriched words outside all test scopes are eligible. Do not redo that fix; preserve its regression and complete cross-kind discovery below.
P1 — Fix the contractual correctness defects
1. Apply public visibility to every projection
Reproduction A: create article→person about, reject the relationship. Claims.incoming returns nothing, but the person's biography section still displays the article.
Reproduction B: create definition→word defines and definition→person authored_by, reject defines. The author's definition summary still displays the rejected word association.
Starting points: Encyclopedia.content_about/2 (lib/devils_dictionary/encyclopedia.ex), EntityPage.targets/2 (lib/devils_dictionary/encyclopedia/entity_page.ex). These query current/active assertions without the shared editorial filter.
- Centralize/reuse public visibility for specialized sections, counts, pagination, linked titles, credits, publication relationships and previews. Internal reviewer/import reads must remain explicit.
- Test rejected/withdrawn relationships through both endpoints, direct/historical claim URLs and every specialized page projection. Also verify disputed/needs-review behavior matches the documented display policy.
- Preserve the working content-lifecycle filter: a genuinely withdrawn current content revision is already hidden from biography rendering. Do not confuse source withdrawal, editorial rejection and rights restrictions.
- Define and enforce rights/display restrictions in content, excerpts, evidence and historical views while preserving permissible identity/provenance metadata. A rights metadata column alone is not display enforcement.
2. Complete identity merge/split/retirement behavior
Reproduction C: merge two person identities, then call Registry.resolve(old_id). It raises an Ecto enum cast error because lifecycle :merged is used where operation :merge is required.
Reproduction D: after a merge, EntityPage.build(old_id) continues presenting the old identity. Public entity/word readers do not consume identity resolution.
Reproduction E: split an identity used only as context_object_id on a claim. No reconciliation case is opened because the split logic checks only subject/object references.
- Fix lifecycle→operation mapping and integrate merge/split/retired states into public canonical reads and discovery.
- Define safe survivor relationship aggregation without rewriting immutable historical endpoints. Old links must lead to a meaningful survivor, split choice or retirement explanation.
- Validate merge/split operations: compatible kinds, live output identities, nonempty valid outputs, no self/cyclic resolution, repeated operation behavior, concurrent changes, and accountable actor/reason. Test multi-step merges and external-ID/name conflicts.
- Preserve provenance when names/identifiers converge; do not discard supporting evidence merely because labels duplicate.
- Reconcile all semantically relevant split references, including subject, object, context and jurisdiction; audit other reference-bearing tables. Never guess which split output inherits an attachment.
- Provide a minimal authorized reconciliation queue/action: see the old identity and alternatives, choose/decline a mapping with evidence and reason, create appropriate new claim revisions, retain original history. Leave unresolved ambiguity visible and safe.
3. Detect stale review/evidence context
Reproduction F: accept article→person with a pinned displayed article revision, then append different article text without revising the claim. Connection.build displays the new text and old acceptance without a stale-context indication. The existing durability test revises the claim manually after changing source text and misses this interval.
- Retain historical acceptance, but derive and display whether its reviewed context matches the currently shown subject/object/context/evidence revisions. Show exact reviewed text or an explicit stale/re-review state under a documented policy.
- Do not make historical votes/acceptance imply endorsement of changed text. Test endpoint-only changes, claim changes, attribution changes, withdrawn evidence and unchanged replay separately.
- Pin the evidence revision shown in the composer at selection time, or reject/reconfirm if it changes before submission. Currently selection stores an object ID and submission resolves its latest revision.
- Include rendered evidence in review context where applicable. Verify stale-context checks against the exact content the reviewer saw, not only the assertion revision number.
4. Repair output run ownership and reconciliation
Observed corpus: 117,363 source_assertion_outputs.last_seen_run_id values are null: Wiktionary 116,448; Johnson 906; Bierce 9. Source association still exists; this is not total provenance loss.
Concrete code path: Mix.Tasks.Dd.Resolve creates a run but calls Resolver.run without its ID; resolver forwards nil to the materializer; ownership upserts can overwrite last-seen stamps with nil. Reconciliation treats null as unseen.
- Pass/validate appropriate run identity through resolver and linker writes. Define which runs can establish source ownership; bookkeeping-only operations must not clear valid ownership.
- Test import→materialize→resolve→reconcile, interrupted/resumed runs, partial-source visits and multiple attestations. Prove one run/source cannot retire another's support or untouched records.
- Re-measure null stamps and reconciliation outcomes. Expected zero unexplained missing stamps; any legitimate exception must have an explicit model and regression test, not a blanket scorecard exemption.
P1 exit gate: all reproduced failures fixed, direct-write and integrated regression tests green, and no known high-priority violation of visibility, identity continuity, review meaning or ownership left open.
P2 — Make discovery and ingestion subject-independent
Execute #77's full specification as part of this issue.
- Remove Animals header/footer links and prototype topic assumptions from public discovery. Keep population browsing as a deliberate diagnostic/operational surface, with consistent old-route handling.
- Remove implicit Animals defaults from commands, health/admin functions and source coverage navigation. Bounded runs require an explicit population; no missing option should accidentally launch a global network import.
- Move animal-specific category exclusions, roots and coverage thresholds into test/population configuration. Separate global integrity checks from scoped enrichment/coverage reports and denominators.
- Main discovery finds words and entities across supported kinds, with explicit type labels, canonical identity links and collision/disambiguation handling. Do not merge identities on a name or slug match.
- Define Surprise me's public population explicitly: useful word and supported entity entries across subjects, not test-scope membership. Preserve bounds and empty-population behavior. Randomness may still select an animal; the acceptance test is eligibility independence, not a promise that animals never appear.
- Generalize Wikidata acquisition/projection policies beyond taxa: distinguish retained facts, typed projection and next-entity fetch policy, with explicit budgets, resumability, unresolved targets and source evidence. Do not recursively crawl the entire graph.
- Demonstrate typed people, organizations, places, works and events from representative source fixtures, plus a bounded actual non-animal import. Classifying/enriching an existing concept must preserve its identity and attachments through a controlled operation.
- Document omitted/unfetched source properties and retained payload policy. Six adapters existing does not mean all source data is represented.
Bierce discovery gate
Current data: person ID 1 exists; lexical ID 341814 / sense 342580 for the name has no outgoing person mapping. The name is outside all test scopes. The whole database currently has only two typed people, two works and two editions—the historical-author seeds—so seeded anchors are not evidence of general entity ingestion.
- Search “Ambrose Bierce”: distinguish the person from the source's lexical entry for the name.
- Follow an evidenced lexical sense→person relationship or clearly labeled candidate. Do not hardcode Bierce or infer identity from matching names alone.
- From person: biography, work, edition and definitions use the same identity; from nepotism's definition: author link reaches that person.
- Repeat with a non-seeded person and a same-name ambiguity fixture to prove this is a general mechanism.
P3 — Complete the minimum connected contribution experience
#74's existing-object submit/review slice is useful. Finish the remaining #73 experience without building a full social platform.
- Create a minimal local person/work/artifact/event/concept through an authenticated flow with typed fields, duplicate candidates, local stable identity and optional evidenced external ID. No QID or topic scope required; no arbitrary unchecked type/id endpoints.
- Connect existing/new objects using only valid predicate/target combinations, exact sense/concept identity, rationale and relevant context (time, language, jurisdiction where supported).
- Distinguish original claimant from submitter when quoting an external claim. Local contribution defaults to its accountable submitter; uncertain attribution remains explicitly unknown. An account does not automatically become a public person identity.
- Add multiple supporting or contradicting citations with exact source/content/sense revisions and locators. Make citations inspectable, not just printed database IDs.
- Provide content/passage and source-sense destinations or equivalent stable revision views. Historical evidence must open the cited version; a current word page alone is not an exact sense destination.
- Allow authorized claim revision and challenge/counterevidence submission. Changing target, meaning or attribution creates a new revision; do not silently carry approval onto the change. Define authorization for submitter edits versus reviewer decisions.
- Render a useful review/history log: who decided what, why, against which version/context, plus current versus historical/stale state. Reviewer accept/dispute/reject stays accountable and permission-checked.
- Preserve independent relevance votes and source support at model/API level. Comprehensive voting feeds/ranking remain downstream; do not present absent voting UI as completed.
- The same artifact can support multiple distinct claims and contexts without duplicate artifact identity or overbroad subject/predicate/object uniqueness.
Target behavior (wireframe guidance)
Search: Ambrose Bierce
Ambrose Bierce Person →
Ambrose Bierce Name · WordNet entry →
Person: Ambrose Bierce
Biography [source / revision]
Works [More] Definitions authored [More]
Connections [More] [Suggest a connection]
Connect [Existing object | Create local object]
Subject [typed identity] Relation [valid predicate]
Target [exact sense / concept / entity]
Context [optional relevant fields]
Claimant [self / attributed source / unknown]
Rationale [...]
Evidence [+ Support] [+ Counterevidence] [inspect exact version]
Preview: A proposes X as an illustration of Y
[Submit]
Connection
X — illustrates → Y
Submitted by ... · Claimed by ...
Review: accepted / disputed / pending / stale context
Evidence [open cited passage/version]
[Revise, if authorized] [Challenge]
History: actor · action · reason · cited/displayed revision
Stack these flows on narrow screens without losing identity, attribution or review state. Clearly label illustrative cultural claims; do not manufacture factual characterizations of living people for fixtures.
P4 — Complete bounded navigation and extension evidence
- Expose existing cursor pagination in public incoming/outgoing lists and authored definitions/works/contents. Current entity sections stop at 50 without continuation. Apply visibility before pagination; ensure stable order, no missing/duplicate results for a fixed dataset.
- Fetch/page distinct role sections independently so the first 50 authored definitions cannot crowd all works off a person's page.
- Test high-degree entities, sense-specific links, multiple credits, anonymous attribution and local content outside every scope.
- Add a genuinely new controlled extension after the baseline is fixed (e.g. fictional character/adaptation or an interactive installation), with explicit typed endpoint rules and recorded schema/code cost. The existing translated-poem test uses already registered kinds/predicates: preserve it, but do not call it proof of an unseen extension.
- Show an existing attachment survives adding the new kind/relationship. Small additive schema changes are acceptable; rewriting old identities or hiding contractual meaning in unvalidated metadata is not.
P5 — Integrated verification and closure report
-
mix precommitand all new negative/integration tests pass. Preserve tests at real commit boundaries where deferred constraints require them. - Run a real browser walkthrough at desktop and mobile widths: search→word/person→work/edition, local-object creation→claim with exact evidence→review→edit/challenge→consistent endpoint views. Include rejected specialized projections and stale reviewed text, not just HTTP status checks.
- Re-run the six-source semantic replay and independent rebuild comparison when the adapter/identity changes require it. Freeze corpus writes during measurement. Compare normalized semantics and history guarantees, plus ownership/lifecycle invariants separately.
- Run current scorecards with explicit populations and global integrity checks; CLI health and browser health agree. Do not conflate zero pending score rows with completion of this issue.
- Revalidate high-degree pages, both traversal directions, cross-kind discovery and write/update budgets affected by the changes. Record cache state, population, p50/p95, query count and relevant plans. Prior warm-cache reference: page p95 44.191 ms, traversal 0.418 ms, lexical search 69 ms; these are measurements, not replacements for the agreed budgets.
- Verify rebuild inputs/checksums and documented commands. Clean up the explicitly disposable proof database after evidence is saved, without touching the original/source archives.
- Publish final commit/PR, requirements matrix, source coverage/limitations, reproducible logs, browser evidence and remaining downstream scope. Update Unify encyclopedia identities and typed connections: people, authors, definitions, works and contextual examples #73/Implement the encyclopedia model from a clean database: schema, connected identities, validation and re-import #74/Remove Animals prototype coupling: make the encyclopedia subject-independent #77 with the final evidence and closure recommendation; merge/close only when authorized.
Scenario matrix — required acceptance coverage
| Scenario | Required result |
|---|---|
| Animals population absent | Core discovery/entries and explicitly selected non-animal imports/health still work. |
| Cat / biological taxon | Ordinary animal content and taxonomy remain valid. |
| Bierce and a non-seeded person | One evidenced person identity across names, biography and credits; no name-only merging. |
| Same-name people / Apple versus apple | Separate identities and deliberate disambiguation. |
| Situationship | Source-specific lexical meaning without taxonomy membership; offline slang-source fixture is labeled as such. |
| Nepotism / local artwork or event | Distinct typed objects with contextual attributed example, not an asserted universal truth. |
| Bank / river image versus financial institution | Exact sense selection; no propagation to unrelated meanings. |
| Translated poem / anonymous work / organization author | Role-aware attribution without forcing every creator into a natural-person identity. |
| Rejected biography or defines relationship | Hidden consistently from all public specialized sections and common readers. |
| Source text changes after acceptance | Exact reviewed version remains inspectable; new text does not inherit undisclosed acceptance. |
| Merge / split / context split | Old links meaningful, no resolver crash, ambiguity queued, historical assertions preserved. |
| Two source attestations; one withdraws | Surviving support retained; ownership stamps and editorial decisions remain correct. |
| More than 50 definitions/connections | All permitted records reachable via bounded stable pagination. |
| One new object/relationship kind | Explicit extension path, no corruption of existing identities or claim semantics. |
A+ exit standard
An independent reviewer can complete the matrix from the documented inputs and UI, verify all high-priority invariants through adversarial tests, and find no known unresolved contract violation. Source coverage and deliberately deferred integrations are explicit. All remaining parent-issue requirements are either evidenced here or explicitly accepted as downstream scope—not silently renamed “done”.
Do not optimize for an A+ label by weakening tests, changing score thresholds after measurement, hiding unsupported cases, or counting seeded examples as general ingestion. The outcome is a reliable, understandable, subject-independent encyclopedia foundation ready for subsequent content layers.
Activity
holden commentedon Sep 10, 2026
Execution order now lives in #79 (MVP-1 tracker, numbered sessions). This issue stays open as the detailed specification; #79 sequences it against the usability work and states what is deferred.
Taken from here into #79:
3cca71d, now onmainDeferred, explicitly: P3 in full (new-local-object creation, claim edit/challenge, counterevidence, original-claimant selection, voting UI) and P4's unseen-kind extension proof. Those answer is the model sound — and the audit says it is. They do not answer is this usable, and they gate contributors we do not have yet (
usersis empty).Two corrections to the baseline in the body, both verified 2026-09-10:
docs/audits/2026-09-09-issue73/, onmainas of the Build and verify the encyclopedia identity and relationship model #76 merge (f4dd10a). P0's publish item is satisfied.mix test docs/audits/2026-09-09-issue73/reproductions.exsstill gives 6 failures, 1 pass atmain. The six named reproductions are unchanged; only run ownership was fixed.The gap this issue does not cover, and which #79 puts first: 12.7 % of the 1,541,668 indexed words have any definition at all. Wiktionary was only ever absorbed for the three test scopes (22,521 records). Running this issue top to bottom yields a provably correct encyclopedia with 195,322 defined words, two people, and Animals in the navbar.
holden commentedon Sep 11, 2026
Superseded at the owner’s request by #84: #84
The replacement distinguishes completed #82 reader work from the remaining correctness, general-entity and connection-authoring work. It includes before/after examples, four wireframes, four delivery checkpoints, acceptance scenarios and a mapping of the old requirements.
Closing this issue as superseded, not because its outstanding requirements are implemented. Continue execution and audit against #84. #73/#74 remain parent specifications; #72 and #67/#66 retain their separate source/deployment and cultural-wall work.