Repository navigation
Description
Two sessions. Self-contained — everything needed is in this issue. No question should need asking; where a decision could be argued, it has already been made below and the reasoning is given so you can implement it rather than relitigate it.
Supersedes the session breakdown in #79 for the remaining MVP-1 work. #77 and #78 stay open as background specification; you do not need to read them to do this.
What this finishes
MVP-1: a stranger can use this. At the end of session B, someone who has never seen the project can arrive, search for a word or a person, get a real definition, and navigate without hitting a dead end or an empty page.
What this explicitly does not finish
General Wikidata entity ingestion (people/places/orgs beyond the 6 seeded rows) and the six correctness defects in docs/audits/2026-09-09-issue73/reproductions.exs. Those are real and they are downstream of a usable product; do not start them here, and do not let them expand these two sessions.
State of the world — verified on main at c942d19, 2026-09-10
752 tests, 0 failures. Working tree clean. Dev DB devils_dictionary_v2, ~5.3 GB. Dev server port 4007 (.claude/launch.json, mix config).
Already done, do not redo:
- W1 (
7a3b813,f5f9e4b) — corpus budgets measured;mix dd.absorb wiktionary --sample-every Nexists;Materializer.bounded_key/1fixes anorigin_keyoverflow. Evidence:docs/rebuild/w1-2026-09-10/README.md. - N1 (
c942d19) — the prototype surface is gone. Nav is a logo and a theme toggle. Consoles live at/ops/{scopes/:slug,health,imports}with 302s from the old paths. All 15\\ "animals"production defaults removed; four mix tasks require--scope. Homepage and word-page copy name no population.
| now | |
|---|---|
| Indexed lexemes | 1,541,668 |
| With any definition | 282,098 (18.3 %) |
| Wiktionary records absorbed | 116,930 of 1,487,639 |
| Coverage ceiling (measured) | 1,470,121 lexemes carry wiktionary — 95.4 % |
Decisions already made
Implement these. Do not re-open them.
- The word stays
scope. Remove Animals prototype coupling: make the encyclopedia subject-independent #77/Complete the encyclopedia foundation: close remaining #73/#74 gaps and verify the full connected experience #78 say "population" as prose; it is not a rename request. A rename touches the table, the schema, the params, 40+ call sites and six documented mix flags. No. - One import pass, no banding by frequency. W1 measured ~15 min. Banding existed against a guess it would take hours.
- Sessions may open more than one PR. CodeRabbit silently skips any PR over 150 files — that is what happened on Build and verify the encyclopedia identity and relationship model #76, which was never reviewed at all. Keep each PR reviewable.
- Never
mix dd.reset. Never write todevils_dictionary_dev— it is the untouched baseline. Work only indevils_dictionary_v2. - Do not weaken a scorecard bar to make a row pass. If a row fails, report it with its measured value.
Session A — The corpus
Goal: coverage from 18.3 % to the low-to-mid 90s, with search still landing on the right word.
Branch impl-a-corpus. Expect 2–3 PRs.
A1 — Canonical form resolution. This blocks everything else in the session.
W1 absorbed 6.3 % of the corpus and broke scorecard row X3 — forms and variants — from 4/4 to 1/4. Session A absorbs the other ~93 %. Fix this first or the import makes search worse, not better.
The evidence it is a regression (do not skip this — it is also how you will verify the fix):
docs/audits/evidence-2026-09-07/animals-scorecard.txt: PASS X3 4 / 4 probes land
docs/rebuild/evidence-2026-09-09/score-animals.txt: PASS X3 4 / 4 probes land
mix dd.score --scope animals # today: X3 1 / 4 — monkeys → monkeys, cats → cats, geese → geese
The mechanism. Lexicon.lookup/1 (lib/devils_dictionary/lexicon.ex:119) opens with:
matches != [] and Enum.any?(matches, &enriched?/1) -> resolve_canonical(matches, word, :lemma)and enriched?/1 is enriched_at != nil or canonical_lexeme_id != nil. The S1b design only ever worked because inflected rows stayed bare: a bare form_of row is rejected from headwords, falls through to by_form/2, and lands on the headword. Absorbing Wiktionary gives those rows senses and an enriched_at, so they now win the first branch and keep their own page.
cats is the sharpest case and the one to test against: cats itself is still bare, but W1 enriched CATs, and by_lemma_or_slug/2 matches on slug, so an enriched sibling drags the whole match set into the wrong branch. This is exactly the CATS collision S1b was written to prevent.
Two ways to fix it. Pick one, say which and why:
- Have
dd.resolvefillcanonical_lexeme_idonform_ofrows. S1b intended this —geese/unknownandspat/unknownalready have theirs; the noun rows do not. - Have
lookup/1's first branch prefer non-form_ofmatches.
Exit: X3 back to 4/4, plus a unit regression test that fails when an inflected row is enriched. Today's X3 only catches this after a corpus-wide import, which is why it went unnoticed for a whole session. The test is the deliverable, not the green row.
Do not fix this by deleting or un-enriching rows. monkeys having a Wiktionary sense is correct data; the lookup is what is wrong.
A2 — The full Wiktionary records pass
Add a third mode to Wiktionary.absorb/2 (lib/devils_dictionary/absorb/sources/wiktionary.ex) beside --index and --scope: full trimmed records, unscoped.
Read sample/1 in that file first — it is the closest thing and it shares scoped/2's stream, trim, insert and materialize path. But it is a measurement path: it makes no completeness claim and it materializes with a bare Batch.run and no reason ordering, because a sample has no scope to order by. Decide your ordering and write down why. materialize_in_reason_order/3 shows the pattern and the reason it exists.
Resumability is required and must be proven by interrupting a real run and restarting it — not argued. A 15-minute job that cannot resume is a job you run twice.
W1's budget for the remaining ~1,370,709 records: ~15 min, +3.09 GB, 1,601 rec/s after a constant 19.4 s dump walk. If you land wildly outside that, stop and report — W1's budget being wrong is itself a finding.
A3 — Materialize, resolve, re-score
Resolve (mix dd.resolve --source wiktionary): W1 measured 14.3 s per 288 k relations; expect ~2.91 M relations, ~2.4 min, +1.25 GB. Projected DB ~9.5 GB.
The trap most likely to bite you. X2's warm-cache page p95 of 66–72 ms was measured on a corpus where most words had no senses. That stops being true in this session. Re-measure /define/cat, /define/run, /define/set and the worst hub you can find. If the budget breaks, fix the query — do not move the bar.
The second trap. origin_key is varchar(255) and W1 bounded the four builders that overflow it. W1 did not sweep the rest of the write path. If you hit ERROR 22001 string_data_right_truncation anywhere, you have found a fifth site; fix it the same way (Materializer.bounded_key/1) and say so.
Exit: coverage measured and posted; mix dd.score green on all three populations with X3 at 4/4; page budget held or a new bar argued for with numbers.
Session B — The reader
Goal: the pages a stranger lands on are navigable and finishable. Branch impl-b-reader. Expect 2 PRs. Session B does not depend on session A's data and may start in parallel — it touches lib/devils_dictionary_web/ and entity_page.ex, which session A does not.
B1 — The person page
/entities/1/ambrose-bierce currently renders 16,908 px — 21 phone screens: 50 complete Bierce definitions with their full verse in arbitrary descending order, then the same 50 repeated as raw DECIDE — authored by → this rows.
@section_cap 50 at lib/devils_dictionary/encyclopedia/entity_page.ex:61. Bierce has 998 authored_by claims, so 948 are unreachable. Worse, line 82 takes the first 50 before works and definitions are split, so a high-degree person can lose their books entirely.
Claims already has the whole cursor API — incoming/2 and outgoing/2 accept :after, and next_cursor/1 exists (lib/devils_dictionary/claims.ex:263–299). EntityLive simply never calls it. This is wiring, not new machinery.
- Fetch and page each role section independently so authored definitions cannot crowd out works.
- Collapse a definition to headword + first line; the full text already lives on the word page.
- Drop the duplicate raw connections list.
- Apply visibility before pagination, and keep the order stable — no missing or duplicated rows across pages for a fixed dataset.
Test a high-degree entity in both traversal directions.
B2 — Auth pages
/users/log-in and /users/register are unstyled phx.gen.auth output — grep -c 'Kit\.' lib/devils_dictionary_web/live/user_live/login.ex returns 0 — and the layout is visibly broken, with the header pushed to the bottom of the viewport. Restyle onto the Oatmeal kit (DevilsDictionaryWeb.Kit). Seed an account so /connect is reachable.
B3 — Cross-kind discovery
Search "Ambrose Bierce" today and you get five word entries and no person. /entities/1/ambrose-bierce exists; nothing in the UI reaches it. HomeLive calls Lexicon.search/2, which is lexemes only (home_live.ex:55).
entities.preferred_label already has a trigram index (entities_label_trgm_index), so this is a second query, not a schema change.
- Return entities alongside lexemes with an explicit type label (Person, Work, Place …) and canonical links.
- Never merge identities on a name or slug match. The word Ambrose Bierce and the person Ambrose Bierce are two results, deliberately distinct.
- Establish an evidenced lexical-sense → person link rather than hardcoding Bierce. Prove it generalises by repeating with a second, non-seeded person.
- While here: the lexeme
Ursus(the bear genus) carries alexeme_entity_candidate → Ambrose Bierceat confidence 0.4. The linker emits cross-kind nonsense. Fix the class, not the row.
Rules for both sessions
Find your own targets. Do not trust a hand-written list — including one in this issue. The last session's brief enumerated 20 sites and the real number was 22. Derive the list:
grep -rn 'pattern' lib/ # then classify every hit before changing it
mix precommit # must stay greenVerify in the browser, not only in tests. A URL test is not a click test. Drive the real app on port 4007 at 1280 px and 375 px. mix dd.score must stay green on all three populations.
Compare regressions against the committed evidence in docs/, never against main. main already contains the change you might be regressing. This is exactly how W1's X3 break was reported as pre-existing and nearly shipped.
Report what you measured, not what you hoped. If something is worse, say so with the number. A finding is a successful session outcome.
Report an overclaim you made earlier in the same session. The N1 session downgraded its own dd.health bug description once it read the rendered line; that is the standard.
Done means
- Coverage ≥ 90 % of 1,541,668 indexed words have at least one definition, measured and posted
-
mix dd.scoregreen on animals, culture and emotions — X3 at 4/4 - A regression test that fails when an inflected row is enriched
- A person page finishable on a phone, with every one of Bierce's 998 definitions reachable
- Searching a person's name returns the person and the word as distinct, labelled results
-
/users/log-instyled, an account seeded,/connectreachable -
mix precommitgreen; browser evidence at both widths retained - Page p95 re-measured at dense-sense scale, budget held or renegotiated with numbers
One open item inherited from N1: dd.scope.categories' denylist now reads scope.rules["wiktionary_category_denylist"] from priv/scopes/animals.json, but the dev DB's scopes.rules has no such key until mix dd.scope.new animals runs. Whoever is comfortable writing to the dev DB should run it; until then that config is code-only.
Activity
holden commentedon Sep 11, 2026
Owner-requested clarification: usable reader first; original MVP still includes participation
Keep #82 as the immediate execution plan, superseding #79's old session sequence. This comment clarifies its acceptance boundaries and the next release steps; it does not expand Sessions A/B into all of #78.
What completing #82 actually delivers
A useful, subject-independent reader milestone: broad definition coverage, correct form lookup, readable/paginated person pages, cross-kind search, evidenced identity links and usable account screens. It does not by itself complete the original layered/participatory encyclopedia in #64/#66/#67.
The original MVP combines:
Words, people, works, editions and examples remain distinct identities linked through evidence. Public contributions must not rewrite editorial/source definitions. No subject, including Animals, is privileged in product navigation or identity eligibility.
Immediate work: retain Session A → Session B
Two important corrections to the release claims
1. ≥90% definition coverage is not “no empty pages.” Keep the measured coverage target. For the remainder, provide an honest, useful missing-content state with recovery navigation and appropriate source links; do not fabricate a definition or silently label an index-only entry as complete. Measure coverage from actual definition/sense content, not an enrichment timestamp alone.
2. Account pages and
/connectbeing reachable are not authorization to open contributions publicly. The outstanding visibility, merge/split and stale-review defects remain correctness blockers for a public contribution release. Internal testing may proceed, but publicly writable entry points must be gated server-side until those guarantees hold. Merely hiding a button is not gating. Read-only review defects also remain known limitations; do not describe this milestone as fully satisfying #73/#74.Do not assume “zero current users” is a durable safety boundary once registration becomes usable.
What follows #82 to complete the original MVP
Release step C — correctness (from #78). Fix and independently reproduce public visibility across specialized sections, identity merge/split/context reconciliation, and stale review/evidence presentation. Preserve the already-landed run-ownership fix and recheck it through full-import→resolve→reconcile. Use the audit cases as regressions, not as permanently excluded known failures. Final health and UI evidence must agree.
Release step D — one complete cultural-example workflow (from #78/#67). Register → choose an exact meaning → create/submit a URL or text/media-reference example → provide rationale and inspectable source evidence → review → display in the mixed cultural-example wall → filter by type and vote on relevance. Distinguish editorial interpretation from public examples and preserve attribution/review status. Start with reliable link/text cards and existing media references; elaborate platform-specific unfurling and integrations can follow. Include a minimal correction/challenge path so misleading examples are actionable.
Release step E — representative general entities. Demonstrate a small non-seeded set of people, a work, an organization and an event imported/created through reusable mechanisms. Verify identity continuity, ambiguity and connected navigation. We do not need all of Wikidata to launch; two seeded authors are not sufficient evidence of general support. Coordinate this work with #72/#78 rather than starting another schema redesign.
These are a bounded follow-through on existing specifications, not another planning layer. Full Artsy/ConceptNet production imports, every social feature, historical curator bots and exhaustive ontology coverage remain separate unless explicitly brought into release scope.
Implementation ownership and independent audit
Recommended implementation model: GPT-5.6 Sol, high reasoning, in a separate task. This is a task-specific recommendation: the work combines long import runs, SQL/identity correctness and Phoenix UI integration, while the execution plan is already concrete. Use one implementation owner initially; avoid concurrent edits or corpus writers that invalidate measurements. Sessions A/B are checkpoints, not guaranteed time estimates.
Keep the current conversation as the independent audit task. Return reviewable PRs and evidence at:
Each handoff must identify the commit, changed files, tests, measured results, reused evidence and known gaps. Keep PRs small enough for actual review; confirm review occurred rather than inferring it from PR creation. Do not merge/close parent issues without the agreed audit and authorization.
Completion wording: closing #82 means the reader milestone is delivered. Claiming the original MVP is usable in full additionally requires the bounded correctness and cultural-contribution release above.
holden commentedon Sep 11, 2026
Independent audit: exact completion patch required before closing #82
Keep this issue open. Implement the following as the final #82 patch, then return it for independent audit. The audited implementation is
b04cf52oncodex/issue-82-reader-milestone. Grade: B. The target below is an A+ reader milestone, not a claim that the original participatory encyclopedia is complete. Do not create a replacement issue for these unfinished acceptance criteria.Starting evidence and constraints
The independent audit verified:
mix precommit: 766 tests, zero failures.The audit files are currently untracked local artifacts, not yet accessible through a GitHub commit, at
docs/audits/2026-09-11-issue82/in/Users/holden/Code/projects-2026/dictionary. ReadREADME.md,reader_regressions_test.exs,corpus.sql, and the retained logs. Preserve relevant evidence in the delivery; do not assume another worktree contains these files. This comment includes the essential reproductions if the local files are unavailable.Use the current implementation as the starting point; first check the branch/HEAD and existing changes. One implementation owner, no concurrent corpus writers. No reset, no writes to
devils_dictionary_dev, no legacy-data migration, and no schema redesign. Work againstdevils_dictionary_v2. Do not repeat the full import simply to finish UI work. Use a bounded source replay/link pass when needed, and document any broader replay justified by an actual importer change.1. Restore relationship navigation in both directions — required
Cause:
EntityPage.other_connections/3inlib/devils_dictionary/encyclopedia/entity_page.exexcludesabout,authored_by,edition_of, andpublished_infrom both incoming and outgoing connections. The specialized role sections replace incoming relationships, but do not replace the outgoing author/parent-work links.Reproduction: create a person, a work and an edition through
Registry; assertwork —authored_by→ personandedition —edition_of→ workthroughClaims.assert/3. On the work LiveView, assert an anchor to the person's canonical entity route. On the edition LiveView, assert an anchor to the work's canonical route. Both currently fail. The real work page also lacks the author link.Implement either explicit outgoing role panels or filtering that suppresses only relationships actually represented elsewhere. Account for direction and endpoint role; do not simply restore a duplicate raw relationship wall. Preserve the existing public-read policy and independent cursors.
Acceptance:
2. Complete the real, evidenced lexical-sense → person connection — required
Typed search is implemented correctly, but it is not the entire B3 requirement. In the audited database:
Ambrose Bierce, object 341814.refers_toorlexeme_entity_candidateassertion targeting person 1. The inspected sense only has two outgoingotherrelations.These IDs identify the observed reproduction, not constants to put in code. Re-discover records by their durable identities when running the fix.
Inspect the stored source record, its trimmed payload, sense metadata and linker selection. Determine whether explicit identity evidence is present but lost, not selected, or genuinely absent. Preserve/use an explicit source identifier or other documented, inspectable evidence through the general adapter/linker or existing attributed assertion path. Do not manufacture evidence, infer identity merely from matching names/slugs, merge the word with the person, or add a Bierce-specific SQL insert/conditional.
The existing QID-backed
Ada Examplenon-seeded test intest/devils_dictionary/absorb/linker_test.exsis useful; retain it. Extend tests at the layer actually responsible for the missing real link. If a direct linker fixture passes while ingestion drops the needed metadata, add an adapter-to-linker regression.Acceptance:
refers_toassertion to the separate person identity with inspectable provenance/evidence.3. Finish the compact reader presentation — required for the requested A+ polish
The mobile Bierce first page still measured 8,214 px at 375×812.
first_line/1only splits on newlines, so an entire long paragraph remains a “first line”; source Markdown markers can appear literally.Search ranking (an exact person currently follows ten lexical results) is optional polish, not a reason to expand this patch into a search redesign.
4. Verification and delivery — required
mix precommit; report actual commands, counts and commit SHA. Do not leave the new reader regressions outside the normal suite.mix dd.score --scope animals,--scope emotions, and--scope cultureagainst the final code and record outputs. X3 must remain 4/4; do not weaken budgets./connect, while anonymous and ordinary registered accounts cannot submit through it. Keep the server-side contribution gate; do not publish credentials.docs/rebuild/issue-82-reader-milestone-2026-09-11.mdwith final evidence and corrected counts. Distinguish fresh measurements from reused import/restart/replay evidence.Explicitly deferred: do not expand this patch
The old #73 audit was rerun: 7 cases, 5 failures, covering stale acceptance after endpoint changes, merged-input page identity, split-context reconciliation, rejected
definesleakage, and the:mergedversus:mergeresolver enum error. These remain Release C/#78 work. Keep them visible; do not fix them opportunistically here or describe the original MVP as finished.After #82 passes, proceed to that correctness release, then the complete cultural-example contribution workflow, then representative general entities. No new architecture-planning cycle is needed.
holden commentedon Sep 11, 2026
Reader milestone merged; owner-requested close-out
PR #83 is merged into
mainatab0cc0a34fd3fd4d655d4f2c405010f84ab58233. The final branch commit was7e172802b5f58edf31b23e6f1c967c5de5304efe; all local review changes and audit evidence were committed and pushed before merging.Final validation:
mix precommit774 tests, zero failures; actual active/non-retired sense coverage 99.50%; isolated warm-cache dense-page p95 36–73 ms, below the 150 ms budget. Prior independent browser/pagination and full scorecard evidence is retained with its original revision, rather than relabelled as newly run.Closing #82 as the reader milestone, at the owner's request. This is not a claim that the original participatory MVP or every foundation guarantee is finished. In particular:
The final review patch addresses the posted review findings. CodeRabbit's earlier review completed; the new-head rerun was still pending when the owner-authorized merge proceeded. Local verification is recorded separately from external review status.
The complete inventory of the remaining open issues, stale/superseded roadmaps, code evidence and proposed build order is committed here:
https://github.com/razrfly/dictionary/blob/ab0cc0a34fd3fd4d655d4f2c405010f84ab58233/docs/audits/2026-09-11-mvp-closeout/README.md
Only this issue is being closed by this action. #73/#74/#78 and other unfinished specifications are not implicitly complete.