Repository navigation
Description
Purpose
The execution tracker for MVP-0. The spec is #69 (schema, sources, scope, scorecard). This issue answers the practical questions: where to build, how many sessions, what each session must produce, and how the code is laid out. Tick boxes here as sessions land; paste the numbers each session produced as a comment.
Session protocol. Start a session by reading #69 and this issue. End it by ticking its boxes, posting its numbers, and committing with the session prefix (S1: …). A session that runs long splits at the marked seam, never in the middle of a source. A session may commit straight to main or on a branch s<N>-…; after the audit the branch is fast-forwarded into main and pushed. Edit issue bodies from the live version, never from a local copy (a stale copy clobbered the S0b tick once).
Practical answers
Same directory, same repo, reset in place. Nothing is lost: the old Rails app is already on origin as the annotated tag ruby-rails-archive → 36d3b153 ("fixes for lyrics", 2025-02-07), and the current Phoenix skeleton gets its own tag before the wipe. A second directory would split one repo across two checkouts and lose this project's Claude memory (keyed by path). Rename the folder only when the product name is decided.
Not one session. Six sessions minimum, eight likely. Reasons: the two dumps take real time to download and stream; the link rate must be measured before pages are worth building; and every session ends with numbers, which is the whole point of the scorecard. Two sessions (S2 and S4) have a marked seam and may become two each.
Environment as found on 2026-09-05.
| Thing | Found | Note |
|---|---|---|
| Elixir / OTP | 1.19.0 on OTP 28.1, pinned in .tool-versions (mise had them installed; 1.18.3/OTP 27 was the global default) |
mise picks the pin up automatically in this directory |
| Phoenix generator | phx_new 1.8.13 (upgraded from 1.8.5; latest on Hex) → Phoenix 1.8.13, LiveView 1.2.11 |
done: commit 58fdb81 |
| Postgres | 18.2 via Postgres.app, listening on 5432 | postgres / postgres, matches the default dev config |
| Disk | 22 GB free | enough: stream the Wiktionary dump, never decompress it to disk (23 GB) |
| History | ruby-rails-archive tag on origin; local object store has the Rails commit |
nothing to rescue |
| Tracked but not wanted in the reset | assets/vendor/oatmeal (415 files) |
a Tailwind Plus template committed to a public repo; it stays reachable at the skeleton tag, and comes back in the design pass from tmp/ (ignored) rather than from git |
Pre-download before S0 (background, into data/, which the new .gitignore will exclude):
data/english-wordnet-2025-json.zip 10 MB https://github.com/globalwordnet/english-wordnet/releases/tag/2025-edition
data/raw-wiktextract-data.jsonl.gz 2.6 GB https://kaikki.org/dictionary/rawdata.html (record the dump + extraction dates)
priv/sources/bierce/pg972.txt 394 KB https://www.gutenberg.org/cache/epub/972/pg972.txt (this one is committed)
Reset procedure (S0a — done 2026-09-05 as commit 58fdb81)
cd /Users/holden/Code/projects-2026/dictionary
# 1. protect the current skeleton (the Rails app is already tagged: ruby-rails-archive)
git tag phoenix-skeleton 7c6dd33 && git push origin phoenix-skeleton
# 2. generate a fresh app next to the repo (module/app names stay; decision #1 in #69)
cd .. && mix phx.new devils_dictionary_fresh --app devils_dictionary --module DevilsDictionary --no-mailer --no-install
# 3. wipe the working tree except README.md, docs/, and the ignored dirs; copy the fresh app in
cd dictionary
git rm -r -q --cached .
find . -mindepth 1 -maxdepth 1 ! -name .git ! -name README.md ! -name docs ! -name tmp ! -name _build ! -name deps -exec rm -rf {} +
rsync -a --exclude .git --exclude README.md ../devils_dictionary_fresh/ ./ && rm -rf ../devils_dictionary_fresh _build deps
# NOTE: phx.new 1.8.13 runs `git init` in the new directory; without --exclude .git the rsync merges its empty
# .git over yours and wipes the remote from .git/config (objects and refs survive). Repair: git remote add origin …
printf '\n# local data dumps\n/data/\n' >> .gitignore
# 4. deps and smoke test
mix deps.get && mix ecto.create && mix compile && mix test
# add to mix.exs: {:oban, "~> 2.19"}, {:req, "~> 0.5"}, {:floki, "~> 0.36"}, {:cachex, "~> 4.0"}, {:slugify, "~> 1.3"}, {:credo, only: [:dev, :test]}
# 5. commit the reset before writing any domain code
git add -A && git commit -m "S0a: reset to a fresh Phoenix 1.8 app for MVP-0 (#69, #70)"
# also: printf 'erlang 28.1\nelixir 1.19.0-otp-28\n' > .tool-versions ; append /tmp/ and .DS_Store to .gitignore (the old tree kept the Oatmeal template in tmp/)mix phx.new regenerates AGENTS.md (ours is the stock file, so nothing is lost). README.md is the living record and is kept as is.
What S0a left in the tree (commit 58fdb81): the fresh app; .tool-versions; deps oban 2.24, req 0.7, floki 0.38, cachex 4.1, slugify 1.3, credo; DevilsDictionary.Absorb.Source (the behaviour from #69 §5, incl. trim/1); moduledoc-only context modules Absorb, Lexicon, Encyclopedia, Sources, Health; the directories below with .gitkeep; priv/sources/bierce/pg972.txt; a local data/README.md (ignored). mix test: 5 tests, 0 failures. Before the first mix ecto.create in S0b, run mix ecto.drop: the skeleton's dev database still exists with the old tables.
Directory structure (what each session creates)
lib/devils_dictionary/
├── absorb/ S0 the import layer
│ ├── source.ex S0 behaviour: slug/0 absorb/2 enrich/2 materialize/1 trim/1 rate_limit_ms/0
│ ├── materializer.ex S0 writes materialize/1 output + materialized_at in one transaction; idempotent upserts
│ ├── scope_builder.ex S0 rules → scope_lexemes with reasons
│ ├── resolver.ex S1 to_lemma → to_lexeme_id; canonical variants
│ ├── linker.ex S2 the ladder → concept_links (method, confidence, status)
│ ├── clients/ S2 Req clients: wikidata.ex, wikipedia.ex (user-agent, rate limit, absent markers)
│ └── sources/
│ ├── wordnet.ex S0
│ ├── wiktionary.ex S0 index pass · S1 scoped materialize + trim
│ ├── wikidata.ex S2
│ ├── wikipedia.ex S2 incl. disambiguation candidates
│ ├── bierce.ex S3
│ └── johnson_1755.ex S5 the extensibility proof
├── lexicon/ S0 context + schemas: lexeme.ex sense.ex entry.ex lexical_relation.ex scope.ex scope_lexeme.ex
├── encyclopedia/ S2 context + schemas: concept.ex concept_relation.ex concept_link.ex
├── sources/ S0 context + schemas: source.ex source_record.ex person.ex import_run.ex
├── health/ S3 coverage.ex parity.ex score.ex (the scorecard rows as functions)
└── workers/ S2 absorb_worker.ex enrich_worker.ex (thin Oban wrappers)
lib/devils_dictionary_web/
├── live/ S4 search_live.ex word_live.ex scope_live.ex source_live.ex admin/imports_live.ex
└── components/ S4 source_card.ex (tier-tinted, ↗ + ⓘ) graph_panel.ex provenance_drawer.ex coverage_badges.ex
lib/mix/tasks/ S0…S3 dd.absorb.ex dd.scope.build.ex dd.resolve.ex dd.link.ex dd.materialize.ex dd.health.ex dd.score.ex
priv/repo/migrations/…_create_lexicon_schema.exs S0 the single baseline migration (#69 §4)
priv/repo/seeds.exs S0 the five sources rows (+ community/bot rows are NOT seeded yet)
priv/sources/bierce/pg972.txt S3
test/support/fixtures/{wordnet,wiktionary,wikidata,wikipedia,bierce}/{cat,dog,oyster}.json S0…S3 real raw records, trimmed
test/devils_dictionary/absorb/ S0… materialize/1 and trim/1 as pure functions; resolver; linker
test/devils_dictionary_web/live/ S4
data/ ignored: the dumps
Contexts: Absorb (import), Lexicon (words), Encyclopedia (things), Sources (provenance), Health (numbers). Nothing in lib/devils_dictionary_web touches source_records.raw except the provenance drawer.
Sessions
S0 — Reset & measure
- Goal: a fresh app with the schema, the behaviour, the first two sources absorbed, and the first real numbers.
- Inputs: the pre-downloaded dumps; 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §4.
- Work: reset procedure above → baseline migration →
Sources/Lexiconschemas and contexts →Absorb.Source+Materializer(tested on fixtures) →sourcesseed →Absorb.Sources.Wordnet(full) →Absorb.Sources.Wiktionaryindex pass (bare lexemes + forms, streamed from the gz) →ScopeBuilderwith the three rules → mix tasksdd.absorb,dd.scope.build. - Exit: A2, A3, A4 measured.
mix testgreen offline. - Numbers to post: synsets, lemmas, index size (and how many are form-of entries), scope size by reason, time for each absorb.
- Commit:
S0: reset, schema, WordNet + Wiktionary index, animals scope (#70) - Seam: if the index pass is slow, stop after WordNet + schema and finish the index as S0b.
S1 — Wiktionary and WordNet, scoped
- Goal: the dictionary side complete for Animals.
- Work: Wiktionary scoped absorb with
trim/1→ senses, pronunciations, etymology, forms, relations withto_lemma→Resolverincl.canonical_lexeme_id→dd.resolve,dd.materialize --dry-runparity → source records carryurl. - Exit: A5, A9, M1, M4 (measured on the real scoped records, not only the fixtures), R2, X3. R1 is already green from S0.
- Numbers: Wiktionary coverage of scope, unresolved targets by type, trim savings, records per second.
- Commit:
S1: Wiktionary scoped, relations resolved, parity (#70) - Notes from the S0 audit (2026-09-05):
- The scope is 21,277 lexemes, kept whole by decision. Absorb the scoped Wiktionary records in reason order:
wordnet_closure(7,692, incl. the 2,027 overlap) first, then the 13,585 category-only tail. The flagship wordscat/dog/oysterare in the first group. - The index pass already wrote
forms,metadata.wikt_categories,origin_source_idandsource_idsfor every English entry; the scoped pass fills senses, pronunciations, etymology, relations andenriched_at, and stores the trimmed raw (trim/1drops translations, descendants, templates; 81 % smaller on the fixtures). Materializermerges lexemes rather than clobbering them (forms fill empty slots, metadata merges, origin stays); keep that contract when Wiktionary and WordNet both touch a word.- Wiktionary relations arrive with
to_lemmaonly;dd.resolvefillsto_lexeme_idbylower(to_lemma)+ pos, andcanonical_lexeme_idfromalt_of/form_ofwhere the target exists. WordNet rows are already resolved, leave them alone. import_runsrows 1–2 are failed dev retries of the WordNet absorb;dd.scoreA1 must countdoneruns only.
- The scope is 21,277 lexemes, kept whole by decision. Absorb the scoped Wiktionary records in reason order:
S1b — Hash before trim, form-of flag (done 2026-09-05, 5b9ee82 + b356876)
- Goal: close the two design nits from the S1 audit before S2 builds two more sources on the same record contract.
- Work:
Wiktionary.scoped_row/1takescontent_hashon the payload as fetched, beforetrim/1;Sources.insert_records/3accepts the hash and only computes it fromrawfor sources whose trim is the identity; regression tests (SourcesTest,WiktionaryTest). The index pass storesmetadata.form_ofon inflected-form entries;Lexicon.lookup/1lets a bare headword keep its page only on an exact-case match and offers forms claimants under:also; a bare form-of row, or a bare row in another casing, yields to the word that lists the string among its forms. - Data: index re-run (1m 23s, 533,218 rows flagged), scope re-absorb (42.5 s), one-time reset of the 19,250 false
changed_atstamps left by S1's trim change → 0. - Exit: Health rows unchanged — A5 17,773 / 21,277 raw (92.2 % amended), A9 100 %, M1 0 gaps, M4 67.7 %, R2 86.4 %.
cats→ cat,CATS→ the musical with cat underalso,monkeys→ monkey,spatkeeps its page. - Lesson: the first S1b commit's message described the exact-case rule before the edit had landed (a scripted edit aborted on a doc anchor and the shell carried on);
b356876makes it true. Scripted edits must gate the commit on their own exit status.
S2 — Things (may split at the seam)
- Goal: the encyclopedia side and the word ↔ thing bridge.
- Work (S2a):
Encyclopediaschemas → Req clients with rate limits and absent markers →Absorb.Sources.Wikidata(concepts, images, taxonomy,taxon_concept_id) →Absorb.Sources.Wikipedia(summaries, thumbnails, title probes, disambiguation candidates) →EnrichWorker, snooze on 429. - Work (S2b):
Linkerladder →dd.linkwith histogram, conflicts, candidates. - Exit: A6, A7, A10, L1, L2, L3, L4.
- Numbers: link rate, disambiguation hits, taxon reach, API calls made and cached, on-demand wall clock (O2b).
- Commit:
S2: Wikidata + Wikipedia, linking ladder (#70) - Seam: between S2a and S2b.
- Notes from the S0 audit (2026-09-05):
- Add the
wordnet_wikidatarung (🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §5, 0.90): 22,036 WordNet synsets carry awikidataQID insenses.metadata. Add:wordnet_wikidatatoConceptLink's method enum (string column, no migration). - Enrich in reason order (WordNet-closure lexemes first). At 5 req/s the whole scope is ≈ 2 h per API; O2 now budgets that separately (O2b, report), so do not shrink the scope to fit a clock.
- After concepts exist, run
mix dd.scope.build animalswithout--reset: reasons union, so thewikidata_taxonrule only adds. - Oban arrives here (queues
absorb: 1,enrich: 3,link: 2,maintenance: 1;AbsorbWorker,EnrichWorker; snooze on 429). S0/S1 tasks are synchronous by design.
- Add the
- Notes from the S1 audit (2026-09-05, merged as
dc6c326):- Only 752 of 31,116 Wiktionary senses carry a QID, so the
wiktionary_qidrung is sparse for Animals. Expectwordnet_wikidata(22,036 synsets) andtitle_matchto carry L1; report links per rung. - The 1,995 Linnaean binomials A5 now excludes are yours. Wikidata taxon items carry them as
P225; report scope coverage forwikidatanext to Wiktionary's (A6 now asks for the union). - The index pass already stores
metadata.wikt_categories;Lexicon.lookup/1,Absorb.Batch(keyset loop honouring the needs-materialization predicate,--allfor M2) andSources.insert_records/3(maintainschanged_atby hash) exist. Use them; do not add a second record-insert path. content_hashis over the trimmed payload, so a trim change looks like a content change (🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §10). Do not build the feeds onchanged_atuntil that is fixed.- Wikipedia and Wikidata clients go under
absorb/clients/; every fetch is anenrich/2that lands asource_recordsrow withurl, thenMaterializer.run/2.
- Only 752 of 31,116 Wiktionary senses carry a QID, so the
S3 — Bierce, health, score
- Goal: the layer proof and the scorecard itself.
- Work: Bierce parser incl. "See X" →
see_also→Health(coverage, parity, drift) →dd.health→dd.scoreprinting 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §7 with actuals → atomicity test (M3) → offline re-materialize (M2). - Exit: A8, M2, M3, O1, O3, O4.
- Numbers: first full
dd.scoreoutput (expect red rows for U*/X*/E*). - Commit:
S3: Bierce, health, dd.score (#70) - Notes from the S0 audit: A1 counts
doneimport runs per source (failed dev retries exist); A2's second metric is distinct lexemes (lemma, pos); O2 is two measures (dump absorbs ≤ 2 h, on-demand report). - Notes from the S2 audit (2026-09-05):
- Already done by the audit, no action: the scope had grown by 4,116 lexemes after S1's Wiktionary pass, so ~2,000 scope words had an encyclopedia side and no dictionary side. Re-ran
dd.absorb wiktionary --scope animals(+2,176 records, +3,367 senses),dd.resolve,dd.absorb wikidata --scope animals(+66),dd.absorb wikipedia --scope animals --concepts,dd.link. Rule for every future scope growth: re-run the dictionary half too. Current rows: L1 58.4 % raw / 88.5 % of reachable, any 68.2 %; A6 union 97.4 %; A7 100 %; A10 93.6 %; L2 1,180; L3 77.2 %; L4 100 %; R2 86.4 % (96,148 / 111,304); A9 100 % (291,230); M4 67.6 %; M1 0 gaps across 242,359 records. The stalerunningrun (Bump aws-sdk-s3 from 1.176.1 to 1.182.0 #33) is closed. - Fix first, small: WordNet synsets can carry an array of QIDs (
panther→["Q35255", "Q109647288"]).Wikidata.wordnet_qids/1and thewordnet_wikidatarung accept onlyjsonb_typeof = 'string', so those 265 scope references never seed a concept or a link. Normalize with the samejsonb_arrayguard rung 1 uses, thendd.absorb wikidata --scope animalsanddd.link. Wikipedia.concept_targets/2skips concepts with an entry or an unexpired absent marker but not concepts that were fetched and yielded no entry (disambiguation pages, empty extracts), so every--conceptsrun re-fetches ~3,800 pages (191 requests). Skip any concept with an existingconcept:<QID>record unless--refresh.EnrichWorkerpaces with a per-process sleep, soenrich: 3can reach 15 req/s against one API. Runenrich: 1until a shared limiter exists, or one queue per source.- The Wikidata absorb's
concept_relations_unresolvedcounts recorded-but-unchasedP279/P31targets (35,942); split it so a realP171residual (which must be 0) cannot hide behind it. dd.scoreA1 countsdoneruns only; twofailedlink runs and the closed run 33 are dev history.
- Already done by the audit, no action: the scope had grown by 4,116 lexemes after S1's Wiktionary pass, so ~2,000 scope words had an encyclopedia side and no dictionary side. Re-ran
- Bierce source, verified 2026-09-05 (Holden asked): Project Gutenberg #972 is the right and only public-domain source of the complete 1911 text (the 2000 Unabridged is a copyrighted compilation; Wikisource carries the same text per letter). Parse the HTML edition, now checked in as
priv/sources/bierce/972-h.htm, exactly as the old Rails seed did with Nokogiri (db/seeds/devils_dictionary.rbat tagruby-rails-archive); the plain textpg972.txtstays as a cross-check. Measured on the file:- parse the
<p>/<pre>nodes between the<h2>A</h2>chapter and the Gutenberg end marker; a headword paragraph matches^([A-Z][A-Z'\- ]+), ([a-z\.]+)\.\s+→ 959 entries (the text has 960:EUCHARISTsits inside a<pre>in the HTML, so a<pre>that starts with the headword pattern is an entry, not verse); - part-of-speech markers seen:
n, v, v.i, v.t, adj, adv, pron, pro, pp, p.p, n.pl, x— keep them verbatim, normalize to the lexicon's pos when creating the lexeme (x= "no definition", HASH); - 248
<pre>verse blocks and 215 short attribution paragraphs (initials and invented poets) follow their entry: keep both inentries.bodyin order (the verse is part of the joke), verse as a blockquote inbody_format: :markdown, and notemetadata.verse: true; - 118 continuation paragraphs: prose that runs on until the next headword paragraph belongs to the current entry;
- alternate headwords in one paragraph (
BABE or BABY, n.,CONFIDANT, CONFIDANTE, n.): the entry attaches to the first, the others becomealt_ofrelations pointing at it; See X(BRUTE, n. See HUSBAND.) and[from ACADEME]becomesee_also/relatedrelations withto_lemma, resolved bydd.resolve;- 18 Animals headwords (alligator, ass, camel, cat, dog, frog, goose, hog, hyena, magpie, monkey, mouse, ostrich, oyster, pig, seal, tortoise, turkey); A8 counts all 959 entries and their attachment to lexemes (lowercase the headword, create the lexeme if the index lacks it).
- parse the
Notes from the S3 audit (2026-09-05), for whoever touches the data next
- Grade A. Verified against the database and by running
mix dd.scoremyself: 24 / 24 graded rows pass, 261 tests, precommit green, pushed tomaincleanly. Bierce 997 entries with verse, attributions and continuations intact (STORYkeeps its 26 paragraphs,PIEits two-paragraph attribution,EUCHARISTits<pre>); the S2 carry-overs are fixed and thewordnet_wikidatarung moved 6,119 → 6,663; 7 scope QIDs remain conceptless and all 7 have absent records (deleted or redirected items). - One parse nit, three entries: the alternate-headword entries keep the remainder of the headword phrase at the start of their body —
BABEbegins "or BABY, n. A misshapen…",CONFIDANT"CONFIDANTE, n. One entrusted…",TZETZE FLY"TZETZE (or TSETSE) FLY, n. An African…". Strip the whole printed headword phrase, alternates included, inmaterialize/1, thenmix dd.materialize --source bierce --all(the M2 path; no re-absorb). Do it before U1 renders Bierce cards. - 125 Wikipedia records now carry
changed_at, stamped during the full lemma re-probe: real page changes since S2's probe, which is the feature working. Leave them. - Cost note, not a defect: the concept pass now fetches a summary for every concept with a sitelink, including ~28,000 disambiguation candidates that will mostly never be asserted (22 min, 1,389 requests, and A7's denominator is 69,681). For the next scope consider limiting the concept pass to asserted concepts plus candidates promoted to 0.6, and reporting A7 on that population.
enrich: 1stays until a shared limiter exists.dd.scoreA1 countsdoneruns only; runs 56, 65, 66 are dev iterations.- U0 (🖼️ MVP-0 UI: the minimum word-hopping interface on the Oatmeal theme — theme boilerplate first, use case, wireframes, fake-data mode #71) can start now: Bierce cards exist for
cat,dog,oyster, and the data every page needs is materialized.
S4 — Pages (may split at the seam)
Split (2026-09-05): the reader-facing pages (home/search, word page, hop, provenance, sparse states, fake-data mode) are specified and built on the Oatmeal theme by #71 (sessions U0–U3); S4b keeps the developer surfaces (scope browse, source page, import dashboard, health), plain. U0 (theme boilerplate) can run right after S3.
-
Goal: six plain, structurally complete pages.
-
Work (S4a): search (incl. forms) → word page: source cards by tier with ↗ and ⓘ, graph panel, image, "may refer to", disagreement plaque → provenance drawer.
-
Work (S4b): scope browse with tree, badges, filters → source page → import dashboard → mobile check → LiveView tests.
-
Exit: X1, X2, U1–U6, O2.
-
Numbers: search p95, full-absorb wall clock, pages tested.
-
Commit:
S4: pages (#70) -
Seam: between S4a and S4b.
-
Notes from the S0 audit: WordNet relations are per sense and expanded per member pair (≈ 4 rows per synset edge). The graph panel must group by
from_sense_id/to_group_key, otherwisecatshows tracked vehicle and whip among its hypernyms (they belong to other senses). Synonyms come fromsenses.group_key, not from relation rows. -
Notes from the S4 audit (2026-09-06, U0 + S4b +
890a834, grade B+): every page driven in Chrome at 1280 px and 375 px — theme, fonts and dark mode correct, no console errors, no sideways scroll, Commons thumbnails load fromupload.wikimedia.org,?q=,?has=,?missing=,?state=,?taxon=and paging all correct from the URL, the five source pages, imports and the scorecard all agree with the database; 322 tests,mix precommitgreen,dd.score26 / 26 graded rows. Two defects a click finds and a URL does not, both fixed in S4c below: the has/missing chips never apply, and the health page abandons its own mount. Not counted against the session: thewikidata 0strip (a data-model truth shown badly) andpage=falsein taxon links.
S4c — Fixes from the S4 audit ✅ (900ea5d, 2026-09-06, done in the audit session)
- Chips: LiveView's
extractMetasetsmeta.value = el.valueafter readingphx-value-*, so on a<button>aphx-value-valueis replaced by the button's empty.valueandtogglereceivesvalue: "". Rename the key (phx-value-slug,%{"kind" => kind, "slug" => slug}), never usevalueas a phx-value key again, and add a LiveView test that clicks#filter-has-bierceand asserts the 66-row count (thestatechips work because their key isstate). - Health page:
Score.rows/1takes 4.5 s and runs inmount/3— on the dead render, again on the connected mount, and on every reconnect.longPollFallbackMs: 2500abandons the websocket mid-mount (thePostgrex … client exitedlines), so every visit costs three scorecards and connects over long-poll; two overlapping visits loop on "Something went wrong! Attempting to reconnect". Compute the scorecard inassign_async(or only whenconnected?/1), cache it in Cachex for ten minutes, keep the dead render under 200 ms. Rule from now on: nothing slower than the long-poll fallback runs inmount/3. - Wikidata is not a word badge:
lexemes.source_idsnever contains Wikidata, because it attests concepts, not words — 17,373 scope words are linked to a Wikidata concept throughconcept_links. Replace the📚 wikidata 0line with linked · 17,373 read fromconcept_links(status ≠ rejected), drop the two dead chips, and let the word page (U1) show Wikidata through the concept card rather than a source badge. page=false:to_params/1emitsfalsefor page 1 and~pserialises it into the taxon links; reject falsy values there too.- A5 v2 (🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 v12): the exclusion widens from the binomial regex to every scientific name, any rank — the lemma equals a linked concept's
taxon.scientific_nameor matches the regex — with the bar raised to 90 %. Wiktionary files genus, family and order names (Archilochus, Paguridae, Therapsida) as Translingual exactly as it files binomials; today's 85.0 % sits on the bar only because those are counted as misses. - Exit: the two LiveView tests above;
/healthconnects over websocket in under a second on a warm cache;dd.scoreA5 v2 reported;.claude/launch.json(committed in the audit) still starts the server for the next audit. - Done (verified in Chrome on the socket): the chip click now pushes
toggle {kind: "has", slug: "bierce"}and the page reads 66 words with bierce;/healthdead-renders in 3 ms, joins over websocket, the scorecard arrives async (4.5 s cold, cached ten minutes, recompute button), the detail underneath still async; A5 v2 reads 78.3 % raw · 93.0 % excl. 4,004 scientific names (bar 90 %); the browse strip reads linked to 📚 wikidata 16,014 from the rows' ownconcept_linksrule (auto/confirmed, ≥ 0.7) and the Wikidata chips are gone; taxon and pager links carry nopage=falseor empty keys;/sources/wiktionary391 ms. One self-inflicted regression caught on the way: the scientific-name set was computed once per miss (an 81 s source page) — now once per call. 322 tests,mix precommitgreen. Cache is off in test (cache_scorecard: false).
S5 — Prove extensibility, close
- Goal: show that the next source, the next scope and the community layer cost what 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 says they cost.
- Work: Johnson 1755 from the LEME TEI-XML as the sixth source →
emotionstoy scope from one WordNet root → write theexamples+votesmigration from 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §4's sketch and confirm it touches no existing table, then delete it → finalmix dd.score. - Exit: E1, E2, E3; every scorecard row green.
- Numbers: the full scorecard, pasted as the closing comment on 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69.
- Decisions from S5 planning (2026-09-06): (1) Johnson's LEME TEI-XML is committed gzipped at
priv/sources/johnson/johnson-1755-leme.xml.gz(~8.8 MB in git, CC BY 4.0 with attribution on thesourcesrow, pinned by edition and a sha256 inconfig), gunzipped in memory at absorb — the Bierce rule generalised: a static source is a committed file when it is under ~10 MB compressed; larger dumps live indata/with a pinned checksum. (2) Make the entry points generic first, as S5a (about half a session):health_live.exreadsSources.list_sources/0,dd.fixtures.capturedispatches through the source registry, amix dd.scope.new <slug>task creates a scope row frompriv/scopes/<slug>.json, and L3 keys on the scope's root concept (reported n/a when the scope has none). Then E1 is scored as onesourcesrow, one module, one registry line, fixtures and tests, zero migrations, and E2 as zero.ex/.exstouches — theemotionsscope arrives as a JSON file plus task runs, using the existingwordnet_closureandwiktionary_categoryrules (awikidata_classrule over P279 is a separate, counted item if wanted). Any touch outside those lists is reported on the row, not hidden. - Notes from the S5 audit (2026-09-06,
9db7805+270e48a, grade A−): every number re-measured — 356 tests,mix precommitgreen,dd.score --scope animalswith parity 29 / 29,schema_migrationsstill 2, six sources and two scopes in the database, Johnson 42,726 records / 42,726 entries / 6,780 introduced / 1,313 in Animals / 435 in Emotions,emotions809 lexemes with 0 unreasoned, the sketch atdocs/sketches/;/s/emotions,/sources/johnson,/health(six parity rows, E rows, websocket) and/admin/importsdriven in Chrome with no console errors. E1's touch list is honest but not the whole cost: Johnson also forced two shared-code fixes (resolve_canonical/3no longer lets a bare form-of row veto, and a general snapshot pin), both improvements, both migration-free. E2 is the proof Holden asked for: a scope with no taxonomy runs the same pipeline unchanged — and it shows the measures are Animals-calibrated (dd.score --scope emotionsfails A4's ≥ 7,500, A10's ≥ 80 % images at 61.8 %, L1's ≥ 70 % at 49.2 % of reachable, and L3 at 0 % because it asks for Animalia). Deductions: the unscoped concept seed was found and reported but left, at 16 minutes of API and ~20 k concepts nobody links to; the bars and L3 were not generalised when the second scope made the problem visible;sources.idran to 21 for six rows during the rollback attempt. All three are S5c below.
S5c — Fixes from the S5 audit ✅ done 2026-09-07, with #71's U1b in commit 30a1e8c
Measured after: seed_qids for emotions 414 (was 72,108) fetching 345 records in 15 s (was 28,084 in 16 min); animals 11,660 (was 92,947). dd.score --scope emotions 33 / 33 graded rows pass (was 30 / 33), --scope animals unchanged at 34 / 34. No migration. Orphans left alone as this list asks; the seed fix stops the growth. Full write-up: #71 §8b.5.
-
Scope the concept seed.
Wikidata.seed_qids/2unionsconcept_qids/0=Repo.all(from c in Concept, select: c.qid), the whole table, so scope N re-walks every concept scopes 1..N−1 introduced (emotions: 72,108 seeds, 28,084 records, 569 requests, 16 min for 809 words). Seed only the concepts this scope's lexemes link to atauto/confirmedplus candidates promoted to ≥ 0.6, and stop chasing 0.40 disambiguation candidates until the linker promotes one. Validate without fetching: compareseed_qids/2sizes foranimalsandemotionsbefore and after. -
Orphans. 35,719 concepts have no link and no relation (92,947 total). Do not delete derived rows alone — M1 would report them as gaps; trim the
source_recordsand the rows together, or leave them and only stop growing. -
Bars are data. A4's ≥ 7,500, A10's ≥ 80 %, L1's ≥ 70 % are Animals' numbers; read them from
scope.rules["bars"]with today's values as defaults, and make L3 report n/a when the scope has nowikidata_rootinstead of failing at 0 %. -
Pages take a scope.
/sources/:slughard-wiresscope: "animals"and/healthhard-wiresscope = "animals";/admin/importsalready reads?scope=. Same parameter on all three. -
Exit:
dd.score --scope emotionsfails nothing it should not;seed_qidsforemotionsis hundreds, not tens of thousands; no migration. ✅ all three. -
Opened by the fix, for its own session — chase
P279, bounded. Re-absorbingemotionson the scoped seed gave 265 concepts their claims for the first time, and joy its card (a kind of happiness). But onlyP171andP13176are chased to closure, so asubclass_ofedge usually points at a QID with noconceptsrow: 97 of 322 emotions concepts reach one chain step, 44 reach two, and ~36k edges are countedskipped_unchased. ChasingP279for asserted concepts at a fixed depth would deepen every non-taxonomic chain on the word page. Deliberately not slipped into U1b: growing the concept graph is what S5c has just stopped doing, and this needs its own budget and its own A7 answer. -
Commit:
S5: Johnson, toy scope, scorecard green (#70)
What can run in parallel: Bierce (S3) needs only S0; the page scaffolding in S4a can start after S1 with WordNet + Wiktionary data. Everything else is sequential.
Checklist
- S0a — Reset in place, toolchain pin, deps, scaffold (commit
58fdb81, 2026-09-05) - S0b — Schema,
Absorb.Source+Materializertests, WordNet, Wiktionary index, Animals scope, first numbers (commit19bccaf, 2026-09-05 — A2/A3/A4 and R1/M4 green; audited) - S1 — Wiktionary scoped, Resolver, parity, first Health rows (commit
dc6c326, 2026-09-05 — A9/M1/M4/R2/X3 green; A5 red at 83.5 %, amendment proposed on 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69) - S1b — hash before trim,
form_offlag on index rows, headword-first lookup (commits5b9ee82+b356876, 2026-09-05 — Health rows unchanged, 123 tests, zero migrations) - S2a — Wikidata + Wikipedia (commit
4a8cfcf, 2026-09-05 — A6/A7/A10 green; 44,194 concepts, 38,206 taxonomy edges) - S2b — Linker (commit
4a8cfcf, 2026-09-05 — L2/L3/L4 green; L1 red at 57.7 %, ceiling 68.1 %, amendment proposed on 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69) - S3 — Bierce, health, score (commit
3fcd293, 2026-09-05 — A8 ✅ 997 entries, 93.0 % already in the index; M2 ✅ identical over 271,296 records; M3/O1/O3/O4 ✅ — 24 / 24 graded rows pass, 12 pending for S4/S5) - S4a — Search, word page, provenance — taken over by 🖼️ MVP-0 UI: the minimum word-hopping interface on the Oatmeal theme — theme boilerplate first, use case, wireframes, fake-data mode #71 (U1, U2)
- U0 — Oatmeal theme ported by hand, daisyUI removed,
/kitverified (commit0430e17, 2026-09-06 — 🖼️ MVP-0 UI: the minimum word-hopping interface on the Oatmeal theme — theme boilerplate first, use case, wireframes, fake-data mode #71 §8's U0; ran first so S4b was not built on daisyUI) - S4b — Browse, source page, dashboard, health (commits
234306a+08fadbc, 2026-09-06 — U5 ✅ 5/5 sources agree withdd.health, X2 ✅ p95 76 ms, U4-browse ✅ at 375 px, U1 5/6 routes; 26 / 26 graded rows pass, 322 tests, zero migrations) - S4b nit — Bierce
strip_headwordwas wrong twice, in 9 entries not 3 (commit890a834, 2026-09-06) - S4 audited 2026-09-06 — U0 A, S4b B+: every page driven in Chrome; two click-level defects (chips, health mount) → S4c
- S4c — Fixes from the S4 audit (commit
900ea5d, 2026-09-06 — chip click verified on the socket,/healthdead render 3 ms over websocket with the scorecard async + cached, A5 v2 93.0 % at a 90 % bar, Wikidata linked · 16,014, nopage=false; 322 tests) - S5 — Extensibility proof (commits
9db7805+270e48a, 2026-09-06 — E1 ✅ Johnson 1755, 42,726 entries, 91.9 % already in the index, 6,780 words of his own, 0 migrations; E2 ✅emotions, 809 lexemes from one WordNet root, 0 code changes; E3 ✅users+examples+votesmigrated, schema-diffed, rolled back, unshipped — 29 / 29 graded rows pass, 356 tests. Two findings: A7 v2 graded on asserted concepts, and an unscopedconcept_qids/0left for its own session. MVP-0 closes when 🖼️ MVP-0 UI: the minimum word-hopping interface on the Oatmeal theme — theme boilerplate first, use case, wireframes, fake-data mode #71 lands U1–U3: R3 X1 U1 U2 U3 U4 U6 are still pending) - S5 audited 2026-09-06 — A−: every number re-measured, E1–E3 hold, the second scope exposed the unscoped concept seed and the Animals-calibrated bars → S5c
- S5c — Fixes from the S5 audit, done 2026-09-07 with 🖼️ MVP-0 UI: the minimum word-hopping interface on the Oatmeal theme — theme boilerplate first, use case, wireframes, fake-data mode #71's U1b (
30a1e8c): scoped concept seed and concept pass, per-scope bars inscopes.rules["bars"], L3 n/a without a root,?scope=on/healthand/sources/:slug; orphans left as decided. Opened: chaseP279for asserted concepts, bounded
Activity
Living record: README rewrite (vision, decision log, roadmap, naming)…
S0a: reset in place to a fresh Phoenix 1.8.13 app for MVP-0 (#69, #70)
holden commentedon Sep 5, 2026
S0a done (2026-09-05). The repo is reset in place and pushed; the next agent starts at S0b.
58fdb81onmainphoenix-skeleton→2ea8124(also carries the README rewrite)ruby-rails-archive→36d3b153phx_new1.8.13 (latest on Hex),--no-mailer.tool-versions(mise)Absorb.Sourcebehaviour;Absorb/Lexicon/Encyclopedia/Sources/Healthcontext stubs; directories from the structure above with.gitkeep;priv/sources/bierce/pg972.txtmix test→ 5 tests, 0 failuresNext agent, first three things: (1)
mix ecto.droponce, the skeleton's dev database still has the old tables; (2) start the two dump downloads intodata/in the background (10 MB WordNet, 2.6 GB Wiktionary, stream the latter, never decompress); (3) write the baseline migration from #69 §4 and theMaterializer, then the WordNet source.One procedure fix learned the hard way, now in the body above:
phx.new1.8.13 runsgit init, so the rsync step must--exclude .gitor it wipes the remote from.git/config.holden commentedon Sep 5, 2026
S0b — done (commit
19bccaf, 2026-09-05)Thirteen-table baseline,
Sources/Lexicon/Encyclopediaschemas and contexts, theMaterializer, WordNet absorbed in full, the Wiktionary index pass, and the Animals scope built. 64 tests, 0 failures, offline.mix precommitgreen.Numbers
lexemeswithlang = enenriched_atnil)scope_lexemesfor animalstrim/1saving on the fixturesScope by reason —
wordnet_closure7,692 ·wiktionary_category15,612 · overlap 2,027 ·wikidata_taxonskipped (S2).The closure figure lands within 11 of the 7,681 predicted in the spec.
Relation types — hypernym 344,049 · hyponym 337,176 · other 151,730 · holonym 105,374 · meronym 104,921 · see_also 4,668.
Wall clock — WordNet absorb 1m40s · Wiktionary index 1m19s (10,806,865 lines streamed) · scope build 0.9s · fixture capture 54s. Well inside O2's 2 h budget.
Five findings that amend the spec
The pinned WordNet asset was wrong.
english-wordnet-2025-json.zipholds 107,519 synsets / 128,009 lemmas / 135,969 (lemma, pos) pairs and fails A2.english-wordnet-2025-plus-json.zipholds 120,564 / 153,888 / 161,875 — which is exactly where 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69's "~120k synsets, ~161k words" came from. Now pinned inSources.Cataloganddata/README.md. Note also that A2's "distinct lemmas" must be read as distinct lexemes (lemma, pos): the plus edition's distinct lemma strings are 153,888, which would fail the ≥155,000 bar.WordNet ships no
hyponymorholonymedges — onlyhypernym(93,395) andmero_*(22,312). Both inverse directions have to be derived, and inversion is global whilematerialize/1must stay pure and per record. Soabsorb/2inverts once and writes an expanded_edgeslist into each record'sraw. That is also what keeps M2 honest:rawalone is enough to rebuild every derived row.22,036 synsets carry a
wikidataQID. Not a rung in §5's ladder, but free and high-signal, so it is kept insenses.metadatafor S2's linker. Given L1 (≥70% link rate) is the flagged unknown, this may be the row that saves it.The
en:Animalscategory tree is not in the Kaikki dump — records carry flat category strings with no hierarchy. Walked once over the Wiktionary API (mix dd.scope.categories animals, depth 4, 12 subtrees denylisted as about-animals-but-not-animals) and pinned as 231 categories inSources.Catalog, so scope building stays offline and reproducible. The index pass stores each entry'sen:-prefixed categories on the lexeme, which turns the category rule into one SQL query instead of a second 2.6 GB pass — and makes every future scope free of re-streaming, which is what E2 actually promises. Small deviation from 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69's "a bare row withforms", and worth it.WordNet relations resolve inside the absorb (§4's "resolved at absorb"), as one set-based
UPDATEafter all senses exist. R1 is therefore green in S0b rather than S1.Smaller decisions
Sources.Catalogholds the source/scope definitions rather thanseeds.exs, because seeds do not run in:test(the alias isecto.create, ecto.migrate, test). Seeds and test fixtures share one definition; S3'sdd.scoreA1 row has something to check against.index(lexical_relations, [:from_sense_id]), which §4's index list omits — the scope closure CTE and the word-page graph panel both join on it.elixirrequirement bumped~> 1.17→~> 1.19to match.tool-versions.coalesce(from_sense_id, 0),coalesce(sense_id, 0)) needconflict_target: {:unsafe_fragment, "…"}ininsert_all; noted in theMaterializermoduledoc since it must match the migration character for character.Jason.decode!(line, strings: :copy)in the index pass is not optional: without it every retained lemma is a sub-binary pinning its whole ~2 MB parent chunk.What is now runnable
Spot check
/define/cathas the material it needs: 10 WordNet senses across 10 synsets, and the hypernym chain walkscat → feline → carnivore → placental mammal → mammal → vertebrate → … → entity, withanimalon the path.Absorb.Source+Materializertests, WordNet, Wiktionary index, Animals scope, first numbersS0b: baseline schema, WordNet + Wiktionary index, animals scope (#70)
holden commentedon Sep 5, 2026
S0 audited (2026-09-05). Verified directly against the database and the spec: schema matches #69 §4 column for column, no Postgres enums, raw not auto-loaded, materializer transactional and idempotent, A2/A3/A4/R1 as reported, 64 offline tests, precommit green. Commit
19bccafis now pushed. The five S0b amendments and the scope decision (keep all 21,277 lexemes, enrich in reason order, O2 split into dump absorbs ≤ 2 h and on-demand report) are folded into #69 (v6). Notes for S1, S2, S3 and S4 are in their session sections above. S1 can start.Decision log: S0 audited, keep the 21k Animals scope, split the O2 bu…
holden commentedon Sep 5, 2026
S1 — done (commit
dc6c326, 2026-09-05)The Wiktionary scoped absorb,
Absorb.Resolver,Absorb.Batch,mix dd.resolve,mix dd.materializeand the firstHealthfunctions. 114 tests, 0 failures, offline.mix precommitgreen. Zero migrations — E1's promise, exercised early.Numbers
dd.materialize --dry-rungaps, both sourcestrim/1saving on the real scoped recordsto_lexeme_idmonkeys→ monkey,oystre→ oyster,Pekinese→ Pekingese,gunnel→ gunwaleWhat landed — 19,251 source records · 31,116 senses · 105,731 relations · 930
canonical_lexeme_idlinks · 18,110 lexemes touched, of which 12,776 newlyenriched_at. 16,497 of 19,987 scope lemmas matched a record in the dump.R2 by type — derived 48,394/58,909 · synonym 26,161/27,886 · hyponym 4,481/5,829 · related 5,052/5,601 · hypernym 3,331/3,374 · coordinate 1,756/1,859 · alt_of 936/979 · antonym 811/826 · meronym 231/243 · form_of 100/102 · holonym 77/82 · other 41/41.
The unresolved tail is genuine, not a bug:
African bush elephant,male elephant,divvoare red links or Thesaurus-only entries with no English headword. About 30 rows are dump dirt — Thesaurus section headers (Formal terms,All are derogatory unless otherwise labeled) and a gloss (Alternative spelling of dumbass.) leaking into a linkage'swordfield. Left as they are: #69 §4 says relations keepto_lemmaforever, and R2 reports them.Wall clock — scoped absorb 41 s (10.8 M lines streamed a second time) ·
dd.resolve2.4 s ·dd.materialize --dry-run10 s over both sources · WordNet re-absorb 1m 21s. Still nowhere near O2(a)'s 2 h.Spot check —
cat/nounnow carries 8 WordNet senses and 28 Wiktionary senses across three etymologies, 642 Wiktionary relations, IPA and etymology.dog/noun7 + 23 and 928 relations.oyster/noun3 + 7 and 127.Three findings that amend the spec
1. A5 cannot pass as written, and nothing we build changes that. 83.5 % is the ceiling, and the 3,504 misses break down as 1,995 Linnaean binomials (
Cimex lectularius,Perdix perdix,Bufo marinus), 1,299 other multiword names (woodland caribou,bird-footed dinosaur) and 210 single words (soup-fin,dickeybird). Wiktionary files binomials under Translingual, not English, and the index pass filterslang_code == "en"— so the English index can never hold them, whatever we do to the absorb. This is exactly the row #69 §7 lists under "what we expect to learn: Wiktionary coverage of WordNet's scientific and multiword names". An amendment is proposed on #69; S2's Wikidata taxon pass is what will actually cover these words.2. M4 measured 45.6 % on real records before the trim was made precise. The three fixtures are unusually translation-heavy, so 81 % on them was never representative. Profiling 2,000 stored payloads found entry- and sense-level
categorieswere 36 % of everything kept — a hundred strings per record of the shape "Requests for translations into Aiton" and "English 3-letter words", of which only theen:-prefixed ones are ever read.trim/1now narrows categories instead of only dropping whole keys, and also dropshyphenations,wikipedia,senses[].linksand thebold_text_offsetsarrays inside examples. That takes M4 to 67.7 %. It is no longer "four top-level keys"; what keeps it honest is the invariantmaterialize(trim(r)) == materialize(r), which the M4 test pins across all three fixtures.3. X3's first half cannot come from the scoped absorb.
monkeyscarries noen:categories, so it is not in the Animals scope and no record is ever fetched for it — but the index pass gave it a bare row of its own (534,780 form-of entries did), so a plain lemma lookup finds it and renders an empty page.Lexicon.lookup/1therefore treats a bare match as not good enough: if another word claims the string among itsforms, that word wins. This needs no new data and uses thelexemes_forms_indexthat already existed — which is what #69 §4's annotation onlexemes.formsalways intended.Four bugs fixed on the way, each of which would have corrupted these numbers
Materializer.stamp_enriched_at/2was all-or-nothing per batch. It marked every lexeme in a batch enriched if the batch produced any sense at all. Fine for WordNet, where every synset yields a sense for every member; wrong for a scoped Wiktionary batch that mixes words which gained senses with words that only had a lexeme row declared. It would have inflated A3 and put empty cards on word pages. Now per lexeme.Health.Paritycompared twoDateTimestructs with<. That is Erlang term order, which compares thedayfield beforemonthbeforeyear, and it reported 113,564 of 120,564 WordNet records as needing materialization. NowDateTime.compare/2, matching the SQL predicateAbsorb.Batchuses. There is a regression test with a record fetched on 30 January and materialized on 2 February.Materializer.collect/2turned an{:error, _}from a source into aMatchErrorinstead of the clean{:error, {step, reason}}its contract promises.source_recordswith its own{:replace, …}list and so never maintainedchanged_at, which the "changed this week" feed will read. Both bulk paths now go throughSources.insert_records/3and shareSources.record_conflict/0.Smaller decisions
Absorb.Batch, shared by both sources and bydd.materialize. It honours 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §5's "needs materialization" predicate (materialized_at IS NULL OR < fetched_at) instead of re-materializing every record on every run;--allforces the full rebuild, which is the lever M2 will pull in S3.Healthstarts now with the S1 rows only (coverage/2,links_back/0,resolution/1,parity/1,trim_saving/1). S3 adds the remaining rows anddd.scoreon top rather than re-deriving numbers that have already been printed once.wordnet_closurefirst, then the category-only tail) as the S0 audit asks. The phase filter readsraw->>'word'rather than splittingexternal_id, because lemmas likeand/orandkm/hcontain the separator.Wiktionary.pos/1is now one function called by both the index pass and the scoped pass. If they disagreed by so much as a default, the scoped absorb would create a second lexeme beside the index row and the word page would show the word twice.external_idis"word/pos/etymology_number#position"; the record's is"word/pos/etymology_number"per 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §4. Wiktionary QIDs ride insenses.metadata["wikidata"]—concept_links.concept_idis NOT NULL and noconceptsrow exists before S2 — same convention WordNet already uses.Sources.Catalog's wiktionaryconfig["trim"]now readsWiktionary.trimmed_keys/0instead of repeating the list.What is now runnable
S1: Wiktionary scoped, relations resolved, parity (#70)
Decision log: S1 audited and merged, A5 amendment accepted (#69 v7, #70)
holden commentedon Sep 5, 2026
S1 audited (2026-09-05). Grade: A−. Merged.
dc6c326fast-forwarded intomain(branch deleted), followed by the README decision-log row.Verified directly, not from the report: A5 17,769 / 21,277 = 83.5 % with the misses bucketing to 1,995 binomials; A9 243,775 / 243,775; M1 0 records needing materialization across both sources; R2 91,371 / 105,731 = 86.4 %; R1 still 1,047,918 / 1,047,918; 930 canonical variants; membership by lemma honoured (
catanddoghave adj/noun/verb records); reason order honoured;enriched_aton every Wiktionary-enriched lexeme; raw fully trimmed (0 records still carrytranslations, mean 1 KB);X3live formonkeys,Pekinese,mice,geese,cats,gray wolf. 114 tests andmix precommitgreen on the branch. Zero migrations. Health functions are pure queries; parity compares by natural key; the resolver is deterministic with a pos preference and a cycle guard.Why not an A: the branch was left unpushed (work at risk on one laptop), and two design nits surfaced that are now #69 open items —
content_hashis taken on the trimmed payload, so tighteningtrim/1marked 19,250 records "changed"; andLexicon.lookup/1lets a form-listing word beat a bare headword. Neither blocks S2.The A5 amendment is accepted and folded into #69 (v7): the row now excludes lemmas Wiktionary files as Translingual and measures 92.2 %, with the raw 83.5 % and the binomials still reported; A6 now asks for the Wikidata-plus-Wiktionary union. S2 notes are in the S2 section above. S2a can start.
S1b: hash records before trim, flag form-of index rows (#70)
holden commentedon Sep 5, 2026
S1b — done (
5b9ee82+b356876, 2026-09-05)The two design nits from the S1 audit, closed before S2 adds sources on the same record contract. 123 tests,
mix precommitgreen, zero migrations.changed_atstampsform_offlagged index rowsContract change for S2:
content_hashis the hash of the payload as fetched, beforetrim/1. A source that trims passescontent_hashtoSources.insert_records/3; a source whose trim is the identity may omit it.changed_atnow moves only when the source moved.Lookup:
cats→ cat (forms),CATS→ the musical with cat underalso,monkeys→ monkey,spatkeeps its own page. A bare headword outranks a forms match only on an exact-case match.Honesty note:
5b9ee82's message described the exact-case rule before the edit had actually landed (a scripted edit aborted on a doc anchor and the shell carried on);b356876makes it true and adds the regression test.S2a can start.
15 remaining items
S4c: fixes from the S4 audit (#70)
holden commentedon Sep 6, 2026
S4c done —
900ea5d(2026-09-06)Judged minor enough to do inside the audit session rather than hand back. Five fixes, each verified in Chrome on the live socket, not from URLs:
phx-value-slug; the click pushestoggle {kind: "has", slug: "bierce"}and the page reads 66 words with bierce. A test guards against anyphx-value-valuecoming back.assign_asyncbehind a ten-minute Cachex cache (:health) with a recompute button. Dead render 3 ms, join over websocket, rows in about 4.5 s cold and instantly warm. Cache off in test.scientific_name(binomial by shape, or equal to a concept'staxon.scientific_name), bar 90 %. Animals: 78.3 % raw · 93.0 % excl. 4,004 scientific names ✅. The source pages' misses by kind table follows.concept_linksrule (auto/confirmed, ≥ 0.7,Lexicon.Browse.linked_count/1), the row glyph reads linked / unlinked, and the two dead chips are gone.page=false, no empty keys in taxon or pager links.One regression of my own caught before commit: the scientific-name set was evaluated once per missed word inside a capture, which made
/sources/wiktionaryan 81 s request; it is one query per call now (391 ms page). 322 tests,mix precommitgreen.Scorecard after S4c: 26 / 26 graded rows pass with parity, A5 at 93.0 % against 90. Next: U1 (word page and the hop) on #71.
holden commentedon Sep 6, 2026
State of MVP-0 after S4c (2026-09-06) — what Animals has proved, what is left
Short version. Four sessions remain, not one: S5 (the extensibility proof) and U1–U3 (the reader pages on #71). The backbone is proved on the whole English index; Animals only scopes the enrichment. What is not yet proved is a non-taxonomic scope (E2), a sixth source at zero migrations (E1), and the product itself (the word page). Recommended order: S5 next, because it answers the "does this generalise" question in one session, then U1–U3, then scale.
Where we are
instance_of· 18,712parent_taxon· 1,809subclass_of)Proved, and not animal-specific
materialize/1, atomic writes, parity 0 gaps over 271,296 records, links back 100 %.wordnet_closureandwiktionary_categoryare general rules;wikidata_taxonis the animal one.Animal-flavoured, and to be proved by E2
parent_taxonis one of three relation types and taxa are 23 % of concepts. The browse tree and the L3 row are taxon features. A general scope getssubclass_of/instance_ofand WordNet's hypernym graph instead — and the hop (🖼️ MVP-0 UI: the minimum word-hopping interface on the Oatmeal theme — theme boilerplate first, use case, wireframes, fake-data mode #71) is defined onlexical_relations, the word graph, not on taxonomy.wikidata_classscope rule (P279 closure) besidewikidata_taxon; L3 generalised to "reaches the scope's root concept"; a non-taxon fallback for the browse tree; hard-codedanimalsdefaults in the health and imports pages and the mix tasks (a?scope=parameter).What is left
emotionsscope from one WordNet root with no code change (E2 — the direct answer to "words that are not animals"); the community migration written and shown to touch no existing table, then not shipped (E3)MVP-0 closes when every row is green — its own exit rule.
Scaling to everything (MVP-1), as an estimate
page_propsdumps and the Wikidata dump would make it an afternoon, as a new access mode for two sources with no schema change — E1 rehearses exactly that shape.S5 planning decisions: committed static sources under ~10 MB, generic…
Dev server on port 4007: 4000–4005 belong to the sibling projects (#70)
holden commentedon Sep 6, 2026
S5 — the extensibility proof
The three claims #69 §7 makes about the architecture rather than the data, turned from prose into measurements. E1, E2 and E3 pass. Two things the proof itself found are at the bottom, and one of them is an amendment to #69 that needs your assent.
E1 — a new source is cheap ✅
Johnson 1755, from the LEME TEI-XML: TSpace handle
1807/124274, edited by Ian Lancashire, CC BY 4.0 (the 1755 text is public domain; the transcription is not, so the attribution is a licence condition, not a courtesy). Committed gzipped atpriv/sources/johnson/johnson-1755-leme.xml.gz— 8.8 MB packed, 29 MB open — and pinned by edition and SHA-256, which the absorb verifies before it reads a byte.<wordentry type="h">), 17.9 s to absorbalt_of, 707see_also, 109form_ofWhat it cost: one
sourcesrow, onepeoplerow, one line inAbsorb.@modules, one module, seven fixtures and 27 tests.schema_migrationsstayed at 2 — the baseline and Oban's job table — which is what E1 now measures rather than asserts. Every enum-like column is a plain string backed byEcto.Enum, so a new tier or relation type would have cost none either.Before Johnson landed, the three places that hard-coded the five slugs were made generic, so the claim needs no footnote:
health_live.exreadsSources.list_sources/0(its parity table already did),dd.fixtures.capturedispatches any static book source through onecapture_static/3, andCoverage's snapshot-pin list is now general (sha256,edition) rather than ending at Bierce'sgutenberg_id.The headword rule is the part worth reading twice. Not "everything before the first period" — wrong for
To SINK pret I sunk, which has no period after the headword, and forA. Bp., which is all periods. Not "everything before the first lower-case word" either — that truncates all 565 phrase headwords (CAT in the pan,LILY of the Valley,SENSITIVE Plant,HALF-SEAS over). The run ends at a full stop or at a grammar marker, and the marker vocabulary was read off the file rather than imagined:n(972),adj(639),v(399),adv(240) leaking out of<class type="pos">, thenpret,part,plur,pronoun. A comma-separated part is an alternate only if it survives the rule whole and every word of it is capitalised — otherwiseMETHO'UGHT, the preterite of methinksputs that sentence in the lexicon as a word, which the first pass did, 167 times.Three transcription conventions decide the lemma, all measured on the file: the apostrophe is a stress mark (
ABA'CKE→ abacke;’in 30,005 forms,'in 4,549), soft hyphens mark where the printed line broke inside a word (27,339 of them —the lowest or\r\nderis the lowest order), and a leadingTois the verb marker of the period (7,793 entries).One thing Johnson broke and then fixed: X3 went red. He defines
GEESE("The plural of goose"), sogeesestopped being a bare index row andLexicon.lookup/1rightly refused to send a word with a definition of its own somewhere else. The fix is to read what Johnson is telling us — 109 entries name their base word in the printed headword or the first line of the body — and emit aform_ofedge, whichdd.resolveturns into acanonical_lexeme_id. One further nit surfaced underneath:resolve_canonical/3required every matching lexeme to agree, so Wiktionary's baregeeserow, which has no opinion because nobody wrote one, vetoed Johnson's. A bare form-of row no longer votes.spatstill keeps its own page, becausespat/nounis a real headword rather than a bare row.E2 — a new scope is data ✅
emotions, 809 lexemes from one WordNet root (oewn-00026390-n, feeling), 42 ms to build.wiktionary_categoryandwikidata_taxonskipped with reasons rather than contributing silent zeros. Then the scoped absorbs unchanged: Wiktionary 1,132 records / 3,325 senses, Wikidata, Wikipedia's 809 lemma probes,dd.link— L1 37.2 % at ≥ 0.8, 74.2 % at any confidence, honestly lower than Animals because abstract nouns often have no article of their own. L3 reads 0 %, which is the right answer for a scope with no taxonomy.Zero code changed to add it. Every scope is now a
priv/scopes/<slug>.jsonfile read byCatalog.scopes/0—animalsmoved there too, 231 pinned Wiktionary categories and all — so there is no scope defined in Elixir to point at, and E2 measures what it claims.mix dd.scope.new <slug> --name … --rules '<json>'writes the file and the row; with no options it reads a file someone else committed.ScopeLive's taxonomy rail was the one page-level assumption: it hard-coded@root_qid "Q729", so/s/emotionswould have drawn Animalia with fifteen zero-count children. It now readsscope.rules["wikidata_root"]and hides itself when there is none.E3 — the community layer fits ✅
users+examples+voteswritten from #69 §4's sketch, generated withmix ecto.gen.migration, applied to the full development database, diffed against apg_dump -staken before them, and rolled back.The diff adds three tables, seven indexes and two check constraints, and changes no column, index or constraint of the thirteen. Four foreign keys point into
lexemes,senses,conceptsandsourceswithout adding anything to them.usersis included deliberately: there is no auth scaffolding at all, soexamples.submitted_by_idandvotes.user_idwould have had nothing to point at, and a proof that skipped the awkward half would not be one.The file now lives at
docs/sketches/community_layer_migration.exs, wheremix ecto.migratecannot see it andmix precommitcannot apply it to the test database. Deleting it outright, as this issue said, would have made the claim unfalsifiable.What the second scope found
Two things, and only the first is fixed here.
1. A7 v2 — an amendment to #69, for your assent
Building
emotionsgrew the concept table from ~44 k to 90,481 sitelinked rows. Of those, 54,273 have no link to any scope word at all — they are things a disambiguation page merely mentioned, and every summary fetched names more of them, so A7's denominator grows faster than any pass can fill it. One concept pass fetched 274 gaps and introduced 20,526 new ones.The S3 audit predicted exactly this and wrote the remedy on this issue: "for the next scope consider limiting the concept pass to asserted concepts plus candidates promoted to 0.6, and reporting A7 on that population." This is the next scope, so A7 is now graded on the concepts a scope word actually links to at
auto/confirmed:10,412 / 10,412 = 100 %, with 70,224 / 90,481 = 77.6 % including candidates reported beside it.
Same shape as the A5 v2 amendment S4c landed. If you would rather A7 kept its old denominator, say so and it goes back — but then it is a row that cannot be closed by fetching, only by not growing the graph, which is really the item below.
2. Open defect — an unscoped concept seed, left for its own session
Wikidata.seed_qids/2unions an unscopedconcept_qids/0:The other three seeds (
wordnet_qids,wiktionary_qids,root_qids) are all scoped throughscope_lexemes. This one is not, so scope N re-walks every concept scopes 1..N−1 introduced. An 809-lexemeemotionsscope seeded 72,108 QIDs and fetched 28,084 records in 16 minutes.With one scope the bug is invisible, because the whole table is that scope. The extensibility proof is what made it visible — which is the proof working, not the proof failing. Two candidate fixes: seed only the concepts this scope's own probes introduced, and stop chasing 0.40 disambiguation candidates until the linker promotes one. Both change the animals data path, so I have left them out of S5's tail rather than slipping them in.
Numbers
mix dd.score: 29 / 29 graded rows pass, 1 reported, 7 pending — R3, X1, U1, U2, U3, U4 and U6, every one of them #71's U1–U3. MVP-0 closes when those land; this issue's own work is done.Alongside the E rows: A1 6 / 6 absorbed and 6 pinned · M1 0 gaps over 344,290 records · M2 row counts identical across all six sources · A9 365,037 / 365,037 links back · X3 4 / 4 · U5 6 / 6 sources agree with
dd.health.mix precommitgreen, 356 tests, zero migrations, six sources, two scopes.Every page driven in Chrome at 4007:
/sources/johnsonexists with no page code (👑 aristocracy, static, CC BY 4.0, attribution, pinned by edition, 42,726 records / 0 stale / 6,780 words introduced);/s/emotionsrenders with no taxonomy rail; thejohnsonhas-chip clicks through to 435 words with johnson on emotions and 1,313 on animals;/healthlists six sources in both its tables.S5: Johnson 1755, the emotions scope, the community-layer sketch (#70)
S5b: A7 v2, Johnson's form-of edges, a general snapshot pin (#70)
S5 audit: status and decision-log rows (#70)
holden commentedon Sep 6, 2026
S5 audit —
9db7805+270e48a(2026-09-06): A−, and the backbone is doneEverything the report claims, re-measured: 356 tests,
mix precommitgreen,dd.score --scope animalswith parity 29 / 29 graded rows,schema_migrationsstill 2, six sources and two scopes, Johnson 42,726 records → 42,726 entries, 6,780 words of his own, 1,313 in Animals and 435 in Emotions;emotions809 lexemes with no unreasoned row; the community migration sitting indocs/sketches/wheremix ecto.migratecannot see it. Driven in Chrome at 4007:/s/emotionsrenders with no taxonomy rail and a working search,/sources/johnsonexists with no page code,/healthlists six parity rows and the three E rows over a websocket,/admin/importsshows six sources, no console errors.What passes, and one honesty note
resolve_canonical/3, which is what putgeeseback for X3; and the snapshot pin list became general). Both are improvements, both migration-free, and both are exactly the kind of thing a sixth source should flush out.auto/confirmed, the sitelinked population reported beside it. The S3 audit wrote that remedy in advance; the second scope made it necessary.Why A− and not A
emotionsseeded 72,108 QIDs and fetched 28,084 records over 16 minutes for 809 words, and the concept table now carries 35,719 rows with no link and no relation. Leaving the animals data path alone in S5's tail was defensible; the cost was not small.dd.score --scope emotionsfails A4 at 809 < 7,500, A10, L1 and L3 at 0 % because it asks for Animalia) and the session reported L3's 0 % as "the right answer" rather than making it n/a. The generalisation was one step further than the session took.sources.idran to 21 for six rows during the rollback attempt. Cosmetic, but it is the trace of a 12-minute cascade that should not have been started.All three are a short S5c in the body, to ride with U1 or run on its own.
Where this leaves MVP-0
#70's own work is complete: every session S0–S5 has landed and been audited, and the scorecard's 7 pending rows (R3 X1 U1 U2 U3 U4 U6) are all #71's U1–U3. This issue can close; #69 stays open until the closing scorecard lands with U3, and #71 is the tracker from here.
U1b: the thing side, and S5c (#71, #70)
holden commentedon Sep 6, 2026
S5c done —
30a1e8c(2026-09-07), landed with #71's U1bAll five items, no migration. The write-up is #71 §8b.5; the numbers:
seed_qidsforemotionsseed_qidsforanimalsdd.score --scope emotionsdd.score --scope animalsconcept_links— asserted, plus candidates at ≥ 0.6 the linker could still promote, and not the 0.40 disambiguation floor. A run with no scope still walks the whole table, because that is what a full refresh means. Validated by comparing seed sizes before any fetch, as this issue asked.concept_targets/2selected every row with awikipedia_title— which is the same defect wearing different clothes.scopes.rules["bars"]with today's Animals numbers as defaults.emotionsdeclares 500 / 55 % / 45 %, each just under what it measures (809, 61.8 %, 49.2 %) so the row catches a regression rather than announcing a pass.wikidata_root— and names the root it walked when there is one (taxonomy reaches Q729).mix dd.health,mix dd.linkand the/healthdetail panel all handle the rootless answer; the last of those raisedKeyError :pctinside its own async render until it did.?scope=on/healthand/sources/:slug, both viahandle_params/3in/admin/imports's shape./sources/johnson?scope=emotionsreads 435 of 809 scope words where the animals page reads 1,313 of 25,402.One thing the fix opened, added to the S5c block above rather than done here: only
P171andP13176are chased to closure, so asubclass_ofedge usually points at a QID with noconceptsrow. The re-absorb gave joy its card (a kind of happiness), but 97 of 322 emotions concepts reach one chain step and 44 reach two. ChasingP279for asserted concepts at a fixed depth would deepen every non-taxonomic chain on the word page — and it needs its own budget and its own answer for A7, so it is its own session. Slipping it into U1b would have undone the thing S5c just achieved.U3: fake-data mode, the mobile pass, the chip tests, and the close of…
holden commentedon Sep 7, 2026
Closed with #71's U3 and the
mvp-0tag. S0–S5 are complete, S4c and S5c landed with U1b, and the scorecard reads animals 36 / 36 with parity, emotions 35 / 35, 0 pending on 487 tests. The full run is on #69.What this issue leaves open, and where it goes: chasing
P279for asserted concepts — the concept chain is one step deep for most non-taxa, which is why joy reaches happiness and stops — and the twoconcept_relationstypes (:part_of,:related) that sit in the enum with nothing writing them. Both are MVP-1 work and neither blocks anything shipped.