Skip to content

🧱 MVP-0 build plan: reset in place, sessions S0–S5 with expectations, directory structure #70

Closed

Description

@holden

Purpose

The execution tracker for MVP-0. The spec is #69 (schema, sources, scope, scorecard). This issue answers the practical questions: where to build, how many sessions, what each session must produce, and how the code is laid out. Tick boxes here as sessions land; paste the numbers each session produced as a comment.

Session protocol. Start a session by reading #69 and this issue. End it by ticking its boxes, posting its numbers, and committing with the session prefix (S1: …). A session that runs long splits at the marked seam, never in the middle of a source. A session may commit straight to main or on a branch s<N>-…; after the audit the branch is fast-forwarded into main and pushed. Edit issue bodies from the live version, never from a local copy (a stale copy clobbered the S0b tick once).


Practical answers

Same directory, same repo, reset in place. Nothing is lost: the old Rails app is already on origin as the annotated tag ruby-rails-archive → 36d3b153 ("fixes for lyrics", 2025-02-07), and the current Phoenix skeleton gets its own tag before the wipe. A second directory would split one repo across two checkouts and lose this project's Claude memory (keyed by path). Rename the folder only when the product name is decided.

Not one session. Six sessions minimum, eight likely. Reasons: the two dumps take real time to download and stream; the link rate must be measured before pages are worth building; and every session ends with numbers, which is the whole point of the scorecard. Two sessions (S2 and S4) have a marked seam and may become two each.

Environment as found on 2026-09-05.

Thing Found Note
Elixir / OTP 1.19.0 on OTP 28.1, pinned in .tool-versions (mise had them installed; 1.18.3/OTP 27 was the global default) mise picks the pin up automatically in this directory
Phoenix generator phx_new 1.8.13 (upgraded from 1.8.5; latest on Hex) → Phoenix 1.8.13, LiveView 1.2.11 done: commit 58fdb81
Postgres 18.2 via Postgres.app, listening on 5432 postgres / postgres, matches the default dev config
Disk 22 GB free enough: stream the Wiktionary dump, never decompress it to disk (23 GB)
History ruby-rails-archive tag on origin; local object store has the Rails commit nothing to rescue
Tracked but not wanted in the reset assets/vendor/oatmeal (415 files) a Tailwind Plus template committed to a public repo; it stays reachable at the skeleton tag, and comes back in the design pass from tmp/ (ignored) rather than from git

Pre-download before S0 (background, into data/, which the new .gitignore will exclude):

data/english-wordnet-2025-json.zip        10 MB   https://github.com/globalwordnet/english-wordnet/releases/tag/2025-edition
data/raw-wiktextract-data.jsonl.gz        2.6 GB  https://kaikki.org/dictionary/rawdata.html   (record the dump + extraction dates)
priv/sources/bierce/pg972.txt             394 KB  https://www.gutenberg.org/cache/epub/972/pg972.txt   (this one is committed)

Reset procedure (S0a — done 2026-09-05 as commit 58fdb81)

cd /Users/holden/Code/projects-2026/dictionary

# 1. protect the current skeleton (the Rails app is already tagged: ruby-rails-archive)
git tag phoenix-skeleton 7c6dd33 && git push origin phoenix-skeleton

# 2. generate a fresh app next to the repo (module/app names stay; decision #1 in #69)
cd .. && mix phx.new devils_dictionary_fresh --app devils_dictionary --module DevilsDictionary --no-mailer --no-install

# 3. wipe the working tree except README.md, docs/, and the ignored dirs; copy the fresh app in
cd dictionary
git rm -r -q --cached .
find . -mindepth 1 -maxdepth 1 ! -name .git ! -name README.md ! -name docs ! -name tmp ! -name _build ! -name deps -exec rm -rf {} +
rsync -a --exclude .git --exclude README.md ../devils_dictionary_fresh/ ./ && rm -rf ../devils_dictionary_fresh _build deps
# NOTE: phx.new 1.8.13 runs `git init` in the new directory; without --exclude .git the rsync merges its empty
# .git over yours and wipes the remote from .git/config (objects and refs survive). Repair: git remote add origin …
printf '\n# local data dumps\n/data/\n' >> .gitignore

# 4. deps and smoke test
mix deps.get && mix ecto.create && mix compile && mix test
# add to mix.exs: {:oban, "~> 2.19"}, {:req, "~> 0.5"}, {:floki, "~> 0.36"}, {:cachex, "~> 4.0"}, {:slugify, "~> 1.3"}, {:credo, only: [:dev, :test]}

# 5. commit the reset before writing any domain code
git add -A && git commit -m "S0a: reset to a fresh Phoenix 1.8 app for MVP-0 (#69, #70)"
# also: printf 'erlang 28.1\nelixir 1.19.0-otp-28\n' > .tool-versions ; append /tmp/ and .DS_Store to .gitignore (the old tree kept the Oatmeal template in tmp/)

mix phx.new regenerates AGENTS.md (ours is the stock file, so nothing is lost). README.md is the living record and is kept as is.

What S0a left in the tree (commit 58fdb81): the fresh app; .tool-versions; deps oban 2.24, req 0.7, floki 0.38, cachex 4.1, slugify 1.3, credo; DevilsDictionary.Absorb.Source (the behaviour from #69 §5, incl. trim/1); moduledoc-only context modules Absorb, Lexicon, Encyclopedia, Sources, Health; the directories below with .gitkeep; priv/sources/bierce/pg972.txt; a local data/README.md (ignored). mix test: 5 tests, 0 failures. Before the first mix ecto.create in S0b, run mix ecto.drop: the skeleton's dev database still exists with the old tables.


Directory structure (what each session creates)

lib/devils_dictionary/
├── absorb/                       S0  the import layer
│   ├── source.ex                 S0  behaviour: slug/0 absorb/2 enrich/2 materialize/1 trim/1 rate_limit_ms/0
│   ├── materializer.ex           S0  writes materialize/1 output + materialized_at in one transaction; idempotent upserts
│   ├── scope_builder.ex          S0  rules → scope_lexemes with reasons
│   ├── resolver.ex               S1  to_lemma → to_lexeme_id; canonical variants
│   ├── linker.ex                 S2  the ladder → concept_links (method, confidence, status)
│   ├── clients/                  S2  Req clients: wikidata.ex, wikipedia.ex (user-agent, rate limit, absent markers)
│   └── sources/
│       ├── wordnet.ex            S0
│       ├── wiktionary.ex         S0 index pass · S1 scoped materialize + trim
│       ├── wikidata.ex           S2
│       ├── wikipedia.ex          S2  incl. disambiguation candidates
│       ├── bierce.ex             S3
│       └── johnson_1755.ex       S5  the extensibility proof
├── lexicon/                      S0  context + schemas: lexeme.ex sense.ex entry.ex lexical_relation.ex scope.ex scope_lexeme.ex
├── encyclopedia/                 S2  context + schemas: concept.ex concept_relation.ex concept_link.ex
├── sources/                      S0  context + schemas: source.ex source_record.ex person.ex import_run.ex
├── health/                       S3  coverage.ex parity.ex score.ex (the scorecard rows as functions)
└── workers/                      S2  absorb_worker.ex enrich_worker.ex (thin Oban wrappers)
lib/devils_dictionary_web/
├── live/                         S4  search_live.ex word_live.ex scope_live.ex source_live.ex admin/imports_live.ex
└── components/                   S4  source_card.ex (tier-tinted, ↗ + ⓘ) graph_panel.ex provenance_drawer.ex coverage_badges.ex
lib/mix/tasks/                    S0…S3  dd.absorb.ex dd.scope.build.ex dd.resolve.ex dd.link.ex dd.materialize.ex dd.health.ex dd.score.ex
priv/repo/migrations/…_create_lexicon_schema.exs   S0  the single baseline migration (#69 §4)
priv/repo/seeds.exs               S0  the five sources rows (+ community/bot rows are NOT seeded yet)
priv/sources/bierce/pg972.txt     S3
test/support/fixtures/{wordnet,wiktionary,wikidata,wikipedia,bierce}/{cat,dog,oyster}.json   S0…S3  real raw records, trimmed
test/devils_dictionary/absorb/    S0…  materialize/1 and trim/1 as pure functions; resolver; linker
test/devils_dictionary_web/live/  S4
data/                             ignored: the dumps

Contexts: Absorb (import), Lexicon (words), Encyclopedia (things), Sources (provenance), Health (numbers). Nothing in lib/devils_dictionary_web touches source_records.raw except the provenance drawer.


Sessions

S0 — Reset & measure

  • Goal: a fresh app with the schema, the behaviour, the first two sources absorbed, and the first real numbers.
  • Inputs: the pre-downloaded dumps; 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §4.
  • Work: reset procedure above → baseline migration → Sources/Lexicon schemas and contexts → Absorb.Source + Materializer (tested on fixtures) → sources seed → Absorb.Sources.Wordnet (full) → Absorb.Sources.Wiktionary index pass (bare lexemes + forms, streamed from the gz) → ScopeBuilder with the three rules → mix tasks dd.absorb, dd.scope.build.
  • Exit: A2, A3, A4 measured. mix test green offline.
  • Numbers to post: synsets, lemmas, index size (and how many are form-of entries), scope size by reason, time for each absorb.
  • Commit: S0: reset, schema, WordNet + Wiktionary index, animals scope (#70)
  • Seam: if the index pass is slow, stop after WordNet + schema and finish the index as S0b.

S1 — Wiktionary and WordNet, scoped

  • Goal: the dictionary side complete for Animals.
  • Work: Wiktionary scoped absorb with trim/1 → senses, pronunciations, etymology, forms, relations with to_lemma → Resolver incl. canonical_lexeme_id → dd.resolve, dd.materialize --dry-run parity → source records carry url.
  • Exit: A5, A9, M1, M4 (measured on the real scoped records, not only the fixtures), R2, X3. R1 is already green from S0.
  • Numbers: Wiktionary coverage of scope, unresolved targets by type, trim savings, records per second.
  • Commit: S1: Wiktionary scoped, relations resolved, parity (#70)
  • Notes from the S0 audit (2026-09-05):
    • The scope is 21,277 lexemes, kept whole by decision. Absorb the scoped Wiktionary records in reason order: wordnet_closure (7,692, incl. the 2,027 overlap) first, then the 13,585 category-only tail. The flagship words cat/dog/oyster are in the first group.
    • The index pass already wrote forms, metadata.wikt_categories, origin_source_id and source_ids for every English entry; the scoped pass fills senses, pronunciations, etymology, relations and enriched_at, and stores the trimmed raw (trim/1 drops translations, descendants, templates; 81 % smaller on the fixtures).
    • Materializer merges lexemes rather than clobbering them (forms fill empty slots, metadata merges, origin stays); keep that contract when Wiktionary and WordNet both touch a word.
    • Wiktionary relations arrive with to_lemma only; dd.resolve fills to_lexeme_id by lower(to_lemma) + pos, and canonical_lexeme_id from alt_of/form_of where the target exists. WordNet rows are already resolved, leave them alone.
    • import_runs rows 1–2 are failed dev retries of the WordNet absorb; dd.score A1 must count done runs only.

S1b — Hash before trim, form-of flag (done 2026-09-05, 5b9ee82 + b356876)

  • Goal: close the two design nits from the S1 audit before S2 builds two more sources on the same record contract.
  • Work: Wiktionary.scoped_row/1 takes content_hash on the payload as fetched, before trim/1; Sources.insert_records/3 accepts the hash and only computes it from raw for sources whose trim is the identity; regression tests (SourcesTest, WiktionaryTest). The index pass stores metadata.form_of on inflected-form entries; Lexicon.lookup/1 lets a bare headword keep its page only on an exact-case match and offers forms claimants under :also; a bare form-of row, or a bare row in another casing, yields to the word that lists the string among its forms.
  • Data: index re-run (1m 23s, 533,218 rows flagged), scope re-absorb (42.5 s), one-time reset of the 19,250 false changed_at stamps left by S1's trim change → 0.
  • Exit: Health rows unchanged — A5 17,773 / 21,277 raw (92.2 % amended), A9 100 %, M1 0 gaps, M4 67.7 %, R2 86.4 %. cats → cat, CATS → the musical with cat under also, monkeys → monkey, spat keeps its page.
  • Lesson: the first S1b commit's message described the exact-case rule before the edit had landed (a scripted edit aborted on a doc anchor and the shell carried on); b356876 makes it true. Scripted edits must gate the commit on their own exit status.

S2 — Things (may split at the seam)

  • Goal: the encyclopedia side and the word ↔ thing bridge.
  • Work (S2a): Encyclopedia schemas → Req clients with rate limits and absent markers → Absorb.Sources.Wikidata (concepts, images, taxonomy, taxon_concept_id) → Absorb.Sources.Wikipedia (summaries, thumbnails, title probes, disambiguation candidates) → EnrichWorker, snooze on 429.
  • Work (S2b): Linker ladder → dd.link with histogram, conflicts, candidates.
  • Exit: A6, A7, A10, L1, L2, L3, L4.
  • Numbers: link rate, disambiguation hits, taxon reach, API calls made and cached, on-demand wall clock (O2b).
  • Commit: S2: Wikidata + Wikipedia, linking ladder (#70)
  • Seam: between S2a and S2b.
  • Notes from the S0 audit (2026-09-05):
  • Notes from the S1 audit (2026-09-05, merged as dc6c326):
    • Only 752 of 31,116 Wiktionary senses carry a QID, so the wiktionary_qid rung is sparse for Animals. Expect wordnet_wikidata (22,036 synsets) and title_match to carry L1; report links per rung.
    • The 1,995 Linnaean binomials A5 now excludes are yours. Wikidata taxon items carry them as P225; report scope coverage for wikidata next to Wiktionary's (A6 now asks for the union).
    • The index pass already stores metadata.wikt_categories; Lexicon.lookup/1, Absorb.Batch (keyset loop honouring the needs-materialization predicate, --all for M2) and Sources.insert_records/3 (maintains changed_at by hash) exist. Use them; do not add a second record-insert path.
    • content_hash is over the trimmed payload, so a trim change looks like a content change (🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §10). Do not build the feeds on changed_at until that is fixed.
    • Wikipedia and Wikidata clients go under absorb/clients/; every fetch is an enrich/2 that lands a source_records row with url, then Materializer.run/2.

S3 — Bierce, health, score

  • Goal: the layer proof and the scorecard itself.
  • Work: Bierce parser incl. "See X" → see_also → Health (coverage, parity, drift) → dd.health → dd.score printing 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §7 with actuals → atomicity test (M3) → offline re-materialize (M2).
  • Exit: A8, M2, M3, O1, O3, O4.
  • Numbers: first full dd.score output (expect red rows for U*/X*/E*).
  • Commit: S3: Bierce, health, dd.score (#70)
  • Notes from the S0 audit: A1 counts done import runs per source (failed dev retries exist); A2's second metric is distinct lexemes (lemma, pos); O2 is two measures (dump absorbs ≤ 2 h, on-demand report).
  • Notes from the S2 audit (2026-09-05):
    • Already done by the audit, no action: the scope had grown by 4,116 lexemes after S1's Wiktionary pass, so ~2,000 scope words had an encyclopedia side and no dictionary side. Re-ran dd.absorb wiktionary --scope animals (+2,176 records, +3,367 senses), dd.resolve, dd.absorb wikidata --scope animals (+66), dd.absorb wikipedia --scope animals --concepts, dd.link. Rule for every future scope growth: re-run the dictionary half too. Current rows: L1 58.4 % raw / 88.5 % of reachable, any 68.2 %; A6 union 97.4 %; A7 100 %; A10 93.6 %; L2 1,180; L3 77.2 %; L4 100 %; R2 86.4 % (96,148 / 111,304); A9 100 % (291,230); M4 67.6 %; M1 0 gaps across 242,359 records. The stale running run (Bump aws-sdk-s3 from 1.176.1 to 1.182.0 #33) is closed.
    • Fix first, small: WordNet synsets can carry an array of QIDs (panther → ["Q35255", "Q109647288"]). Wikidata.wordnet_qids/1 and the wordnet_wikidata rung accept only jsonb_typeof = 'string', so those 265 scope references never seed a concept or a link. Normalize with the same jsonb_array guard rung 1 uses, then dd.absorb wikidata --scope animals and dd.link.
    • Wikipedia.concept_targets/2 skips concepts with an entry or an unexpired absent marker but not concepts that were fetched and yielded no entry (disambiguation pages, empty extracts), so every --concepts run re-fetches ~3,800 pages (191 requests). Skip any concept with an existing concept:<QID> record unless --refresh.
    • EnrichWorker paces with a per-process sleep, so enrich: 3 can reach 15 req/s against one API. Run enrich: 1 until a shared limiter exists, or one queue per source.
    • The Wikidata absorb's concept_relations_unresolved counts recorded-but-unchased P279/P31 targets (35,942); split it so a real P171 residual (which must be 0) cannot hide behind it.
    • dd.score A1 counts done runs only; two failed link runs and the closed run 33 are dev history.
  • Bierce source, verified 2026-09-05 (Holden asked): Project Gutenberg #972 is the right and only public-domain source of the complete 1911 text (the 2000 Unabridged is a copyrighted compilation; Wikisource carries the same text per letter). Parse the HTML edition, now checked in as priv/sources/bierce/972-h.htm, exactly as the old Rails seed did with Nokogiri (db/seeds/devils_dictionary.rb at tag ruby-rails-archive); the plain text pg972.txt stays as a cross-check. Measured on the file:
    • parse the <p> / <pre> nodes between the <h2>A</h2> chapter and the Gutenberg end marker; a headword paragraph matches ^([A-Z][A-Z'\- ]+), ([a-z\.]+)\.\s+ → 959 entries (the text has 960: EUCHARIST sits inside a <pre> in the HTML, so a <pre> that starts with the headword pattern is an entry, not verse);
    • part-of-speech markers seen: n, v, v.i, v.t, adj, adv, pron, pro, pp, p.p, n.pl, x — keep them verbatim, normalize to the lexicon's pos when creating the lexeme (x = "no definition", HASH);
    • 248 <pre> verse blocks and 215 short attribution paragraphs (initials and invented poets) follow their entry: keep both in entries.body in order (the verse is part of the joke), verse as a blockquote in body_format: :markdown, and note metadata.verse: true;
    • 118 continuation paragraphs: prose that runs on until the next headword paragraph belongs to the current entry;
    • alternate headwords in one paragraph (BABE or BABY, n., CONFIDANT, CONFIDANTE, n.): the entry attaches to the first, the others become alt_of relations pointing at it;
    • See X (BRUTE, n. See HUSBAND.) and [from ACADEME] become see_also / related relations with to_lemma, resolved by dd.resolve;
    • 18 Animals headwords (alligator, ass, camel, cat, dog, frog, goose, hog, hyena, magpie, monkey, mouse, ostrich, oyster, pig, seal, tortoise, turkey); A8 counts all 959 entries and their attachment to lexemes (lowercase the headword, create the lexeme if the index lacks it).

Notes from the S3 audit (2026-09-05), for whoever touches the data next

  • Grade A. Verified against the database and by running mix dd.score myself: 24 / 24 graded rows pass, 261 tests, precommit green, pushed to main cleanly. Bierce 997 entries with verse, attributions and continuations intact (STORY keeps its 26 paragraphs, PIE its two-paragraph attribution, EUCHARIST its <pre>); the S2 carry-overs are fixed and the wordnet_wikidata rung moved 6,119 → 6,663; 7 scope QIDs remain conceptless and all 7 have absent records (deleted or redirected items).
  • One parse nit, three entries: the alternate-headword entries keep the remainder of the headword phrase at the start of their body — BABE begins "or BABY, n. A misshapen…", CONFIDANT "CONFIDANTE, n. One entrusted…", TZETZE FLY "TZETZE (or TSETSE) FLY, n. An African…". Strip the whole printed headword phrase, alternates included, in materialize/1, then mix dd.materialize --source bierce --all (the M2 path; no re-absorb). Do it before U1 renders Bierce cards.
  • 125 Wikipedia records now carry changed_at, stamped during the full lemma re-probe: real page changes since S2's probe, which is the feature working. Leave them.
  • Cost note, not a defect: the concept pass now fetches a summary for every concept with a sitelink, including ~28,000 disambiguation candidates that will mostly never be asserted (22 min, 1,389 requests, and A7's denominator is 69,681). For the next scope consider limiting the concept pass to asserted concepts plus candidates promoted to 0.6, and reporting A7 on that population.
  • enrich: 1 stays until a shared limiter exists. dd.score A1 counts done runs only; runs 56, 65, 66 are dev iterations.
  • U0 (🖼️ MVP-0 UI: the minimum word-hopping interface on the Oatmeal theme — theme boilerplate first, use case, wireframes, fake-data mode #71) can start now: Bierce cards exist for cat, dog, oyster, and the data every page needs is materialized.

S4 — Pages (may split at the seam)

Split (2026-09-05): the reader-facing pages (home/search, word page, hop, provenance, sparse states, fake-data mode) are specified and built on the Oatmeal theme by #71 (sessions U0–U3); S4b keeps the developer surfaces (scope browse, source page, import dashboard, health), plain. U0 (theme boilerplate) can run right after S3.

  • Goal: six plain, structurally complete pages.

  • Work (S4a): search (incl. forms) → word page: source cards by tier with ↗ and ⓘ, graph panel, image, "may refer to", disagreement plaque → provenance drawer.

  • Work (S4b): scope browse with tree, badges, filters → source page → import dashboard → mobile check → LiveView tests.

  • Exit: X1, X2, U1–U6, O2.

  • Numbers: search p95, full-absorb wall clock, pages tested.

  • Commit: S4: pages (#70)

  • Seam: between S4a and S4b.

  • Notes from the S0 audit: WordNet relations are per sense and expanded per member pair (≈ 4 rows per synset edge). The graph panel must group by from_sense_id / to_group_key, otherwise cat shows tracked vehicle and whip among its hypernyms (they belong to other senses). Synonyms come from senses.group_key, not from relation rows.

  • Notes from the S4 audit (2026-09-06, U0 + S4b + 890a834, grade B+): every page driven in Chrome at 1280 px and 375 px — theme, fonts and dark mode correct, no console errors, no sideways scroll, Commons thumbnails load from upload.wikimedia.org, ?q=, ?has=, ?missing=, ?state=, ?taxon= and paging all correct from the URL, the five source pages, imports and the scorecard all agree with the database; 322 tests, mix precommit green, dd.score 26 / 26 graded rows. Two defects a click finds and a URL does not, both fixed in S4c below: the has/missing chips never apply, and the health page abandons its own mount. Not counted against the session: the wikidata 0 strip (a data-model truth shown badly) and page=false in taxon links.

S4c — Fixes from the S4 audit ✅ (900ea5d, 2026-09-06, done in the audit session)

  • Chips: LiveView's extractMeta sets meta.value = el.value after reading phx-value-*, so on a <button> a phx-value-value is replaced by the button's empty .value and toggle receives value: "". Rename the key (phx-value-slug, %{"kind" => kind, "slug" => slug}), never use value as a phx-value key again, and add a LiveView test that clicks #filter-has-bierce and asserts the 66-row count (the state chips work because their key is state).
  • Health page: Score.rows/1 takes 4.5 s and runs in mount/3 — on the dead render, again on the connected mount, and on every reconnect. longPollFallbackMs: 2500 abandons the websocket mid-mount (the Postgrex … client exited lines), so every visit costs three scorecards and connects over long-poll; two overlapping visits loop on "Something went wrong! Attempting to reconnect". Compute the scorecard in assign_async (or only when connected?/1), cache it in Cachex for ten minutes, keep the dead render under 200 ms. Rule from now on: nothing slower than the long-poll fallback runs in mount/3.
  • Wikidata is not a word badge: lexemes.source_ids never contains Wikidata, because it attests concepts, not words — 17,373 scope words are linked to a Wikidata concept through concept_links. Replace the 📚 wikidata 0 line with linked · 17,373 read from concept_links (status ≠ rejected), drop the two dead chips, and let the word page (U1) show Wikidata through the concept card rather than a source badge.
  • page=false: to_params/1 emits false for page 1 and ~p serialises it into the taxon links; reject falsy values there too.
  • A5 v2 (🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 v12): the exclusion widens from the binomial regex to every scientific name, any rank — the lemma equals a linked concept's taxon.scientific_name or matches the regex — with the bar raised to 90 %. Wiktionary files genus, family and order names (Archilochus, Paguridae, Therapsida) as Translingual exactly as it files binomials; today's 85.0 % sits on the bar only because those are counted as misses.
  • Exit: the two LiveView tests above; /health connects over websocket in under a second on a warm cache; dd.score A5 v2 reported; .claude/launch.json (committed in the audit) still starts the server for the next audit.
  • Done (verified in Chrome on the socket): the chip click now pushes toggle {kind: "has", slug: "bierce"} and the page reads 66 words with bierce; /health dead-renders in 3 ms, joins over websocket, the scorecard arrives async (4.5 s cold, cached ten minutes, recompute button), the detail underneath still async; A5 v2 reads 78.3 % raw · 93.0 % excl. 4,004 scientific names (bar 90 %); the browse strip reads linked to 📚 wikidata 16,014 from the rows' own concept_links rule (auto/confirmed, ≥ 0.7) and the Wikidata chips are gone; taxon and pager links carry no page=false or empty keys; /sources/wiktionary 391 ms. One self-inflicted regression caught on the way: the scientific-name set was computed once per miss (an 81 s source page) — now once per call. 322 tests, mix precommit green. Cache is off in test (cache_scorecard: false).

S5 — Prove extensibility, close

  • Goal: show that the next source, the next scope and the community layer cost what 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 says they cost.
  • Work: Johnson 1755 from the LEME TEI-XML as the sixth source → emotions toy scope from one WordNet root → write the examples + votes migration from 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 §4's sketch and confirm it touches no existing table, then delete it → final mix dd.score.
  • Exit: E1, E2, E3; every scorecard row green.
  • Numbers: the full scorecard, pasted as the closing comment on 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69.
  • Decisions from S5 planning (2026-09-06): (1) Johnson's LEME TEI-XML is committed gzipped at priv/sources/johnson/johnson-1755-leme.xml.gz (~8.8 MB in git, CC BY 4.0 with attribution on the sources row, pinned by edition and a sha256 in config), gunzipped in memory at absorb — the Bierce rule generalised: a static source is a committed file when it is under ~10 MB compressed; larger dumps live in data/ with a pinned checksum. (2) Make the entry points generic first, as S5a (about half a session): health_live.ex reads Sources.list_sources/0, dd.fixtures.capture dispatches through the source registry, a mix dd.scope.new <slug> task creates a scope row from priv/scopes/<slug>.json, and L3 keys on the scope's root concept (reported n/a when the scope has none). Then E1 is scored as one sources row, one module, one registry line, fixtures and tests, zero migrations, and E2 as zero .ex/.exs touches — the emotions scope arrives as a JSON file plus task runs, using the existing wordnet_closure and wiktionary_category rules (a wikidata_class rule over P279 is a separate, counted item if wanted). Any touch outside those lists is reported on the row, not hidden.
  • Notes from the S5 audit (2026-09-06, 9db7805 + 270e48a, grade A−): every number re-measured — 356 tests, mix precommit green, dd.score --scope animals with parity 29 / 29, schema_migrations still 2, six sources and two scopes in the database, Johnson 42,726 records / 42,726 entries / 6,780 introduced / 1,313 in Animals / 435 in Emotions, emotions 809 lexemes with 0 unreasoned, the sketch at docs/sketches/; /s/emotions, /sources/johnson, /health (six parity rows, E rows, websocket) and /admin/imports driven in Chrome with no console errors. E1's touch list is honest but not the whole cost: Johnson also forced two shared-code fixes (resolve_canonical/3 no longer lets a bare form-of row veto, and a general snapshot pin), both improvements, both migration-free. E2 is the proof Holden asked for: a scope with no taxonomy runs the same pipeline unchanged — and it shows the measures are Animals-calibrated (dd.score --scope emotions fails A4's ≥ 7,500, A10's ≥ 80 % images at 61.8 %, L1's ≥ 70 % at 49.2 % of reachable, and L3 at 0 % because it asks for Animalia). Deductions: the unscoped concept seed was found and reported but left, at 16 minutes of API and ~20 k concepts nobody links to; the bars and L3 were not generalised when the second scope made the problem visible; sources.id ran to 21 for six rows during the rollback attempt. All three are S5c below.

S5c — Fixes from the S5 audit ✅ done 2026-09-07, with #71's U1b in commit 30a1e8c

Measured after: seed_qids for emotions 414 (was 72,108) fetching 345 records in 15 s (was 28,084 in 16 min); animals 11,660 (was 92,947). dd.score --scope emotions 33 / 33 graded rows pass (was 30 / 33), --scope animals unchanged at 34 / 34. No migration. Orphans left alone as this list asks; the seed fix stops the growth. Full write-up: #71 §8b.5.

  • Scope the concept seed. Wikidata.seed_qids/2 unions concept_qids/0 = Repo.all(from c in Concept, select: c.qid), the whole table, so scope N re-walks every concept scopes 1..N−1 introduced (emotions: 72,108 seeds, 28,084 records, 569 requests, 16 min for 809 words). Seed only the concepts this scope's lexemes link to at auto/confirmed plus candidates promoted to ≥ 0.6, and stop chasing 0.40 disambiguation candidates until the linker promotes one. Validate without fetching: compare seed_qids/2 sizes for animals and emotions before and after.

  • Orphans. 35,719 concepts have no link and no relation (92,947 total). Do not delete derived rows alone — M1 would report them as gaps; trim the source_records and the rows together, or leave them and only stop growing.

  • Bars are data. A4's ≥ 7,500, A10's ≥ 80 %, L1's ≥ 70 % are Animals' numbers; read them from scope.rules["bars"] with today's values as defaults, and make L3 report n/a when the scope has no wikidata_root instead of failing at 0 %.

  • Pages take a scope. /sources/:slug hard-wires scope: "animals" and /health hard-wires scope = "animals"; /admin/imports already reads ?scope=. Same parameter on all three.

  • Exit: dd.score --scope emotions fails nothing it should not; seed_qids for emotions is hundreds, not tens of thousands; no migration. ✅ all three.

  • Opened by the fix, for its own session — chase P279, bounded. Re-absorbing emotions on the scoped seed gave 265 concepts their claims for the first time, and joy its card (a kind of happiness). But only P171 and P13176 are chased to closure, so a subclass_of edge usually points at a QID with no concepts row: 97 of 322 emotions concepts reach one chain step, 44 reach two, and ~36k edges are counted skipped_unchased. Chasing P279 for asserted concepts at a fixed depth would deepen every non-taxonomic chain on the word page. Deliberately not slipped into U1b: growing the concept graph is what S5c has just stopped doing, and this needs its own budget and its own A7 answer.

  • Commit: S5: Johnson, toy scope, scorecard green (#70)

What can run in parallel: Bierce (S3) needs only S0; the page scaffolding in S4a can start after S1 with WordNet + Wiktionary data. Everything else is sequential.


Checklist

Activity

  1. added
    documentationImprovements or additions to documentation
    enhancementNew feature or request
    on Sep 5, 2026
  2. holden commented on Sep 5, 2026

    @holden
    MemberAuthor

    S0a done (2026-09-05). The repo is reset in place and pushed; the next agent starts at S0b.

    Reset commit 58fdb81 on main
    Previous tree tag phoenix-skeleton → 2ea8124 (also carries the README rewrite)
    Old Rails app tag ruby-rails-archive → 36d3b153
    Generator phx_new 1.8.13 (latest on Hex), --no-mailer
    Framework Phoenix 1.8.13 · LiveView 1.2.11 · ecto_sql 3.14
    Toolchain Elixir 1.19.0 / OTP 28.1, pinned in .tool-versions (mise)
    Backbone deps oban 2.24.1 · req 0.7.4 · floki 0.38.4 · cachex 4.1.1 · slugify 1.3.1 · credo
    Scaffold Absorb.Source behaviour; Absorb / Lexicon / Encyclopedia / Sources / Health context stubs; directories from the structure above with .gitkeep; priv/sources/bierce/pg972.txt
    Tests mix test → 5 tests, 0 failures

    Next agent, first three things: (1) mix ecto.drop once, the skeleton's dev database still has the old tables; (2) start the two dump downloads into data/ in the background (10 MB WordNet, 2.6 GB Wiktionary, stream the latter, never decompress); (3) write the baseline migration from #69 §4 and the Materializer, then the WordNet source.

    One procedure fix learned the hard way, now in the body above: phx.new 1.8.13 runs git init, so the rsync step must --exclude .git or it wipes the remote from .git/config.

  3. holden commented on Sep 5, 2026

    @holden
    MemberAuthor

    S0b — done (commit 19bccaf, 2026-09-05)

    Thirteen-table baseline, Sources/Lexicon/Encyclopedia schemas and contexts, the Materializer, WordNet absorbed in full, the Wiktionary index pass, and the Animals scope built. 64 tests, 0 failures, offline. mix precommit green.

    Numbers

    Row Measure Result Pass
    A2 synset groups 120,564 ≥ 120,000 ✅
    A2 distinct WordNet lexemes 161,814 ≥ 155,000 ✅
    WordNet senses 212,659
    A3 lexemes with lang = en 1,534,818 ≥ 1,200,000 ✅
    A3 lexemes with forms 566,239 report
    form-of entries in the dump 534,780 of 1,487,639 English records (36%) report
    bare rows (enriched_at nil) 1,373,004
    distinct slugs (page keys) 1,374,880
    attested by 2+ sources 97,117
    A4 scope_lexemes for animals 21,277 ≥ 7,500 ✅
    A4 rows without a reason 0 0 ✅
    R1 WordNet relations resolved 1,047,918 / 1,047,918 (100%) 100% ✅
    M4 trim/1 saving on the fixtures 81% ≥ 50% ✅

    Scope by reason — wordnet_closure 7,692 · wiktionary_category 15,612 · overlap 2,027 · wikidata_taxon skipped (S2).
    The closure figure lands within 11 of the 7,681 predicted in the spec.

    Relation types — hypernym 344,049 · hyponym 337,176 · other 151,730 · holonym 105,374 · meronym 104,921 · see_also 4,668.

    Wall clock — WordNet absorb 1m40s · Wiktionary index 1m19s (10,806,865 lines streamed) · scope build 0.9s · fixture capture 54s. Well inside O2's 2 h budget.

    Five findings that amend the spec

    1. The pinned WordNet asset was wrong. english-wordnet-2025-json.zip holds 107,519 synsets / 128,009 lemmas / 135,969 (lemma, pos) pairs and fails A2. english-wordnet-2025-plus-json.zip holds 120,564 / 153,888 / 161,875 — which is exactly where 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69's "~120k synsets, ~161k words" came from. Now pinned in Sources.Catalog and data/README.md. Note also that A2's "distinct lemmas" must be read as distinct lexemes (lemma, pos): the plus edition's distinct lemma strings are 153,888, which would fail the ≥155,000 bar.

    2. WordNet ships no hyponym or holonym edges — only hypernym (93,395) and mero_* (22,312). Both inverse directions have to be derived, and inversion is global while materialize/1 must stay pure and per record. So absorb/2 inverts once and writes an expanded _edges list into each record's raw. That is also what keeps M2 honest: raw alone is enough to rebuild every derived row.

    3. 22,036 synsets carry a wikidata QID. Not a rung in §5's ladder, but free and high-signal, so it is kept in senses.metadata for S2's linker. Given L1 (≥70% link rate) is the flagged unknown, this may be the row that saves it.

    4. The en:Animals category tree is not in the Kaikki dump — records carry flat category strings with no hierarchy. Walked once over the Wiktionary API (mix dd.scope.categories animals, depth 4, 12 subtrees denylisted as about-animals-but-not-animals) and pinned as 231 categories in Sources.Catalog, so scope building stays offline and reproducible. The index pass stores each entry's en:-prefixed categories on the lexeme, which turns the category rule into one SQL query instead of a second 2.6 GB pass — and makes every future scope free of re-streaming, which is what E2 actually promises. Small deviation from 🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69's "a bare row with forms", and worth it.

    5. WordNet relations resolve inside the absorb (§4's "resolved at absorb"), as one set-based UPDATE after all senses exist. R1 is therefore green in S0b rather than S1.

    Smaller decisions

    • Sources.Catalog holds the source/scope definitions rather than seeds.exs, because seeds do not run in :test (the alias is ecto.create, ecto.migrate, test). Seeds and test fixtures share one definition; S3's dd.score A1 row has something to check against.
    • Oban tables stay deferred to S2, with the workers that need them. S0b's tasks are synchronous.
    • Added index(lexical_relations, [:from_sense_id]), which §4's index list omits — the scope closure CTE and the word-page graph panel both join on it.
    • elixir requirement bumped ~> 1.17 → ~> 1.19 to match .tool-versions.
    • Two expression unique indexes (coalesce(from_sense_id, 0), coalesce(sense_id, 0)) need conflict_target: {:unsafe_fragment, "…"} in insert_all; noted in the Materializer moduledoc since it must match the migration character for character.
    • Jason.decode!(line, strings: :copy) in the index pass is not optional: without it every retained lemma is a sub-binary pinning its whole ~2 MB parent chunk.

    What is now runnable

    mix dd.absorb wordnet
    mix dd.absorb wiktionary --index [--limit N] [--rebuild-indexes]
    mix dd.scope.categories animals [--depth N] [--dry-run]
    mix dd.scope.build animals [--reset]
    mix dd.fixtures.capture [--source S] [--lemma L] [--force]
    

    Spot check

    /define/cat has the material it needs: 10 WordNet senses across 10 synsets, and the hypernym chain walks cat → feline → carnivore → placental mammal → mammal → vertebrate → … → entity, with animal on the path.

    • S0b — Schema, Absorb.Source + Materializer tests, WordNet, Wiktionary index, Animals scope, first numbers
  4. holden commented on Sep 5, 2026

    @holden
    MemberAuthor

    S0 audited (2026-09-05). Verified directly against the database and the spec: schema matches #69 §4 column for column, no Postgres enums, raw not auto-loaded, materializer transactional and idempotent, A2/A3/A4/R1 as reported, 64 offline tests, precommit green. Commit 19bccaf is now pushed. The five S0b amendments and the scope decision (keep all 21,277 lexemes, enrich in reason order, O2 split into dump absorbs ≤ 2 h and on-demand report) are folded into #69 (v6). Notes for S1, S2, S3 and S4 are in their session sections above. S1 can start.

  5. holden commented on Sep 5, 2026

    @holden
    MemberAuthor

    S1 — done (commit dc6c326, 2026-09-05)

    The Wiktionary scoped absorb, Absorb.Resolver, Absorb.Batch, mix dd.resolve, mix dd.materialize and the first Health functions. 114 tests, 0 failures, offline. mix precommit green. Zero migrations — E1's promise, exercised early.

    Numbers

    Row Measure Result Pass
    A5 Wiktionary coverage of the Animals scope 17,773 / 21,277 = 83.5 % ≥ 85 % ❌ see below
    A9 senses with a url of their own, a record url, or a source template 243,775 / 243,775 = 100 % 100 % ✅
    M1 dd.materialize --dry-run gaps, both sources 0 over 139,815 records 0 ✅
    M4 trim/1 saving on the real scoped records 72.1 MB → 23.3 MB = 67.7 % ≥ 50 % ✅
    R2 Wiktionary edges with to_lexeme_id 91,371 / 105,731 = 86.4 % ≥ 80 % ✅
    X3 forms and variants resolve monkeys → monkey, oystre → oyster, Pekinese → Pekingese, gunnel → gunwale both ✅
    R1 WordNet edges resolved (re-checked after the refactor) 1,047,918 / 1,047,918 = 100 % 100 % ✅

    What landed — 19,251 source records · 31,116 senses · 105,731 relations · 930 canonical_lexeme_id links · 18,110 lexemes touched, of which 12,776 newly enriched_at. 16,497 of 19,987 scope lemmas matched a record in the dump.

    R2 by type — derived 48,394/58,909 · synonym 26,161/27,886 · hyponym 4,481/5,829 · related 5,052/5,601 · hypernym 3,331/3,374 · coordinate 1,756/1,859 · alt_of 936/979 · antonym 811/826 · meronym 231/243 · form_of 100/102 · holonym 77/82 · other 41/41.

    The unresolved tail is genuine, not a bug: African bush elephant, male elephant, divvo are red links or Thesaurus-only entries with no English headword. About 30 rows are dump dirt — Thesaurus section headers (Formal terms, All are derogatory unless otherwise labeled) and a gloss (Alternative spelling of dumbass.) leaking into a linkage's word field. Left as they are: #69 §4 says relations keep to_lemma forever, and R2 reports them.

    Wall clock — scoped absorb 41 s (10.8 M lines streamed a second time) · dd.resolve 2.4 s · dd.materialize --dry-run 10 s over both sources · WordNet re-absorb 1m 21s. Still nowhere near O2(a)'s 2 h.

    Spot check — cat/noun now carries 8 WordNet senses and 28 Wiktionary senses across three etymologies, 642 Wiktionary relations, IPA and etymology. dog/noun 7 + 23 and 928 relations. oyster/noun 3 + 7 and 127.

    Three findings that amend the spec

    1. A5 cannot pass as written, and nothing we build changes that. 83.5 % is the ceiling, and the 3,504 misses break down as 1,995 Linnaean binomials (Cimex lectularius, Perdix perdix, Bufo marinus), 1,299 other multiword names (woodland caribou, bird-footed dinosaur) and 210 single words (soup-fin, dickeybird). Wiktionary files binomials under Translingual, not English, and the index pass filters lang_code == "en" — so the English index can never hold them, whatever we do to the absorb. This is exactly the row #69 §7 lists under "what we expect to learn: Wiktionary coverage of WordNet's scientific and multiword names". An amendment is proposed on #69; S2's Wikidata taxon pass is what will actually cover these words.

    2. M4 measured 45.6 % on real records before the trim was made precise. The three fixtures are unusually translation-heavy, so 81 % on them was never representative. Profiling 2,000 stored payloads found entry- and sense-level categories were 36 % of everything kept — a hundred strings per record of the shape "Requests for translations into Aiton" and "English 3-letter words", of which only the en:-prefixed ones are ever read. trim/1 now narrows categories instead of only dropping whole keys, and also drops hyphenations, wikipedia, senses[].links and the bold_text_offsets arrays inside examples. That takes M4 to 67.7 %. It is no longer "four top-level keys"; what keeps it honest is the invariant materialize(trim(r)) == materialize(r), which the M4 test pins across all three fixtures.

    3. X3's first half cannot come from the scoped absorb. monkeys carries no en: categories, so it is not in the Animals scope and no record is ever fetched for it — but the index pass gave it a bare row of its own (534,780 form-of entries did), so a plain lemma lookup finds it and renders an empty page. Lexicon.lookup/1 therefore treats a bare match as not good enough: if another word claims the string among its forms, that word wins. This needs no new data and uses the lexemes_forms_index that already existed — which is what #69 §4's annotation on lexemes.forms always intended.

    Four bugs fixed on the way, each of which would have corrupted these numbers

    • Materializer.stamp_enriched_at/2 was all-or-nothing per batch. It marked every lexeme in a batch enriched if the batch produced any sense at all. Fine for WordNet, where every synset yields a sense for every member; wrong for a scoped Wiktionary batch that mixes words which gained senses with words that only had a lexeme row declared. It would have inflated A3 and put empty cards on word pages. Now per lexeme.
    • Health.Parity compared two DateTime structs with <. That is Erlang term order, which compares the day field before month before year, and it reported 113,564 of 120,564 WordNet records as needing materialization. Now DateTime.compare/2, matching the SQL predicate Absorb.Batch uses. There is a regression test with a record fetched on 30 January and materialized on 2 February.
    • Materializer.collect/2 turned an {:error, _} from a source into a MatchError instead of the clean {:error, {step, reason}} its contract promises.
    • WordNet bulk-inserted source_records with its own {:replace, …} list and so never maintained changed_at, which the "changed this week" feed will read. Both bulk paths now go through Sources.insert_records/3 and share Sources.record_conflict/0.

    Smaller decisions

    What is now runnable

    mix dd.absorb wiktionary --scope animals [--reason wordnet_closure] [--limit N]
    mix dd.resolve [--source wiktionary]
    mix dd.materialize [--source S] [--dry-run] [--all]
    
    • S1 — Wiktionary and WordNet, scoped
  6. holden commented on Sep 5, 2026

    @holden
    MemberAuthor

    S1 audited (2026-09-05). Grade: A−. Merged. dc6c326 fast-forwarded into main (branch deleted), followed by the README decision-log row.

    Verified directly, not from the report: A5 17,769 / 21,277 = 83.5 % with the misses bucketing to 1,995 binomials; A9 243,775 / 243,775; M1 0 records needing materialization across both sources; R2 91,371 / 105,731 = 86.4 %; R1 still 1,047,918 / 1,047,918; 930 canonical variants; membership by lemma honoured (cat and dog have adj/noun/verb records); reason order honoured; enriched_at on every Wiktionary-enriched lexeme; raw fully trimmed (0 records still carry translations, mean 1 KB); X3 live for monkeys, Pekinese, mice, geese, cats, gray wolf. 114 tests and mix precommit green on the branch. Zero migrations. Health functions are pure queries; parity compares by natural key; the resolver is deterministic with a pos preference and a cycle guard.

    Why not an A: the branch was left unpushed (work at risk on one laptop), and two design nits surfaced that are now #69 open items — content_hash is taken on the trimmed payload, so tightening trim/1 marked 19,250 records "changed"; and Lexicon.lookup/1 lets a form-listing word beat a bare headword. Neither blocks S2.

    The A5 amendment is accepted and folded into #69 (v7): the row now excludes lemmas Wiktionary files as Translingual and measures 92.2 %, with the raw 83.5 % and the binomials still reported; A6 now asks for the Wikidata-plus-Wiktionary union. S2 notes are in the S2 section above. S2a can start.

  7. holden commented on Sep 5, 2026

    @holden
    MemberAuthor

    S1b — done (5b9ee82 + b356876, 2026-09-05)

    The two design nits from the S1 audit, closed before S2 adds sources on the same record contract. 123 tests, mix precommit green, zero migrations.

    Row Result
    A5 (raw / amended) 17,773 / 21,277 = 83.5 % · 92.2 % excl. binomials unchanged ✅
    A9 243,775 / 243,775 = 100 % unchanged ✅
    M1 wiktionary 0 gaps over 19,251 records unchanged ✅
    M4 72.1 MB → 23.3 MB = 67.7 % unchanged ✅
    R2 91,371 / 105,731 = 86.4 % unchanged ✅
    false changed_at stamps 19,250 → 0 fixed
    form_of flagged index rows 533,218 new

    Contract change for S2: content_hash is the hash of the payload as fetched, before trim/1. A source that trims passes content_hash to Sources.insert_records/3; a source whose trim is the identity may omit it. changed_at now moves only when the source moved.

    Lookup: cats → cat (forms), CATS → the musical with cat under also, monkeys → monkey, spat keeps its own page. A bare headword outranks a forms match only on an exact-case match.

    Honesty note: 5b9ee82's message described the exact-case rule before the edit had actually landed (a scripted edit aborted on a doc anchor and the shell carried on); b356876 makes it true and adds the regression test.

    S2a can start.

  8. 15 remaining items

  9. added a commit that references this issue on Sep 6, 2026
  10. holden commented on Sep 6, 2026

    @holden
    MemberAuthor

    S4c done — 900ea5d (2026-09-06)

    Judged minor enough to do inside the audit session rather than hand back. Five fixes, each verified in Chrome on the live socket, not from URLs:

    • Chips: the has/missing chips now carry phx-value-slug; the click pushes toggle {kind: "has", slug: "bierce"} and the page reads 66 words with bierce. A test guards against any phx-value-value coming back.
    • Health page: the scorecard runs in assign_async behind a ten-minute Cachex cache (:health) with a recompute button. Dead render 3 ms, join over websocket, rows in about 4.5 s cold and instantly warm. Cache off in test.
    • A5 v2 (🔁 MVP-0: absorption-first word graph, walking skeleton (WordNet · Wiktionary · Wikidata · Wikipedia · Bierce, Animals scope) — decisions, schema, pipeline, scorecard #69 v12): miss kind scientific_name (binomial by shape, or equal to a concept's taxon.scientific_name), bar 90 %. Animals: 78.3 % raw · 93.0 % excl. 4,004 scientific names ✅. The source pages' misses by kind table follows.
    • Wikidata is linked, not attesting: the browse strip reads linked to 📚 wikidata 16,014 from the rows' own concept_links rule (auto/confirmed, ≥ 0.7, Lexicon.Browse.linked_count/1), the row glyph reads linked / unlinked, and the two dead chips are gone.
    • URLs: no page=false, no empty keys in taxon or pager links.

    One regression of my own caught before commit: the scientific-name set was evaluated once per missed word inside a capture, which made /sources/wiktionary an 81 s request; it is one query per call now (391 ms page). 322 tests, mix precommit green.

    Scorecard after S4c: 26 / 26 graded rows pass with parity, A5 at 93.0 % against 90. Next: U1 (word page and the hop) on #71.

  11. holden commented on Sep 6, 2026

    @holden
    MemberAuthor

    State of MVP-0 after S4c (2026-09-06) — what Animals has proved, what is left

    Short version. Four sessions remain, not one: S5 (the extensibility proof) and U1–U3 (the reader pages on #71). The backbone is proved on the whole English index; Animals only scopes the enrichment. What is not yet proved is a non-taxonomic scope (E2), a sixth source at zero migrations (E1), and the product itself (the word page). Recommended order: S5 next, because it answers the "does this generalise" question in one session, then U1–U3, then scale.

    Where we are

    Scorecard 26 / 26 graded rows pass, 1 report (L2), 10 pending — 7 need the word page, 3 are S5's
    Sessions S0–S3, U0, S4b, S4c done and audited; S5, U1, U2, U3 left
    Index 1,538,784 lexemes — all of English, every part of speech (879k nouns · 220k verbs · 196k names · 188k adjectives · 28k adverbs)
    Enriched 176,502 lexemes with senses (WordNet in full, Wiktionary for the scope); 25,402 in Animals
    Things 72,079 concepts (55,612 things · 16,467 taxa), 66,951 links, 40,001 relations (19,480 instance_of · 18,712 parent_taxon · 1,809 subclass_of)
    Database 2.8 GB; zero migrations since S0

    Proved, and not animal-specific

    • The absorption pattern across four access kinds (dump, API, static book, graph): raw kept and trimmed, hashed before trim, idempotent materialize/1, atomic writes, parity 0 gaps over 271,296 records, links back 100 %.
    • The index is already "everything"; only enrichment is scoped. Wiktionary senses, forms and relations for the full 1.5 M is an offline dump pass — the index pass over the same dump takes under nine minutes.
    • The linking ladder with confidence, status and candidates; disambiguation handled on 100 % of hits; conflicts surfaced.
    • Scope is data: wordnet_closure and wiktionary_category are general rules; wikidata_taxon is the animal one.
    • Every page reads the same functions the scorecard grades (U5), and the pages now survive a click test.

    Animal-flavoured, and to be proved by E2

    • The taxonomy is an extra, not the spine. parent_taxon is one of three relation types and taxa are 23 % of concepts. The browse tree and the L3 row are taxon features. A general scope gets subclass_of / instance_of and WordNet's hypernym graph instead — and the hop (🖼️ MVP-0 UI: the minimum word-hopping interface on the Oatmeal theme — theme boilerplate first, use case, wireframes, fake-data mode #71) is defined on lexical_relations, the word graph, not on taxonomy.
    • Two things animals made easy: the link rate (animals are things with articles; abstract words hit disambiguation pages and polysemy, so expect L1 lower and L4 heavier — no sense alignment, by decision) and images (92 % of taxa have one, 47 % of things).
    • Small generality gaps, all additive: a wikidata_class scope rule (P279 closure) beside wikidata_taxon; L3 generalised to "reaches the scope's root concept"; a non-taxon fallback for the browse tree; hard-coded animals defaults in the health and imports pages and the mix tasks (a ?scope= parameter).

    What is left

    Session Gives Rows
    S5 Johnson 1755 as a sixth source at zero migrations (E1); the emotions scope from one WordNet root with no code change (E2 — the direct answer to "words that are not animals"); the community migration written and shown to touch no existing table, then not shipped (E3) E1 E2 E3
    U1 the word page and the hop — the product R3 X1 U1 U2 U3 U6
    U2 home, search, provenance slide-over, sparse states X2 (word page)
    U3 fake-data mode, mobile, tests; the closing scorecard U4

    MVP-0 closes when every row is green — its own exit rule.

    Scaling to everything (MVP-1), as an estimate

    • Dictionary half: already indexed; senses, forms and relations for all 1.5 M is a dump pass of hours and roughly 10 GB.
    • Encyclopedia half: the cost driver. At today's pacing (200 ms, 20 titles per request, about 2 s end to end measured), probing the 600k–1 M headwords that could be things is one to two days of polite fetching; the Wikipedia titles / page_props dumps and the Wikidata dump would make it an afternoon, as a new access mode for two sources with no schema change — E1 rehearses exactly that shape.
    • Storage: 15–20 GB estimated; fine for the Argus NUC.
    • Not before S5 (the proof) and U1 (the word page is the real integration test of the read model, and any change it forces is cheaper on 25k words than on 1.5 M). Planning MVP-1 can start now; building it waits.
  12. holden commented on Sep 6, 2026

    @holden
    MemberAuthor

    S5 — the extensibility proof

    The three claims #69 §7 makes about the architecture rather than the data, turned from prose into measurements. E1, E2 and E3 pass. Two things the proof itself found are at the bottom, and one of them is an amendment to #69 that needs your assent.

    E1 — a new source is cheap ✅

    Johnson 1755, from the LEME TEI-XML: TSpace handle 1807/124274, edited by Ian Lancashire, CC BY 4.0 (the 1755 text is public domain; the transcription is not, so the attribution is a licence condition, not a courtesy). Committed gzipped at priv/sources/johnson/johnson-1755-leme.xml.gz — 8.8 MB packed, 29 MB open — and pinned by edition and SHA-256, which the absorb verifies before it reads a byte.

    entries 42,726 (<wordentry type="h">), 17.9 s to absorb
    distinct lemmas 37,235
    index hit rate 34,206 / 37,235 = 91.9 % already attested by another source
    words Johnson alone attests 6,780 — his own eighteenth-century spellings (domestick, chuse, ake)
    quotations 114,787 across 36,864 entries, kept in order as blockquotes with his own citations
    etymologies 36,542
    relations 114 alt_of, 707 see_also, 109 form_of
    animals-scope words he defines 1,313 — a second 👑 card on the browse page
    emotions-scope words he defines 435 of 809

    What it cost: one sources row, one people row, one line in Absorb.@modules, one module, seven fixtures and 27 tests. schema_migrations stayed at 2 — the baseline and Oban's job table — which is what E1 now measures rather than asserts. Every enum-like column is a plain string backed by Ecto.Enum, so a new tier or relation type would have cost none either.

    Before Johnson landed, the three places that hard-coded the five slugs were made generic, so the claim needs no footnote: health_live.ex reads Sources.list_sources/0 (its parity table already did), dd.fixtures.capture dispatches any static book source through one capture_static/3, and Coverage's snapshot-pin list is now general (sha256, edition) rather than ending at Bierce's gutenberg_id.

    The headword rule is the part worth reading twice. Not "everything before the first period" — wrong for To SINK pret I sunk, which has no period after the headword, and for A. Bp., which is all periods. Not "everything before the first lower-case word" either — that truncates all 565 phrase headwords (CAT in the pan, LILY of the Valley, SENSITIVE Plant, HALF-SEAS over). The run ends at a full stop or at a grammar marker, and the marker vocabulary was read off the file rather than imagined: n (972), adj (639), v (399), adv (240) leaking out of <class type="pos">, then pret, part, plur, pronoun. A comma-separated part is an alternate only if it survives the rule whole and every word of it is capitalised — otherwise METHO'UGHT, the preterite of methinks puts that sentence in the lexicon as a word, which the first pass did, 167 times.

    Three transcription conventions decide the lemma, all measured on the file: the apostrophe is a stress mark (ABA'CKE → abacke; ’ in 30,005 forms, ' in 4,549), soft hyphens mark where the printed line broke inside a word (27,339 of them — the lowest or­\r\nder is the lowest order), and a leading To is the verb marker of the period (7,793 entries).

    One thing Johnson broke and then fixed: X3 went red. He defines GEESE ("The plural of goose"), so geese stopped being a bare index row and Lexicon.lookup/1 rightly refused to send a word with a definition of its own somewhere else. The fix is to read what Johnson is telling us — 109 entries name their base word in the printed headword or the first line of the body — and emit a form_of edge, which dd.resolve turns into a canonical_lexeme_id. One further nit surfaced underneath: resolve_canonical/3 required every matching lexeme to agree, so Wiktionary's bare geese row, which has no opinion because nobody wrote one, vetoed Johnson's. A bare form-of row no longer votes. spat still keeps its own page, because spat/noun is a real headword rather than a bare row.

    E2 — a new scope is data ✅

    emotions, 809 lexemes from one WordNet root (oewn-00026390-n, feeling), 42 ms to build. wiktionary_category and wikidata_taxon skipped with reasons rather than contributing silent zeros. Then the scoped absorbs unchanged: Wiktionary 1,132 records / 3,325 senses, Wikidata, Wikipedia's 809 lemma probes, dd.link — L1 37.2 % at ≥ 0.8, 74.2 % at any confidence, honestly lower than Animals because abstract nouns often have no article of their own. L3 reads 0 %, which is the right answer for a scope with no taxonomy.

    Zero code changed to add it. Every scope is now a priv/scopes/<slug>.json file read by Catalog.scopes/0 — animals moved there too, 231 pinned Wiktionary categories and all — so there is no scope defined in Elixir to point at, and E2 measures what it claims. mix dd.scope.new <slug> --name … --rules '<json>' writes the file and the row; with no options it reads a file someone else committed.

    ScopeLive's taxonomy rail was the one page-level assumption: it hard-coded @root_qid "Q729", so /s/emotions would have drawn Animalia with fifteen zero-count children. It now reads scope.rules["wikidata_root"] and hides itself when there is none.

    E3 — the community layer fits ✅

    users + examples + votes written from #69 §4's sketch, generated with mix ecto.gen.migration, applied to the full development database, diffed against a pg_dump -s taken before them, and rolled back.

    The diff adds three tables, seven indexes and two check constraints, and changes no column, index or constraint of the thirteen. Four foreign keys point into lexemes, senses, concepts and sources without adding anything to them. users is included deliberately: there is no auth scaffolding at all, so examples.submitted_by_id and votes.user_id would have had nothing to point at, and a proof that skipped the awkward half would not be one.

    The file now lives at docs/sketches/community_layer_migration.exs, where mix ecto.migrate cannot see it and mix precommit cannot apply it to the test database. Deleting it outright, as this issue said, would have made the claim unfalsifiable.


    What the second scope found

    Two things, and only the first is fixed here.

    1. A7 v2 — an amendment to #69, for your assent

    Building emotions grew the concept table from ~44 k to 90,481 sitelinked rows. Of those, 54,273 have no link to any scope word at all — they are things a disambiguation page merely mentioned, and every summary fetched names more of them, so A7's denominator grows faster than any pass can fill it. One concept pass fetched 274 gaps and introduced 20,526 new ones.

    The S3 audit predicted exactly this and wrote the remedy on this issue: "for the next scope consider limiting the concept pass to asserted concepts plus candidates promoted to 0.6, and reporting A7 on that population." This is the next scope, so A7 is now graded on the concepts a scope word actually links to at auto/confirmed:

    10,412 / 10,412 = 100 %, with 70,224 / 90,481 = 77.6 % including candidates reported beside it.

    Same shape as the A5 v2 amendment S4c landed. If you would rather A7 kept its old denominator, say so and it goes back — but then it is a row that cannot be closed by fetching, only by not growing the graph, which is really the item below.

    2. Open defect — an unscoped concept seed, left for its own session

    Wikidata.seed_qids/2 unions an unscoped concept_qids/0:

    defp concept_qids do
      Repo.all(from c in Concept, select: c.qid)   # the whole table
    end

    The other three seeds (wordnet_qids, wiktionary_qids, root_qids) are all scoped through scope_lexemes. This one is not, so scope N re-walks every concept scopes 1..N−1 introduced. An 809-lexeme emotions scope seeded 72,108 QIDs and fetched 28,084 records in 16 minutes.

    With one scope the bug is invisible, because the whole table is that scope. The extensibility proof is what made it visible — which is the proof working, not the proof failing. Two candidate fixes: seed only the concepts this scope's own probes introduced, and stop chasing 0.40 disambiguation candidates until the linker promotes one. Both change the animals data path, so I have left them out of S5's tail rather than slipping them in.


    Numbers

    mix dd.score: 29 / 29 graded rows pass, 1 reported, 7 pending — R3, X1, U1, U2, U3, U4 and U6, every one of them #71's U1–U3. MVP-0 closes when those land; this issue's own work is done.

    Alongside the E rows: A1 6 / 6 absorbed and 6 pinned · M1 0 gaps over 344,290 records · M2 row counts identical across all six sources · A9 365,037 / 365,037 links back · X3 4 / 4 · U5 6 / 6 sources agree with dd.health.

    mix precommit green, 356 tests, zero migrations, six sources, two scopes.

    Every page driven in Chrome at 4007: /sources/johnson exists with no page code (👑 aristocracy, static, CC BY 4.0, attribution, pinned by edition, 42,726 records / 0 stale / 6,780 words introduced); /s/emotions renders with no taxonomy rail; the johnson has-chip clicks through to 435 words with johnson on emotions and 1,313 on animals; /health lists six sources in both its tables.

  13. holden commented on Sep 6, 2026

    @holden
    MemberAuthor

    S5 audit — 9db7805 + 270e48a (2026-09-06): A−, and the backbone is done

    Everything the report claims, re-measured: 356 tests, mix precommit green, dd.score --scope animals with parity 29 / 29 graded rows, schema_migrations still 2, six sources and two scopes, Johnson 42,726 records → 42,726 entries, 6,780 words of his own, 1,313 in Animals and 435 in Emotions; emotions 809 lexemes with no unreasoned row; the community migration sitting in docs/sketches/ where mix ecto.migrate cannot see it. Driven in Chrome at 4007: /s/emotions renders with no taxonomy rail and a working search, /sources/johnson exists with no page code, /health lists six parity rows and the three E rows over a websocket, /admin/imports shows six sources, no console errors.

    What passes, and one honesty note

    Why A− and not A

    1. The unscoped concept seed was found, explained, and left: emotions seeded 72,108 QIDs and fetched 28,084 records over 16 minutes for 809 words, and the concept table now carries 35,719 rows with no link and no relation. Leaving the animals data path alone in S5's tail was defensible; the cost was not small.
    2. The second scope showed the scorecard's bars are Animals' numbers (dd.score --scope emotions fails A4 at 809 < 7,500, A10, L1 and L3 at 0 % because it asks for Animalia) and the session reported L3's 0 % as "the right answer" rather than making it n/a. The generalisation was one step further than the session took.
    3. sources.id ran to 21 for six rows during the rollback attempt. Cosmetic, but it is the trace of a 12-minute cascade that should not have been started.

    All three are a short S5c in the body, to ride with U1 or run on its own.

    Where this leaves MVP-0

    #70's own work is complete: every session S0–S5 has landed and been audited, and the scorecard's 7 pending rows (R3 X1 U1 U2 U3 U4 U6) are all #71's U1–U3. This issue can close; #69 stays open until the closing scorecard lands with U3, and #71 is the tracker from here.

  14. added a commit that references this issue on Sep 6, 2026
  15. holden commented on Sep 6, 2026

    @holden
    MemberAuthor

    S5c done — 30a1e8c (2026-09-07), landed with #71's U1b

    All five items, no migration. The write-up is #71 §8b.5; the numbers:

    before after
    seed_qids for emotions 72,108 414
    records fetched 28,084 345
    time 16 min 15 s
    seed_qids for animals 92,947 11,660
    dd.score --scope emotions 30 / 33 33 / 33
    dd.score --scope animals 34 / 34 34 / 34
    • Concept seed scoped through concept_links — asserted, plus candidates at ≥ 0.6 the linker could still promote, and not the 0.40 disambiguation floor. A run with no scope still walks the whole table, because that is what a full refresh means. Validated by comparing seed sizes before any fetch, as this issue asked.
    • The Wikipedia concept pass is scoped too. It took no scope at all — concept_targets/2 selected every row with a wikipedia_title — which is the same defect wearing different clothes.
    • Bars are data. A4/A10/L1/L3 read scopes.rules["bars"] with today's Animals numbers as defaults. emotions declares 500 / 55 % / 45 %, each just under what it measures (809, 61.8 %, 49.2 %) so the row catches a regression rather than announcing a pass.
    • L3 reports, not fails, when a scope has no wikidata_root — and names the root it walked when there is one (taxonomy reaches Q729). mix dd.health, mix dd.link and the /health detail panel all handle the rootless answer; the last of those raised KeyError :pct inside its own async render until it did.
    • ?scope= on /health and /sources/:slug, both via handle_params/3 in /admin/imports's shape. /sources/johnson?scope=emotions reads 435 of 809 scope words where the animals page reads 1,313 of 25,402.
    • Orphans left alone, as this list decided: deleting derived rows alone would make M1 report them as gaps, and the seed fix stops the growth.

    One thing the fix opened, added to the S5c block above rather than done here: only P171 and P13176 are chased to closure, so a subclass_of edge usually points at a QID with no concepts row. The re-absorb gave joy its card (a kind of happiness), but 97 of 322 emotions concepts reach one chain step and 44 reach two. Chasing P279 for asserted concepts at a fixed depth would deepen every non-taxonomic chain on the word page — and it needs its own budget and its own answer for A7, so it is its own session. Slipping it into U1b would have undone the thing S5c just achieved.

  16. holden commented on Sep 7, 2026

    @holden
    MemberAuthor

    Closed with #71's U3 and the mvp-0 tag. S0–S5 are complete, S4c and S5c landed with U1b, and the scorecard reads animals 36 / 36 with parity, emotions 35 / 35, 0 pending on 487 tests. The full run is on #69.

    What this issue leaves open, and where it goes: chasing P279 for asserted concepts — the concept chain is one step deep for most non-taxa, which is why joy reaches happiness and stops — and the two concept_relations types (:part_of, :related) that sit in the enum with nothing writing them. Both are MVP-1 work and neither blocks anything shipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions