Repository navigation
Description
Status: decided 2026-09-05. This body is the MVP-0 implementation spec (v13). v1, the directional draft (source research, probe log, wireframes, deferred layers), is the first comment below. Amendment history: v3 backbone rules and media; v4 tracker #70; v6 S0b findings; v7 A5; v10 S2 (batched clients, record keying, corroboration, the L1/L3/A10/L4/A7 populations); v11 S3 (Bierce 997, A8 metric, L1 denominator computed, A1 pins, M2 from runs); v12 S4 audit (A5 v2, the mount rule); v13 S5 audit (A7 v2: graded on asserted concepts; E1βE3 measured). The living record of vision, decisions and roadmap is the repo README.
Decisions (made with Holden, 2026-09-05)
| # | Decision | Choice |
|---|---|---|
| 1 | App identity | Internal name stays devils_dictionary. Working product title Wordhoard, provisional (README β Naming). Core contexts are named neutrally: Absorb, Lexicon, Encyclopedia. |
| 2 | One app or two | One Phoenix app, one Postgres database. The contexts are the seam if it ever splits. |
| 3 | How much lexicon | Full index: every English Wiktionary headword becomes a bare lexemes row from day one. Enrichment (senses, entries, links) is scoped. |
| 4 | Sense alignment | None. Each source keeps its own senses. Sources join only through concepts. |
| 5 | Commercial | Non-commercial. MVP-0 uses only open sources. Urban Dictionary, Merriam-Webster and Datamuse come later under their non-commercial terms, on demand, never bulk-stored or exported. |
| 6 | Page identity | The word. /define/:slug shows every part of speech and every source. Concept pages later. |
| 7 | Tier | A theme, not the core. tier is a column on sources; the UI inherits it. |
| 8 | Freshness | Pinned snapshots recorded on the source row. Re-absorb by hand. No daily sync in MVP-0. |
| 9 | Environment | Reset in place (#70), done as 58fdb81: Phoenix 1.8.13 / LiveView 1.2.11, Elixir 1.19 / OTP 28 pinned in .tool-versions; stay on the current Phoenix release. Local Postgres for dev; Kamal later (Β§9). |
| 10 | Existing issues | Leave open. This issue references what it uses (Β§11). |
| 11 | Links, not content | Living sources store a short gloss/summary and a link back; full text is imported only from public-domain dictionaries; raw records are trimmed to the fields we use. |
| 12 | Humans and bots are sources | Community and bot contributions get sources rows (kind crowd / bot) plus a submitter column. Examples and media are edges from words to things/URLs with provenance and votes. |
| 13 | Product shape | Backend first, then a JSON API over the same contexts, then a progressive web app, then iOS. The LiveView site is one client, not the product. |
| β | Test scope | Animals. Only a filter. Nothing in the schema or pipeline is animal-specific. |
History: tags ruby-rails-archive (the old Rails app, 36d3b153) and phoenix-skeleton (2ea8124); main starts from the reset 58fdb81.
0. What the backbone must support (the long-term product, in one place)
Not being built now. Written down so every MVP-0 table is checked against it.
- Aggregator, not author. Every word from every open source; for each source a short definition and a link to where it came from. We never write definitions.
- Complete lexicon, living. New words arrive from any source, including crowd sources that define words no institution has. Re-absorbing a source surfaces new and changed words as feeds.
- Words β things. Monkey the word links to monkey the thing (Wikidata), which links to its taxonomy, its Wikipedia summary, its image.
- Examples from the culture. People and bots attach examples to a word: nepotism β a public figure (a thing we already have), a news story (a URL), a sentence. Votes rank them.
- Any media on top. TikToks, YouTube, posts, songs, GIFs, photos, movie scenes attach to words and things as links plus metadata and thumbnails, never as files.
- Feeds and an app. New words, changed senses, top examples, word of the day. Backend β API β PWA β iOS.
Backbone rules derived from that (each one is enforced by a column or a table in Β§4):
- Every content row points at its
source_record, which carries the canonical URL at the source. Sources also carry a URL template, so a link out always exists. - Short by default. Living sources materialize a gloss or summary + URL. Raw records are trimmed to the fields we use (for Wiktionary: no translations, descendants or templates). Only public-domain dictionaries are imported in full.
- Any source may introduce a word. Identity is
(lang, lemma, pos);posdefaults to"unknown"for crowd sources. The lexeme records which source introduced it and which sources attest it. - Provenance accepts humans and bots. They are
sourcesrows; community tables addsubmitted_by. - Everything that connects a word to a thing, a URL or a piece of media is an edge with a method or role, a confidence or score, a status and provenance.
concept_linksis the first such edge; examples, attachments and votes follow the same shape and add tables, not columns. - Media is links + metadata + thumbnails. A provider is a
sourcesrow; an unfurl or a provider API call is anenrich/2; the payload is asource_record. - Feeds are queries. Every row is timestamped; records carry a content hash and a
changed_at. - Variants resolve. Inflected forms live on the lexeme; spelling variants point at a canonical lexeme; a word with many things has candidate links from its disambiguation page.
1. What MVP-0 is
A walking skeleton of the absorption-first word graph: every mechanism the final system needs, exercised once, on real data, deliberately plain to look at. It must be functionally complete, not pretty. When it is done, adding a new source is one row, one module and one task run, and adding a new scope is one task run.
| Mechanism | Proven by |
|---|---|
| Bulk dump absorb, streamed, scoped | Wiktionary (2.6 GB gz) |
| Full-corpus absorb | Open English WordNet, Bierce |
| On-demand API absorb, rate-limited, with "source had nothing" markers | Wikidata, Wikipedia |
| Raw record β idempotent materialization β parity check | all five |
| Lexical relations with unresolved targets, resolved later | Wiktionary, WordNet |
| Word β thing linking with method + confidence, including disambiguation candidates | Wiktionary sense QIDs, Wikidata, Wikipedia |
| Encyclopedic relations (taxonomy) and images | Wikidata, Wikipedia |
| Multi-source disagreement shown, not smoothed | Wiktionary vs WordNet senses |
| A dead author as a layer | Bierce |
| Scope as data | Animals (+ one toy scope to prove generality) |
What MVP-0 is not: no auth, no votes, no examples, no media, no design pass, no Johnson/Webster/EB1911 (one is added after acceptance as the extensibility proof), no Urban Dictionary or Merriam-Webster, no Wikidata lexeme dump, no daily sync, no API, no deploy.
2. Sources to start with (five)
| Source | Role | Exact input (pinned in sources.config) |
Coverage in MVP-0 | License |
|---|---|---|---|---|
| Open English WordNet 2025, plus edition | the canonical lexical graph: synsets, glosses, hypernym and meronym edges (the hyponym/holonym inverses are derived once at absorb), similar, ILI ids, and a Wikidata QID on 22,036 synsets | english-wordnet-2025-plus-json.zip (11 MB) from the 2025-edition GitHub release. The base english-wordnet-2025-json.zip holds only 107,519 synsets and fails A2 |
full (measured: 120,564 synsets, 161,814 lexemes) | CC BY 4.0 |
| Wiktionary via Kaikki | headwords, senses (glosses/tags/topics/examples), IPA, etymology, inflected forms, derived/related/synonym/hypernym/hyponym linkages, sense-level Wikidata QIDs, topical categories | raw-wiktextract-data.jsonl.gz (2.6 GB gz, all languages), filtered to lang_code == "en"; dump and extraction dates recorded; raw trimmed (drop translations, descendants, *_templates) |
index full (every English entry β bare lexeme with forms and its en: topical categories in metadata), records scoped (Animals) |
CC BY-SA 4.0 |
| Wikidata items | concepts keyed by QID; description; sitelinks; image (P18); taxonomy (P171, P105, P225, P1843); P31/P279; the P13176 bridge from an everyday concept to its taxon item | wbgetentities, 50 QIDs per request, languages=en|mul, sitefilter=enwiki, trimmed to 15 properties (S2); parents walked to closure over P171 and P13176 only; the per-item Special:EntityData URL stays as each record's link back |
scoped | CC0 |
| Wikipedia | the encyclopedia layer: extract, short description, thumbnail, wikibase_item, disambiguation flag and candidates |
the Action API (20 titles per request, redirects resolved server-side; S2) in two passes: every scope lemma as a title probe, then every concept with an enwiki sitelink (--concepts, which is what makes A7 100 %); disambiguation pages read once for up to 100 candidate links |
scoped | CC BY-SA 4.0 |
| Bierce, The Devil's Dictionary (1911) | the layer proof; the brand | Gutenberg #972 HTML edition, checked in at priv/sources/bierce/972-h.htm; absorb/2 segments the document, materialize/1 parses each entry's own markup |
full (measured: 997 entries, 996 headwords, 247 verse blocks; 64 in Animals) | public domain |
Zero API keys, zero accounts. Every input is a file we pin or a public endpoint hit politely (identifying User-Agent, β€ 5 req/s, cached in source_records).
Deferred, in order: Wikidata lexemes dump (P5137 bridge, 614 MB) Β· Johnson 1755 (LEME TEI-XML, CC BY 4.0) Β· Webster 1913 (GCIDE 0.54) Β· EB1911 (Britannica11 corpus, CC BY-SA) Β· Merriam-Webster (1k/day, non-commercial) Β· Urban Dictionary (on demand, attributed, never stored in bulk) Β· Datamuse Β· Voltaire Β· Guardian (#63) Β· media providers (Β§5, "Media"). Details and probe results are in the v1 comment.
3. Scope: Animals
A lexeme is in animals if any rule matches. Each match is recorded in scope_lexemes.reasons.
| Rule | Reason tag | Expected size |
|---|---|---|
lemma of any synset in the OEWN hyponym closure of animal.n.01 (oewn-00015568-n), run as a recursive CTE over the stored hyponym edges |
wordnet_closure |
predicted 7,681 lemmas; measured 7,692 lexemes (5,665 only by this rule) |
Wiktionary entry carries one of the 231 categories of the en:Animals tree. The dump only has flat category strings, so the tree is walked once to depth 4 over the Wiktionary API by mix dd.scope.categories (12 subtrees denylisted: body parts, animal products, equestrian gear and sport, veterinary medicine, a toy franchise, the thesaurus mirror) and pinned in scopes.rules, so scope building stays offline |
wiktionary_category |
measured 15,612 (13,585 only by this rule) |
English P1843 common name of a Wikidata taxon whose P171 chain reaches Animalia (Q729) and that has an enwiki sitelink |
wikidata_taxon |
measured in S2 |
Membership is by lemma: if dog is in scope, its noun, verb and adjective entries are all absorbed. Measured total: 21,277 lexemes (5,665 by WordNet only, 13,585 by category only, 2,027 by both; the Wikidata rule lands in S2 and only adds reasons). The estimate was 8kβ12k; the category tree's long tail (breeds, livestock, fossil groups, species names) is the difference. Decision 2026-09-05: keep the whole scope. Enrich in reason order (WordNet closure first, category-only tail second) so the flagship numbers arrive early, and budget on-demand fetch time separately from dump absorbs (O2).
Full index, scoped enrichment. Every English Wiktionary headword exists as a lexeme row (β1.4M, including form-of entries such as plurals). WordNet is absorbed in full. Only Wiktionary records, Wikidata concepts and Wikipedia entries are limited to the scope. A word outside the scope still has a page; it is just sparse.
4. Schema
Conventions. Bigint identity ids. timestamps(type: :utc_datetime_usec). Enum-like columns are strings with Ecto.Enum, never Postgres enum types. Lemmas are case-sensitive text with a lower(lemma) index (so Turkey and turkey never merge); slugs are lowercase. Raw payloads are JSONB with load_in_query: false. One baseline migration create_lexicon_schema; pg_trgm enabled in it.
erDiagram
SOURCES ||--o{ SOURCE_RECORDS : "absorbs into"
SOURCES ||--o{ SENSES : asserts
SOURCES ||--o{ ENTRIES : publishes
SOURCES ||--o{ LEXICAL_RELATIONS : asserts
SOURCES ||--o{ CONCEPT_RELATIONS : asserts
SOURCES ||--o{ CONCEPT_LINKS : asserts
SOURCES ||--o{ IMPORT_RUNS : runs
SOURCE_RECORDS ||--o{ SENSES : materializes
SOURCE_RECORDS ||--o{ ENTRIES : materializes
LEXEMES ||--o{ SENSES : has
LEXEMES ||--o{ ENTRIES : has
LEXEMES ||--o{ LEXICAL_RELATIONS : "from"
LEXEMES ||--o{ CONCEPT_LINKS : links
LEXEMES ||--o{ SCOPE_LEXEMES : "is in"
LEXEMES o|--o| LEXEMES : "canonical variant"
SCOPES ||--o{ SCOPE_LEXEMES : contains
SENSES ||--o{ LEXICAL_RELATIONS : "from (optional)"
SENSES ||--o{ CONCEPT_LINKS : "links (optional)"
CONCEPTS ||--o{ CONCEPT_LINKS : "linked by"
CONCEPTS ||--o{ ENTRIES : "described by"
CONCEPTS ||--o{ CONCEPT_RELATIONS : "from"
CONCEPTS o|--o| CONCEPTS : "taxon item"
PEOPLE ||--o{ ENTRIES : authored
SOURCES {
bigint id PK
string slug UK
string tier "aristocracy | middle | plebs"
string kind "dictionary | encyclopedia | lexical_db | knowledge_graph | media_provider | crowd | bot | corpus"
string access "dump | api | static | user"
string url_template "link out when a record has no url"
string license
jsonb config "dump url, snapshot date, rate limit"
}
SOURCE_RECORDS {
bigint id PK
bigint source_id FK
string external_id "synset id, word/pos/etym, QID, pageid, HEADWORD/pos"
string url "canonical url at the source"
jsonb raw "trimmed, load_in_query false"
string content_hash
datetime fetched_at
datetime changed_at "hash changed on refetch"
datetime materialized_at
datetime absent_until "set on source-had-nothing markers"
}
LEXEMES {
bigint id PK
text lemma "case-sensitive, lower() index"
string pos "default unknown"
string lang "en"
string slug "not unique, the page key"
jsonb forms "inflections with tags"
jsonb pronunciations
text etymology
bigint canonical_lexeme_id FK "spelling variant to canonical"
bigint origin_source_id FK "who introduced the word"
int_array source_ids "who attests it, cached"
jsonb metadata "disambiguation title, etc"
datetime enriched_at "nil = bare index row"
}
SENSES {
bigint id PK
bigint lexeme_id FK
bigint source_id FK
bigint source_record_id FK
string external_id
string group_key "WordNet synset id, nil for Wiktionary"
text gloss
string url "optional anchor, else record url"
int position
string_array tags
string_array topics
jsonb examples
jsonb metadata "etymology_number, QIDs, ili"
}
ENTRIES {
bigint id PK
bigint source_id FK
bigint source_record_id FK
bigint lexeme_id FK "nullable"
bigint concept_id FK "nullable"
bigint author_id FK "nullable"
string headword
string pos
text body
string body_format "text | markdown | html"
string url
string thumbnail_url
int year
int position
jsonb metadata
}
LEXICAL_RELATIONS {
bigint id PK
bigint source_id FK
bigint from_lexeme_id FK
bigint from_sense_id FK "nullable"
text to_lemma
string to_pos
bigint to_lexeme_id FK "nullable until resolved"
bigint to_sense_id FK "nullable"
string to_group_key
string type "hypernym | hyponym | meronym | holonym | synonym | antonym | derived | related | coordinate | form_of | alt_of | see_also | other"
string subtype
float weight
jsonb metadata
}
CONCEPTS {
bigint id PK
string qid UK
string label
text description
string kind "taxon | thing | other"
string wikipedia_title
int wikipedia_pageid
string image_url "Wikipedia thumbnail or Wikidata P18"
string image_attribution
string wordnet_ili
jsonb taxon "rank, scientific_name, common_names"
bigint taxon_concept_id FK "everyday concept to taxon item"
jsonb metadata
}
CONCEPT_RELATIONS {
bigint id PK
bigint source_id FK
bigint from_concept_id FK
bigint to_concept_id FK
string type "parent_taxon | subclass_of | instance_of | part_of | related"
string property "P171, P279, P31"
}
CONCEPT_LINKS {
bigint id PK
bigint lexeme_id FK
bigint sense_id FK "nullable"
bigint concept_id FK
bigint source_id FK "nullable"
string method "wiktionary_qid | wordnet_wikidata | wikidata_p5137 | wordnet_ili | title_match | disambiguation | manual"
float confidence
string status "auto | candidate | confirmed | rejected"
jsonb metadata
}
SCOPES {
bigint id PK
string slug UK
jsonb rules
jsonb stats
}
SCOPE_LEXEMES {
bigint scope_id PK
bigint lexeme_id PK
string_array reasons
}
IMPORT_RUNS {
bigint id PK
bigint source_id FK
bigint scope_id FK "nullable"
string task
string status "running | done | failed"
jsonb stats
text error
datetime started_at
datetime finished_at
}
PEOPLE {
bigint id PK
string name
string slug UK
date birth_date
date death_date
string wikidata_id
bigint source_id FK "nullable"
}
Tables, column by column
# sources β one row per provider, human channel or bot. Tier, kind, access and license live HERE, never on content rows.
schema "sources" do
field :slug, :string # "wordnet" "wiktionary" "wikidata" "wikipedia" "bierce" Β· later "community" "youtube" "bierce-bot"
field :name, :string
field :tier, Ecto.Enum, values: [:aristocracy, :middle, :plebs]
field :kind, Ecto.Enum, values: [:dictionary, :encyclopedia, :lexical_db, :knowledge_graph, :media_provider, :crowd, :bot, :corpus]
field :access, Ecto.Enum, values: [:dump, :api, :static, :user]
field :era_year, :integer # 1911, 2025 β¦
field :license, :string # "CC BY-SA 4.0"
field :license_url, :string
field :homepage, :string
field :url_template, :string # "https://en.wiktionary.org/wiki/{lemma}#English" β link out when a record has no url
field :attribution, :string # the line we must show
field :active, :boolean, default: true
field :config, :map, default: %{} # dump url, snapshot/dump date, rate_limit_ms, budget, trim rules
has_many :source_records, SourceRecord
timestamps(type: :utc_datetime_usec)
end
# unique(slug)# source_records β the truth we fetched, trimmed to what we use. One row per (source, external_id). Replaced, never edited.
schema "source_records" do
belongs_to :source, Source
field :external_id, :string # see the mapping table below
field :url, :string # canonical URL of this record at the source (the "link back")
field :raw, :map, load_in_query: false # payload, trimmed per sources.config.trim
field :content_hash, :string # sha256(raw)
field :fetched_at, :utc_datetime_usec # last fetch
field :changed_at, :utc_datetime_usec # set when a refetch changes content_hash β "changed this week" feeds
field :materialized_at, :utc_datetime_usec # nil or < fetched_at = needs materialization
field :absent_until, :utc_datetime_usec # "source had nothing" marker (raw = %{}); retry after this
timestamps(type: :utc_datetime_usec) # inserted_at = first seen β "new this week" feeds
end
# unique(source_id, external_id); index(source_id, materialized_at); index(absent_until); index(changed_at)# lexemes β a word as a lexical unit: (lang, lemma, pos). The full English index lives here. Any source may create one.
schema "lexemes" do
field :lang, :string, default: "en"
field :lemma, :string # case-sensitive text; lookups use lower(lemma)
field :pos, :string, default: "unknown" # noun verb adj adv name phrase prefix β¦ normalized; "unknown" for crowd sources
field :slug, :string # slugify(lemma). NOT unique: /define/:slug shows every pos and every casing
field :forms, {:array, :map}, default: [] # [%{"form" => "monkeys", "tags" => ["plural"]}] β "monkeys" resolves here
field :pronunciations, {:array, :map}, default: [] # [%{"ipa" => "/ΛkΓ¦t/", "audio" => url, "source" => "wiktionary"}]
field :etymology, :string
field :etymology_source_id, :integer
belongs_to :canonical_lexeme, Lexeme # spelling variant β canonical (e-mail β email); #60's redirect in the new model
belongs_to :origin_source, Source # who introduced this word (wiktionary for the index; urban for a crowd-only word)
field :source_ids, {:array, :integer}, default: [] # who attests it; maintained by the materializer β "crowd-only words" is a filter
field :metadata, :map, default: %{} # wikipedia_disambiguation_title, etc.
field :enriched_at, :utc_datetime_usec # nil = bare index row (no senses/entries yet)
has_many :senses, Sense
has_many :entries, Entry
timestamps(type: :utc_datetime_usec)
end
# unique(lang, lemma, pos); index(lower(lemma)); index(slug); gin(lemma gin_trgm_ops); gin(forms); index(canonical_lexeme_id); index(enriched_at)# senses β one meaning as asserted by ONE source. Structured dictionaries land here. Always short; always linked.
schema "senses" do
belongs_to :lexeme, Lexeme
belongs_to :source, Source
belongs_to :source_record, SourceRecord
field :external_id, :string # wiktionary: "word/pos/etym#n" Β· wordnet: "oewn-02124272-n#cat"
field :group_key, :string # "same meaning" inside one source: WordNet synset id; nil for Wiktionary
field :gloss, :string # the short definition we show
field :url, :string # optional anchor (e.g. Wiktionary senseid); falls back to the record url / source template
field :position, :integer, default: 0
field :tags, {:array, :string}, default: [] # archaic slang figurative transitive β¦
field :topics, {:array, :string}, default: [] # zoology nautical β¦
field :examples, {:array, :map}, default: []
field :metadata, :map, default: %{} # etymology_number, raw_glosses, wiktionary QIDs, wordnet ili
timestamps(type: :utc_datetime_usec)
end
# unique(source_id, external_id); index(lexeme_id, source_id, position); index(group_key)# entries β a text a source published about a word or a thing. Prose sources (Bierce, Wikipedia summaries) land here.
schema "entries" do
belongs_to :source, Source
belongs_to :source_record, SourceRecord
belongs_to :lexeme, Lexeme # dictionaries attach to the word
belongs_to :concept, Concept # encyclopedias attach to the thing; both allowed
belongs_to :author, Person
field :headword, :string # as printed: "CAT"
field :pos, :string
field :body, :string # what we render: a summary for living sources, full text for public-domain ones
field :body_format, Ecto.Enum, values: [:text, :markdown, :html], default: :text
field :url, :string # canonical link out
field :thumbnail_url, :string # Wikipedia summary thumbnail β feeds
field :year, :integer
field :position, :integer, default: 0
field :metadata, :map, default: %{} # redirect_from, canonical title, extract length β¦
timestamps(type: :utc_datetime_usec)
end
# check(lexeme_id IS NOT NULL OR concept_id IS NOT NULL); unique(source_record_id, position)
# index(lexeme_id, source_id); index(concept_id, source_id)# lexical_relations β typed edges between words, with provenance, tolerant of targets we have not seen yet.
schema "lexical_relations" do
belongs_to :source, Source
belongs_to :from_lexeme, Lexeme # always set
belongs_to :from_sense, Sense # set when the source is sense-precise (WordNet, Wiktionary sense-level)
field :to_lemma, :string # always set, exactly as the source wrote it
field :to_pos, :string
belongs_to :to_lexeme, Lexeme # filled by dd.resolve; nil until then
belongs_to :to_sense, Sense # WordNet: the target synset's sense for to_lemma
field :to_group_key, :string # WordNet target synset id β UI can expand to all synonyms
field :type, Ecto.Enum, values: [:hypernym, :hyponym, :meronym, :holonym, :synonym, :antonym,
:derived, :related, :coordinate, :form_of, :alt_of, :see_also, :other]
field :subtype, :string # the source's label when :other ("instance_hypernym", "entails" β¦)
field :weight, :float, default: 1.0
field :metadata, :map, default: %{} # Wiktionary "sense" qualifier, tags
timestamps(type: :utc_datetime_usec)
end
# unique(source_id, from_lexeme_id, coalesce(from_sense_id, 0), to_lemma, type)
# index(lower(to_lemma)) WHERE to_lexeme_id IS NULL; index(from_lexeme_id, type); index(to_lexeme_id, type)# concepts β a thing, keyed by Wikidata QID. Encyclopedias attach here; words reach here through concept_links.
schema "concepts" do
field :qid, :string # "Q146"
field :label, :string
field :description, :string # Wikidata description / Wikipedia short description
field :kind, Ecto.Enum, values: [:taxon, :thing, :other], default: :thing
field :wikipedia_title, :string
field :wikipedia_pageid, :integer
field :image_url, :string # Wikipedia summary thumbnail, else Wikidata P18 via Commons β feeds, cards
field :image_attribution, :string # Commons license/author line
field :wordnet_ili, :string
field :taxon, :map # %{"rank" => "species", "scientific_name" => "Felis catus", "common_names" => [...]}
belongs_to :taxon_concept, Concept # dog (Q144, everyday concept) β Canis lupus familiaris (Q26972265, taxon item)
field :metadata, :map, default: %{}
has_many :entries, Entry
timestamps(type: :utc_datetime_usec)
end
# unique(qid); index(wikipedia_title); index(taxon_concept_id)# concept_relations β encyclopedic edges. In MVP-0: the Wikidata taxonomy.
schema "concept_relations" do
belongs_to :source, Source
belongs_to :from_concept, Concept
belongs_to :to_concept, Concept
field :type, Ecto.Enum, values: [:parent_taxon, :subclass_of, :instance_of, :part_of, :related]
field :property, :string # "P171" "P279" "P31"
timestamps(type: :utc_datetime_usec)
end
# unique(from_concept_id, to_concept_id, type, source_id); index(to_concept_id, type)# concept_links β the word β thing bridge. Every link records how we know and how sure we are. The template for every later edge.
schema "concept_links" do
belongs_to :lexeme, Lexeme # always set
belongs_to :sense, Sense # set when the method is sense-precise
belongs_to :concept, Concept
belongs_to :source, Source # who asserted it (wiktionary, wikidata); nil for title_match / disambiguation
field :method, Ecto.Enum, values: [:wiktionary_qid, :wordnet_wikidata, :wikidata_p5137, :wordnet_ili, :title_match, :disambiguation, :manual]
field :confidence, :float # 0.95 0.90 0.85 0.70 0.40β0.60 1.00 by method, adjusted by checks
field :status, Ecto.Enum, values: [:auto, :candidate, :confirmed, :rejected], default: :auto
field :metadata, :map, default: %{} # disambiguation page, competing QIDs, gloss-match score
timestamps(type: :utc_datetime_usec)
end
# unique(lexeme_id, coalesce(sense_id, 0), concept_id, method); index(concept_id); index(lexeme_id, status)# scopes + scope_lexemes β a scope is data. "animals" is one row.
schema "scopes" do
field :slug, :string # "animals"
field :name, :string
field :rules, :map # %{"wordnet_roots" => ["oewn-00015568-n"], "wiktionary_categories" => ["en:Animals"], "wikidata_root" => "Q729"}
field :stats, :map, default: %{} # counts per reason, last built
timestamps(type: :utc_datetime_usec)
end
# unique(slug)
schema "scope_lexemes" do # composite primary key (scope_id, lexeme_id)
belongs_to :scope, Scope
belongs_to :lexeme, Lexeme
field :reasons, {:array, :string}, default: [] # "wordnet_closure" "wiktionary_category" "wikidata_taxon"
timestamps(type: :utc_datetime_usec)
end
# index(lexeme_id)# import_runs β every task run leaves a row. The dashboard and dd.score read these.
schema "import_runs" do
belongs_to :source, Source
belongs_to :scope, Scope
field :task, :string # "index" "absorb" "link" "resolve" "materialize" "score"
field :status, Ecto.Enum, values: [:running, :done, :failed], default: :running
field :stats, :map, default: %{} # counts, durations, snapshot pins, dry-run gaps
field :error, :string
field :started_at, :utc_datetime_usec
field :finished_at, :utc_datetime_usec
end
# index(source_id, started_at)# people β authors of layers. Bierce in MVP-0; Johnson, Webster later.
schema "people" do
field :name, :string
field :slug, :string
field :birth_date, :date
field :death_date, :date
field :bio, :string
field :wikidata_id, :string
belongs_to :source, Source # the source this person authored, if any
timestamps(type: :utc_datetime_usec)
end
# unique(slug)Designed for, not built: the community and media layers (sketch)
Additive tables only. Every one of them is an edge with provenance, the same shape as concept_links.
# users β when auth arrives (Clerk or phx.gen.auth). Roles: reader, contributor, editor, admin.
# examples β "X is an example of this word." X is a thing we already have, a URL, a media item, or text.
schema "examples" do
belongs_to :lexeme, Lexeme # nepotism
belongs_to :sense, Sense # optional: which meaning
field :kind, Ecto.Enum, values: [:concept, :url, :media, :text]
belongs_to :concept, Concept # a public figure, a company β¦ (Wikidata-keyed)
belongs_to :media_item, MediaItem # a video, a post, a song β¦
field :url, :string # a news story
field :body, :string # the sentence, or why it fits
belongs_to :source, Source # "community" or a bot row β tier plebs, provenance intact
belongs_to :submitted_by, User
field :status, Ecto.Enum, values: [:pending, :approved, :rejected], default: :pending
field :score, :float, default: 0.0 # cached from votes
timestamps(type: :utc_datetime_usec)
end
# media_items β links + metadata + thumbnails, never files. A provider is a sources row (kind: media_provider).
schema "media_items" do
belongs_to :source, Source # youtube Β· tiktok Β· instagram Β· x Β· giphy Β· unsplash Β· spotify Β· genius Β· tmdb (via Cinegraph) Β· news Β· web
belongs_to :source_record, SourceRecord # the unfurl (oEmbed / OpenGraph) or provider API payload
field :kind, Ecto.Enum, values: [:image, :video, :audio, :article, :post, :song, :gif, :scene]
field :url, :string
field :embed_url, :string # official embed only
field :title, :string
field :author, :string # handle, channel, artist
field :thumbnail_url, :string
field :metrics, :map, default: %{} # likes, views, plays as reported; refreshed by enrich/2
field :license, :string
field :published_at, :utc_datetime_usec
timestamps(type: :utc_datetime_usec)
end
# unique(source_id, url)
# attachments β a media item β a word, a sense or a thing, with a role. (#67's evidence wall is role: :evidence.)
schema "attachments" do
belongs_to :media_item, MediaItem
field :target_type, :string # "lexeme" | "sense" | "concept"
field :target_id, :integer
field :role, Ecto.Enum, values: [:evidence, :illustration, :example, :quote_source]
belongs_to :source, Source # community, a bot, or a provider search ("giphy search: monkey")
belongs_to :submitted_by, User
field :status, Ecto.Enum, values: [:pending, :approved, :rejected], default: :pending
field :score, :float, default: 0.0
timestamps(type: :utc_datetime_usec)
end
# unique(media_item_id, target_type, target_id, role); index(target_type, target_id, status)
# votes β polymorphic; humans and bots (#17) both vote.
schema "votes" do
field :votable_type, :string # "example" | "attachment" | "sense" | "entry"
field :votable_id, :integer
belongs_to :user, User
belongs_to :bot_source, Source # a bot is a sources row
field :value, :integer # +1 / -1
timestamps(type: :utc_datetime_usec)
end
# unique(votable_type, votable_id, user_id); unique(votable_type, votable_id, bot_source_id)examples and attachments may fold into one table when built; the point here is that nothing in the thirteen MVP-0 tables changes to support them.
How each source lands
| Source | source_records.external_id / url |
lexemes |
senses |
entries |
lexical_relations |
concepts / concept_links |
|---|---|---|---|---|---|---|
| WordNet | synset id / en-word.net synset url; raw carries the synset plus a derived _edges list, because the corpus ships only hypernym and meronym edges and the inverses are computed once at absorb |
every lemma of every synset | one per (synset, lemma); group_key = synset id; no synonym rows, synonymy is co-membership of a group |
β | hypernym/hyponym/meronym/holonym, and similar/instance_hypernym/β¦ as :other with subtype; sense-precise; resolved at absorb in one set-based update (R1 green in S0) |
ili and the synset's wikidata QID in senses.metadata |
| Wiktionary | "word/pos/etymology_number" / wiktionary page url |
index pass: every English entry as a bare row with forms and metadata.wikt_categories; origin_source = wiktionary |
one per sense, scoped words only (trimmed raw) | β | entry- and sense-level linkages with to_lemma; form-of entries β form_of; resolved by dd.resolve |
sense wikidata QIDs β concept_links(wiktionary_qid, 0.95) |
| Wikidata | QID / wikidata entity url | β | β | β | β | concepts + image_url (P18) + concept_relations (P171 chased; P279/P31 recorded) + taxon + taxon_concept_id via P13176 |
| Wikipedia | the probed lemma, or concept:<QID> for the concept pass (not the pageid: a miss has no pageid, and two lemmas may share a page) / canonical article url |
β | β | one per concept (extract, description, thumbnail, url) | β | sets wikipedia_title/pageid; title_match links (0.70); a disambiguation hit stores the title on the lexeme and each candidate article as a disambiguation link at 0.40, raised to 0.60 when a Wiktionary sense gloss matches the candidate's description |
| Bierce | "HEADWORD/pos" / gutenberg url + letter anchor; raw = the entry's own <p>/<pre> fragments with their kinds, segmented once at absorb |
creates the lexeme if the index lacks it (70 words are his alone) | β | one per entry (prose + verse as markdown, attribution kept), author by person slug resolved in the same transaction |
alternates (BABE or BABY) β alt_of; "See X" β see_also; [from X] β related |
β |
Design rules. Tier is inherited from the source. Words and things are different tables joined only by concept_links. Every normalized row points at its source_record, which carries the link back; re-materialization needs no network. Relations keep to_lemma forever. A scope is a row plus a filter. Enums are strings. Humans and bots are sources.
5. Pipeline
flowchart LR
subgraph inputs["Pinned inputs"]
WN["OEWN 2025 json.zip<br/>10 MB, full"]
WK["Kaikki raw-wiktextract<br/>2.6 GB gz, lang=en, trimmed"]
BI["Bierce, Gutenberg 972<br/>static text"]
end
subgraph ondemand["On demand, Oban enrich queue, β€5 req/s"]
WD["Wikidata EntityData<br/>per QID, image P18"]
WP["Wikipedia REST summary<br/>per title, thumbnail,<br/>disambiguation candidates"]
end
IDX["dd.absorb wiktionary --index<br/>~1.4M bare lexemes + forms"]
SC["dd.scope.build animals<br/>scope_lexemes + reasons"]
RAW[("source_records<br/>trimmed raw, url, hash")]
LEX[("lexemes Β· senses Β· entries<br/>lexical_relations")]
CON[("concepts Β· concept_relations<br/>concept_links")]
WK --> IDX --> SC
WN --> SC
WN --> RAW
WK -->|"scoped records"| RAW
BI --> RAW
WD --> RAW
WP --> RAW
RAW -->|"materialize/1<br/>one transaction"| LEX
RAW --> CON
LEX -->|"QIDs from senses,<br/>title probes"| WD
CON -->|"enwiki sitelinks"| WP
LEX --> LNK["dd.link<br/>ladder β concept_links"]
CON --> LNK
LEX --> RES["dd.resolve<br/>to_lexeme_id, canonical variants"]
LNK --> SCORE["dd.score Β· dd.health<br/>PASS / FAIL table"]
RES --> SCORE
The Absorb.Source behaviour (mirrors Cinegraph's ApiProcessors.Behaviour):
@callback slug() :: String.t()
@callback absorb(scope :: Scope.t() | nil, opts) :: {:ok, stats} | {:error, term} # dump/static: stream β source_records
@callback enrich(target, opts) :: {:ok, SourceRecord.t()} | {:absent, until} | {:error, term} # api: one fetch (a lexeme, a concept, later a URL to unfurl)
@callback materialize(SourceRecord.t()) :: {:ok, %{lexemes: [], senses: [], entries: [], relations: [], concepts: [], links: []}} # pure, idempotent
@callback trim(raw :: map) :: map # drop what we never use before storing (Wiktionary: translations, descendants, templates)
@callback rate_limit_ms() :: non_neg_integer()materialize/1 is pure; the Materializer writes its output and materialized_at inside one Repo.transaction with the record. Upserts on the unique keys above; running it twice is a no-op. The same enrich/2 shape later unfurls a pasted URL (oEmbed/OpenGraph) for the media layer.
Terminal states per (source Γ target), copied from razrfly/cinegraph#1053:
| State | Definition | Action |
|---|---|---|
| Fetched | source_records row exists |
materialize; no network |
| Source-absent | marker row with absent_until in the future |
skip until then |
| Needs fetch | in scope, no row, no live marker | enqueue EnrichWorker (api) or wait for the next absorb (dump) |
| Needs materialization | materialized_at IS NULL OR < fetched_at, or a parity gap |
re-run materialize/1 locally |
Oban queues: absorb: 1, enrich: 3 (per-source rate_limit_ms inside the worker), link: 2, maintenance: 1. Two workers: AbsorbWorker and EnrichWorker. Quota or 429 β {:snooze, seconds}, never discard.
Mix tasks (every one prints numbers and writes an import_runs row):
mix dd.absorb wordnet # full
mix dd.absorb wiktionary --index # bare lexemes + forms for every English entry
mix dd.scope.build animals # scope_lexemes; prints counts per reason
mix dd.absorb wiktionary --scope animals # trimmed raw + senses + relations for scoped lemmas
mix dd.absorb wikidata --scope animals # concepts, images, taxon chains for discovered QIDs
mix dd.absorb wikipedia --scope animals # summaries + thumbnails for concepts; title probes; disambiguation candidates
mix dd.absorb bierce # full
mix dd.resolve # fill to_lexeme_id and canonical variants; prints unresolved by type
mix dd.link --scope animals # ladder; prints confidence histogram + conflicts + candidates
mix dd.materialize --source wiktionary --dry-run # parity report, raw vs derived, no network
mix dd.health # coverage per source, unresolved, parity, disputed links
mix dd.score # Β§7 as a PASS/FAIL table with actuals
Linking ladder (each link stores method + confidence; conflicts are surfaced, never resolved silently):
| # | Method | Signal | Confidence |
|---|---|---|---|
| 1 | wiktionary_qid |
Wiktionary sense carries wikidata: ["Q146"] (sparse: 752 of the 31,116 Animals senses carry one, measured in S1, so the load falls on rungs 2 and 4) |
0.95 |
| 2 | wordnet_wikidata |
the OEWN synset itself carries a wikidata QID (22,036 synsets, kept in senses.metadata) |
0.90 |
| 3 | wordnet_ili |
OEWN ILI β Wikidata P8814 where present |
0.85 |
| 4 | title_match |
Wikipedia title == lemma, page not a disambiguation, wikibase_item present, pos = noun or name (proper nouns are things too) |
0.70 |
| 5 | disambiguation |
title probe hit a disambiguation page: each linked article becomes a candidate at 0.40; raised to 0.60 when a Wiktionary sense gloss matches the candidate's description; the page title is stored on the lexeme for a "may refer to" panel |
0.40 β 0.60 |
| 6 | manual |
editor override | 1.00 |
(wikidata_p5137 joins when the lexeme dump is absorbed.) Concept β taxon: P13176 bridges an everyday concept to its taxon item (sparse: 109 in Animals); the P171 chain is walked to closure and recorded; P279/P31 are recorded, never chased.
Corroboration (S2). The QID rungs reach only ~21 % of the scope and title_match sits below L1's 0.8 bar, so a corroboration pass raises a title match when an independent signal agrees, recording which in metadata.corroboration: the concept is a taxon whose P225 or English P1843 name is the lemma (β 0.90); a QID rung already links the same pair (β 0.90, confirmed); the article's description or extract shares β₯ 2 content words with a WordNet or Wiktionary gloss (β 0.85). mix dd.link prints L1 both strict and corroborated.
Media: how any kind of media attaches (later, same machinery). A provider (YouTube, TikTok, Instagram, X, Giphy, Unsplash, Spotify/Genius, TMDb via Cinegraph, news) is a sources row of kind media_provider with its license and rate limit. Intake has two paths: a provider search run by a bot or an editor ("GIFs for monkey"), and a person pasting a link, which enrich/2 on a web source unfurls through oEmbed or OpenGraph. Either way the payload is a source_record, the normalized row is a media_items row (kind, url, official embed url, title, author, thumbnail, metrics, license), and the connection to a word, sense or thing is an attachments edge with a role, a submitter, a status and votes. We store links, thumbnails and metadata; never the file. Ranking is votes plus provider metrics, refreshed by enrich/2 on a schedule. The v1 comment's #67 flow (paste β unfurl β preview β submit) is this path.
Tests. Real raw records for cat, dog, oyster from all five sources are checked in as fixtures. Context tests never touch the network; materialize/1 and trim/1 are tested as pure functions; LiveView tests key on element ids.
6. Pages
Six pages, deliberately plain, structurally complete. Visual direction comes in the design pass after MVP-0, using #66. Tier tint on cards is the only styling MVP-0 commits to. Every card links out (β) and opens provenance (β).
2026-09-05: the reader-facing pages (home/search, word page and the hop, provenance, sparse states, a labelled fake-data mode) are specified in #71 on the Oatmeal theme (Tailwind Plus kit), theme boilerplate first. The developer surfaces stay here and plain. #66's three-tier treatment follows #71.
| Route | Page | Must show |
|---|---|---|
/ |
search | trigram search over all ~1.4M lexemes and their forms; AβZ |
/define/:slug |
word page | headword block (all pos, IPA); source cards grouped by tier, each with β and β; word-graph panel (broader / narrower / family / synonyms / taxon); image from the linked concept; "may refer to" panel when disambiguation candidates exist; "sources disagree" plaque; dashed card for not-yet-fetched on-demand sources |
/s/animals |
scope browse | taxonomy/WordNet tree; lexeme list with coverage badges (one glyph per source, tier-tinted); filters: has source X, missing source Y, disputed, crowd-only, bare |
/sources/:slug |
source page | tier, kind, license, attribution, snapshot pin, counts, coverage vs scope, sample entries |
/admin/imports |
dashboard | per source: fetched / absent / needs-fetch / needs-materialization; last run; health list; buttons for the tasks |
| drawer | provenance | raw (trimmed) JSON, external id, url, license, fetched/changed/materialized timestamps, links with method + confidence + status |
W1, word page (desktop). Real absorbed content shown.
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β β° wordhoard [ search a wordβ¦ ] Animals β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β CAT in: Animals βΊ Mammals β
β /ΛkΓ¦t/ Β· noun (also verb) 4 of 5 sources Β· 1 disagree β
β [ All ] [ π Dead ] [ π Institutions ] [ π± Crowd ] [ β provenance ] β
βββββββββββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ€
β π ARISTOCRACY OF THE DEAD β ββββββββββββββββββββββββββββ β
β βββββββββββββββββββββββββββββββββββββββββββββ β β (image Β· Wikipedia/ β β
β β A soft, indestructible automaton provided β β β Commons Β· attribution) β β
β β by nature to be kicked when things go β β ββββββββββββββββββββββββββββ β
β β wrong in the domestic circle. β β THE WORD GRAPH β
β β β Ambrose Bierce, 1911 β β β β broader β
β βββββββββββββββββββββββββββββββββββββββββββββ β cat βΊ feline βΊ carnivore βΊ β
βββββββββββββββββββββββββββββββββββββββββββββββββ€ mammal βΊ vertebrate βΊ animalβ
β π THE INSTITUTIONS β (WordNet β Wiktionary β) β
β βββββββββββββββββββββββββββββββββββββββββββββ β narrower (118) β
β β 1 Β· A mammal of the family Felidae. β β domestic cat Β· wildcat Β· β
β β 2 Β· A domesticated species (Felis catus)β¦ β β lion Β· tiger Β· ocelot β¦ β
β β β¦ 16 more senses β Wiktionary β β β β word family (402) β
β βββββββββββββββββββββββββββββββββββββββββββββ β catnap Β· catwalk Β· catcall β¦β
β βββββββββββββββββββββββββββββββββββββββββββββ β forms: cats β
β β feline mammal usually having thick soft β β synonyms β
β β fur and no ability to roar β¦ 8 synsets β β grimalkin Β· housecat Β· puss β
β β β Open English WordNet β β β β taxon (Wikidata) β
β βββββββββββββββββββββββββββββββββββββββββββββ β Animalia βΊ Chordata βΊ β
β βββββββββββββββββββββββββββββββββββββββββββββ β Mammalia βΊ Carnivora βΊ β
β β Small domesticated carnivorous mammal. β β Felidae βΊ Felis βΊ F. catus β
β β The cat, also called domestic cat and β β may also refer to β
β β house cat, is a small β¦ β β CAT scan Β· Cat (Unix) Β· β
β β β Wikipedia Β· Q146 β β β β Caterpillar Inc. (candidates)β
β βββββββββββββββββββββββββββββββββββββββββββββ β β sources disagree β
βββββββββββββββββββββββββββββββββββββββββββββββββ€ senses: Wiktionary 18 Β· β
β π± THE CROWD β WordNet 8 β
β (no crowd source in MVP-0; slot reserved) β ON THIS WORD Bierce β β
β β Wiktionary β WordNet β β
β β Wikipedia β Wikidata β β
βββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ
W3, scope browse (/s/animals).
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ANIMALS 8,412 words Β· 5 sources Β· 71% linkedβ
β [ search within animalsβ¦ ] sort: coverage βΎ filter: [has Bierce] [missing β
β Wikipedia] [disputed] [crowd-only] [bare rows] β
ββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β TREE (taxon/WordNet) β πΌ cat 5/5 π ππππ Felidae β
β βΎ Animalia β πΌ dog 5/5 π ππππ Canidae β
β βΎ Chordata β πΌ oyster 5/5 π ππππ Ostreidae β
β βΎ Mammalia β πΌ hyena 5/5 π ππππ Hyaenidae β
β βΎ Carnivora β πΌ aardvark 4/5 ππππ Orycteropodidae β
β βΈ Felidae 41 β axolotl 3/5 πππ Ambystomatidae β
β βΈ Canidae 38 β β¦ β
β βΈ Rodentia 212 β ββ no dead author wrote about 8,394 of these ββ β
β βΈ Aves 1,102 β βΉ 1 2 3 βΊβ
ββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
W5, import dashboard (/admin/imports).
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β IMPORTS Β· scope: [ animals βΎ ] β oban ok β
ββββββββββββββββββ¬βββββββββ¬ββββββββββ¬ββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ€
β source β access β fetched β absent β needs β needs β last run β
β β β β β fetch β material.β β
ββββββββββββββββββΌβββββββββΌββββββββββΌββββββββββΌβββββββββββΌβββββββββββΌβββββββββββ€
β wordnet β dump β 120,630 β β β 0 β 0 β 2h ago β β
β wiktionary β dump β 8,102 β β β 310 β 0 β 2h ago β β
β wikidata β api β 6,220 β 41 β 1,120 β 12 β β runningβ¦ β
β wikipedia β api β 5,980 β 120 β 1,300 β 0 β runningβ¦ β
β bierce β static β 966 β β β 0 β 0 β 3d ago β β
ββββββββββββββββββ΄βββββββββ΄ββββββββββ΄ββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ€
β HEALTH β 71% lexemes linked β 2,113 unresolved targets β 38 disputed links β
β β 412 disambiguation candidates β parity: 0 gaps β
β [absorb βΎ] [resolve] [link] [materialize --dry-run] [score] β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Mobile word page, source page and provenance drawer wireframes are in the v1 comment (W2, W4, W6).
7. What "done" means: the scorecard
mix dd.score computes every row that can be computed and prints PASS / FAIL with the actual number. MVP-0 is done when every row passes. Rows marked report have no threshold; their number is the finding.
| ID | Check | Measured by | Pass |
|---|---|---|---|
| A1 | all five sources absorbed | import_runs with status done + a pin per source: a dump pins its file date, a book its Gutenberg id, an API the day of its last finished run |
5 / 5 Β· measured 5 / 5 β |
| A2 | WordNet is full (plus edition) | synset groups / distinct lexemes (lemma, pos) with senses | β₯ 120,000 / β₯ 155,000 Β· measured 120,564 / 161,814 β |
| A3 | Wiktionary index is full | lexemes with lang = en; lexemes with forms |
β₯ 1,200,000; report Β· measured 1,534,818 / 566,239 β |
| A4 | scope built with reasons | scope_lexemes for animals; every row has β₯ 1 reason; counts per reason |
β₯ 7,500 and report Β· measured 21,277, 0 without reason β |
| A5 | Wiktionary coverage of scope, English headwords | scope lexemes attested by Wiktionary, excluding lemmas Wiktionary files as Translingual: a Linnaean binomial (^[A-Z][a-z]+ [a-z]+$) or a scientific name of any rank (the lemma equals a linked concept's taxon.scientific_name); the excluded set and the other misses counted and listed by kind |
β₯ 90 % of the remainder Β· S1: 92.2 % excl. binomials; after S2's scope growth 85.0 %, on the bar only because genus, family and order names (Archilochus, Paguridae, Therapsida) counted as misses. v12 (S4 audit): excluding every scientific name gives 93.0 % (4,004 of 5,506 misses excluded); bar raised to 90 %; β
S4c 900ea5d |
| A6 | Wikidata coverage | every QID referenced by a link or relation has a concepts row; plus scope coverage by Wikidata reported next to Wiktionary's, the union of the two expected to approach 100 % because the binomials A5 excludes are Wikidata taxon names |
100 % Β· union report Β· measured 42,753 referenced, 0 dangling β ; union 97.4 % |
| A7 | Wikipedia coverage | v13: concepts a scope word links to at auto/confirmed that were answered (an entry, a disambiguation page, no extract, or an absent marker); the sitelinked population reported beside it β a 0.40 candidate is a thing a page merely mentioned, and every summary names more |
100 % of asserted Β· S5: 10,412 / 10,412 β , 77.6 % incl. candidates |
| A8 | Bierce is full and attached | entries (the text holds 997, not 966); index hit rate = headwords another source already attested (attachment is 100 % by construction) | 997; hit rate report Β· measured 997, 93.0 % β |
| A9 | links back everywhere | senses and entries with a url of their own, a record url, or a source url_template |
100 % |
| A10 | images | concepts with an image_url among the concepts the scope's words link to by asserted links (auto/confirmed); the all-concepts figure reported |
β₯ 80 % Β· measured 9,607 / 10,261 = 93.6 % β (70 % over every concept with an entry) |
| M1 | parity | dd.materialize --dry-run gaps, all five sources |
0 |
| M2 | idempotent, offline | mix dd.materialize --all per source records the row counts it saw before and after in import_runs; the score reads the latest --all of every source |
identical Β· measured identical over 271,296 records, all five sources β |
| M3 | atomic writes | test injects a failure inside materialize; no record without materialized_at and no derived rows for it |
passes |
| M4 | trimmed raw | mean trimmed Wiktionary record size vs untrimmed on the fixtures; no materialized field lost | β₯ 50 % smaller; 0 lost |
| R1 | WordNet edges resolved | lexical_relations from WordNet with to_sense_id |
100 % Β· measured 1,047,918 / 1,047,918 in S0 β |
| R2 | Wiktionary edges resolved | Wiktionary edges with to_lexeme_id after dd.resolve |
β₯ 80 %, unresolved report by type |
| R3 | chains render | word page for cat shows a hypernym chain reaching animal from β₯ 2 sources |
yes |
| L1 | link rate (the unknown) | scope lexemes with β₯ 1 concept_link at confidence β₯ 0.8, as a share of the scope lexemes for which an English Wikipedia article exists; the raw share, the share with any link and the strict-ladder share reported beside it |
β₯ 70 % of the reachable set (a scope lexeme whose Wikipedia record is not an absent marker, computed by Health.links/2) Β· measured after S3: 82.5 % of 18,028 reachable β
(raw 58.6 %; any link 68.4 %; strict 25.4 %). Amended 2026-09-05: 6,840 scope lemmas have no English article at all |
| L2 | conflicts surfaced | links where two concepts score β₯ 0.7 for one lexeme/sense are listed on the dashboard | listed, count report |
| L3 | taxonomy reaches | concepts linked by asserted links (auto/confirmed) with a parent_taxon path to Animalia (Q729), directly or via taxon_concept_id; the with-candidates figure reported |
β₯ 60 % Β· measured 7,918 / 10,261 = 77.2 % β (26.6 % incl. candidates) |
| L4 | disambiguation handled | scope lemmas with a nominal lexeme whose title probe hit a disambiguation page: flag stored on the lexeme, candidates stored as candidate links (up to 100 per page) |
100 % of hits Β· measured 1,753 / 1,753 β , 39,143 candidates |
| X1 | every word has a page | 200 random index lexemes rendered via /define/:slug |
0 errors |
| X2 | search is fast | trigram prefix + fuzzy search over the full index, p95 locally | < 150 ms |
| X3 | forms and variants resolve | named probes through Lexicon.lookup/1: monkeys β monkey, cats β cat, geese β goose, spat keeps its page |
all probes Β· measured 4 / 4 β |
| U1 | six pages exist | routes + LiveView tests | all |
| U2 | the flagship words | cat, dog, oyster: β₯ 4 source cards across β₯ 2 tiers; graph panel with β₯ 3 relation types; taxon chain; image |
all three |
| U3 | provenance everywhere | every card opens the drawer with raw JSON, external id, license, timestamps | 100 % of cards |
| U4 | mobile | word page and browse at 375 px, no horizontal scroll | passes |
| U5 | coverage is legible | badges + filters on /s/animals, counts match dd.health |
passes |
| U6 | every card links out | β on every sense/entry card resolves to the source url | 100 % |
| E1 | a new source is cheap | add Johnson 1755 after acceptance: touched = 1 sources row, 1 module, fixtures, tests; migrations = 0 |
0 migrations |
| E2 | a new scope is data | build a toy scope from one WordNet root (e.g. emotions) with dd.scope.build; run the scoped absorbs |
no code change |
| E3 | the community layer fits | write the examples + votes migration from Β§4's sketch against the MVP-0 schema; do not ship it |
0 changes to existing tables |
| O1 | the scorecard runs itself | mix dd.score prints this table with actuals |
yes |
| O2 | it is fast enough | (a) empty DB β WordNet, Wiktionary index and scoped records, Bierce, scope build; laptop, dumps pre-downloaded. (b) on-demand Wikipedia + Wikidata fetches for the whole scope at β€ 5 req/s | (a) β€ 2 h wall clock (S0 took under 5 min); (b) report, expected β 2 h per source for 21k lexemes, cached forever after |
| O3 | clean and offline-testable | mix precommit green; test suite runs with no network on the checked-in fixtures |
green |
| O4 | health page | coverage per source, unresolved targets, link histogram, candidates, parity, disputed links | all present |
What you will actually be able to do when it passes: type any English word (or an inflected form of it) and get a page. For the ~8kβ12k animal words, see Bierce, Wiktionary, WordNet and Wikipedia side by side, ordered by class, with an image, the hypernym chain, the derived-word family, synonyms, the taxon chain, and a "may also refer to" list where the word is ambiguous, and click β to go to the source or β to see exactly which record it came from and how sure we are. Browse animals as a tree with badges showing who bothered to define what. Watch the dashboard while a source absorbs. Turn the network off and rebuild every derived row from raw. Add the next dead author with one module. Write the examples-and-votes migration and watch it touch nothing.
What we expect to learn (reported by dd.score): scope size (A4), index size (A3), Wiktionary coverage of WordNet's scientific and multiword names (A5), the link rate (L1), how many scope words are ambiguous on Wikipedia (L4), and how much trimming saves (M4). If L1 lands far below 70 %, the linker gets another rung before any layer is added.
8. Build order
Tracked in #70 (reset procedure, per-session expectations, directory structure, checklist). Six sessions minimum, eight likely; every session ends with numbers posted there.
| Session | Produces | Scorecard rows |
|---|---|---|
S0 Reset & measure β
(58fdb81, 19bccaf) |
fresh app in place, baseline migration, Absorb.Source + Materializer, WordNet full, Wiktionary index, Animals scope |
A2 A3 A4, plus R1 and M4-on-fixtures |
| S1 Wiktionary & WordNet, scoped | senses, forms, relations, resolver, parity, trimmed raw, links back | A5 A9 M1 M4 R1 R2 X3 |
S2a/b Things β
(4a8cfcf) |
Wikidata + Wikipedia + enrich worker, then the linking ladder | A6 A7 A10 L1βL4 |
S3 Bierce, health, score β
(3fcd293) |
Bierce layer, dd.health, dd.score, atomicity + offline re-materialize |
A8 M2 M3 O1 O3 O4 Β· first full scorecard: 24 / 24 graded rows pass |
S4 U0 + S4b β
(0430e17, 08fadbc; audited B+; S4c β
900ea5d) |
Oatmeal theme; browse, source page, dashboard, health, mobile Β· rule: nothing slower than the long-poll fallback in mount/3 |
X2 U4 U5 O2 β Β· X1 U1βU3 U6 with #71's U1βU3 |
S5 Extensibility β
(9db7805, 270e48a; audited Aβ) |
Johnson 1755, emotions, unshipped community migration; scopes as priv/scopes/*.json |
E1 E2 E3 β Β· closing scorecard β #71's U3 |
9. Hosting recommendation (for later)
Develop locally on plain Postgres (no Supabase features are used). When it deploys, use Kamal, the same as Argus and Cinegraph, and put it on the Argus Intel NUC as a second database on that Postgres instance. The dictionary DB is small (single-digit GB even with trimmed raw Wiktionary records for a few scopes), its load is bursty absorbs rather than a constant stream, and keeping it off the Cinegraph Mac Mini avoids sharing that box's PgBouncer pool and heavy Oban queues. Move to its own machine only if we go full-English with raw Wiktionary and Wikipedia, or add media at scale, or open it to the public.
10. Open items
- Product name.
devils_dictionaryis internal; Wordhoard is the working title. Shortlist: Glossa, Marginalia, Polysemy, Necrolex. The obvious.com/.app/.iodomains for all of them were already registered on 2026-09-05, so the brainstorm must start from availability. Decide before the app. - App direction. JSON API (
/api/v1/words/:slug,/api/v1/scopes/:slug, feeds) after MVP-0; PWA = the LiveView site plus a manifest and a service worker; iOS = a native client over the API. Cinegraph's Absinthe setup is the reference if GraphQL is wanted. - Media providers to evaluate when the media layer starts: YouTube Data API, TikTok oEmbed, Instagram oEmbed (Meta app review), X API (paid tiers), Giphy, Unsplash, Spotify + Genius, TMDb via Cinegraph's catalogue, news via RSS. Each is a
sourcesrow; unfurl covers the rest. - Unknowns measured in S0βS3: every figure is in Β§7.
- Resolved in S1b:
content_hashis taken on the payload as fetched, beforetrim/1. A change totrim/1therefore re-marks every record as changed (19,250 Wiktionary records gotchanged_atwhen the trim was tightened). Before feeds ship, hash the untrimmed payload, or version the trim and exclude trim-version changes fromchanged_at. - Resolved in S1b: index rows carry
metadata.form_of;Lexicon.lookup/1prefers a genuine bare headword only on an exact-case match, forms claimants underalso.Lexicon.lookup/1(S1) lets a word that lists the string among itsformswin over a bare row, which is right formonkeysand wrong for a bare headword that happens to be a form of another word. The index pass already computesform_of?; storing it asmetadata.form_ofcosts nothing and lets the lookup prefer genuine headwords. Do it in S4 if a case shows up. - Wiktionary multi-etymology words (
bankhas two noun entries): one lexeme row per (lemma, pos); senses keepetymology_numberin metadata. Revisit if the word page needs to group by etymology. - Concepts without a QID (later sources such as EB1911 headwords that match nothing):
qidstays required in MVP-0; relax to a partial unique index when needed.
11. References
- Living record of vision, decisions and roadmap: the repo
README.md. - Build tracker with the reset procedure, sessions and directory structure: π§± MVP-0 build plan: reset in place, sessions S0βS5 with expectations, directory structureΒ #70.
- Uses: Define layered encyclopedia vision and MVP planΒ #64 (three-layer vision), Architecture: Topic Relationships, Word Forms & DisambiguationΒ #60 (relationship types, empty-topic fallbacks and redirects, now
lexical_relations+canonical_lexeme_id), Design: Definition page β three-tier definitions + community evidence wall (wireframes, UI, UX)Β #66 / Evidence Wall β masonry feed of mixed culture (tweets, lyrics, reels, TikToks, YouTube) with type filtersΒ #67 (design direction and the evidence wall =attachmentswith roleevidence), Quote layer with provenance scoringΒ #65 (quotes attach tosenses), Import the Guardian's Gen Z dating glossary (1 Jan 2026, 49 terms) as a π± source: every term defines a word, 24 new word pages, the wording held until the Guardian says yesΒ #63 (Guardian as a later source row), Curator BotsΒ #17 (bots aresourcesrows that vote), π State of the project + MVP build order (dictionary β layered encyclopedia)Β #68 (state of the project; its build order is superseded by this issue). - Patterns copied from Cinegraph:
ApiProcessors.Behaviour,load_in_query: false, Maintenance + Sweeper, terminal-state predicates and atomic blob+derived writes (Backfill OMDb completely, as fast as possible (and stop the gap from silently reopening)Β cinegraph#1053), Health/Drift, import dashboard, Absinthe API. - Everything researched but deferred (Johnson, Webster, EB1911, Merriam-Webster, Urban Dictionary, Datamuse, Wikimedia Enterprise, ConceptNet, Diderot, Voltaire), the probe log, the licensing table, the Cinegraph mapping and wireframes W2/W4/W6: v1 comment below.
Activity
holden commentedon Sep 5, 2026
v1 draft (expand)
TL;DR
Change gears: stop hand-building definitions, absorb the word-graph first, then layer every dictionary and encyclopedia on top of it. This is the Cinegraph move (absorb the whole TMDb catalogue, then layer OMDb, festivals, lists on top) applied to words.
topics / definitions / topic_relationshipsmodel puts the tier on the definition row and the source in a free-text column, has no raw-record store, no notion of a sense, and no encyclopedic entity. It can't hold multiple sources honestly. There is one commit, demo seeds, and zero workers, so a reset costs nothing. (Β§2)Sourcebehaviour per provider; Oban queues; terminal-state predicates; import dashboard + health checks. (Β§6βΒ§8)This issue supersedes the build order in #68 Β§4 (which was API-first, per-word fan-out). The vision (#64), the tier joke (#46/#66), relationships UX (#60), evidence wall (#67) and quotes (#65) all still apply and sit on top of what's proposed here.
1. Where we actually are
One commit (
7c6dd33). Phoenix 1.8 / LiveView 1.1 / Postgres / Tailwind v4 / Oban / Req / Floki / Cachex, all wired. Four domain tables (topics,definitions,people,topic_relationships) +users+ Oban. Three LiveViews (HomeLive,TopicLiveat/define/:slug,BrowseLive).DictionaryAPIis an empty stub. Zero Oban workers. Seeds: 5 topics, 17 definitions. Tests: Phoenix defaults only. Full audit already in #68 Β§1; nothing has changed since.Found while researching this: the old Rails app is not lost. Its final commit is still on GitHub (dangling, but fetchable by SHA). It has all 14 API service objects and 37 models. Pin it before GitHub garbage-collects it:
app/servicesat that SHA: artsy, brainy_quotes, concept_net, datamuse, genius, giphy, knowledge_graph, open_library, poetry_db, samuel_johnson, tmdb, unsplash, urban_dictionary, wiki_quotes. Closes the "where is the Rails repo" question from #68 Β§5.2. Start over? Analysis and verdict
users+ Clerk-shapedAccountspeoplesource_id, keepwikidata_idtopicslexemes+conceptscat, noun) with the thing (encyclopedic entity: Q146). Dictionaries attach to words; encyclopedias attach to things. They need different keys (lemma+pos vs QID) and different relations (hypernym vs parent taxon)definitionsentries+senses+source_recordstopic_relationshipssense_relations+concept_relationscat β feline?) and unresolved targets (Wiktionary says hypernymcarnivorebefore we have acarnivorelexeme). #60's relationship types and UI patterns survive as a view over the new tablesTopicLive/DictionaryComponentstier stylingsource.tiercreate_clean_cri_schema). Don't carry six migrations of a model we're abandoningcat,dog,oyster)Verdict: reset the domain layer, keep the shell. Roughly two days of throwaway. The alternative (bolting
source_idandrawontodefinitions) leaves the word/thing conflation in place and we would be re-doing this in three months with data in the tables.3. The Cinegraph pattern we're copying
What worked there, translated:
source_recordsmovies.tmdb_dataraw JSONB,load_in_query: falsesource_records.rawJSONB,load_in_query: false, one row per (source, external_id)external_metrics(one row per movie Γ source Γ metric)entries/senses/*_relations(one row per lexeme Γ source Γ fact)ApiProcessors.Behaviour(process_movie,can_process?,has_data?,rate_limit_ms)Absorb.Sourcebehaviour (absorb/2bulk,enrich/2on-demand,materialize/1pure,has_data?/1,rate_limit_ms/0)run/1returning stats, thin cron Oban wrapperfetch_attempt+ cooldown) vs needs-fetch vs needs-materializationRepo.transactionaround raw + materialized writes from day oneHealth.Driftchecks scoped to canonical moviesscope(Animals)/admin/imports(Β§10 W5)4. Source research
Everything below was probed on 2026-09-05. Full probe log in the collapsible at the end of this section.
4a. The spine (bulk, open, durable)
en:Cats,en:Dogs,en:Mammals). ~1.39M English forms (964k nouns)raw-wiktextract-data.jsonl.gz(2.6 GB gz / 22.9 GB) filtered tolang_code == "en"dognoun: 23 senses, 8 hypernyms, 759 derived terms, sense QIDQ144, IPA.catnoun: 18 senses, 118 hyponyms, 402 derivedanimal.n.01closure: 4,027 synsets, 7,681 lemmas (2,607 single-word), 4,061 hyponym edgeslatest-lexemes.json.gz614 MB. Live counts: 74,860 English lexemes, 59,818 senses (P5137 coverage not yet measured; query times out, measure from dump in Phase 0)dogL1122: 3/3 senses linked (Q144 dog, Q5288153 tool, Q10538/Q9305 constellations).cat= L7Special:EntityData/Q146.jsonon demand, or SPARQL batches. Full item dump is ~100+ GB, don'tdogQ144 (everyday concept) βCanis lupus familiarisQ26972265 (taxon, P171 β Q18498 Canis lupus). Linking must handle bothwikibase_item, redirects, disambiguation flagspage/summary(200 req/s soft ceiling, returns QID + extract). Bulk: Wikimedia Enterprise free tier (since 2026-06-02: 50,000 on-demand req/month, 30 snapshot downloads/month, Structured Contents included). SQL tables if needed:page_props471 MB,redirect187 MBCatβ Q146, "Small domesticated carnivorous mammal", extract.DEPRECATED.txtin the listing). Don't build on them4b. Dictionary layers (definitions, attach to lexemes)
gcide-0.54.tar.xz14.8 MB, SGML-ish markup (<ent>,<hw>,<pos>,<def>,[1913 Webster]), ~99k words / ~160k definitions. A separate GCIDE_XML exists4c. Encyclopedia layers (articles, attach to concepts)
Category:EB1911:Science:Biology:Animals= 901 articles, a ready-made animal scope. CAT: "properly the name of the well-known domesticated feline animal usually termed by naturalists Felis domesticaβ¦"4d. Relationship helpers (weak signals, on demand)
rel_gen= hyponyms,rel_spc= hypernyms,rel_com/rel_par= holonym/meronym,rel_trg= associations,md=d= definitionsrel_spc=catβ "trounce, pass" because of verb senses). Weak signal for related chips only, never for the graphconceptnet-assertions-5.7.0.csv.gz498 MB, 34M edges, frozen since 2019relatedsignal. Not in MVP4e. Plebs layer (attach to lexemes; unlicensed, link out)
api.urbandictionary.com/v0/define)thumbs_up/down,author,written_on,permalink4f. Verdict
Spine: Kaikki + OEWN + Wikidata + Wikipedia. They are the only sources that are simultaneously complete, bulk-downloadable, openly licensed, and maintained. They give us words, senses, pronunciations, etymology, the lexical graph, the taxonomy, and the encyclopedia article without a single API key.
First layers: Bierce, Johnson 1755 (LEME), Webster 1913 (GCIDE), EB1911 (Britannica11). All public domain or CC BY(-SA), all static files, all "dead aristocracy" in the brand's sense. Then the on-demand living layers: Wikipedia summaries (π), Merriam-Webster (π, 1k/day), Urban Dictionary (π±).
Probe log (2026-09-05)
5. The MVP vertical slice: Animals
Why animals: bounded (thousands, not millions), every source has coverage, the encyclopedic relationships are a real tree (taxonomy) rather than fuzz, the dictionary relationships are rich (cat β feline β carnivore β mammal β animal; 402 derived terms), and Bierce, Johnson and Britannica all wrote about them. It exercises every layer and every relationship type in the schema.
Scope definition (a headword is "in Animals" if any of):
animal.n.01hyponym closure (7,681 lemmas).en:Animalscategory tree (en:Mammals,en:Birds,en:Fish,en:Dogsβ¦).P1843common name of a Wikidata taxon under Animalia (Q729) that has an enwiki sitelink.Each membership is recorded with its reason (
scope_lexemes.reason). Expect ~8β12k headwords after dedupe; the exact number is Phase 0's first measurement. Layers (Bierce, Johnson, Webster, EB1911, WordNet) are small enough to absorb in full; only Wiktionary and Wikidata are filtered to scope.Worked example,
cat, everything below is real absorbed data:cat.n.01"feline mammal usually having thick soft fur and no ability to roar: domestic cats; wildcats"; hypernym feline; hyponyms domestic cat, wildcat; 8 noun synsets totalThe page for
catshould show all of that, in tier order, with provenance one click away. That is the MVP.6. Architecture
Oban queues (mirror Cinegraph's DB-protective defaults):
absorb: 1(bulk streams, one at a time),enrich: 3(per-source rate limits inside the worker),link: 2,maintenance: 1.Mix tasks (every component must be runnable and produce a number, the Argus/Cinegraph rule):
7. Schema
erDiagram SOURCES ||--o{ SOURCE_RECORDS : "absorbs into" SOURCES ||--o{ ENTRIES : publishes SOURCES ||--o{ SENSES : asserts SOURCES ||--o{ SENSE_RELATIONS : asserts SOURCES ||--o{ CONCEPT_RELATIONS : asserts SOURCES ||--o{ SENSE_CONCEPT_LINKS : asserts SOURCES ||--o{ IMPORT_RUNS : runs SOURCE_RECORDS ||--o{ ENTRIES : materializes SOURCE_RECORDS ||--o{ SENSES : materializes LEXEMES ||--o{ ENTRIES : has LEXEMES ||--o{ SENSES : has LEXEMES ||--o{ SCOPE_LEXEMES : "is in" SCOPES ||--o{ SCOPE_LEXEMES : contains SENSES ||--o{ SENSE_RELATIONS : "from" SENSES ||--o{ SENSE_CONCEPT_LINKS : "links to" CONCEPTS ||--o{ SENSE_CONCEPT_LINKS : "linked by" CONCEPTS ||--o{ ENTRIES : "described by" CONCEPTS ||--o{ CONCEPT_RELATIONS : "from" PEOPLE ||--o{ ENTRIES : authored SOURCES { int id PK string slug "wiktionary, wordnet, bierce, johnson-1755 ..." enum tier "aristocracy | middle | plebs" enum kind "dictionary | encyclopedia | lexical_db | knowledge_graph | crowd | corpus" enum access "dump | api | static" int era_year string license map config } SOURCE_RECORDS { int id PK int source_id FK string external_id "L1122, oewn-02124272-n, Q146, 'cat' ..." jsonb raw "load_in_query false" datetime fetched_at datetime materialized_at string content_hash } LEXEMES { int id PK citext lemma string pos string slug string lang "en" jsonb pronunciations text etymology } SENSES { int id PK int lexeme_id FK int source_id FK int source_record_id FK string external_id "senseid, synset id, L7-S1" text gloss int position string_array tags string_array topics } ENTRIES { int id PK int source_id FK int source_record_id FK int lexeme_id FK "nullable" int concept_id FK "nullable" int author_id FK "nullable" string headword string pos text body string body_format "text | markdown | html" string url int year int position } SENSE_RELATIONS { int id PK int from_sense_id FK int to_sense_id FK "nullable until resolved" citext to_lemma string to_pos enum type "hypernym | hyponym | meronym | holonym | synonym | antonym | derived | related | coordinate | form_of | alt_of | see_also" int source_id FK float weight } CONCEPTS { int id PK string qid UK string label text description enum kind "taxon | thing | other" string wikipedia_title string wordnet_ili jsonb taxon "rank, name, common_names" } CONCEPT_RELATIONS { int id PK int from_concept_id FK int to_concept_id FK enum type "parent_taxon | subclass_of | instance_of | part_of | related" int source_id FK } SENSE_CONCEPT_LINKS { int id PK int sense_id FK int concept_id FK enum method "wiktionary_qid | wikidata_p5137 | title_match | wordnet_ili | manual" float confidence int source_id FK } SCOPES { int id PK string slug "animals" jsonb rules } SCOPE_LEXEMES { int scope_id FK int lexeme_id FK string reason "wordnet_closure | wiktionary_category | wikidata_taxon" } IMPORT_RUNS { int id PK int source_id FK string scope enum status jsonb stats datetime started_at datetime finished_at } PEOPLE { int id PK string name date birth_date date death_date string wikidata_id }Design rules baked into this:
sources. A definition is aristocratic because Bierce is, not because someone set a column.entries.tieris a view/virtual field. (π State of the project + MVP build order (dictionary β layered encyclopedia)Β #68 Β§3's mapping aristocracy = canonical, middle = curated, plebs = public still holds.)lexemesare words,conceptsare things. Dictionaries produceentrieswithlexeme_id; encyclopedias produceentrieswithconcept_id; some (EB1911 headwords) get both. The word page joins acrosssense_concept_links.source_records.rawis the truth we fetched. Every normalized row points back to its record.mix dd.materialize --dry-runreports parity per fact type, exactly likebackfill_from_jsonbin Cinegraph. Re-materialization needs no network.to_lemmais always set;to_sense_idis filled by a maintenance sweeper once the target lexeme exists (ResolveRelationTargets). Architecture: Topic Relationships, Word Forms & DisambiguationΒ #60'sword_form / synonym / related / see_also / antonym / broader / narrowerare a presentation mapping overtype(derived|form_of β word form,hypernym β broader,hyponym β narrower).mix dd.scope.build emotions, not a refactor.evidence(Evidence Wall β masonry feed of mixed culture (tweets, lyrics, reels, TikToks, YouTube) with type filtersΒ #67) βlexeme_id;quotes(Quote layer with provenance scoringΒ #65) βsense_id/lexeme_id;votesβ polymorphic;bots(Curator BotsΒ #17) β votes.Indexes that matter:
lexemes (lemma, pos, lang)unique;lexemes lemma gin_trgm;source_records (source_id, external_id)unique;entries (lexeme_id, source_id),entries (concept_id, source_id);sense_relations (from_sense_id, type),(to_lemma)for resolution;concepts (qid)unique;sense_concept_links (sense_id, concept_id, method)unique.Migration plan: one baseline migration
create_absorption_schemareplacing the four domain migrations; keepusers,people(altered), Oban, pg extensions. Seedsources(12 rows) in a migration orseeds.exs.8. Pipeline
flowchart LR subgraph bulk["Bulk dumps (static or weekly)"] K["Kaikki Wiktionary<br/>502 MB gz"] W["OEWN 2025<br/>10 MB"] WL["Wikidata lexemes<br/>614 MB gz"] ST["Bierce Β· Johnson Β· Webster Β· EB1911<br/>static files"] end subgraph scope["Scope"] S["scope_lexemes<br/>(animals)"] end subgraph raw["Raw (truth we fetched)"] R[("source_records<br/>raw JSONB")] end subgraph norm["Materialized (derived, idempotent)"] L[("lexemes Β· senses<br/>entries Β· relations")] C[("concepts Β·<br/>concept_relations")] end subgraph od["On demand (per word, rate-limited)"] WI["Wikipedia REST summary"] WD["Wikidata EntityData / SPARQL"] MW["Merriam-Webster 1k/day"] UD["Urban Dictionary TTL"] DM["Datamuse weak rels"] end W --> S K --> S S --> K K --> R W --> R WL --> R ST --> R R --> |"materialize/1 in one transaction"| L L --> |"QIDs discovered<br/>(Wiktionary sense, P5137, sitelink)"| WD WD --> R R --> C C --> WI WI --> R L --> |"mix dd.enrich"| MW L --> UD L --> DM MW --> R UD --> R DM --> R L --> LK["Linker: resolution ladder<br/>sense_concept_links + confidence"] C --> LK LK --> H["Health: coverage per source,<br/>unresolved targets, parity, drift"]Terminal states per (source Γ lexeme), copied from razrfly/cinegraph#1053 because they were load-bearing there:
source_recordsrow exists for this source and external idfetch_attemptmarker row (raw{}+absent_until) inside the cooldownenrich(API sources) or wait for the next bulk absorb (dump sources)materialized_at < fetched_ator parity gapRaw and derived writes commit in one transaction so the "orphaned blob" class of bug never exists here.
9. Linking words to things (the hard part, done honestly)
The dictionary side keys on (lemma, pos); the encyclopedia side keys on QID. Resolution ladder, highest confidence first, every link stored with
methodandconfidence:wiktionary_qidwikidata: ["Q144"]dogsense 1 β Q144, sense 2 β Q25324wikidata_p5137P5137β item; align to our sense by gloss similaritywordnet_iliP8814(WordNet 3.1 synset id) where presentdog.n.01ILI i46360title_matchwikibase_itemfrom REST/page_props; POS must be nounCatβ Q146; needs disambiguation guardmanualThen concept β taxon: Wikidata models
cat(Q146, the everyday concept) separately from Felis catus (the taxon). WalkP279/P31from the concept and matchP1843English common names against the lemma to attach the taxon and itsP171chain. Store the taxon onconcepts.taxonand the chain inconcept_relations.Disagreement is content, not an error. Wiktionary says 18 senses, WordNet says 8, Wikidata lexeme says 1; the UI shows the count per source and lets the reader pick a lens. Conflicting links (two concepts with confidence β₯ 0.7 for one sense) are surfaced in
/admin/importsand on the page as "sources disagree".10. Wireframes and UX
Low-fidelity, orientation only. Visual direction stays #66's: gilded π, corporate π, group-chat π±. What changes is that every card is a source card, and the word graph is a first-class panel, not a footer.
W1 β Word page, desktop (
/define/cat)Notes: tier filter chips at the top swap the stack in place (#66 option B) while the graph panel stays put (#67's "stays put while the definitions argue"). Every card has β β W6. Dashed card = a source that exists but hasn't been fetched for this word (on-demand budget). "Sources disagree" is a feature, styled as a small warning plaque.
W2 β Word page, mobile
The graph panel becomes a bottom sheet triggered by the
[Graph]chip.W3 β Browse a scope (
/animals)Coverage badges (one glyph per source, tinted by tier) make the shape of knowledge visible at a glance: dead men wrote about cats and oysters, nobody dead wrote about the axolotl. That's the joke, in data.
W4 β Source page (
/sources/johnson-1755)W5 β Import dashboard (
/admin/imports, dev-only until auth exists)W6 β Provenance drawer (any β)
UX principles
sourcesrow and a parser; the page already knows how to dress him.[fetch], never a spinner that hides a 1,000/day limit./), AβZ browse and Architecture: Topic Relationships, Word Forms & DisambiguationΒ #60's empty-topic fallbacks carry over unchanged.11. Build order
Phase 0 β Decide & reset (β2 days)
rails-final) so it can't be garbage-collected.users,people, Oban, extensions.Absorb.Sourcebehaviour +sourcesseed (12 rows with tier/kind/access/license/era).cat,dog,oysterfrom every source (checked in, tiny) so tests never touch the network.mix dd.scope.build animalsβ first real number for scope size; measure P5137 coverage and Wiktionaryen:Animalstree size from the dumps.Phase 1 β Absorb the spine, scoped to Animals
lang_code=enβ§ lemma β scope; keep the all-language raw path since the per-language file is deprecated) β lexemes, senses, pronunciations, etymology, relations (with unresolvedto_lemma), sense QIDs, categories./admin/importsv0: counts per source per state.Phase 2 β Relations and linking
ResolveRelationTargetssweeper (fillsto_sense_id).TopicRelationship-style presentation view (Architecture: Topic Relationships, Word Forms & DisambiguationΒ #60 types) oversense_relations.Phase 3 β The dead speak (static layers, full corpora)
see_also, as Architecture: Topic Relationships, Word Forms & DisambiguationΒ #60 asked).<ent>in the C file alone).wikidata_id.Phase 4 β UI
lexemes(pg_trgm), AβZ browse re-pointed.Phase 5 β Living layers, budgets, health
{:snooze, 3600}).mix dd.health+ drift checks + parity watchdog;mix dd.enrich <word>demo path.Deferred (unchanged from #68)
Clerk auth Β· votes Β· evidence wall backend (#67) Β· quotes with provenance (#65) Β· curator bots (#17) Β· Voltaire/Diderot layers Β· media sources (TMDB/Giphy/Unsplash) Β· other scopes (emotions, politics, food are one
scope.buildeach).12. MVP acceptance criteria
mix dd.scope.build animalsyields β₯ 8,000 lexemes, each with a recorded reason.sense_concept_linkwith confidence β₯ 0.8; conflicts are listed, not hidden./define/cat,/define/dog,/define/oystereach render β₯ 7 source cards across all three tiers, the word graph, and the taxon chain, with β provenance on every card.mix dd.materialize --dry-runreports 0 parity gaps and re-materialization reproduces every normalized row fromsource_records./animalsbrowses the taxonomy tree with coverage badges and filters./admin/importsshows per-source terminal-state counts and health is green for the spine.sourcesrow + one module implementingAbsorb.Source+ one mix task run. Proven by adding Voltaire last.13. Open decisions (recommendations inline)
aristocracy / middle / plebsassources.tier? Yes. It's the brand; π State of the project + MVP build order (dictionary β layered encyclopedia)Β #68 already decided this.14. Licensing summary
15. Issue map
sense_relations), Design: Definition page β three-tier definitions + community evidence wall (wireframes, UI, UX)Β #66 (definition page look and feel), Evidence Wall β masonry feed of mixed culture (tweets, lyrics, reels, TikToks, YouTube) with type filtersΒ #67 (evidence wall attaches tolexemes), Quote layer with provenance scoringΒ #65 (quotes attach tosenses), Import the Guardian's Gen Z dating glossary (1 Jan 2026, 49 terms) as a π± source: every term defines a word, 24 new word pages, the wording held until the Guardian says yesΒ #63 (Guardian is one moresourcesrow), Curator BotsΒ #17 (bots vote on rows that now have provenance).referenceso day-to-day is this issue + Design: Definition page β three-tier definitions + community evidence wall (wireframes, UI, UX)Β #66/Evidence Wall β masonry feed of mixed culture (tweets, lyrics, reels, TikToks, YouTube) with type filtersΒ #67.ApiProcessors.Behaviour, Maintenance + Sweeper,load_in_query: false, terminal-state predicates (Backfill OMDb completely, as fast as possible (and stop the gap from silently reopening)Β cinegraph#1053), Health/Drift.[-]π MVP reset: absorb the word-graph first (Wiktionary Β· WordNet Β· Wikidata Β· Wikipedia), layer dictionaries on top β Animals vertical slice (sources, schema, pipeline, wireframes)[/-][+]π MVP-0: absorption-first word graph, walking skeleton (WordNet Β· Wiktionary Β· Wikidata Β· Wikipedia Β· Bierce, Animals scope) β decisions, schema, pipeline, scorecard[/+]Living record: README rewrite (vision, decision log, roadmap, naming)β¦
S0a: reset in place to a fresh Phoenix 1.8.13 app for MVP-0 (#69, #70)
holden commentedon Sep 5, 2026
Spec amendments from S0b (2026-09-05, commit
19bccaf)Five things the real data contradicted. Tracker and full numbers: #70.
Β§2 / Β§7 A2 β the pinned WordNet asset was wrong.
english-wordnet-2025-json.zipholds 107,519 synsets, 128,009 lemmas, 135,969 (lemma, pos) pairs, and fails A2 by ~13k synsets.english-wordnet-2025-plus-json.zip(11 MB, same CC BY 4.0, same format) holds 120,564 / 153,888 / 161,875 β which is exactly this issue's "~120k synsets, ~161k words". Now pinned. A2's second metric must also be read as distinct lexemes (lemma, pos), not distinct lemma strings: the plus edition's distinct lemma strings are 153,888, which would fail the β₯155,000 bar, while its (lemma, pos) pairs are 161,875. Measured: 120,564 groups and 161,814 lexemes (the 61-row gap is adjective satellites collapsing ontoadj).Β§4 "How each source lands" β WordNet has no
hyponymorholonymedges. The corpus carries onlyhypernym(93,395) andmero_*(22,312). Both inverse directions must be derived. Inversion is inherently global, whilematerialize/1is pure and per record, so the absorb inverts once and writes an expanded_edgeslist intosource_records.raw. That is also what makes M2 true in the strong sense:rawalone rebuilds every derived row. Result: 1,047,918 relations, 100% resolved (R1 green in S0b, not S1).Β§5 linking ladder β WordNet ships 22,036 synsets with a
wikidataQID. Not listed as a rung. It costs one line to keep, so it now lives insenses.metadata. Given L1 (β₯70% link rate) is the flagged unknown of this project, this is plausibly the rung that saves it; suggest addingwordnet_wikidatato the ladder at ~0.90 when S2 writes theLinker.Β§3 rule 2 β the
en:Animalscategory tree is not in the Kaikki dump. Records carry a flat array of category strings ("en:Corvids" next to "English countable nouns") with no hierarchy. The tree has to come from the Wiktionary API. It is now walked once (mix dd.scope.categories animals, depth 4, 12 subtrees denylisted as about-animals-but-not-animals: body parts, animal products, equestrian equipment and sport, veterinary medicine, a toy franchise) and pinned as 231 categories inscopes.rules, so scope building itself stays offline (O3).Two consequences worth recording in Β§4:
en:-prefixed categories inlexemes.metadata["wikt_categories"]. Β§4 describes an index row as carrying onlyforms. Without the categories, every scope build needs a second full 2.6 GB pass; with them, the rule is one SQL query against a GIN index β which is what E2 ("a new scope is data, no code change") actually promises.lexical_relationsneedsindex(from_sense_id), which Β§4's index list omits. The scope closure CTE and the word page's graph panel both join on it.Β§7 M4 β trimming is much better than the bar. Dropping
translations,descendants,etymology_templates,head_templatessaves 81% on the checked-in fixtures (needs β₯50%). Fixtures are stored untrimmed on purpose, so the denominator survives.Measured, from Β§10's "unknowns to measure in S0βS2"
wordnet_closure7,692 (spec predicted 7,681),wiktionary_category15,612, overlap 2,027Still open: A5 (Wiktionary coverage of WordNet's multiword and scientific names) in S1, L1/L4 in S2.
S0b: baseline schema, WordNet + Wiktionary index, animals scope (#70)
Decision log: S0 audited, keep the 21k Animals scope, split the O2 buβ¦
holden commentedon Sep 5, 2026
Proposed amendment to scorecard row A5 (from S1, commit
dc6c326)A5 as written cannot pass, and no work on our side changes that. Measured on the real data after the scoped Wiktionary absorb:
Why
All 3,504 misses are nouns. Bucketed:
Cimex lectularius,Perdix perdix,Bufo marinus,Mugil cephaluswoodland caribou,bird-footed dinosaur,dusky-footed wood ratsoup-fin,dickeybird,percoidean,groenendaelThe binomials are the story. They enter the scope through the
wordnet_closurerule β WordNet lists scientific names as lemmas β and Wiktionary files binomials under Translingual, not English. Decision #3 and the index pass both key onlang_code == "en", so a binomial can never acquire an Englishlexemesrow and can never be covered by this source.domestic catis in Wiktionary and is covered;Felis catusis Translingual and is not. Chasing the last 1.5 % would mean adding a second language to an index the spec defines as "every English Wiktionary headword", for words S2 is about to cover properly from Wikidata anyway.Β§7 already anticipated this: A5 is listed under "what we expect to learn: Wiktionary coverage of WordNet's scientific and multiword names". The number is the finding.
Proposed wording
Two things that follow, if this is accepted:
P225scientific name,P171parent taxon) covers well, so S2 should report scope coverage forwikidataalongsidewiktionaryand the two together should approach 100 %. Worth adding as a report line rather than a new row.mix dd.scoreprints both numbers, raw and adjusted, so the exclusion is never invisible:A5 83.5% raw Β· 92.2% excluding 1,995 Translingual binomials.If the preference is instead to leave A5 at β₯ 85 % of everything and let it sit red until S2, that is also a defensible read β the row would then be the thing that forces the Wikidata pass to be good. Either way S1 leaves the number measured, bucketed and reproducible (
DevilsDictionary.Health.coverage("animals", "wiktionary")).Full S1 numbers and the other two spec amendments (
trim/1narrowing rather than dropping, which takes M4 from 45.6 % to 67.7 % on real records; and X3 resolving throughlexemes.formsrather than the scoped absorb) are on #70.S1: Wiktionary scoped, relations resolved, parity (#70)
Decision log: S1 audited and merged, A5 amendment accepted (#69 v7, #70)
S2: Wikidata + Wikipedia, linking ladder (#70)
holden commentedon Sep 5, 2026
S2 measured L1, and it cannot pass as written (2026-09-05,
4a8cfcf). Numbers and reasoning in #70; this is the amendment Β§7 needs.L1 as specified: β₯ 70 % of scope lexemes with a
concept_linkat confidence β₯ 0.8. Measured: 57.7 % (14,643 / 25,393), with a ceiling of 68.1 % β that is how many scope lexemes have any link at all.The 31.9 % with no link are not a linker failure: 6,840 of them have no English Wikipedia article at all (
soup-fin,trochid,prophaethontid,sandprawnβ theen:Animalscategory tail's fossil groups, dialect names and breed variants). 1,209 have an article that no rung matched, and 39 were never probed. Β§7's clause "if L1 lands far below 70 %, the linker gets another rung" was written for exactly this, but another rung cannot conjure an article that does not exist.The ladder itself was never going to reach the bar either, and this was measurable before a line of the linker was written: only 4,216 scope lexemes have a WordNet sense carrying a QID and 546 a Wiktionary one β about 21 % β while Β§5 pins
title_matchat 0.70, below L1's 0.8 threshold. Β§5 already anticipates the fix ("0.95 0.90 0.85 0.70 0.40β0.60 1.00 by method, adjusted by checks"), so a corroboration pass raises a title match when a second independent signal agrees, recording which one inmetadata["corroboration"]:P225binomial or EnglishP1843common name is the lemmastatus: :confirmedThat is what takes L1 from 25.2 % (strict ladder) to 57.7 %. Both figures are printed by
mix dd.linkevery run.Proposed amendment to Β§7's L1 row:
On that reading: 14,643 / 16,553 = 88.5 % β , with 57.7 % raw and 68.1 % reachable reported. The shape mirrors the A5 amendment accepted after S1 β exclude what the source structurally cannot cover, report the excluded set and the raw figure, and keep the finding visible.
Two smaller amendments the same session earned, both about denominators rather than thresholds:
auto/confirmed), not every link. An Animals scope carries ~38,000:candidaterows from "may refer to" pages and BYD Seal was never going to reach Animalia. On asserted links: 77.3 % β ; including candidates, 26.7 %. Both reported.A6's union came out as predicted: Wiktionary 78.3 % βͺ Wikidata 68.1 % = 97.4 % of the scope, and the 2,008 Linnaean binomials A5 structurally cannot cover are linked by Wikidata β
Cimex lectulariusβ Q940337 at 0.90. A7 100 % and M1 0 gaps across all four sources.Three deviations from Β§2/Β§4/Β§5, each recorded in the commit and on #70: the batched endpoints (
wbgetentitiesat 50 ids, the Action API at 20 titles) instead of the pinned per-item URLs, which turned 100,201 fetches into 7,380 requests and O2(b)'s predicted β 4 h into 1 h 37 m; a Wikipedia record keyed by the thing we asked about (the probed lemma, orconcept:<QID>) rather than by pageid, because an absent marker for a lemma with no article has no pageid to key on; andtitle_matchaccepting Wiktionary'snamepart of speech alongsidenoun, since it files proper nouns separately and Wild Turkey is a thing.Also worth pinning in Β§3:
wikidata_taxongrew the Animals scope from 21,277 to 25,393 lexemes. The enwiki-sitelink requirement is applied (8,870 matched) and the count without it reported (9,290), because the article usually sits on the everyday concept rather than on the taxon item β Q146 Cat has one, Q20980826 Felis catus has none.Decision log: S2 audited, completion pass run, amendments accepted (#69β¦
2 remaining items
Decision log: S3 audited (grade A), amendments accepted (#69 v11, #70)
U0: the Oatmeal theme, ported by hand; daisyUI removed (#71)
S4b: the read layer for the developer surfaces (#70)
S4b: the four developer surfaces (#70)
S4c: fixes from the S4 audit (#70)
holden commentedon Sep 6, 2026
S5 closed. 29 / 29 graded rows pass; the seven that remain are #71's.
mix dd.scorein full, against the development database β six sources, two scopes, 344,290 raw records. M1 is shown as the full parity run gave it (0 gaps over 344,290 records, 24.6 s); everything else is verbatim from this run.What this means, in the words of Β§7
The three extensibility rows, which S5 existed to answer
1807/124274) as the sixth source: 42,726 entries, 37,235 lemmas, 91.9 % already in the index, 6,780 words of his own, 114,787 quotations. Cost: onesourcesrow, onepeoplerow, one registry line, one module, fixtures and tests.schema_migrationsstayed at 2 β which is now what the row measures, not what it asserts.emotions, 809 lexemes from one WordNet root, built and absorbed with zero code changed. Every scope is apriv/scopes/<slug>.jsonfile now,animalsincluded, so there is no scope defined in Elixir left to point at.users+examples+votesgenerated, applied to a full database, schema-diffed, rolled back, and moved todocs/sketches/. Three tables, seven indexes, two check constraints; no column, index or constraint of the thirteen changed.Detail, and the two things the second scope found β A7 v2, and an unscoped
concept_qids/0that makes scope N re-walk scopes 1..Nβ1 β are in the S5 comment on #70.One amendment for your assent: A7 v2 (v13)
Building the second scope grew the concept table to 90,481 sitelinked rows, 54,273 of which have no link to any scope word at all β things a disambiguation page merely mentioned. Every summary fetched names more of them, so the old denominator grows faster than any pass can fill it: one concept pass closed 274 gaps and introduced 20,526.
The S3 audit called this in advance and wrote the remedy on #70 β "for the next scope consider limiting the concept pass to asserted concepts plus candidates promoted to 0.6, and reporting A7 on that population." This is the next scope, so A7 is now graded on the concepts a scope word links to at
auto/confirmed: 10,412 / 10,412 = 100 %, with 77.6 % including candidates reported beside it. Same shape as A5 v2 in S4c.S5: Johnson 1755, the emotions scope, the community-layer sketch (#70)
S5b: A7 v2, Johnson's form-of edges, a general snapshot pin (#70)
U1b: the thing side, and S5c (#71, #70)
U3: fake-data mode, the mobile pass, the chip tests, and the close ofβ¦
holden commentedon Sep 7, 2026
The closing scorecard β MVP-0 passes on both scopes
a2ef867(2026-09-07), the U3 commit on #71. Both runs are with parity β no--skip-parity,so M1 re-runs
materialize/1over all 344,635 records and diffs the result against the database.Every row of Β§7 is green, on both scopes, with nothing pending. That was the exit condition for
MVP-0, written into this issue in v1 and unchanged since.
mix dd.score --scope animalsmix dd.score --scope emotionsThe three rows that are reported rather than graded
A count, not a threshold: disagreement is content, and the word page shows it as a plaque.
wikidata_root. The scope is not under Animalia and never will be,so the row reports instead of grading (S5c's rule).
What is behind it
mix precommit: 487 tests, 0 failures, on checked-in fixtures with the network off (O3).access kinds (E1).
a JSON file (E2).
docs/sketches/, sothe claim that it touches none of the thirteen is falsifiable (E3).
end, every card carrying β and β. Posted in full on πΌοΈ MVP-0 UI: the minimum word-hopping interface on the Oatmeal theme β theme boilerplate first, use case, wireframes, fake-data modeΒ #71.
What MVP-0 did not settle
The culture layer is still a rolled-back sketch;
P279is recorded and never chased, so the conceptchain is one step deep for most non-taxa (#70); Surprise me counts its population on every click;
and the design is #66's, with #67's evidence wall behind it.
?demo=1on any word page draws bothof those as labelled samples, which is the handover.
Closing this with #70 and #71. Tag:
mvp-0.holden commentedon Sep 7, 2026
Closed. MVP-0 passes: animals 36 / 36 graded rows with parity, 1 reported, 0 pending; emotions 35 / 35, 2 reported, 0 pending, on 487 tests with the network off. The full table is in the comment above; the build history is #70 and the UI history #71. Tag:
mvp-0, commita2ef867.What you can actually do now, which is what Β§7 promised: type any English word or an inflected form of it and get a page β 1,541,669 of them. For the enriched scopes, read Bierce, Johnson, Wiktionary, WordNet and Wikipedia side by side in tier order, with the thing the word names, its chain, its kinds, its may refer to and its disagreements; hop by any relation the sources assert; click β on any card for the record it came from. Turn the network off and rebuild every derived row from raw.
Next: #66 the design pass, #67 the evidence wall, #65 quotes. None of them changes the thirteen tables.
P1: the port onto the encyclopedia model, and the seven correctness gβ¦
The preflight checked one provider of twelve, and the ledger had no rβ¦