Repository navigation
Description
Coordination update — 26 September 2026
This remains the broad persona-free assessor issue; its provider-spectrum, per-type evidence and evaluation gates remain in force. #193's delivery can begin with manually reviewed candidate packets for its ten-page editorial experiment without asserting that this assessor is built or waiving this issue's gates. Once available, these assessments feed the same bounded candidate interface.
#195 owns the persistent local generative runtime on the Studio (later #199 Mini), with Req, pinned model versions, structured output and a global inference budget. Bumblebee/Nx or a separate embedding/classification runtime requires a supported-architecture benchmark; it is not a dependency of HTTP curation. Bumblebee's documentation is architecture-specific.
#196 separates page composition from specific semantic claims. Reuse its model lineage where useful without making this assessor persona-dependent. #190 must land before any bot candidate-claim writes, for every nominated illustrates subject kind; composition approval never substitutes for a claim review. #197 internal ballots are not #189 human votes. #156 renders approved saved editorial choices above the existing shelves; it does not replace assessor relevance labels.
The design below is still a proposal, not shipped implementation; source-status inventories are dated research and must be rechecked during its spike.
Purpose
Put an objective relation assessor inside the discovery pipeline: a bounded, inspectable stage that looks at what a provider returned for a word and decides which meaning it concerns, what kind of relation it is, how sure we are, and what to do with it (rank, label, route to review, hide, abstain). It sits between the two things we have today: tagging (CineGraph keyword hits, GIPHY search, and every provider proposed in #100, all term-level with the "relevance unverified" plaque) and curation (#97's persona editors and #75's Dead Editors, which are deliberately subjective and voiced). The assessor is persona-free, runs before any curator exists, and later becomes the candidate pool those curators read from (#97 steps 2 and 3 already sketch a shared retrieve-and-rerank stage; this issue makes it a first-class discovery stage now).
This is a research and design issue. Nothing here is implemented or claimed; every model name is a candidate to benchmark, every number an expectation to measure.
Refs #100 (next providers; the source of the concrete cases below and of the minimum spectrum), #97 (local AI curation; shares runtime, budgets and idempotence keys), #75, #85 (three presentation levels; real-person rule), #88 (discovery MVP: "show real match reasons, not model-generated explanations"), #92, #93 (identity: titles are never identity evidence), #96 (composition), #99 (Met tags and reviewed mappings), #72 (ConceptNet; "heuristic weights are not calibrated probabilities"), #65, #67, #84.
Why tagging stops being enough as #100 lands
CineGraph's exact-keyword rule works because TMDb keywords are a curated vocabulary. The providers in #100 are weaker or stranger on purpose, and the live probes there produced exactly the cases a tag-only shelf gets wrong:
| Observed on 2026-09-13 | What a tag-only shelf would do | What the reader actually needs decided |
|---|---|---|
| iTunes and MusicBrainz return several songs titled "Nepotism" | Show them as matches for nepotism | Titled after the word (named_after) versus about the concept (about). A song title is not a subject. |
MusicBrainz tag:grief returns instrumental tracks tagged sorrowful, cathartic, grief by their uploader |
Show them as "grief" matches | A folksonomy mood tag: weak about, worth ranking below works whose description says grief. |
Commons depicts=war returns 2023 Gaza photographs |
Show them under "war" | They depict an instance of war (an event), not the concept, and they are real-world violence. Relation depicts via the instance chain, with a safety gate. |
Commons depicts=nepotism returns a portrait of a living politician |
Show it under "nepotism" | An adverse interpretation of a real person (#85's rule): never automatic, review only. |
| Open Library subject "nepotism" returns Il nipotismo di Roma (1667) and What Is Sustainable Journalism? | Show both equally | Rank the strong subject assignment above the weak one from the work's description. |
AIC q=grief returns a Daumier print whose caption contains "Good grief" |
Show it as art about grief | A lexical false positive: the caption mentions the word as an idiom. Hide by default, keep reachable. |
| TVmaze returns "Good Grief" (two shows) for grief | Show them | Title only; the idiom is not the concept. |
| PoetryDB returns 305 poems with "grief" in a line; Chronicling America returns 12,850 OCR pages with "nepotism" | Show the first twelve | Usage evidence (attests): which sense, and does the OCR snippet really contain the token in context. |
| Wikiquote "Nepotism" theme page returns six bullets of wikitext | Show a text blob | Structured extraction of quotation, author, source and year, verbatim, with Misattributed and Disputed sections kept as provenance (#65). |
CineGraph bank returns films matched on the keyword |
"Relevance to this meaning is unverified" | Which meaning: the heist thriller is the financial sense; the fishing documentary is the river. |
Wikidata P921 main subject returns works about nepotism |
Not available today | Exact by identifier. No model needed, and a free source of labelled positives for evaluating everything above. |
The pattern: the relations are more varied than "matched a tag" (about, depicts, named_after, attests, title_only, mentions, unrelated), and the meaning has to be chosen from the senses we already store. Both are objective questions with checkable answers, which is what separates this from persona curation.
Prerequisite: a minimum provider spectrum
An assessor proven on films alone would be a film classifier. Before its gates are evaluated, a minimum set of providers from #100 must be live (not fixtures), chosen so that every matching model, both modalities, both retention postures and both target kinds are represented. Proposed minimum: nine providers across eight content types.
| Content type | Provider(s) | Matching model | Modality | Retention | Status today | What it stresses in the assessor |
|---|---|---|---|---|---|---|
| Film | CineGraph | B (TMDb keyword) | poster + title + keywords | durable | live | The baseline; sense split on bank, run, set |
| Memes and GIFs | GIPHY; Wikidata items typed internet meme (980 items, 129 with a main subject); Commons Internet memes category tree | D (search) / A (QID) | animated image + title + alt text | transient (GIPHY) / durable | GIPHY live; the Wikidata and Commons sources are proposed | The thinnest text, an image-only signal, and a browser-only provider the server never persists |
| Art | The Met via #99 (offline seeding, tag QIDs); AIC or Cleveland as feeders | B (tag QID) / D | image + title + tags | durable, image gated by public-domain flag | #99 in progress | Exact tag-to-sense rule; image-text similarity; depicts versus about |
| Poems | PoetryDB | C (a line contains the word) | text | durable | #100 wave 1 | Usage attestation and sense choice inside a verse line |
| Quotes | Wikiquote theme pages | A (sitelink) + C | text (wikitext) | durable | #100 wave 1 | Verbatim structured extraction; Misattributed and Disputed provenance |
| News | The Guardian | C/D (search) | text | transient, 24 h | #100 wave 2, #63 | Assessing without retaining: labels must be recomputable from a fingerprint |
| Music | MusicBrainz, with iTunes previews in the browser | B (tags) / D (title) | title + tags + artwork + 30 s preview | durable metadata, transient preview | #100 wave 2 | named_after versus about; folksonomy tags; audio we cannot assess |
| Books | Open Library | B (subjects) / D | title + subjects + description + cover | durable metadata, cover reference-only | #100 wave 1 | Weak versus strong subject assignments |
| Photos | Openverse; Commons depicts | C / A (QID) | image + title + tags | durable, per-file licence | #100 wave 1 and 2 | Image-text similarity; instance-of-event photos; real-person gate |
| Works about the thing | Wikidata P921 |
A (QID) | text (labels, descriptions) | durable | #100 wave 1 | The rule tier; also the silver-label source |
Coverage rule: at least eight providers across at least six content types; all four matching models (A identifier-native, B tag vocabulary, C full-text usage, D title); both modalities with at least one animated, browser-only case; both retention postures; both target kinds (term-level and sense-level). The Guardian may be swapped for Chronicling America (OCR usage, public domain) if the key is not in hand; the spectrum must still contain one transient provider.
Two contract additions the spectrum forces on providers, small and provider-agnostic:
- A minimal assessable payload. Every provider's
preview_metadatamust carry a title, a bounded description or snippet when the source has one, tags or subjects when it has them, an image reference when it has one, the language, and the source URL. A provider that can supply only a title and alt text (GIPHY) declares it, so the assessor knows the image tier is required rather than optional. - A transient assessment path. Browser-only providers never reach the server today. For them the LiveView receives the item metadata from the browser, assesses it in memory, returns labels, and stores only the provider id, our decision and a fingerprint hash. No media URL or asset is stored, which keeps GIPHY's caching rule intact; whether even that is acceptable under its terms is an owner decision below.
Gate rule. Every evaluation gate in this issue is computed per content type and passes only if it holds on every type in the spectrum. No averaging across types: a 0.95 on films must not hide a 0.6 on GIFs.
Where it sits in the lifecycle
retrieve (provider) → normalize items
→ ASSESS (this issue)
tier 0 rules free, deterministic
tier 1 lite ML in-process, milliseconds per item
tier 2 LLM judge local, seconds, only for the ambiguous slice
→ persist results + assessments
→ reader: rank · label · hide-by-default · route candidates to review
→ later: #97 personas read the assessed pool
Tiers 0 and 1 run inline in the existing Oban discovery run, inside the run's time budget, so the reader sees an ordered shelf on first paint. Tier 2 runs as a separate bounded batch (nightly window, one generation in flight, the #97 runtime) and updates assessments when it finishes; the page never waits for it. A discovery result never becomes an illustrates claim through this stage: the assessor can only rank, label, hide-by-default or propose a candidate claim for review.
The three-level table from #88 gains a middle row:
| Discovery (today) | Assessment (this issue) | Curation (#85, #97) | |
|---|---|---|---|
| Statement | Provider returned this item for this query | An automatic assessor judged the item relates to this meaning as about (score 0.83) |
A claimant interprets the item as illustrating the meaning |
| Provenance | Provider, mapping version, exact query | Assessor id, model digest, template version, input fingerprint, quoted evidence excerpt | Claimant, rationale, evidence, review history |
| Reader label | "Found through the keyword war" | "Likely about war · automatic" | "Selected by Alex · reason" |
| Refresh | Provider policy | Recomputed when the item, the sense text or the assessor version changes | Never rewritten by refresh |
Wireframes
W1: one discovery run with the assess stage
VISIT /define/bank ──► definition renders immediately (unchanged)
│
▼
Oban discovery run (per provider, existing budgets)
┌──────────────────────────────────────────────────────────┐
│ retrieve ──► normalize 12 items │
│ │
│ ASSESS (budget: ≤ 500 ms for 12 items, else defer T1) │
│ T0 rules exact QID tag? monosemous? title==word? │
│ person + adverse concept? OCR token ok? │
│ T1 lite ML sense by gloss embedding (margin) │
│ rerank (cross-encoder) │
│ relation by NLI hypotheses │
│ image ↔ concept similarity (CLIP) │
│ ──► decision per item: │
│ rank | label | hide_by_default | route | abstain │
│ │
│ persist results (disposable, 7 d) │
│ persist assessments (kept, keyed by fingerprint) │
│ PubSub ──► "In culture" section updates │
└──────────────────────────────────────────────────────────┘
│ ambiguous or high-value items only
▼
nightly batch (T2 LLM judge, one generation in flight)
──► updates assessments ──► PubSub if the page is open
│ score ≥ route threshold
▼
review queue: candidate claim (about / depicts / attests / named_after)
W2: the word page shelf after assessment
BANK Meanings: [financial institution] [river bank] [row of things]
IN CULTURE
Films · CineGraph · keywords: TMDb 3 weaker matches hidden · show
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ poster │ │ poster │ │ poster │ │ poster │ │ (no img) │
└──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘
Inside Man Heist The River Salmon Run Bank Job
2006 · Film 2001 · Film 1984 · Film 2019 · Film 1998 · Film
Likely about Likely about Likely about Likely about Titled "Bank"
financial financial river bank river bank · automatic
· automatic · automatic · automatic · automatic
[details ▾] [details ▾] [details ▾] [details ▾] [details ▾]
Matches for "bank" · About these results ▾
Ordering is by assessment score within the provider. The plaque "relevance to this meaning is unverified" is what an item shows when the assessor abstains; the "Likely …" label appears only above the calibrated threshold. Hidden items are reachable in one click and are never deleted.
W3: the details disclosure for one card
▾ Inside Man · why it appears here
Found through CineGraph keyword "bank" (TMDb 3411) ← discovery, unchanged
Assessed as about · financial institution (sense 2) ← automatic assessment
Confidence high (0.86; calibrated bucket 4 of 5)
Because "…a bank robbery in Manhattan turns into a hostage
standoff…" (TMDb overview, quoted)
How rules: no exact tag · gloss match 0.71 (margin 0.32)
· rerank 0.86 · NLI "about" 0.91 · LLM judge: not run
Assessor gloss_embed:bge-m3@a1f3 · rerank:bge-v2-m3@77c0 · nli:bart-mnli@0e2b
Not a curated example. [Report a wrong match] [Propose as an example →]
W4: a routed candidate in the review queue
REVIEW QUEUE · automatic candidates filter: [about] [depicts] [attests] [named_after]
┌────────────────────────────────────────────────────────────────────────────┐
│ Proposed: Inside Man (film, TMDb 3411) ── about ──► bank · financial (sense 2)│
│ Method model_relation:nli@0e2b confidence 0.70 (capped for tier 1) │
│ Evidence TMDb overview excerpt (copied into assertion_evidence) │
│ Origin actor: discovery assessor process · submitted by: same │
│ Checks identity resolved (tmdb_movie → local film #4821) · no person gate │
│ │
│ [Accept] [Request changes] [Reject] Rejected candidates never return │
└────────────────────────────────────────────────────────────────────────────┘
W5: the transient path for browser-only providers
BROWSER (GIPHY search, direct) SERVER (LiveView)
8 GIFs: id, title, alt_text, url ──► "assess" event carries id + title + alt_text only
T0/T1 on text; CLIP on the first frame fetched by
the browser? NO: the server never fetches media.
Text-only assessment, image tier marked unavailable.
labels per id ◄──────────────────── {id, decision, label, fingerprint}
shelf reorders, weak matches fold stored: provider id · decision · fingerprint hash
NOT stored: media URLs, assets, GIPHY text
W6: the evaluation ledger, printed by mix dd.assess --report
assessor P1 candidate set · gold 2026-10 · per content type (gate passes only if every row passes)
type n p@5 p@12 wrong-sense abstain hidden-gold-pos person-leaks ms/item(T1)
film 144 0.93 0.88 0.03 0.08 0 0 41
gif 96 0.71 0.64 0.12 0.31 1 ◄ FAIL 0 12
art 120 0.90 0.85 0.04 0.10 0 0 58
poems 132 0.95 0.91 0.02 0.05 0 0 23
quotes 108 0.92 0.90 0.03 0.06 0 0 27
news 84 0.88 0.83 0.06 0.12 0 0 33
music 120 0.81 0.77 0.09 0.15 0 0 39
books 120 0.94 0.89 0.02 0.07 0 0 30
photos 132 0.86 0.80 0.07 0.14 0 1 ◄ FAIL 66
label gate (p@5 ≥ 0.90): FAIL (gif, news, music, photos) hide gate: FAIL (gif) leak gate: FAIL (photos)
The numbers above are invented to show the shape of the report and the per-type gate rule. Real ones come from P0.
The three tiers
Tier 0: rules (deterministic, free)
- Exact identifier match: a provider tag or subject whose Wikidata QID equals the sense's concept (Met tag QIDs in Met Open Access: durable public-domain art corpus, tag-driven candidates, and Artsy overlap rules #99, Wikidata
P921/P180, TMDb keyword ids resolved through a reviewed mapping) →aboutordepicts, confidence at the top of the assessor band. - Monosemous headword (one sense in the registry) → sense choice is trivial; skip disambiguation.
- Exact title equality with the headword →
named_aftercandidate, neverabouton its own. - Blocklists and gates: any item resolving to an
entity/personunder an adverse concept → review-only; NSFW ratings and provider withdrawal flags respected. - OCR sanity: the token must appear in the returned snippet with word boundaries, else
mentions:unverified.
Tier 1: lite ML (benchmark supported in-process candidates)
The asset that makes this cheap is already in the database: every sense has a gloss (WordNet and Wiktionary in sense_revisions), every concept a Wikidata description, and every provider item a title, description, tags or snippet.
- Sense choice by gloss embedding. Embed each candidate sense's gloss (plus its hypernym chain) and the item's text; cosine similarity picks the sense with a margin, abstaining below a margin threshold. This is the classic definition-embedding approach to zero-shot word-sense disambiguation and needs no training data.
- Relevance rerank. A cross-encoder scores (sense gloss, item text) pairs and orders the shelf.
- Relation typing by NLI. Zero-shot classification with one hypothesis per relation: "This work is about nepotism", "This image depicts grief", "This text uses bank in the sense of a riverbank", "This title merely contains the word". Once a few hundred reviewed decisions exist, add a small classifier trained on the stored embeddings (logistic regression via
scholar, or SetFit-style), which also gives calibration for free. - Image-text similarity. For art, photos and GIF stills, CLIP-style image and text embeddings compared against the sense gloss and the concept label;
depictsneeds this tier, text tiers cannot see a painting. - Precomputed relatedness. ConceptNet Numberbatch (CC BY-SA 4.0) gives term-to-tag relatedness offline (sorrowful ↔ grief) with no service dependency. The public ConceptNet API answered 502 to every request today, which is one more reason Harden the encyclopedia backbone: durable word identities, correct source relationships, safe refresh, and ConceptNet integration #72's bounded ConceptNet pilot should absorb the dumps rather than call the API.
- Storage. Embeddings for sense glosses and assessed items in Postgres via
pgvector(the Elixir library is maintained, HNSW supported) so reruns are cheap and Explore local AI curation: persona editors, schema, models and compute budgets #97's retrieval stage reuses them instead of re-embedding.
Benchmark supported Tier 1 models before choosing their runtime. Bumblebee/Nx is an option for architectures its pinned version supports, not a guarantee that every model below loads or meets the budget. Verify exact model/tokenizer support and CPU/GPU performance in P0; a separate embedding/classification service is justified only by that comparison. Pin model revisions, digests and licences. The generative judge uses #195's persistent Ollama HTTP runtime; it need not move into Elixir or share the assessor's dependencies. Latency expectations are hypotheses, not measurements.
Tier 2: LLM judge (local, bounded, last)
Only for items that tier 1 leaves ambiguous (small margin, conflicting relation hypotheses) or that are high-value (the top of a shelf on a frequently visited page). Reuses #97's runtime choices (a local model on Ollama, LM Studio or MLX; structured JSON validated in Elixir; one generation in flight; nightly window; cloud fallback disabled) and its idempotence key: (item identity, sense, model digest, template version, input fingerprint).
Its jobs are objective and checkable: pick the sense, name the relation, quote the provider text that supports it, and, for Wikiquote and PoetryDB items, extract quotation, author, source and year verbatim (any extracted string that is not a substring of the source text is discarded). It writes a one-sentence factual rationale, never a voice. Provider text is untrusted input: the judge sees bounded, sanitised fields, resolves ids only against the supplied candidate set, and has no tools.
Model candidates
Sizes and licences are from the model cards or #97 as of September 2026; verify marks anything not read today. The final tier 2 choice belongs to #97's benchmark; this table exists so P0 tests more than one option per role.
| Role | Model | Size | Licence | Runs where | Job here | Notes |
|---|---|---|---|---|---|---|
| Text embedding | sentence-transformers/all-MiniLM-L6-v2 |
22M | Apache-2.0 | Bumblebee text_embedding |
Gloss and item embeddings; fastest baseline | Milliseconds on CPU; English-centric |
| Text embedding | BAAI/bge-m3 |
568M | MIT | Bumblebee (XLM-RoBERTa) | Stronger multilingual embeddings | Heavier; good for Wiktionary's languages |
| Text embedding | Qwen/Qwen3-Embedding-0.6B |
0.6B | Apache-2.0 | Bumblebee (verify) or external | #97's retrieval pick; shared with personas | Same family as the reranker |
| Reranker | cross-encoder/ms-marco-MiniLM-L-6-v2 |
22M | Apache-2.0 | Bumblebee cross_encoding |
Cheap (gloss, item) scoring | Baseline for the latency budget |
| Reranker | BAAI/bge-reranker-base / bge-reranker-v2-m3 |
278M / 568M | Apache-2.0 | Bumblebee (XLM-RoBERTa) | Multilingual reranking | v2-m3 is the open self-host default in 2026 comparisons |
| Reranker | Qwen/Qwen3-Reranker-0.6B (4B if quality demands) |
0.6B / 4B | Apache-2.0 | Bumblebee text_reranking_qwen3 |
Reranking with an instruction prompt | Already in #97's shortlist |
| Relation typing (NLI) | facebook/bart-large-mnli |
407M | MIT | Bumblebee zero_shot_classification |
Hypothesis-per-relation typing | The documented Bumblebee path |
| Relation typing (NLI) | roberta-large-mnli |
355M | MIT | Bumblebee | Alternative NLI | Slightly smaller |
| Relation typing (NLI) | MoritzLaurer/deberta-v3-base-zeroshot-v2.0 |
184M | MIT | Not in Bumblebee (verify); ONNX or a Python sidecar | Best small zero-shot classifier | Only if the BART/RoBERTa path underperforms |
| Trained classifier | Logistic regression over stored embeddings (scholar) or SetFit-style |
tiny | n/a | Elixir | Relation and usability classifier once ~300 labels exist; calibration | Replaces or corrects NLI over time |
| Image-text | openai/clip-vit-base-patch32 |
151M | MIT | Bumblebee image_embedding + CLIP text |
Image ↔ sense similarity for art, photos, GIF stills | Documented Bumblebee example |
| Image-text | google/siglip2-base |
~375M | Apache-2.0 | Bumblebee support verify; else external | Better zero-shot image-text matching | Candidate, not baseline |
| Vision-language judge | Qwen/Qwen2.5-VL-7B or Qwen3-VL |
7B+ | Apache-2.0 | External (Ollama/MLX) | Tier 2 for images: "does this painting depict grief?" | Frames sampled, per #97's GIF note |
| LLM judge | Qwen3.5 9B | 9B (6.6 GB artifact) | verify card | External | #97's baseline; sense, relation, verbatim extraction | Cheapest tier 2 |
| LLM judge | Qwen3.6-27B (dense) | 27B (~17 GB) | Apache-2.0 | External | Harder cases; JSON-schema structured output | Latency may outweigh gains; measure |
| LLM judge | Qwen3.6-35B-A3B (MoE) | 35B total, 3B active | Apache-2.0 | External | Speed versus quality comparison | Verify a pinned quantisation and runtime |
| LLM judge | gpt-oss-20b (MoE) |
21B total, 3.6B active | Apache-2.0 | External | Alternative family for agreement checks | Two prompts on one model are not independent; two families are closer |
| LLM judge | Phi-4 14B | 14B | MIT | External | Small, strong on classification-shaped tasks | Good second family |
| LLM judge | Gemma 3 12B / 27B | 12B / 27B | Gemma terms (not Apache) | External | Multimodal option | Licence review before use |
| LLM judge | Mistral Small 3.x 24B | 24B | Apache-2.0 | External | Throughput-oriented judge with vision | Memory check against the 64 GB budget |
| Hosted frontier model | vendor to decide | n/a | vendor terms | Cloud, off by default | Only if local judges fail the gates on some type | Data leaves the machine; per-call cost; keep tier 2 swappable behind one behaviour |
Memory arithmetic follows #97: reserve 16 to 20 GB for the OS, the app and Postgres; weights, KV cache and image inputs all count; the tier 1 models together stay under 3 GB.
Data model
One small table, discovery_assessments, separate from discovery_results because results are disposable cache (7-day expiry, cleanup every 15 minutes) and assessments are the expensive part worth keeping:
| Column | Rule |
|---|---|
external_namespace, external_id |
The provider identity, as on results |
target_object_id |
The lexeme the discovery targeted |
sense_object_id, concept_object_id |
The chosen meaning, nullable when abstaining |
assessor |
rules:v1, gloss_embed:<digest>, rerank:<digest>, nli:<digest>, clip:<digest>, llm:<digest>@<template> |
relation |
about, depicts, named_after, attests, title_only, mentions, unrelated, abstain |
score, margin, calibration_bucket |
Raw score kept separate from the calibrated bucket, per #72 |
evidence_excerpt |
Bounded, quoted from provider text; never generated; absent for transient providers |
input_fingerprint |
Hash of item preview, sense revision text and assessor version; a changed sense or item invalidates the row |
decision |
rank, label, route_to_review, hide_by_default, abstain |
Unique on (external_namespace, external_id, target_object_id, assessor, input_fingerprint). Assessments survive result cleanup and are recomputed on fingerprint change; withdrawal of the underlying item removes them. No claim references an assessment row. A routed candidate copies its evidence into the existing claim and assertion_evidence structures, so #88's invariant holds: a claim never depends on an expiring cache row.
How candidates enter the claims system
The linker is the precedent: every automatic link records method and confidence, rungs have pinned confidences, corroboration raises them and records why in metadata, reruns never resurrect a rejected link, and editorial state is append-only review. The assessor follows the same ladder rules with its own band below every identifier rung: rules-exact at most 0.90, lite ML at most 0.70, the LLM judge at most 0.75, two independent tiers agreeing recorded as corroboration rather than as a review nobody performed.
Only objective predicates are eligible for automatic candidates, and they already exist as data in priv/predicates/core.json: about (today content/article → entity/*, needing entity/work → entity/concept added as endpoint rows), depicts (content/image → entity/concept exists). Two would be new file entries, not migrations: attests (content/passage|quotation → sense, usage evidence) and named_after (entity/work → entity/concept). illustrates and satirizes stay human- or persona-attributed and are never written by this stage.
Reader impact
Small and reversible, behind configuration:
- Shelf order follows the assessment score within a provider; provider order remains inspectable in details.
- "Relevance to this meaning is unverified" stays the default. It is replaced by "Likely about grief · automatic" only above a calibrated threshold, with the details disclosure showing the assessor, the score and the quoted excerpt.
- Lexical false positives and
title_onlyitems are hidden by default behind "show N weaker matches"; nothing is deleted. - Routed candidates appear in the review queue as
about/depicts/attests/named_afterproposals with evidence; accepted ones show as reviewed connections, exactly like Unify source identities into clickable encyclopedia entries for films, artwork and quotes #93's film page distinguishes them from automatic appearances. - Nothing model-generated is ever presented as a quotation, a definition or a source statement.
Evaluation before anything is enabled
- Gold set. The Artsy: verify access before retirement, seed significant artworks and connect art to meanings #86 ten meanings (nepotism, war, grief, love, power, family, solitude, justice, bank, Pop Art), the Curated culture MVP: film-first discovery, quotes, art, GIFs and AI-assisted curation #85 showcases (situationship, narcissism) and three polysemy stressors (mountain, run, set): about 15 targets, top 12 items from every provider in the minimum spectrum, roughly 1,000 to 1,500 items. Two labellers record sense, relation and usability. Labels are stored as reviews on candidate claims so they stay attributable and reusable.
- Silver set, free. Wikidata
P921pairs are positives forabout, CommonsP180pairs fordepicts, Open Library subject pairs forabout; negatives are items paired with unrelated concepts. Used for calibration and ablations, reported separately from gold. - Metrics, per content type. Precision at 5 and at 12 per meaning; wrong-sense rate on the polysemous targets; abstention rate; a reliability table for calibration; latency and cost per item per tier; review acceptance rate of routed candidates; adverse-person leaks (must be zero).
- Baselines and ablations. Tag-only (today) versus rules plus lite ML versus plus the LLM judge; encoder choice; reranker choice; NLI versus the trained classifier once labels exist; text-only versus text plus CLIP on the image types.
- Gates, each per type. The "likely this meaning" label needs precision of at least 0.9 at its threshold on gold. Routing to review needs at least half of routed candidates accepted in the pilot, or thresholds rise. Hiding by default needs zero gold positives hidden at its threshold. Real-person adverse leaks: zero, or the gate stays closed. A type that fails keeps the plaque and provider order until it passes; the other types may ship.
Runtime and cost
- Candidate dependencies, not a mandatory install list:
nx,exla,bumblebee,pgvector,scholar. Verify architecture support and benchmark the chosen embedding/assessor workload before adding any of them; CPU adequacy is a measurement, not an assumption. Pin model digests/storage explicitly. The generative LLM tier uses the persistent HTTP runtime in Local curation runtime: persistent Ollama service, pinned models and bounded Req inference #195 throughReq; it does not require native-model dependencies merely to call an API. - Budget: tiers 0 and 1 add a measured, capped amount to each discovery run (proposal: 500 ms for 12 items with the small encoder; the reranker and CLIP only when the run has budget left, else deferred to the batch). Tier 2 has its own compute budget and queue, separate from the provider request budget in
Discovery.Budget. - Scale: with Explore local AI curation: persona editors, schema, models and compute budgets #97's formula, a lite tier at well under a second per item makes assessing 10,000 meanings a matter of hours, and the LLM judge sees only the ambiguous slice (expect 10 to 20% of items; measure).
- Electricity, review time and provider requests remain separate line items, as Explore local AI curation: persona editors, schema, models and compute budgets #97 insists.
Safety and integrity (carried over, not new)
- Provider text is untrusted; the judge cannot change tools, budgets or policy, and ids resolve only against supplied candidates.
- Adverse interpretations of real people are review-only (Curated culture MVP: film-first discovery, quotes, art, GIFs and AI-assisted curation #85); fixtures use fictional people.
- Extraction is verbatim or discarded; a model never contributes wording to the quotation pool (Quote layer with provenance scoring #65, Explore local AI curation: persona editors, schema, models and compute budgets #97).
- Assessments are not identity evidence; the assessor may flag "these two items look like the same work" as a reconciliation candidate, never a merge (Unify source identities into clickable encyclopedia entries for films, artwork and quotes #93).
- Rejected candidates stay rejected on rerun, like the linker; reviews are append-only (Make encyclopedia connections reliable: identity changes, exact evidence, general entities and authoring #84).
- Every assessment is inspectable: assessor, digest, inputs, excerpt, score, decision.
Sequence
- Prerequisite: the minimum provider spectrum above is live, with dates, under Explore next discovery providers: quotes, music, books, news, images, games, nature and social APIs #100's child issues. The assessor's P0 does not start on films and GIFs alone.
- P0, spike (no schema): pick two encoders, one reranker, one NLI model and CLIP; run offline over the cached results of every spectrum provider for the gold targets; publish per-type metrics, latencies and a go/no-go. Bounded to one day of compute.
- P1: rules tier, gloss-embedding sense choice, reranking and image similarity inline; the transient path for browser-only providers;
discovery_assessments; ordering and the label behind a config flag; deterministic tests on fixture embeddings; gold-set regression per type. - P2: relation typing; candidate claims to the review queue with the predicate endpoint additions; review UI reuse; acceptance-rate ledger; the trained classifier once labels exist.
- P3: LLM judge for the ambiguous slice, text and vision; verbatim extraction for Wikiquote and PoetryDB; nightly batch; cost ledger.
- P4: hand the assessed pool to Explore local AI curation: persona editors, schema, models and compute budgets #97 and compare persona selection against assessor-only ranking on the same evidence.
Decisions for the owner
- Confirm the minimum spectrum (the nine providers above), or name the substitutions. This decides the order of Explore next discovery providers: quotes, music, books, news, images, games, nature and social APIs #100's waves.
- Is the transient assessment path for GIPHY acceptable under its terms (we would store the GIF id, our decision and a hash; never media URLs or GIPHY text)? If not, GIFs stay unassessed and the spectrum needs another animated, image-only case.
- Reorder and label first, hide later? Recommendation: yes; hide only lexical false positives, always with "show more".
- The precision gate before the sense label replaces the plaque (proposed 0.9, per type).
- Tier 1 runtime: benchmark exact supported Bumblebee/Nx model architectures against any necessary external alternative in P0. No native dependency is required for Local curation runtime: persistent Ollama service, pinned models and bounded Req inference #195 HTTP generation. Choose by measured support, quality and latency, not an assumed all-Elixir requirement.
- Which predicates are objective enough for automatic candidates. Recommendation:
about,depicts,attests,named_after; neverillustratesorsatirizes. - Store embeddings in
pgvectorfrom P1, or defer until Explore local AI curation: persona editors, schema, models and compute budgets #97 needs them. Recommendation: from P1, since the sense-gloss embeddings are the shared asset. - The real-person policy in detail: which concepts count as adverse, and whether any automatic display is ever allowed.
- Whether a hosted frontier model may ever be tried for tier 2, and under what data-handling rule.
Acceptance for this issue
- Owner confirms the boundary (objective assessor now, personas later, no automatic acceptance), the tiering and the minimum spectrum.
- The spectrum is live and listed with dates before P0 runs.
- P0 report published with per-type gold and silver metrics, latencies, model digests and licences, and a go/no-go for P1.
- Child issues for P1 to P3 carrying the per-type gates above.
Non-goals
No automatically accepted claims. No persona voice or marginalia (that is #75/#97). No cloud LLM by default. No fine-tuning before reviewed labels exist. No replacement of the linker ladder or of #93's identity resolver. No change to how definitions render. No assessor shipped on a single content type.
Activity
holden commentedon Oct 21, 2026
Folded into #193, the curation programme's one open tracker, on the owner's word (21 October 2026). The specification here stands; reopen when the work starts. Status is in the consolidation comment on #193.