Skip to content

Discovery relation assessor: lite ML and LLM judging of provider results before persona curation #101

Closed as not planned

Description

@holden

Coordination update — 26 September 2026

This remains the broad persona-free assessor issue; its provider-spectrum, per-type evidence and evaluation gates remain in force. #193's delivery can begin with manually reviewed candidate packets for its ten-page editorial experiment without asserting that this assessor is built or waiving this issue's gates. Once available, these assessments feed the same bounded candidate interface.

#195 owns the persistent local generative runtime on the Studio (later #199 Mini), with Req, pinned model versions, structured output and a global inference budget. Bumblebee/Nx or a separate embedding/classification runtime requires a supported-architecture benchmark; it is not a dependency of HTTP curation. Bumblebee's documentation is architecture-specific.

#196 separates page composition from specific semantic claims. Reuse its model lineage where useful without making this assessor persona-dependent. #190 must land before any bot candidate-claim writes, for every nominated illustrates subject kind; composition approval never substitutes for a claim review. #197 internal ballots are not #189 human votes. #156 renders approved saved editorial choices above the existing shelves; it does not replace assessor relevance labels.

The design below is still a proposal, not shipped implementation; source-status inventories are dated research and must be rechecked during its spike.

Purpose

Put an objective relation assessor inside the discovery pipeline: a bounded, inspectable stage that looks at what a provider returned for a word and decides which meaning it concerns, what kind of relation it is, how sure we are, and what to do with it (rank, label, route to review, hide, abstain). It sits between the two things we have today: tagging (CineGraph keyword hits, GIPHY search, and every provider proposed in #100, all term-level with the "relevance unverified" plaque) and curation (#97's persona editors and #75's Dead Editors, which are deliberately subjective and voiced). The assessor is persona-free, runs before any curator exists, and later becomes the candidate pool those curators read from (#97 steps 2 and 3 already sketch a shared retrieve-and-rerank stage; this issue makes it a first-class discovery stage now).

This is a research and design issue. Nothing here is implemented or claimed; every model name is a candidate to benchmark, every number an expectation to measure.

Refs #100 (next providers; the source of the concrete cases below and of the minimum spectrum), #97 (local AI curation; shares runtime, budgets and idempotence keys), #75, #85 (three presentation levels; real-person rule), #88 (discovery MVP: "show real match reasons, not model-generated explanations"), #92, #93 (identity: titles are never identity evidence), #96 (composition), #99 (Met tags and reviewed mappings), #72 (ConceptNet; "heuristic weights are not calibrated probabilities"), #65, #67, #84.

Why tagging stops being enough as #100 lands

CineGraph's exact-keyword rule works because TMDb keywords are a curated vocabulary. The providers in #100 are weaker or stranger on purpose, and the live probes there produced exactly the cases a tag-only shelf gets wrong:

Observed on 2026-09-13 What a tag-only shelf would do What the reader actually needs decided
iTunes and MusicBrainz return several songs titled "Nepotism" Show them as matches for nepotism Titled after the word (named_after) versus about the concept (about). A song title is not a subject.
MusicBrainz tag:grief returns instrumental tracks tagged sorrowful, cathartic, grief by their uploader Show them as "grief" matches A folksonomy mood tag: weak about, worth ranking below works whose description says grief.
Commons depicts=war returns 2023 Gaza photographs Show them under "war" They depict an instance of war (an event), not the concept, and they are real-world violence. Relation depicts via the instance chain, with a safety gate.
Commons depicts=nepotism returns a portrait of a living politician Show it under "nepotism" An adverse interpretation of a real person (#85's rule): never automatic, review only.
Open Library subject "nepotism" returns Il nipotismo di Roma (1667) and What Is Sustainable Journalism? Show both equally Rank the strong subject assignment above the weak one from the work's description.
AIC q=grief returns a Daumier print whose caption contains "Good grief" Show it as art about grief A lexical false positive: the caption mentions the word as an idiom. Hide by default, keep reachable.
TVmaze returns "Good Grief" (two shows) for grief Show them Title only; the idiom is not the concept.
PoetryDB returns 305 poems with "grief" in a line; Chronicling America returns 12,850 OCR pages with "nepotism" Show the first twelve Usage evidence (attests): which sense, and does the OCR snippet really contain the token in context.
Wikiquote "Nepotism" theme page returns six bullets of wikitext Show a text blob Structured extraction of quotation, author, source and year, verbatim, with Misattributed and Disputed sections kept as provenance (#65).
CineGraph bank returns films matched on the keyword "Relevance to this meaning is unverified" Which meaning: the heist thriller is the financial sense; the fishing documentary is the river.
Wikidata P921 main subject returns works about nepotism Not available today Exact by identifier. No model needed, and a free source of labelled positives for evaluating everything above.

The pattern: the relations are more varied than "matched a tag" (about, depicts, named_after, attests, title_only, mentions, unrelated), and the meaning has to be chosen from the senses we already store. Both are objective questions with checkable answers, which is what separates this from persona curation.

Prerequisite: a minimum provider spectrum

An assessor proven on films alone would be a film classifier. Before its gates are evaluated, a minimum set of providers from #100 must be live (not fixtures), chosen so that every matching model, both modalities, both retention postures and both target kinds are represented. Proposed minimum: nine providers across eight content types.

Content type Provider(s) Matching model Modality Retention Status today What it stresses in the assessor
Film CineGraph B (TMDb keyword) poster + title + keywords durable live The baseline; sense split on bank, run, set
Memes and GIFs GIPHY; Wikidata items typed internet meme (980 items, 129 with a main subject); Commons Internet memes category tree D (search) / A (QID) animated image + title + alt text transient (GIPHY) / durable GIPHY live; the Wikidata and Commons sources are proposed The thinnest text, an image-only signal, and a browser-only provider the server never persists
Art The Met via #99 (offline seeding, tag QIDs); AIC or Cleveland as feeders B (tag QID) / D image + title + tags durable, image gated by public-domain flag #99 in progress Exact tag-to-sense rule; image-text similarity; depicts versus about
Poems PoetryDB C (a line contains the word) text durable #100 wave 1 Usage attestation and sense choice inside a verse line
Quotes Wikiquote theme pages A (sitelink) + C text (wikitext) durable #100 wave 1 Verbatim structured extraction; Misattributed and Disputed provenance
News The Guardian C/D (search) text transient, 24 h #100 wave 2, #63 Assessing without retaining: labels must be recomputable from a fingerprint
Music MusicBrainz, with iTunes previews in the browser B (tags) / D (title) title + tags + artwork + 30 s preview durable metadata, transient preview #100 wave 2 named_after versus about; folksonomy tags; audio we cannot assess
Books Open Library B (subjects) / D title + subjects + description + cover durable metadata, cover reference-only #100 wave 1 Weak versus strong subject assignments
Photos Openverse; Commons depicts C / A (QID) image + title + tags durable, per-file licence #100 wave 1 and 2 Image-text similarity; instance-of-event photos; real-person gate
Works about the thing Wikidata P921 A (QID) text (labels, descriptions) durable #100 wave 1 The rule tier; also the silver-label source

Coverage rule: at least eight providers across at least six content types; all four matching models (A identifier-native, B tag vocabulary, C full-text usage, D title); both modalities with at least one animated, browser-only case; both retention postures; both target kinds (term-level and sense-level). The Guardian may be swapped for Chronicling America (OCR usage, public domain) if the key is not in hand; the spectrum must still contain one transient provider.

Two contract additions the spectrum forces on providers, small and provider-agnostic:

  • A minimal assessable payload. Every provider's preview_metadata must carry a title, a bounded description or snippet when the source has one, tags or subjects when it has them, an image reference when it has one, the language, and the source URL. A provider that can supply only a title and alt text (GIPHY) declares it, so the assessor knows the image tier is required rather than optional.
  • A transient assessment path. Browser-only providers never reach the server today. For them the LiveView receives the item metadata from the browser, assesses it in memory, returns labels, and stores only the provider id, our decision and a fingerprint hash. No media URL or asset is stored, which keeps GIPHY's caching rule intact; whether even that is acceptable under its terms is an owner decision below.

Gate rule. Every evaluation gate in this issue is computed per content type and passes only if it holds on every type in the spectrum. No averaging across types: a 0.95 on films must not hide a 0.6 on GIFs.

Where it sits in the lifecycle

retrieve (provider) → normalize items
                    → ASSESS (this issue)
                        tier 0  rules            free, deterministic
                        tier 1  lite ML          in-process, milliseconds per item
                        tier 2  LLM judge        local, seconds, only for the ambiguous slice
                    → persist results + assessments
                    → reader: rank · label · hide-by-default · route candidates to review
                    → later: #97 personas read the assessed pool

Tiers 0 and 1 run inline in the existing Oban discovery run, inside the run's time budget, so the reader sees an ordered shelf on first paint. Tier 2 runs as a separate bounded batch (nightly window, one generation in flight, the #97 runtime) and updates assessments when it finishes; the page never waits for it. A discovery result never becomes an illustrates claim through this stage: the assessor can only rank, label, hide-by-default or propose a candidate claim for review.

The three-level table from #88 gains a middle row:

Discovery (today) Assessment (this issue) Curation (#85, #97)
Statement Provider returned this item for this query An automatic assessor judged the item relates to this meaning as about (score 0.83) A claimant interprets the item as illustrating the meaning
Provenance Provider, mapping version, exact query Assessor id, model digest, template version, input fingerprint, quoted evidence excerpt Claimant, rationale, evidence, review history
Reader label "Found through the keyword war" "Likely about war · automatic" "Selected by Alex · reason"
Refresh Provider policy Recomputed when the item, the sense text or the assessor version changes Never rewritten by refresh

Wireframes

W1: one discovery run with the assess stage

VISIT /define/bank  ──►  definition renders immediately (unchanged)
                          │
                          ▼
              Oban discovery run (per provider, existing budgets)
              ┌──────────────────────────────────────────────────────────┐
              │ retrieve ──► normalize 12 items                          │
              │                                                          │
              │ ASSESS  (budget: ≤ 500 ms for 12 items, else defer T1)   │
              │   T0 rules      exact QID tag? monosemous? title==word?  │
              │                 person + adverse concept? OCR token ok?  │
              │   T1 lite ML    sense by gloss embedding (margin)        │
              │                 rerank (cross-encoder)                   │
              │                 relation by NLI hypotheses               │
              │                 image ↔ concept similarity (CLIP)        │
              │   ──► decision per item:                                 │
              │       rank | label | hide_by_default | route | abstain   │
              │                                                          │
              │ persist results (disposable, 7 d)                        │
              │ persist assessments (kept, keyed by fingerprint)         │
              │ PubSub ──► "In culture" section updates                  │
              └──────────────────────────────────────────────────────────┘
                          │ ambiguous or high-value items only
                          ▼
              nightly batch (T2 LLM judge, one generation in flight)
              ──► updates assessments ──► PubSub if the page is open
                          │ score ≥ route threshold
                          ▼
              review queue: candidate claim (about / depicts / attests / named_after)

W2: the word page shelf after assessment

BANK                              Meanings: [financial institution] [river bank] [row of things]

IN CULTURE
Films · CineGraph · keywords: TMDb                       3 weaker matches hidden · show

┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ poster   │ │ poster   │ │ poster   │ │ poster   │ │ (no img) │
└──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘
Inside Man    Heist        The River     Salmon Run    Bank Job
2006 · Film   2001 · Film  1984 · Film   2019 · Film   1998 · Film
Likely about  Likely about Likely about  Likely about  Titled "Bank"
financial     financial    river bank    river bank    · automatic
· automatic   · automatic  · automatic   · automatic
[details ▾]   [details ▾]  [details ▾]   [details ▾]   [details ▾]

Matches for "bank" · About these results ▾

Ordering is by assessment score within the provider. The plaque "relevance to this meaning is unverified" is what an item shows when the assessor abstains; the "Likely …" label appears only above the calibrated threshold. Hidden items are reachable in one click and are never deleted.

W3: the details disclosure for one card

▾ Inside Man · why it appears here

  Found through      CineGraph keyword "bank" (TMDb 3411)          ← discovery, unchanged
  Assessed as        about · financial institution (sense 2)       ← automatic assessment
  Confidence         high (0.86; calibrated bucket 4 of 5)
  Because            "…a bank robbery in Manhattan turns into a hostage
                     standoff…"  (TMDb overview, quoted)
  How                rules: no exact tag · gloss match 0.71 (margin 0.32)
                     · rerank 0.86 · NLI "about" 0.91 · LLM judge: not run
  Assessor           gloss_embed:bge-m3@a1f3 · rerank:bge-v2-m3@77c0 · nli:bart-mnli@0e2b
  Not a curated example. [Report a wrong match]  [Propose as an example →]

W4: a routed candidate in the review queue

REVIEW QUEUE · automatic candidates                          filter: [about] [depicts] [attests] [named_after]

┌────────────────────────────────────────────────────────────────────────────┐
│ Proposed: Inside Man (film, TMDb 3411) ── about ──► bank · financial (sense 2)│
│ Method    model_relation:nli@0e2b   confidence 0.70 (capped for tier 1)      │
│ Evidence  TMDb overview excerpt (copied into assertion_evidence)             │
│ Origin    actor: discovery assessor process · submitted by: same             │
│ Checks    identity resolved (tmdb_movie → local film #4821) · no person gate │
│                                                                              │
│ [Accept]  [Request changes]  [Reject]        Rejected candidates never return │
└────────────────────────────────────────────────────────────────────────────┘

W5: the transient path for browser-only providers

BROWSER (GIPHY search, direct)          SERVER (LiveView)
  8 GIFs: id, title, alt_text, url ──►  "assess" event carries id + title + alt_text only
                                         T0/T1 on text; CLIP on the first frame fetched by
                                         the browser? NO: the server never fetches media.
                                         Text-only assessment, image tier marked unavailable.
  labels per id  ◄────────────────────  {id, decision, label, fingerprint}
  shelf reorders, weak matches fold     stored: provider id · decision · fingerprint hash
                                         NOT stored: media URLs, assets, GIPHY text

W6: the evaluation ledger, printed by mix dd.assess --report

assessor P1 candidate set · gold 2026-10 · per content type (gate passes only if every row passes)

type      n     p@5   p@12  wrong-sense  abstain  hidden-gold-pos  person-leaks  ms/item(T1)
film      144   0.93  0.88  0.03         0.08     0                0             41
gif        96   0.71  0.64  0.12         0.31     1  ◄ FAIL        0             12
art       120   0.90  0.85  0.04         0.10     0                0             58
poems     132   0.95  0.91  0.02         0.05     0                0             23
quotes    108   0.92  0.90  0.03         0.06     0                0             27
news       84   0.88  0.83  0.06         0.12     0                0             33
music     120   0.81  0.77  0.09         0.15     0                0             39
books     120   0.94  0.89  0.02         0.07     0                0             30
photos    132   0.86  0.80  0.07         0.14     0                1  ◄ FAIL     66

label gate (p@5 ≥ 0.90): FAIL (gif, news, music, photos)   hide gate: FAIL (gif)   leak gate: FAIL (photos)

The numbers above are invented to show the shape of the report and the per-type gate rule. Real ones come from P0.

The three tiers

Tier 0: rules (deterministic, free)

  • Exact identifier match: a provider tag or subject whose Wikidata QID equals the sense's concept (Met tag QIDs in Met Open Access: durable public-domain art corpus, tag-driven candidates, and Artsy overlap rules #99, Wikidata P921/P180, TMDb keyword ids resolved through a reviewed mapping) → about or depicts, confidence at the top of the assessor band.
  • Monosemous headword (one sense in the registry) → sense choice is trivial; skip disambiguation.
  • Exact title equality with the headword → named_after candidate, never about on its own.
  • Blocklists and gates: any item resolving to an entity/person under an adverse concept → review-only; NSFW ratings and provider withdrawal flags respected.
  • OCR sanity: the token must appear in the returned snippet with word boundaries, else mentions:unverified.

Tier 1: lite ML (benchmark supported in-process candidates)

The asset that makes this cheap is already in the database: every sense has a gloss (WordNet and Wiktionary in sense_revisions), every concept a Wikidata description, and every provider item a title, description, tags or snippet.

  • Sense choice by gloss embedding. Embed each candidate sense's gloss (plus its hypernym chain) and the item's text; cosine similarity picks the sense with a margin, abstaining below a margin threshold. This is the classic definition-embedding approach to zero-shot word-sense disambiguation and needs no training data.
  • Relevance rerank. A cross-encoder scores (sense gloss, item text) pairs and orders the shelf.
  • Relation typing by NLI. Zero-shot classification with one hypothesis per relation: "This work is about nepotism", "This image depicts grief", "This text uses bank in the sense of a riverbank", "This title merely contains the word". Once a few hundred reviewed decisions exist, add a small classifier trained on the stored embeddings (logistic regression via scholar, or SetFit-style), which also gives calibration for free.
  • Image-text similarity. For art, photos and GIF stills, CLIP-style image and text embeddings compared against the sense gloss and the concept label; depicts needs this tier, text tiers cannot see a painting.
  • Precomputed relatedness. ConceptNet Numberbatch (CC BY-SA 4.0) gives term-to-tag relatedness offline (sorrowful ↔ grief) with no service dependency. The public ConceptNet API answered 502 to every request today, which is one more reason Harden the encyclopedia backbone: durable word identities, correct source relationships, safe refresh, and ConceptNet integration #72's bounded ConceptNet pilot should absorb the dumps rather than call the API.
  • Storage. Embeddings for sense glosses and assessed items in Postgres via pgvector (the Elixir library is maintained, HNSW supported) so reruns are cheap and Explore local AI curation: persona editors, schema, models and compute budgets #97's retrieval stage reuses them instead of re-embedding.

Benchmark supported Tier 1 models before choosing their runtime. Bumblebee/Nx is an option for architectures its pinned version supports, not a guarantee that every model below loads or meets the budget. Verify exact model/tokenizer support and CPU/GPU performance in P0; a separate embedding/classification service is justified only by that comparison. Pin model revisions, digests and licences. The generative judge uses #195's persistent Ollama HTTP runtime; it need not move into Elixir or share the assessor's dependencies. Latency expectations are hypotheses, not measurements.

Tier 2: LLM judge (local, bounded, last)

Only for items that tier 1 leaves ambiguous (small margin, conflicting relation hypotheses) or that are high-value (the top of a shelf on a frequently visited page). Reuses #97's runtime choices (a local model on Ollama, LM Studio or MLX; structured JSON validated in Elixir; one generation in flight; nightly window; cloud fallback disabled) and its idempotence key: (item identity, sense, model digest, template version, input fingerprint).

Its jobs are objective and checkable: pick the sense, name the relation, quote the provider text that supports it, and, for Wikiquote and PoetryDB items, extract quotation, author, source and year verbatim (any extracted string that is not a substring of the source text is discarded). It writes a one-sentence factual rationale, never a voice. Provider text is untrusted input: the judge sees bounded, sanitised fields, resolves ids only against the supplied candidate set, and has no tools.

Model candidates

Sizes and licences are from the model cards or #97 as of September 2026; verify marks anything not read today. The final tier 2 choice belongs to #97's benchmark; this table exists so P0 tests more than one option per role.

Role Model Size Licence Runs where Job here Notes
Text embedding sentence-transformers/all-MiniLM-L6-v2 22M Apache-2.0 Bumblebee text_embedding Gloss and item embeddings; fastest baseline Milliseconds on CPU; English-centric
Text embedding BAAI/bge-m3 568M MIT Bumblebee (XLM-RoBERTa) Stronger multilingual embeddings Heavier; good for Wiktionary's languages
Text embedding Qwen/Qwen3-Embedding-0.6B 0.6B Apache-2.0 Bumblebee (verify) or external #97's retrieval pick; shared with personas Same family as the reranker
Reranker cross-encoder/ms-marco-MiniLM-L-6-v2 22M Apache-2.0 Bumblebee cross_encoding Cheap (gloss, item) scoring Baseline for the latency budget
Reranker BAAI/bge-reranker-base / bge-reranker-v2-m3 278M / 568M Apache-2.0 Bumblebee (XLM-RoBERTa) Multilingual reranking v2-m3 is the open self-host default in 2026 comparisons
Reranker Qwen/Qwen3-Reranker-0.6B (4B if quality demands) 0.6B / 4B Apache-2.0 Bumblebee text_reranking_qwen3 Reranking with an instruction prompt Already in #97's shortlist
Relation typing (NLI) facebook/bart-large-mnli 407M MIT Bumblebee zero_shot_classification Hypothesis-per-relation typing The documented Bumblebee path
Relation typing (NLI) roberta-large-mnli 355M MIT Bumblebee Alternative NLI Slightly smaller
Relation typing (NLI) MoritzLaurer/deberta-v3-base-zeroshot-v2.0 184M MIT Not in Bumblebee (verify); ONNX or a Python sidecar Best small zero-shot classifier Only if the BART/RoBERTa path underperforms
Trained classifier Logistic regression over stored embeddings (scholar) or SetFit-style tiny n/a Elixir Relation and usability classifier once ~300 labels exist; calibration Replaces or corrects NLI over time
Image-text openai/clip-vit-base-patch32 151M MIT Bumblebee image_embedding + CLIP text Image ↔ sense similarity for art, photos, GIF stills Documented Bumblebee example
Image-text google/siglip2-base ~375M Apache-2.0 Bumblebee support verify; else external Better zero-shot image-text matching Candidate, not baseline
Vision-language judge Qwen/Qwen2.5-VL-7B or Qwen3-VL 7B+ Apache-2.0 External (Ollama/MLX) Tier 2 for images: "does this painting depict grief?" Frames sampled, per #97's GIF note
LLM judge Qwen3.5 9B 9B (6.6 GB artifact) verify card External #97's baseline; sense, relation, verbatim extraction Cheapest tier 2
LLM judge Qwen3.6-27B (dense) 27B (~17 GB) Apache-2.0 External Harder cases; JSON-schema structured output Latency may outweigh gains; measure
LLM judge Qwen3.6-35B-A3B (MoE) 35B total, 3B active Apache-2.0 External Speed versus quality comparison Verify a pinned quantisation and runtime
LLM judge gpt-oss-20b (MoE) 21B total, 3.6B active Apache-2.0 External Alternative family for agreement checks Two prompts on one model are not independent; two families are closer
LLM judge Phi-4 14B 14B MIT External Small, strong on classification-shaped tasks Good second family
LLM judge Gemma 3 12B / 27B 12B / 27B Gemma terms (not Apache) External Multimodal option Licence review before use
LLM judge Mistral Small 3.x 24B 24B Apache-2.0 External Throughput-oriented judge with vision Memory check against the 64 GB budget
Hosted frontier model vendor to decide n/a vendor terms Cloud, off by default Only if local judges fail the gates on some type Data leaves the machine; per-call cost; keep tier 2 swappable behind one behaviour

Memory arithmetic follows #97: reserve 16 to 20 GB for the OS, the app and Postgres; weights, KV cache and image inputs all count; the tier 1 models together stay under 3 GB.

Data model

One small table, discovery_assessments, separate from discovery_results because results are disposable cache (7-day expiry, cleanup every 15 minutes) and assessments are the expensive part worth keeping:

Column Rule
external_namespace, external_id The provider identity, as on results
target_object_id The lexeme the discovery targeted
sense_object_id, concept_object_id The chosen meaning, nullable when abstaining
assessor rules:v1, gloss_embed:<digest>, rerank:<digest>, nli:<digest>, clip:<digest>, llm:<digest>@<template>
relation about, depicts, named_after, attests, title_only, mentions, unrelated, abstain
score, margin, calibration_bucket Raw score kept separate from the calibrated bucket, per #72
evidence_excerpt Bounded, quoted from provider text; never generated; absent for transient providers
input_fingerprint Hash of item preview, sense revision text and assessor version; a changed sense or item invalidates the row
decision rank, label, route_to_review, hide_by_default, abstain

Unique on (external_namespace, external_id, target_object_id, assessor, input_fingerprint). Assessments survive result cleanup and are recomputed on fingerprint change; withdrawal of the underlying item removes them. No claim references an assessment row. A routed candidate copies its evidence into the existing claim and assertion_evidence structures, so #88's invariant holds: a claim never depends on an expiring cache row.

How candidates enter the claims system

The linker is the precedent: every automatic link records method and confidence, rungs have pinned confidences, corroboration raises them and records why in metadata, reruns never resurrect a rejected link, and editorial state is append-only review. The assessor follows the same ladder rules with its own band below every identifier rung: rules-exact at most 0.90, lite ML at most 0.70, the LLM judge at most 0.75, two independent tiers agreeing recorded as corroboration rather than as a review nobody performed.

Only objective predicates are eligible for automatic candidates, and they already exist as data in priv/predicates/core.json: about (today content/article → entity/*, needing entity/work → entity/concept added as endpoint rows), depicts (content/image → entity/concept exists). Two would be new file entries, not migrations: attests (content/passage|quotation → sense, usage evidence) and named_after (entity/work → entity/concept). illustrates and satirizes stay human- or persona-attributed and are never written by this stage.

Reader impact

Small and reversible, behind configuration:

  • Shelf order follows the assessment score within a provider; provider order remains inspectable in details.
  • "Relevance to this meaning is unverified" stays the default. It is replaced by "Likely about grief · automatic" only above a calibrated threshold, with the details disclosure showing the assessor, the score and the quoted excerpt.
  • Lexical false positives and title_only items are hidden by default behind "show N weaker matches"; nothing is deleted.
  • Routed candidates appear in the review queue as about/depicts/attests/named_after proposals with evidence; accepted ones show as reviewed connections, exactly like Unify source identities into clickable encyclopedia entries for films, artwork and quotes #93's film page distinguishes them from automatic appearances.
  • Nothing model-generated is ever presented as a quotation, a definition or a source statement.

Evaluation before anything is enabled

  • Gold set. The Artsy: verify access before retirement, seed significant artworks and connect art to meanings #86 ten meanings (nepotism, war, grief, love, power, family, solitude, justice, bank, Pop Art), the Curated culture MVP: film-first discovery, quotes, art, GIFs and AI-assisted curation #85 showcases (situationship, narcissism) and three polysemy stressors (mountain, run, set): about 15 targets, top 12 items from every provider in the minimum spectrum, roughly 1,000 to 1,500 items. Two labellers record sense, relation and usability. Labels are stored as reviews on candidate claims so they stay attributable and reusable.
  • Silver set, free. Wikidata P921 pairs are positives for about, Commons P180 pairs for depicts, Open Library subject pairs for about; negatives are items paired with unrelated concepts. Used for calibration and ablations, reported separately from gold.
  • Metrics, per content type. Precision at 5 and at 12 per meaning; wrong-sense rate on the polysemous targets; abstention rate; a reliability table for calibration; latency and cost per item per tier; review acceptance rate of routed candidates; adverse-person leaks (must be zero).
  • Baselines and ablations. Tag-only (today) versus rules plus lite ML versus plus the LLM judge; encoder choice; reranker choice; NLI versus the trained classifier once labels exist; text-only versus text plus CLIP on the image types.
  • Gates, each per type. The "likely this meaning" label needs precision of at least 0.9 at its threshold on gold. Routing to review needs at least half of routed candidates accepted in the pilot, or thresholds rise. Hiding by default needs zero gold positives hidden at its threshold. Real-person adverse leaks: zero, or the gate stays closed. A type that fails keeps the plaque and provider order until it passes; the other types may ship.

Runtime and cost

  • Candidate dependencies, not a mandatory install list: nx, exla, bumblebee, pgvector, scholar. Verify architecture support and benchmark the chosen embedding/assessor workload before adding any of them; CPU adequacy is a measurement, not an assumption. Pin model digests/storage explicitly. The generative LLM tier uses the persistent HTTP runtime in Local curation runtime: persistent Ollama service, pinned models and bounded Req inference #195 through Req; it does not require native-model dependencies merely to call an API.
  • Budget: tiers 0 and 1 add a measured, capped amount to each discovery run (proposal: 500 ms for 12 items with the small encoder; the reranker and CLIP only when the run has budget left, else deferred to the batch). Tier 2 has its own compute budget and queue, separate from the provider request budget in Discovery.Budget.
  • Scale: with Explore local AI curation: persona editors, schema, models and compute budgets #97's formula, a lite tier at well under a second per item makes assessing 10,000 meanings a matter of hours, and the LLM judge sees only the ambiguous slice (expect 10 to 20% of items; measure).
  • Electricity, review time and provider requests remain separate line items, as Explore local AI curation: persona editors, schema, models and compute budgets #97 insists.

Safety and integrity (carried over, not new)

Sequence

  • Prerequisite: the minimum provider spectrum above is live, with dates, under Explore next discovery providers: quotes, music, books, news, images, games, nature and social APIs #100's child issues. The assessor's P0 does not start on films and GIFs alone.
  • P0, spike (no schema): pick two encoders, one reranker, one NLI model and CLIP; run offline over the cached results of every spectrum provider for the gold targets; publish per-type metrics, latencies and a go/no-go. Bounded to one day of compute.
  • P1: rules tier, gloss-embedding sense choice, reranking and image similarity inline; the transient path for browser-only providers; discovery_assessments; ordering and the label behind a config flag; deterministic tests on fixture embeddings; gold-set regression per type.
  • P2: relation typing; candidate claims to the review queue with the predicate endpoint additions; review UI reuse; acceptance-rate ledger; the trained classifier once labels exist.
  • P3: LLM judge for the ambiguous slice, text and vision; verbatim extraction for Wikiquote and PoetryDB; nightly batch; cost ledger.
  • P4: hand the assessed pool to Explore local AI curation: persona editors, schema, models and compute budgets #97 and compare persona selection against assessor-only ranking on the same evidence.

Decisions for the owner

  • Confirm the minimum spectrum (the nine providers above), or name the substitutions. This decides the order of Explore next discovery providers: quotes, music, books, news, images, games, nature and social APIs #100's waves.
  • Is the transient assessment path for GIPHY acceptable under its terms (we would store the GIF id, our decision and a hash; never media URLs or GIPHY text)? If not, GIFs stay unassessed and the spectrum needs another animated, image-only case.
  • Reorder and label first, hide later? Recommendation: yes; hide only lexical false positives, always with "show more".
  • The precision gate before the sense label replaces the plaque (proposed 0.9, per type).
  • Tier 1 runtime: benchmark exact supported Bumblebee/Nx model architectures against any necessary external alternative in P0. No native dependency is required for Local curation runtime: persistent Ollama service, pinned models and bounded Req inference #195 HTTP generation. Choose by measured support, quality and latency, not an assumed all-Elixir requirement.
  • Which predicates are objective enough for automatic candidates. Recommendation: about, depicts, attests, named_after; never illustrates or satirizes.
  • Store embeddings in pgvector from P1, or defer until Explore local AI curation: persona editors, schema, models and compute budgets #97 needs them. Recommendation: from P1, since the sense-gloss embeddings are the shared asset.
  • The real-person policy in detail: which concepts count as adverse, and whether any automatic display is ever allowed.
  • Whether a hosted frontier model may ever be tried for tier 2, and under what data-handling rule.

Acceptance for this issue

  • Owner confirms the boundary (objective assessor now, personas later, no automatic acceptance), the tiering and the minimum spectrum.
  • The spectrum is live and listed with dates before P0 runs.
  • P0 report published with per-type gold and silver metrics, latencies, model digests and licences, and a go/no-go for P1.
  • Child issues for P1 to P3 carrying the per-type gates above.

Non-goals

No automatically accepted claims. No persona voice or marginalia (that is #75/#97). No cloud LLM by default. No fine-tuning before reviewed labels exist. No replacement of the linker ladder or of #93's identity resolver. No change to how definitions render. No assessor shipped on a single content type.

Activity

  1. holden commented on Oct 21, 2026

    @holden
    MemberAuthor

    Folded into #193, the curation programme's one open tracker, on the owner's word (21 October 2026). The specification here stands; reopen when the work starts. Status is in the consolidation comment on #193.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions