Repository navigation
Description
Audit and repair issue, phased. Nine providers now run through the discovery kit, three generations deep (#88 → #109 → #116/#135), and the question is no longer does each source work (it does: docs/audits/2026-09-21-api-saturation-audit.md) but is the kit one thing: are the nine driven the same way, is what they hold dated and refreshed by rule, can a source's terms about retention be honoured, and does the twelfth provider touch nothing shared. Audited 2026-09-21 on main at 71a7983: the pipeline (Discovery, Transport, Budget, Policy, the workers), every provider module side by side, the conformance kit, the ledger in devils_dictionary_v2, and Artsy's history. Nothing here is a new source; it is the kit catching up with what nine sources taught it.
Refs #109 (the kit and the two archetypes), #116 (many sources, one shelf), #135/#140 (the freshness override and the dated locator), #142 (the first provider with a retention obligation, which this issue makes honourable), #143 (the first provider needing a token, which this issue gives a home), #134 (the trending signal, which needs a refresh the kit does not have), #101 (relevance, out of scope here).
What nine providers taught us, in one paragraph
The kit holds. Every server provider is admitted by the same eight-step admit/5, paced and budgeted by the same transport, ledgered in the same table, cached under the same two refresh clocks, cleaned by the same worker, and rendered by a component that knows none of them by name. The third generation (Openverse, Unsplash, Pexels, Bing) is uniform to the line. What has drifted is (1) the first generation — CineGraph, GIPHY, the Met, Artsy — which predates the conventions the third follows; (2) the helpers, copied provider to provider until the one gate the project calls shared exists in three private copies; (3) the contract, which documents more than it enforces; and (4) freshness and retention, where the kit refreshes well but forgets nothing, and cannot yet keep a promise like "not held longer than 24 hours". None of these is a redesign. Each is one bounded phase below.
1. Freshness: what is true today
Measured from the code (discovery.ex, policy.ex, cleanup_worker.ex) and the ledger.
| Question | Answer today | Where |
|---|---|---|
| Does each provider have its own refresh rate? | Yes. positive_refresh_seconds and empty_refresh_seconds per slug via source_policies and via <SLUG>_DISCOVERY_*_SECONDS, derived from the registry so a new provider gets both for free. Defaults 30 days and 24 hours. Bing is the one override, 24 h |
policy.ex:10-15, runtime.exs:40-52, config.exs:124-138 |
| Does a page refresh automatically? | On visit, yes. A run past refresh_after is re-queued on the next connected render; the reader keeps seeing the old items until the new run publishes and broadcasts. Stale-while-revalidate, by construction. Nothing refreshes a page nobody visits, and the only cron is cleanup every 15 minutes |
discovery.ex:682-758, word_live.ex:178-181, config.exs:199-202 |
| Is the reader told how fresh a shelf is? | No. refresh_due is computed for every shelf state and read by one test. The definitions' provenance panel shows fetched · changed · materialized; no culture shelf shows when it was fetched |
discovery.ex:863, provenance.ex:129 |
| Is there a hard expiry? | No, and the field that says so is inert. expires_at is written as now on every success and read by nothing; the ledger shows every run "expired" at completion. runs.display_allowed has a :withdrawn branch nothing can reach because nothing sets it false |
discovery.ex:983, :894 |
| Can a source's retention rule be honoured? | No. One global retention_seconds (7 days); cleanup exempts the current display root from the age cutoff, so a displayed run lives until superseded; and deleting a run cascades to its results but never to source_records, which is where the provider's payload lives. Bing's 24 h is a refresh clock, not a hold. The Guardian's "not longer than 24 hours" (#142) and Artsy's "remove on termination" cannot be kept by the kit today |
discovery.ex:349-399, :1053-1058, migration 20260912103419:33 |
| Is the ledger bounded? | Attempts are deleted only with their run; protected roots are never deleted; 402 rows and growing with no pruning path | budget.ex:89 |
| Are corpora dated? | Yes: generated_at and a checksum on every manifest; they never refresh, by design |
priv/artworks/manifests/*.json |
| Failure handling | Per-position backoff 5 min (global), provider-wide backoff only from a Retry-After header, taking the max and never shortening |
config.exs:141, budget.ex:106-129 |
The one-line verdict: refresh is per-provider and automatic; retention is neither.
2. The contract: documented versus enforced
| Documented | Enforced | Gap |
|---|---|---|
validate_mapping/2 is optional |
Called unconditionally at discovery.ex:260 and :1253, on every render |
A provider without it raises mid-run; Providers.retrievable?/1 gates three callbacks, not four |
capabilities/0 has six required keys |
Nothing checks; bare map access | A missing key is a KeyError deep in admit/5 |
operations in capabilities |
Read by no shared code | Decoration |
match_details["kind"] |
Read by no shared code; MatchReason.from_result/2 dispatches on the presence of "tags", "depicts", "keywords", "lines" |
A twelfth provider with a new reason shape silently becomes :query and fails conformance on any identity shelf. There is no extension point |
| Optional callbacks are probed | covers?, shelf_detail, mapping_identity, retryable_status?, body use Code.ensure_loaded? + function_exported?; identity_record/1 uses function_exported? alone (discovery.ex:1093) |
Under lazy loading a durable identity can silently degrade to insufficient_evidence |
A provider declines with {:deferred, …} |
Conformance scenarios are `:results | :empty |
| Adding a keyed provider touches no shared file | allowed_provider_env is a hand-kept list; an unlisted .env key is silently discarded in dev |
Silent, not loud |
| The reader knows no provider by name | word_live.ex:217-218 names Giphy.browser_config/1 and UrbanDictionary.browser_config/1; culture.ex:271 renders GiphyShelf by name |
A second browser-transport provider is three shared edits |
mix dd.discovery.check is the preflight |
It checks CineGraph and nothing else | Stale since the second provider |
:transient on the server |
Implemented in publish_success, reachable by no registered provider |
Dead path, or GIPHY's future home (K10) |
| An adapter bump invalidates the cache | latest_display_root/2 filters adapter_version; pending_or_failure_state/2 does not |
After a bump a good old run makes the page say :expired |
Smaller: cached_run/4 is evaluated twice in one cond clause (:715-716); Budget.claim's trailing case raises CaseClauseError on any unlisted transaction error (budget.ex:97-102); source_policies keys are never checked against the registry, so a renamed slug keeps a dead override; and poetrydb.ex:507 contains a raw NUL byte in a string literal, so file(1) calls it data and grep -r skips the whole provider.
3. Uniformity: three generations side by side
Kept by the third generation, missing from the first. Overridable pacing pair (request_interval_ms, min_retry_interval_ms through a config-aware interval/2): six providers have it, the Met hard-codes min_retry_interval_ms: 1_500, CineGraph, GIPHY and Artsy declare no pacing at all. A public namespace/0: seven have it, CineGraph and the Met inline the string at the item site, which is why Openverse can call Commons.namespace() and nothing can call CineGraph's. The M4 attribution set (attribution, creator, creator_url, license, license_url): the image providers write all five, the Met writes credit_line and artist, CineGraph none. Sparse preview maps (reject nil |> Map.new): five providers; CineGraph and the Met write nil-bearing maps. active: true in source_attrs: seven have it, four do not. adapter_version as a module attribute: nine; Artsy and GIPHY inline a literal.
Kept by the first generation, missing from the third. identity_record/1 (the SourceIdentity.Adapter behaviour): CineGraph, Met, Commons, Open Library, PoetryDB. mapping_identity/1: Met and Commons only. retryable_status?/1: the Met only. A second identifier namespace from the payload (imdb_title, wikidata): CineGraph and the Met; Unsplash and Pexels write one identifier that restates the external id.
Copied, not shared. The counts are of byte-identical or near-identical private functions:
| Helper | Copies |
|---|---|
presence/1 (trim → nil) |
8 |
headers/0 (the shared User-Agent) |
9 |
interval/2 (config-overridable pacing) |
6, plus a divergent seventh in the Met |
offset/1 (cursor → non-negative int) |
5, in two shapes |
limit/1 |
7, in two shapes |
word_pattern/1 — the whole-word Unicode-boundary gate |
3 (PoetryDB, Open Library, Bing), two of whose moduledocs name PoetryDB's function as the authority for the copy they hold |
media/1 (absolute https URL or nil) |
3 |
upload/1 (per-page {creator, title} fold) |
3 |
label clamp to varchar(255) |
7 places |
| year-from-text | 5 different parsers |
drop_hidden/1 (the display:none regex) |
2, differing in a quote class |
page/6 + next_cursor/4 |
3 structurally identical copies (Openverse, Pexels, Unsplash) |
The only real cross-provider call is Openverse → Commons.namespace(), deliberate and documented. BingNews.normalize/1 and Poetrydb.first_attestation/2 are public, @doc'd, and called by nobody else; #142 plans to call the first from the Guardian, which would make a provider depend on a sibling for a rule that belongs to the kit.
4. Artsy: why it is off, and whether to turn it on
Why it is off. The Artsy public API is being retired (announced 2026-09-11, no date); its terms grant non-commercial use, reasonable caching with freshness obligations, and removal of all API content on termination, which is not the durable licence the encyclopedia needs. The feasibility probe measured 59 artwork hits, 3 hydrated, 56 answering 404, gene-filtered acquisition unverifiable ("do not seed through gene traversal"), and a pagination link that dropped its own type=artwork. #109 Phase 3a (c390c4c) deleted the private client, coordinator and availability check (1,054 lines) and left the module registry-only: background: false, no retrieve/4, a validate_mapping/2 for a parameter shape nothing builds, a 34-line fixture returning empty pages, and no test stanza, so enabled?/0 is false in the suite. Its 43 artworks and gene mappings reach pages through the catalog, as :gene identity, and that works.
Today. The API still answers: token exchange 201, /api/artworks 200, /api/search 200 (one probe each, 2026-09-21).
Can it be enabled? Technically, as an ordinary kit provider rather than the deleted client: a token in request_options/1 (the same token cache #143 builds for Spotify), the shared transport's pacing instead of the old coordinator, automatic_mapping, retrieve, a real fixture. Three things say not to:
- The
:artworkrow admits:identityonly, and a live Artsy search result carries a gene, not a QID. Live Artsy is a:queryprovider on an identity shelf, which conformance refuses by design. - The hydration rate (3 of 59) means a shelf that is mostly empty after spending its budget.
- The terms require removal on termination, which Phase 2 makes possible but which turns Artsy into the only provider whose whole cache can be ordered deleted on a day's notice.
Recommendation. Remove Artsy from :discovery_providers. Keep its source row and its catalog through the seeded rows (they already arrive through Sources.Catalog and the manifests). Its conformance suite, fixture and stanza go with it; the registry-only profile stays for GIPHY. It stops being "a provider that never runs" and becomes what it is: a corpus. Revisit only if the owner wants a disposable Artsy shelf after Phase 2 and #143 land, and then as a :query source on a row that admits one.
Phases
Each phase is one PR, one session; each ends with mix precommit clean, the full suite green, and a short report in the PR naming the shared files touched. Order matters: 0 makes the contract honest, 1 makes providers small, 2 makes retention real, 3 brings the old four in line, 4 gives the owner the view.
Phase 0 — the contract enforces what it documents
Providers.pipeline_callbacksgainsvalidate_mapping: 2(andsource_attrs: 0is checked at registration), soretrievable?/1is the whole truth.- A compile-time or boot-time check of
capabilities/0: the six documented keys present with the right types,content_types ⊆ ContentTypes.known(), pacing keys integers or absent. Refuse to boot on a bad map, the wayContentTypesrefuses a bad row. identity_proposal/2probes withCode.ensure_loaded?like every other probe.MatchReasongets an explicit evidence declaration. A result writesmatch_details["evidence"] ∈ {identity, attestation, query}and a"kind"the reason builder dispatches on; the four magic keys become the legacy path with a deprecation note. Conformance asserts the declaration is present and matches the row. This is the extension point a twelfth provider needs.- A
:throttledscenario inConformance.Fixture: the stub answers429withRetry-Afteron the first request and200on the second; the suite asserts one deferral, one provider-wide backoff row, one success, and the ledger shape. source_policieskeys validated against the registry at boot;operationseither read or removed from the documented shape.- The NUL byte in
poetrydb.ex:507becomes\0. pending_or_failure_state/2filtersadapter_version; the doublecached_run/4becomes one;Budget.claim'scasegets an{:error, other}clause.
Phase 1 — one kit, not eleven copies
DevilsDictionary.Discovery.Provider.Helpers(or defaults injected byuse DevilsDictionary.Discovery.Provider):presence/1,headers/0,config/1,interval/2,offset/1,limit/2,media_url/1,clamp_label/1,year/1,sparse/1(the nil-rejecting map), andwhole_word?/2andword_pattern/1as the attestation gate, with PoetryDB, Open Library and Bing calling it.BingNews.normalize/1andarticle_id/1move to aHelpers.News(or the kit) so The Guardian joins the News shelf: keyed, verified in the text, 24-hour retention (Phase 1 of #134) #142's Guardian depends on the kit and not on Bing.- Every provider migrated; conformance unchanged and green before and after; a test that the three attestation providers produce the same gate result on the same fixture lines.
mix dd.provider.newemits calls to the helpers, not copies.- Acceptance: the copy table above reads 1 across, and the provider line counts fall (expect Openverse, Pexels, Unsplash to lose the
page/6triplet).
Phase 2 — retention is a rule, not a hope
- Per-source
retention_secondsinPolicy.@keys(and the derived<SLUG>_DISCOVERY_RETENTION_SECONDS), honoured byDiscovery.cleanup/0including the current display root: a run past its source's retention is withdrawn from display (runs.display_allowed = false, the branch that exists and nothing reaches) and its results and theirsource_recordsdeleted when no other result references them. A test: a 25-hour-old Guardian-policy result is gone aftercleanup/0, a 25-hour-old Bing result is not, and a source record referenced by a corpus row survives. expires_atbecomes real —completed_at + retention— and is what cleanup reads; or it is dropped with its constraint. Not left inert.- The page whose root was withdrawn by retention shows the honest empty and re-queues on the next visit (today it would say
:withdrawn). discovery_request_attemptspruned toretained_attempts_per_positionfor protected roots too, on the same tick.- Per-source
failure_backoff_secondsjoins@keys(GDELT's "one 429 closes the gate for a minute" needs it, Current words: a trending signal (Wikipedia top-1,000, keyless), a News shelf (Guardian then GDELT) and Urban Dictionary by permission — tested on /define/bestiality, 2026-09-15 #134). - The reader is told the age of a shelf:
refresh_dueand the root'scompleted_atreach the shelf header as fetched 3 days ago · refreshes on your next visit after 21 Oct (wording to taste, Word page height and shelf templates: fold the prose, one template per content type, the world above the fold — measured on /define/love, phased #111's template question), and a corpus shelf says held since its manifest'sgenerated_at. This is the "properly dated" the owner asked for, and it costs one assign. - A seam, not a feature:
Discovery.request/3acceptsrefresh: truetoday; a cron that calls it for the words a trending signal names is Current words: a trending signal (Wikipedia top-1,000, keyless), a News shelf (Guardian then GDELT) and Urban Dictionary by permission — tested on /define/bestiality, 2026-09-15 #134 Phase 0's job and needs nothing here beyond the seam existing.
Phase 3 — the first generation joins the third
- CineGraph, the Met, GIPHY: pacing keys through the shared
interval/2(the Met'smin_retry_interval_msoverridable),namespace/0,active: true, sparse preview maps,@adapter_version, the M4 attribution set where the row wants it (the Met's:creditedrow:attributionfromcreditLine,creator_urlfrom the artist's ULAN or Wikidata where the object carries one). - Browser transport generalised:
browser_config/1becomes an optional callback inProvider;WordLiveiteratesProviders.all()filtered ontransport: :browserand assigns a list;Culturerenders a browser shelf through the content-type row like any other, with the GIF row'sevidence: [:query]already admitting it. GIPHY's hook keeps its client-side accounting; the ledger gains nothing it cannot (K10 stands: nothing is stored), but the reader rule (knows no provider by name) becomes true. Urban Dictionary's card is a definition, not a shelf, and stays where it is under the on-demand registry. - Artsy leaves the provider registry per §4; its source row, catalog rows and mappings stay;
dd.artsy.withdrawstays. - Conformance coverage test asserts every registered provider is either
retrievable?ortransport: :browser— no third state.
Phase 4 — the operator's view
mix dd.discovery.checkchecks every registered provider: enabled, key present whereenabled?/0needs one, endpoint sane, policy keys known. Never prints a key.mix dd.discovery.status: one row per provider from the ledger — runs, last success, results held, stale roots (pastrefresh_after), retention due, budget used in the current window, provider-wide backoff if any. The same numbers on/admin/importsor a sibling page, sincedd.healthalready renders there.docs/discovery/README.mdgains a Freshness and retention section stating what this issue made true, and the promise table's row 6 gains retention beside the ledger.
Acceptance for the issue as a whole
- A twelfth provider (the Guardian, The Guardian joins the News shelf: keyed, verified in the text, 24-hour retention (Phase 1 of #134) #142, or Spotify, Spotify joins the world: a Music shelf from Spotify search, cover art and a link back with the mark (the
:musicrow) #143 — whichever lands first) is written against Phase 0 and Phase 1 and touches: its module, its fixture, its suite, its doc, its config stanzas, andallowed_provider_env. Nothing else. The PR says so. -
grep -c 'defp word_pattern' lib/devils_dictionary/discovery/providers/*.exreturns 0. - A source with
retention_seconds: 86_400holds nothing older than a day, provably, including itssource_records. - Every shelf on
/define/warsays when it was fetched or since when it is held. (Six did; the browser shelf had no display root to read and said nothing — one line in Phase 4, #153.) -
mix dd.discovery.statusruns on a keyless checkout and prints nine rows, not one. (Twelve rows, which is what the registry has now.) - Artsy is a corpus, GIPHY is a browser provider through the row, and
Providers.all()has no member that can neither run nor render.
Activity
The review round on #146: promise 9, and three threads
Merge pull request #147 from razrfly/claude/144-kit-de9e9d
Nine providers, three generations deep, had arrived at the same dozen…
Merge pull request #148 from razrfly/claude/144-kit-phase1
Merge pull request #149 from razrfly/claude/144-kit-phase2
Merge pull request #150 from razrfly/claude/144-kit-phase3
holden commentedon Sep 22, 2026
Phase 3's browser proof — taken, 2026-09-22
/define/warand/define/loveat 1280 px and 375 px, light and dark, showing the GIF shelf rendered through the shared section rather than its old component.Eight shots in
docs/discovery/issue-144-phase3-{word}-{width}[-dark]-2026-09-22.jpg, device scale 1, JPEG q72, 24–66 KB. Driven over CDP withEmulation.setDeviceMetricsOverrideand an emulatedprefers-color-scheme—--window-sizesets the window and not the layout viewport.Measured on all eight, out of the live DOM:
<section id="culture-browser-giphy-…" phx-hook="GiphyShelf" phx-update="ignore">, and#in-culturecontains it:gifrow ofContentTypes;data-column="w-24 sm:w-28"anddata-title-clamp="line-clamp-1"on the section are that row's too, and the cards the hook built carry both<a id="culture-mark-giphy" href="https://giphy.com/">with exactly one visible badge image —/images/giphy-powered-by.pngat 80 px — in both themesdocument.documentElement.scrollWidth === 375at 375 px,=== 1280at 1280 px(Bandit.HTTPError) Read timeoutfrom the dev server's own channel on the first take and nothing on the retake)fetchwent toapi.giphy.comandapi.urbandictionary.com— the two browser sources' two hooks, each to its own API and nowhere else. The GIFs themselves come frommedia0-4.giphy.com, hotlinked as the terms requireThe shots are of the branch's final state, so the GIF shelf's mark in them is the unified one from followup 2 below.
Both other followups are in the same PR: one mark instead of three (
attribution_mark/0with:placement, replacingpreview_metadata["brand_mark"]andbrowser_config/1'sbadge), and the Guardian and Spotify taking Phase 1's kit — eight private copies each, deleted.→ #151. Gate: 1,692 tests, 0 failures.
This issue stays open for Phase 4, the operator's view.
🤖 Generated with Claude Code
Merge pull request #151 from razrfly/claude/issue-144-followups-d2ad47
holden commentedon Sep 22, 2026
Phase 4 is up as #153, and with it the six acceptance boxes are settled. Going through them one at a time, with what says so.
allowed_provider_env, and nothing else. The Guardian (The Guardian joins the News shelf: keyed, verified in the text, 24-hour retention (Phase 1 of #134) #142) and Spotify (Spotify joins the world: a Music shelf from Spotify search, cover art and a link back with the mark (the:musicrow) #143) are the twelfth and thirteenth, and both PRs say so. The honest footnote is that both merged either side of Phase 1 and shipped holding their own copies of the kit;a04530abrought them ontoProvider.Helpers— 73 insertions against 131 deletions across the two modules, conformance untouched.grep -c 'defp word_pattern' lib/devils_dictionary/discovery/providers/*.exreturns 0. 0 for every one of the twelve, onmaintoday.retention_secondsholds nothing older than that, including itssource_records.retention_test.exs: a result an hour old under a half-hour policy is withdrawn from display, its results deleted and its source record deleted with them; the same result under the suite's default is untouched; a source record the encyclopedia is standing on survives. The Guardian's own suite holds its 23-hour policy against clause 5's "whether or not published". 65 tests across the three files, 0 failures./define/warsays when it was fetched or since when it is held. This one was not true onmain, and I checked rather than assumed. Six shelves said it — film, artworks, images, text, news, music, reading Fetched 4 days ago · refreshes on your next visit after…, and the artwork shelf's corpus half adding catalog held since 17 Sep 2026. The GIFs shelf said nothing, becausefreshness/1reads a display root and a browser shelf has none: the request is the reader's browser's and nothing it returns is stored. That is K10 working, but silence reads as nobody knows how old this is rather than this has no age. The operator's view: one preflight for every provider, the ledger in a table, and the freshness a browser shelf can claim (#144 Phase 4) #153 adds the one line, in the shared browser shelf and naming no provider: Fetched by your browser on this visit; nothing is stored here. All seven shelves now answer.mix dd.discovery.statusruns on a keyless checkout and prints nine rows, not one. Twelve rows now, which is what the registry has. With the.envmoved aside: twelve rows, sixready(the keyless providers, the Met among them), sixno key, exit 0.mix dd.discovery.checkon the same checkout prints twelve rows and6/12 ready · 0 switched off · 6 waiting for a credential, also exit 0 — a missing credential is a deployment state, not a preflight failure.Providers.all()has no member that can neither run nor render.conformance_coverage_test.exs, every registered provider can either run or render — there is no third state, asserted againstretrievable?/1andbrowser?/1for the whole registry.What Phase 4 itself added, briefly: one module,
Discovery.Status, whosepreflight/0androws/0are rendered bymix dd.discovery.check,mix dd.discovery.statusand a new/ops/discovery— the arrangementHealth.records/1has withmix dd.healthand/ops/imports. The check task walked CineGraph by name since the second provider registered; it now walksProviders.all(), callsProviders.validate!/0rather than re-asking its questions, and names no provider anywhere. A credential is reported present or missing from a boolean map built inconfig/runtime.exs, which is the only scope that holds both the exported environment and the development.env; no value leaves it, and the check task's suite greps the task's whole output for every configured secret.mix precommit: 1,722 tests, 0 failures, from 1,692 onmain. Screenshots of the page at 1280 and 375 px, light and dark, are in the PR and committed underdocs/discovery/.Closing this with the PR.
Merge pull request #153 from razrfly/claude/phase-4-operators-view-82…