Skip to content

The kit after nine providers: enforce the contract, share the helpers, make retention real, bring the first generation in line, and show the operator the freshness (phased) #144

Closed

Description

@holden

Audit and repair issue, phased. Nine providers now run through the discovery kit, three generations deep (#88 → #109 → #116/#135), and the question is no longer does each source work (it does: docs/audits/2026-09-21-api-saturation-audit.md) but is the kit one thing: are the nine driven the same way, is what they hold dated and refreshed by rule, can a source's terms about retention be honoured, and does the twelfth provider touch nothing shared. Audited 2026-09-21 on main at 71a7983: the pipeline (Discovery, Transport, Budget, Policy, the workers), every provider module side by side, the conformance kit, the ledger in devils_dictionary_v2, and Artsy's history. Nothing here is a new source; it is the kit catching up with what nine sources taught it.

Refs #109 (the kit and the two archetypes), #116 (many sources, one shelf), #135/#140 (the freshness override and the dated locator), #142 (the first provider with a retention obligation, which this issue makes honourable), #143 (the first provider needing a token, which this issue gives a home), #134 (the trending signal, which needs a refresh the kit does not have), #101 (relevance, out of scope here).


What nine providers taught us, in one paragraph

The kit holds. Every server provider is admitted by the same eight-step admit/5, paced and budgeted by the same transport, ledgered in the same table, cached under the same two refresh clocks, cleaned by the same worker, and rendered by a component that knows none of them by name. The third generation (Openverse, Unsplash, Pexels, Bing) is uniform to the line. What has drifted is (1) the first generation — CineGraph, GIPHY, the Met, Artsy — which predates the conventions the third follows; (2) the helpers, copied provider to provider until the one gate the project calls shared exists in three private copies; (3) the contract, which documents more than it enforces; and (4) freshness and retention, where the kit refreshes well but forgets nothing, and cannot yet keep a promise like "not held longer than 24 hours". None of these is a redesign. Each is one bounded phase below.


1. Freshness: what is true today

Measured from the code (discovery.ex, policy.ex, cleanup_worker.ex) and the ledger.

Question Answer today Where
Does each provider have its own refresh rate? Yes. positive_refresh_seconds and empty_refresh_seconds per slug via source_policies and via <SLUG>_DISCOVERY_*_SECONDS, derived from the registry so a new provider gets both for free. Defaults 30 days and 24 hours. Bing is the one override, 24 h policy.ex:10-15, runtime.exs:40-52, config.exs:124-138
Does a page refresh automatically? On visit, yes. A run past refresh_after is re-queued on the next connected render; the reader keeps seeing the old items until the new run publishes and broadcasts. Stale-while-revalidate, by construction. Nothing refreshes a page nobody visits, and the only cron is cleanup every 15 minutes discovery.ex:682-758, word_live.ex:178-181, config.exs:199-202
Is the reader told how fresh a shelf is? No. refresh_due is computed for every shelf state and read by one test. The definitions' provenance panel shows fetched · changed · materialized; no culture shelf shows when it was fetched discovery.ex:863, provenance.ex:129
Is there a hard expiry? No, and the field that says so is inert. expires_at is written as now on every success and read by nothing; the ledger shows every run "expired" at completion. runs.display_allowed has a :withdrawn branch nothing can reach because nothing sets it false discovery.ex:983, :894
Can a source's retention rule be honoured? No. One global retention_seconds (7 days); cleanup exempts the current display root from the age cutoff, so a displayed run lives until superseded; and deleting a run cascades to its results but never to source_records, which is where the provider's payload lives. Bing's 24 h is a refresh clock, not a hold. The Guardian's "not longer than 24 hours" (#142) and Artsy's "remove on termination" cannot be kept by the kit today discovery.ex:349-399, :1053-1058, migration 20260912103419:33
Is the ledger bounded? Attempts are deleted only with their run; protected roots are never deleted; 402 rows and growing with no pruning path budget.ex:89
Are corpora dated? Yes: generated_at and a checksum on every manifest; they never refresh, by design priv/artworks/manifests/*.json
Failure handling Per-position backoff 5 min (global), provider-wide backoff only from a Retry-After header, taking the max and never shortening config.exs:141, budget.ex:106-129

The one-line verdict: refresh is per-provider and automatic; retention is neither.

2. The contract: documented versus enforced

Documented Enforced Gap
validate_mapping/2 is optional Called unconditionally at discovery.ex:260 and :1253, on every render A provider without it raises mid-run; Providers.retrievable?/1 gates three callbacks, not four
capabilities/0 has six required keys Nothing checks; bare map access A missing key is a KeyError deep in admit/5
operations in capabilities Read by no shared code Decoration
match_details["kind"] Read by no shared code; MatchReason.from_result/2 dispatches on the presence of "tags", "depicts", "keywords", "lines" A twelfth provider with a new reason shape silently becomes :query and fails conformance on any identity shelf. There is no extension point
Optional callbacks are probed covers?, shelf_detail, mapping_identity, retryable_status?, body use Code.ensure_loaded? + function_exported?; identity_record/1 uses function_exported? alone (discovery.ex:1093) Under lazy loading a durable identity can silently degrade to insufficient_evidence
A provider declines with {:deferred, …} Conformance scenarios are `:results :empty
Adding a keyed provider touches no shared file allowed_provider_env is a hand-kept list; an unlisted .env key is silently discarded in dev Silent, not loud
The reader knows no provider by name word_live.ex:217-218 names Giphy.browser_config/1 and UrbanDictionary.browser_config/1; culture.ex:271 renders GiphyShelf by name A second browser-transport provider is three shared edits
mix dd.discovery.check is the preflight It checks CineGraph and nothing else Stale since the second provider
:transient on the server Implemented in publish_success, reachable by no registered provider Dead path, or GIPHY's future home (K10)
An adapter bump invalidates the cache latest_display_root/2 filters adapter_version; pending_or_failure_state/2 does not After a bump a good old run makes the page say :expired

Smaller: cached_run/4 is evaluated twice in one cond clause (:715-716); Budget.claim's trailing case raises CaseClauseError on any unlisted transaction error (budget.ex:97-102); source_policies keys are never checked against the registry, so a renamed slug keeps a dead override; and poetrydb.ex:507 contains a raw NUL byte in a string literal, so file(1) calls it data and grep -r skips the whole provider.

3. Uniformity: three generations side by side

Kept by the third generation, missing from the first. Overridable pacing pair (request_interval_ms, min_retry_interval_ms through a config-aware interval/2): six providers have it, the Met hard-codes min_retry_interval_ms: 1_500, CineGraph, GIPHY and Artsy declare no pacing at all. A public namespace/0: seven have it, CineGraph and the Met inline the string at the item site, which is why Openverse can call Commons.namespace() and nothing can call CineGraph's. The M4 attribution set (attribution, creator, creator_url, license, license_url): the image providers write all five, the Met writes credit_line and artist, CineGraph none. Sparse preview maps (reject nil |> Map.new): five providers; CineGraph and the Met write nil-bearing maps. active: true in source_attrs: seven have it, four do not. adapter_version as a module attribute: nine; Artsy and GIPHY inline a literal.

Kept by the first generation, missing from the third. identity_record/1 (the SourceIdentity.Adapter behaviour): CineGraph, Met, Commons, Open Library, PoetryDB. mapping_identity/1: Met and Commons only. retryable_status?/1: the Met only. A second identifier namespace from the payload (imdb_title, wikidata): CineGraph and the Met; Unsplash and Pexels write one identifier that restates the external id.

Copied, not shared. The counts are of byte-identical or near-identical private functions:

Helper Copies
presence/1 (trim → nil) 8
headers/0 (the shared User-Agent) 9
interval/2 (config-overridable pacing) 6, plus a divergent seventh in the Met
offset/1 (cursor → non-negative int) 5, in two shapes
limit/1 7, in two shapes
word_pattern/1 — the whole-word Unicode-boundary gate 3 (PoetryDB, Open Library, Bing), two of whose moduledocs name PoetryDB's function as the authority for the copy they hold
media/1 (absolute https URL or nil) 3
upload/1 (per-page {creator, title} fold) 3
label clamp to varchar(255) 7 places
year-from-text 5 different parsers
drop_hidden/1 (the display:none regex) 2, differing in a quote class
page/6 + next_cursor/4 3 structurally identical copies (Openverse, Pexels, Unsplash)

The only real cross-provider call is Openverse → Commons.namespace(), deliberate and documented. BingNews.normalize/1 and Poetrydb.first_attestation/2 are public, @doc'd, and called by nobody else; #142 plans to call the first from the Guardian, which would make a provider depend on a sibling for a rule that belongs to the kit.

4. Artsy: why it is off, and whether to turn it on

Why it is off. The Artsy public API is being retired (announced 2026-09-11, no date); its terms grant non-commercial use, reasonable caching with freshness obligations, and removal of all API content on termination, which is not the durable licence the encyclopedia needs. The feasibility probe measured 59 artwork hits, 3 hydrated, 56 answering 404, gene-filtered acquisition unverifiable ("do not seed through gene traversal"), and a pagination link that dropped its own type=artwork. #109 Phase 3a (c390c4c) deleted the private client, coordinator and availability check (1,054 lines) and left the module registry-only: background: false, no retrieve/4, a validate_mapping/2 for a parameter shape nothing builds, a 34-line fixture returning empty pages, and no test stanza, so enabled?/0 is false in the suite. Its 43 artworks and gene mappings reach pages through the catalog, as :gene identity, and that works.

Today. The API still answers: token exchange 201, /api/artworks 200, /api/search 200 (one probe each, 2026-09-21).

Can it be enabled? Technically, as an ordinary kit provider rather than the deleted client: a token in request_options/1 (the same token cache #143 builds for Spotify), the shared transport's pacing instead of the old coordinator, automatic_mapping, retrieve, a real fixture. Three things say not to:

  1. The :artwork row admits :identity only, and a live Artsy search result carries a gene, not a QID. Live Artsy is a :query provider on an identity shelf, which conformance refuses by design.
  2. The hydration rate (3 of 59) means a shelf that is mostly empty after spending its budget.
  3. The terms require removal on termination, which Phase 2 makes possible but which turns Artsy into the only provider whose whole cache can be ordered deleted on a day's notice.

Recommendation. Remove Artsy from :discovery_providers. Keep its source row and its catalog through the seeded rows (they already arrive through Sources.Catalog and the manifests). Its conformance suite, fixture and stanza go with it; the registry-only profile stays for GIPHY. It stops being "a provider that never runs" and becomes what it is: a corpus. Revisit only if the owner wants a disposable Artsy shelf after Phase 2 and #143 land, and then as a :query source on a row that admits one.


Phases

Each phase is one PR, one session; each ends with mix precommit clean, the full suite green, and a short report in the PR naming the shared files touched. Order matters: 0 makes the contract honest, 1 makes providers small, 2 makes retention real, 3 brings the old four in line, 4 gives the owner the view.

Phase 0 — the contract enforces what it documents

  • Providers.pipeline_callbacks gains validate_mapping: 2 (and source_attrs: 0 is checked at registration), so retrievable?/1 is the whole truth.
  • A compile-time or boot-time check of capabilities/0: the six documented keys present with the right types, content_types ⊆ ContentTypes.known(), pacing keys integers or absent. Refuse to boot on a bad map, the way ContentTypes refuses a bad row.
  • identity_proposal/2 probes with Code.ensure_loaded? like every other probe.
  • MatchReason gets an explicit evidence declaration. A result writes match_details["evidence"] ∈ {identity, attestation, query} and a "kind" the reason builder dispatches on; the four magic keys become the legacy path with a deprecation note. Conformance asserts the declaration is present and matches the row. This is the extension point a twelfth provider needs.
  • A :throttled scenario in Conformance.Fixture: the stub answers 429 with Retry-After on the first request and 200 on the second; the suite asserts one deferral, one provider-wide backoff row, one success, and the ledger shape.
  • source_policies keys validated against the registry at boot; operations either read or removed from the documented shape.
  • The NUL byte in poetrydb.ex:507 becomes \0.
  • pending_or_failure_state/2 filters adapter_version; the double cached_run/4 becomes one; Budget.claim's case gets an {:error, other} clause.

Phase 1 — one kit, not eleven copies

  • DevilsDictionary.Discovery.Provider.Helpers (or defaults injected by use DevilsDictionary.Discovery.Provider): presence/1, headers/0, config/1, interval/2, offset/1, limit/2, media_url/1, clamp_label/1, year/1, sparse/1 (the nil-rejecting map), and whole_word?/2 and word_pattern/1 as the attestation gate, with PoetryDB, Open Library and Bing calling it. BingNews.normalize/1 and article_id/1 move to a Helpers.News (or the kit) so The Guardian joins the News shelf: keyed, verified in the text, 24-hour retention (Phase 1 of #134) #142's Guardian depends on the kit and not on Bing.
  • Every provider migrated; conformance unchanged and green before and after; a test that the three attestation providers produce the same gate result on the same fixture lines.
  • mix dd.provider.new emits calls to the helpers, not copies.
  • Acceptance: the copy table above reads 1 across, and the provider line counts fall (expect Openverse, Pexels, Unsplash to lose the page/6 triplet).

Phase 2 — retention is a rule, not a hope

Phase 3 — the first generation joins the third

  • CineGraph, the Met, GIPHY: pacing keys through the shared interval/2 (the Met's min_retry_interval_ms overridable), namespace/0, active: true, sparse preview maps, @adapter_version, the M4 attribution set where the row wants it (the Met's :credited row: attribution from creditLine, creator_url from the artist's ULAN or Wikidata where the object carries one).
  • Browser transport generalised: browser_config/1 becomes an optional callback in Provider; WordLive iterates Providers.all() filtered on transport: :browser and assigns a list; Culture renders a browser shelf through the content-type row like any other, with the GIF row's evidence: [:query] already admitting it. GIPHY's hook keeps its client-side accounting; the ledger gains nothing it cannot (K10 stands: nothing is stored), but the reader rule (knows no provider by name) becomes true. Urban Dictionary's card is a definition, not a shelf, and stays where it is under the on-demand registry.
  • Artsy leaves the provider registry per §4; its source row, catalog rows and mappings stay; dd.artsy.withdraw stays.
  • Conformance coverage test asserts every registered provider is either retrievable? or transport: :browser — no third state.

Phase 4 — the operator's view

  • mix dd.discovery.check checks every registered provider: enabled, key present where enabled?/0 needs one, endpoint sane, policy keys known. Never prints a key.
  • mix dd.discovery.status: one row per provider from the ledger — runs, last success, results held, stale roots (past refresh_after), retention due, budget used in the current window, provider-wide backoff if any. The same numbers on /admin/imports or a sibling page, since dd.health already renders there.
  • docs/discovery/README.md gains a Freshness and retention section stating what this issue made true, and the promise table's row 6 gains retention beside the ledger.

Acceptance for the issue as a whole

Activity

  1. added 6 commits that reference this issue on Sep 22, 2026
  2. holden commented on Sep 22, 2026

    @holden
    MemberAuthor

    Phase 3's browser proof — taken, 2026-09-22

    • /define/war and /define/love at 1280 px and 375 px, light and dark, showing the GIF shelf rendered through the shared section rather than its old component.

    Eight shots in docs/discovery/issue-144-phase3-{word}-{width}[-dark]-2026-09-22.jpg, device scale 1, JPEG q72, 24–66 KB. Driven over CDP with Emulation.setDeviceMetricsOverride and an emulated prefers-color-scheme — --window-size sets the window and not the layout viewport.

    Measured on all eight, out of the live DOM:

    • <section id="culture-browser-giphy-…" phx-hook="GiphyShelf" phx-update="ignore">, and #in-culture contains it
    • heading GIFs, from the :gif row of ContentTypes; data-column="w-24 sm:w-28" and data-title-clamp="line-clamp-1" on the section are that row's too, and the cards the hook built carry both
    • the byline column: the name GIPHY, and under it <a id="culture-mark-giphy" href="https://giphy.com/"> with exactly one visible badge image — /images/giphy-powered-by.png at 80 px — in both themes
    • document.documentElement.scrollWidth === 375 at 375 px, === 1280 at 1280 px
    • console clean (one shot logged (Bandit.HTTPError) Read timeout from the dev server's own channel on the first take and nothing on the retake)
    • every fetch went to api.giphy.com and api.urbandictionary.com — the two browser sources' two hooks, each to its own API and nowhere else. The GIFs themselves come from media0-4.giphy.com, hotlinked as the terms require

    The shots are of the branch's final state, so the GIF shelf's mark in them is the unified one from followup 2 below.

    Both other followups are in the same PR: one mark instead of three (attribution_mark/0 with :placement, replacing preview_metadata["brand_mark"] and browser_config/1's badge), and the Guardian and Spotify taking Phase 1's kit — eight private copies each, deleted.

    → #151. Gate: 1,692 tests, 0 failures.

    This issue stays open for Phase 4, the operator's view.

    🤖 Generated with Claude Code

  3. added a commit that references this issue on Sep 22, 2026
  4. holden commented on Sep 22, 2026

    @holden
    MemberAuthor

    Phase 4 is up as #153, and with it the six acceptance boxes are settled. Going through them one at a time, with what says so.

    • 1. A twelfth provider touches its module, fixture, suite, doc, config stanzas and allowed_provider_env, and nothing else. The Guardian (The Guardian joins the News shelf: keyed, verified in the text, 24-hour retention (Phase 1 of #134) #142) and Spotify (Spotify joins the world: a Music shelf from Spotify search, cover art and a link back with the mark (the :music row) #143) are the twelfth and thirteenth, and both PRs say so. The honest footnote is that both merged either side of Phase 1 and shipped holding their own copies of the kit; a04530a brought them onto Provider.Helpers — 73 insertions against 131 deletions across the two modules, conformance untouched.
    • 2. grep -c 'defp word_pattern' lib/devils_dictionary/discovery/providers/*.ex returns 0. 0 for every one of the twelve, on main today.
    • 3. A source with a short retention_seconds holds nothing older than that, including its source_records. retention_test.exs: a result an hour old under a half-hour policy is withdrawn from display, its results deleted and its source record deleted with them; the same result under the suite's default is untouched; a source record the encyclopedia is standing on survives. The Guardian's own suite holds its 23-hour policy against clause 5's "whether or not published". 65 tests across the three files, 0 failures.
    • 4. Every shelf on /define/war says when it was fetched or since when it is held. This one was not true on main, and I checked rather than assumed. Six shelves said it — film, artworks, images, text, news, music, reading Fetched 4 days ago · refreshes on your next visit after…, and the artwork shelf's corpus half adding catalog held since 17 Sep 2026. The GIFs shelf said nothing, because freshness/1 reads a display root and a browser shelf has none: the request is the reader's browser's and nothing it returns is stored. That is K10 working, but silence reads as nobody knows how old this is rather than this has no age. The operator's view: one preflight for every provider, the ledger in a table, and the freshness a browser shelf can claim (#144 Phase 4) #153 adds the one line, in the shared browser shelf and naming no provider: Fetched by your browser on this visit; nothing is stored here. All seven shelves now answer.
    • 5. mix dd.discovery.status runs on a keyless checkout and prints nine rows, not one. Twelve rows now, which is what the registry has. With the .env moved aside: twelve rows, six ready (the keyless providers, the Met among them), six no key, exit 0. mix dd.discovery.check on the same checkout prints twelve rows and 6/12 ready · 0 switched off · 6 waiting for a credential, also exit 0 — a missing credential is a deployment state, not a preflight failure.
    • 6. Artsy is a corpus, GIPHY is a browser provider through the row, and Providers.all() has no member that can neither run nor render. conformance_coverage_test.exs, every registered provider can either run or render — there is no third state, asserted against retrievable?/1 and browser?/1 for the whole registry.

    What Phase 4 itself added, briefly: one module, Discovery.Status, whose preflight/0 and rows/0 are rendered by mix dd.discovery.check, mix dd.discovery.status and a new /ops/discovery — the arrangement Health.records/1 has with mix dd.health and /ops/imports. The check task walked CineGraph by name since the second provider registered; it now walks Providers.all(), calls Providers.validate!/0 rather than re-asking its questions, and names no provider anywhere. A credential is reported present or missing from a boolean map built in config/runtime.exs, which is the only scope that holds both the exported environment and the development .env; no value leaves it, and the check task's suite greps the task's whole output for every configured secret.

    mix precommit: 1,722 tests, 0 failures, from 1,692 on main. Screenshots of the page at 1280 and 375 px, light and dark, are in the PR and committed under docs/discovery/.

    Closing this with the PR.

  5. added a commit that references this issue on Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions