Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion config/config.exs
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ config :devils_dictionary,
# page all read it, and none of them names a provider itself.
config :devils_dictionary, :discovery_providers, [
DevilsDictionary.Discovery.Providers.Artsy,
DevilsDictionary.Discovery.Providers.BingNews,
DevilsDictionary.Discovery.Providers.CineGraph,
DevilsDictionary.Discovery.Providers.Commons,
DevilsDictionary.Discovery.Providers.Giphy,
Expand Down Expand Up @@ -128,7 +129,13 @@ config :devils_dictionary, :discovery,
"giphy" => [request_budget_limit: 100, request_budget_window_seconds: 60 * 60],
# One Met page is a search plus up to `Met.scan_window/0` hydrations, and the
# probe put the sustainable rate near 1 req/s rather than the documented 80.
"met" => [request_budget_limit: 1_000, request_budget_window_seconds: 3_600]
"met" => [request_budget_limit: 1_000, request_budget_window_seconds: 3_600],
# A News shelf cached for the shipped 30 days would be frozen at whatever
# was in the news the day a reader first opened the page. A day is what
# "current" costs (#135). The empty refresh is already 24 h and stays
# there. `BING_NEWS_DISCOVERY_POSITIVE_REFRESH_SECONDS` overrides this at
# runtime, as it does every provider's.
"bing-news" => [positive_refresh_seconds: 24 * 60 * 60]
},
refresh_cooldown_seconds: 60,
failure_backoff_seconds: 5 * 60,
Expand Down Expand Up @@ -229,6 +236,27 @@ config :devils_dictionary, :pexels,
endpoint: "https://api.pexels.com/v1/search",
enabled: true

# Bing's news RSS feed is keyless and undocumented (#135, from #134's live
# shootout). `mkt` is a request parameter and not a preference: the feed infers
# a market from the caller's address, and the probe's inferred Poland — the
# channel came back titled *BingWiadomości* and `/define/war` answered with
# Polish games-site pages about *War Thunder*. `max_age_days` is what makes
# this a News shelf rather than a search shelf; the feed will happily answer a
# word with 2023.
#
# **Off by default.** The feed's own `<copyright>` element restricts its results
# to "rendering Bing results within an RSS aggregator for your personal,
# non-commercial use" and reserves any other use to Microsoft's express written
# permission (quoted in full in `docs/integrations/bing-news.md`). A public word
# page is neither, so the integration ships disabled and the owner turns it on
# with `BING_NEWS_ENABLED=true` (read in `runtime.exs`) once that question is
# settled. CodeRabbit's review of #140 asked for this default; the PR raised it.
config :devils_dictionary, :bing_news,
endpoint: "https://www.bing.com/news/search",
enabled: false,
market: "en-US",
max_age_days: 30

# Import environment specific config. This must remain at the bottom
# of this file so it overrides the configuration defined above.
import_config "#{config_env()}.exs"
20 changes: 18 additions & 2 deletions config/runtime.exs
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,18 @@ config :devils_dictionary, DevilsDictionaryWeb.Endpoint,
# The per-provider policy variables are derived from the same registry the
# override loop below reads, so a provider added to the registry cannot have its
# .env overrides silently discarded by an allowlist nobody remembered to edit.
# A slug's environment-variable prefix. Hyphens become underscores because a
# shell cannot export `OPEN-LIBRARY_DISCOVERY_POSITIVE_REFRESH_SECONDS` —
# `bash` and `zsh` both refuse a name with a hyphen in it — so the
# hyphenated form this used to build was an override no deployment could set.
# `bing-news` reads `BING_NEWS_…` (#135), and `open-library` `OPEN_LIBRARY_…`.
env_prefix = fn slug -> slug |> String.upcase() |> String.replace("-", "_") end

discovery_provider_policy_env =
:devils_dictionary
|> Application.get_env(:discovery_providers, [])
|> Enum.flat_map(fn provider ->
prefix = String.upcase(provider.slug())
prefix = env_prefix.(provider.slug())

[
"#{prefix}_DISCOVERY_POSITIVE_REFRESH_SECONDS",
Expand Down Expand Up @@ -114,6 +121,15 @@ if giphy_api_key = System.get_env("GIPHY_API_KEY") || local_provider_env["GIPHY_
config :devils_dictionary, :giphy, api_key: giphy_api_key
end

# Bing News (#135) ships **disabled** in `config.exs` because the feed's own
# `<copyright>` reserves anything beyond a personal RSS aggregator to
# Microsoft's written permission (`docs/integrations/bing-news.md`). This is the
# one switch that turns the News shelf's first provider on, and it has to say
# so explicitly; a host that has never heard of it stays off.
if System.get_env("BING_NEWS_ENABLED") in ~w(true 1 yes on) do
config :devils_dictionary, :bing_news, enabled: true
end

# Urban Dictionary's kill switch (#136), and the only one of the two that needs
# a deploy — `active: false` on the source row is the other and needs none.
# Default **true**: the variable has to say `false` to turn the card off, so a
Expand Down Expand Up @@ -183,7 +199,7 @@ source_policies =
|> Application.get_env(:discovery_providers, [])
|> Enum.map(& &1.slug())
|> Enum.reduce(source_policies, fn slug, policies ->
prefix = String.upcase(slug)
prefix = env_prefix.(slug)

overrides =
[
Expand Down
13 changes: 13 additions & 0 deletions config/test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -156,3 +156,16 @@ config :devils_dictionary, :pexels,
enabled: true,
request_interval_ms: 0,
min_retry_interval_ms: 0

# Every request in the suite goes through the `Req.Test` stub named after the
# provider module. `now` is the clock the freshness gate reads: the conformance
# fixture is a real capture whose items ran between 15 and 20 September 2026,
# and a gate read against the wall clock would pass that week and fail the
# next. Pinned to the day of the capture, so the 30-day window is a property of
# the fixture and not of the date the suite happens to run.
config :devils_dictionary, :bing_news,
endpoint: "https://bing-news.test/news/search",
enabled: true,
now: ~U[2026-09-21 12:00:00Z],
request_interval_ms: 0,
min_retry_interval_ms: 0
16 changes: 14 additions & 2 deletions docs/discovery/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ finished. Where a claim is not yet true everywhere, the last column says so.
|---|---|---|---|
| 1 | **Association is identity or attestation, never a text search's ranking.** A search generates candidates; an identifier the encyclopedia already asserts, or the word's use at a locator, decides. A search result reaches a shelf only where the row admits `:query`, and is labelled as one. | The `evidence` column of the content-type table; conformance asserts every reason a provider delivers against its row | Stated in full under [*The one rule*](#the-one-rule). Two rows admit a labelled search: `:image` (#116 M6, beside its identity matches) and `:gif` (a search by construction, outside the shared pipeline pending K10). No identity-bearing shelf does, and a future row that admits one says so in its `evidence` |
| 2 | **One shelf per content type, never per source.** A reader sees one rail of images, with every contributing source credited once in its header, items taking turns by tier then slug, one item per identity and one per media URL. | The multi-source conformance check (two stubs on one type → one shelf of 14); the browser proof each phase records | Live before corpus is a fixed order, by decision (K2); tier is the only trust weighting |
| 3 | **Every item says why it is here, in one sentence, from fields and nothing else.** | Conformance: every result carries a reason, it renders, it ends with a full stop, its class is admitted | The locator has two shapes (line, page); a dated locator is the next |
| 3 | **Every item says why it is here, in one sentence, from fields and nothing else.** | Conformance: every result carries a reason, it renders, it ends with a full stop, its class is admitted | The locator has three shapes: a line (PoetryDB), a page (Open Library, which leaves it empty rather than call a page a line) and a **date** — `a headline, Wired, 15 September 2026`, Bing News, #135. It is a `"locator"` string in `match_details` and not a clause in `MatchReason`, which is why the third shape cost that module nothing. One wrinkle it did expose: the attestation sentence's preposition is fixed at *at*, so a dated locator reads *Uses “x” **at** a headline, …* |
| 4 | **Every item carries what its licence owes.** A shelf whose row requires attribution shows a credit beneath every thumbnail, always visible, never on hover, **and linked** where the item names a URL for the creator or the licence. | The `attribution` column; conformance asserts the credit node on every item of a `:required` shelf, that it carries no line clamp, and that its links are the creator's and the licence's | — |
| 5 | **Adding a source touches no shared file.** A provider is its own module, fixture, conformance suite and, for a corpus, manifest; the generator writes every registration. | `conformance_coverage_test.exs`: every registered provider has a suite, every fixture is run, every manifest has a suite; each phase report's *shared files touched* list | Adding a *shelf* is one row in the table, shared by design; adding a corpus *kind* is a seeder clause |
| 6 | **Nothing is spent without a ledger, and no bytes are held.** Every live request is a row in `discovery_request_attempts`; results are URLs and metadata; images are hotlinked from the provider's host. | The ledger; the transport's budget; D14; conformance's coverage gate (a provider declines before anything is spent) | — |
Expand Down Expand Up @@ -276,6 +276,17 @@ envelope for the error before it reads it for results, and return
`{:deferred, code, seconds, request_parameters}`; a transport that only watches
status codes will retry straight back into the lag it was asked to wait out.

Nor is JSON the only body. The transport accepts a `200` whose body Req decoded
to a map or a list, and a provider that declares `body: :xml` in
`capabilities/0` may also be handed a **binary** body, which the transport
gives to that provider's own `parse_body/1` to decode into the map the rest of
the pipeline expects; `:error` from it is the same `"malformed_response"` a bad
JSON envelope gets, and an absent `body` key means JSON as before. Bing's news
feed is RSS 2.0 and is the only such provider (#135). The format a source
speaks is the provider's knowledge, so this is one capability read in one
place rather than a branch per format in shared code — and `parse/1` inside
such a provider therefore never sees the document twice.

### 7. Persistence — cache first, identity only if earned

`persist_results/3` writes, for every item:
Expand Down Expand Up @@ -414,6 +425,7 @@ declares the first.
| `:artwork` | The Met | `met-highlights-v1`, `wikidata-famous-v1`, the Artsy pilot | AIC, Cleveland via corpora (#100) | identity |
| `:image` | Wikimedia Commons, Openverse, Unsplash, Pexels | — | iNaturalist or GBIF, if a taxon QID earns a second identity path | identity (Commons), or a labelled search (the other three) |
| `:text` | PoetryDB, Open Library | `poetrydb-v1`, `open-library-v1` — **identity only, by decision; neither reaches a page** | Chronicling America as a corpus, Gutenberg | attestation, live only |
| `:news` | Bing News | — | the Guardian, keyed (#63); Chronicling America's bulk OCR | attestation, **dated** |
| `:gif` | GIPHY, browser-only, outside this chrome | — | Tenor, after K10 | a labelled search |
| `:quote` | — | — | Wikiquote, Gutenberg extraction, after #65 | identity (`(author_id, body hash)`) |

Expand Down Expand Up @@ -443,7 +455,6 @@ for when it is scheduled.
|---|---|---|---|---|---|
| `:music` | Music | square (cover art), title, artist, link out | identity where a sense's entity has a Wikidata song or work with `P2207` (Spotify track id) or a MusicBrainz id, and `P921` *main subject* names the page's QID; otherwise a labelled `:query` | Spotify (search, cover art, metadata — `SPOTIFY_CLIENT_ID` is held); MusicBrainz (metadata CC0) with the Cover Art Archive (images stay their owners' copyright — an item-level rights decision before any is displayed, not archive membership); Internet Archive audio (public domain) | **Spotify's terms make it single-source on its shelf.** Its Developer Policy forbids a product "integrated with streams or content from another service", requires the Spotify marks and a link back on every piece of metadata or cover art, and forbids offering metadata or cover art as a standalone product. Since 2024-11-27, apps registered after that date and development-mode apps without a pending extension request — which is what this project's app would be — have no Recommendations, Related Artists, Audio Features or Audio Analysis, and no 30-second preview URLs in multi-get responses (single-track responses are widely reported to return `null` too); apps that already held extended access were unaffected. So *related songs* is not a feature a new app gets, and a preview is not something the card can count on; what remains is search (a `:query`) and identity through Wikidata. Lyrics are licensed and never held, so attestation is closed. A Music row therefore admits `identity` and `query`, requires attribution, and — if Spotify is its source — takes no second source, the way `:film` is single-source by decision. MusicBrainz as the source instead gives a many-source shelf, but its cover art needs a per-item rights decision and it has no audio |
| `:video` | Videos | 16:9, playable | identity (`P180` on Commons `video/*` files, which the `:image` gate already sees and drops; `P1651` YouTube id on a sense's entity) | Wikimedia Commons video; Internet Archive moving images | the first non-square, non-portrait aspect; playback on the card |
| `:news` | News | text-first, date and masthead | attestation (the article uses the word) | the Guardian (#63, non-commercial); Chronicling America's bulk OCR (its live API is bot-walled) | attestation with a **date** locator, the third shape after line and page |
| `:game` | Games | portrait cover | identity (IGDB id ↔ `P5794`) | IGDB (non-commercial) | the non-commercial posture on a per-item licence |
| `:place` | Places | map tile | identity (`P625` coordinates on the page's entity) | OpenStreetMap tiles | probably the entity panel's job, not a shelf; listed to say so |
| taxon images | joins `:image` | — | identity (taxon QID) | iNaturalist (per-item media licence, CC BY-NC by default, all rights reserved allowed), GBIF | not a new type: an `:image` source whose identity is a taxon QID, with the licence gate applied to each image's own licence, as Commons's already is |
Expand Down Expand Up @@ -751,6 +762,7 @@ suite red.
| Openverse | discovery (GET, offset, **labelled `:query`** exact-phrase search, per-item licence gate, `commons_file` upstream identifier) | `Culture.section`, the `:image` shelf | #116 Phase 2 |
| Unsplash | discovery (GET, offset, **labelled `:query`** search, `UNSPLASH_ACCESS_KEY`, linked credit with UTM, download trigger held and not fired) | `Culture.section`, the `:image` shelf | #116 Phase 3 |
| Pexels | discovery (GET, offset, **labelled `:query`** search, `PEXELS_API_KEY`) | `Culture.section`, the `:image` shelf | #116 Phase 3 |
| Bing News | discovery (GET, offset, **attestation with a dated locator**, keyless RSS via `body: :xml`, 30-day freshness gate, publisher URL as identity in `news_article`) | `Culture.section`, the `:news` shelf, attribution `:credited` | #135 — and see `docs/integrations/bing-news.md` on the feed's own `<copyright>` |
| Artsy | registered, registry-only — its 43 artworks and their gene mappings reach a page through the catalog; the private client was retired in #109 Phase 3a | `Culture.section` | #86, K9 of #109 |
| GIPHY | registered, browser-only, transient | its own `GiphyShelf` component, **not** `Culture.section` | parked pending caching approval, K10 |

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/discovery/issue-135-war-1280-2026-09-21.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/discovery/issue-135-war-375-2026-09-21.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading