Repository navigation
Description
Implementation issue, one PR. Adds the News shelf — the :news content-type row — with Bing's news RSS feed as its first provider. Keyless, so nothing to register and nothing to wait for. Split out of #134, whose live shootout (2026-09-21) chose it: for bestiality the feed returned 7 items from 7 mastheads (Wired, the Guardian, the Telegraph, NBC…), each with the masthead's name, the date, a one-line description, a thumbnail and the publisher's real URL. Google's feed found more and cannot be linked from; GDELT missed the story.
Refs #134 (the research and the shootout), #116 (many sources, one shelf — the Guardian joins this shelf later, once a key exists), #109 (the kit this is built with), #63 (the Guardian, which follows this), #111/#131 (the word page: this is one more shelf template).
Follow docs/discovery/adding-a-provider.md in order. Probe ceiling for this issue: 150 requests, recorded as a running ledger in docs/integrations/bing-news.md.
What the reader gets
On /define/bestiality, beneath the Texts shelf, a shelf headed News: text-first cards, each a headline, its masthead, its date, and a reason that reads
Uses “bestiality” in a headline, Wired, 15 September 2026.
Every card links out to the publisher's page, not to Bing. On /define/logomachy, no News shelf: the feed's only two items are Word of the day pieces from March 2026, outside the freshness window, and the run records no_results.
The feed, as measured
| Fact | Measured 2026-09-21 |
|---|---|
| Endpoint | GET https://www.bing.com/news/search?q=<term>&format=rss |
| Auth | none; needs a browser-like User-Agent (the project's own UA string was not tested — test it first, and fall back to a generic Mozilla UA only if the project's is refused) |
| Body | application/xml; charset=utf-8, RSS 2.0 with a News: namespace |
| Per item | title, description (one sentence), pubDate, News:Source (the masthead: Wired, The Guardian, HuffPost on MSN), News:Image (a Bing-hosted thumbnail), link |
| The link | http://www.bing.com/news/apiclick.aspx?…&url=<percent-encoded publisher URL>&…. Decode the url= parameter; that is the item's URL and its identity. Never link to the apiclick redirect |
| Size | 7 items for bestiality, 12 with a second word, 9 for nepotism, 2 for logomachy. count= is ignored. qft=interval="7" (freshness) returned fewer than the plain query and dropped the 15 September items — do not use it; filter by pubDate yourself |
| Quoting | q="bestiality" (a quoted phrase) returned the same 7 as bare. Decide in the probe whether to quote; a multi-word lemma needs it |
| Substring noise | not seen for this word (Openverse's Restaurante Bestial problem did not appear), but gate on it anyway |
| Staleness | nepotism returned items from 2025; logomachy from March 2026. The feed is a search, not a wire — a freshness window is what makes it a News shelf |
| Rate | not measured beyond ~10 requests; assume a search engine's tolerance for a polite crawler and pace at 1 request/s with a long retry gap. Measure it in the probe |
| Terms | none published for the feed. It is undocumented and unsupported, like the rest of the keyless feeds in #100. What is kept is metadata (headline, masthead, date, URL, a sentence), which is the posture #100 records for GDELT and NYT |
The spec (the four decisions)
| Decision | Value | Why |
|---|---|---|
| archetype | discovery |
live answers to a page; nothing to seed |
| content type | news — a new row, see below |
a headline is not a Text (no line to cite) and not an image |
| transport | get |
— |
| pagination | offset |
the feed does not page; the provider never returns a cursor |
| identifier | the publisher URL, normalised (scheme + host lowercased, tracking parameters stripped, no fragment), hashed as the external_id in namespace news_article |
it is the only stable identity the feed offers, and a later Guardian provider can carry the same namespace for the same URL so the shelf's identity dedup merges them |
| evidence | attestation | the headline uses the word. match_details is %{"kind" => "attestation", "query" => term, "lines" => [%{"locator" => "a headline, <Masthead>, <D Month YYYY>", "text" => headline}]} — MatchReason.attestations/1 already reads a "locator" string, so the dated locator is data, not a new clause |
| the gate | whole-word match of the term in title (Unicode boundary, as Poetrydb.first_attestation/2 does), else in description; and pubDate within the freshness window |
the search proposes, the text disposes |
| freshness window | config :devils_dictionary, :bing_news, max_age_days: 30, overridable |
30 days is "in the news"; a 2025 article on nepotism is not. Older items are dropped at retrieval, so a word with only old coverage is an honest empty |
| persistence | :persistent, with BING_NEWS_DISCOVERY_POSITIVE_REFRESH_SECONDS=86400 as the shipped default in config/runtime.exs's provider-prefixed overrides |
the shipped 30-day positive refresh would freeze this week's shelf for a month; a day is what "current" costs. Empty refresh stays 24 h |
covers?/1 |
default true |
any word can be searched; the empty is cached negative |
| identity | no identity_record/1 (as Openverse) — every result persists as :insufficient_evidence |
a headline proposes no encyclopedia identity |
| tier | :plebs |
an aggregator of mastheads, the tier #100 gives the Guardian and Hacker News |
| source row | slug bing-news, name Bing News, attribution: "via Bing News", license: "Undocumented public RSS feed; metadata only", homepage https://www.bing.com/news, url_template: nil |
the shelf byline; the card credits the masthead, not Bing |
The :news row
One entry in DevilsDictionary.Discovery.ContentTypes, plus its place in @known (after :text, before :gif):
news: %{
heading: "News",
label: "news",
badge: "News",
aspect: nil, # text-first: no poster frame (README's sketch of this row)
icon: "hero-newspaper",
column: "w-44 sm:w-52", # the text card's width; measure at 375 px
title_clamp: "line-clamp-3",
thumbnail_keys: [], # News:Image is Bing-hosted; not shown in this phase
attribution: :credited, # the masthead is the credit, shown beside the creator line
evidence: [:attestation]
}preview_metadata per item: title (headline), author and artist (the masthead, so the creator line reads it), attribution (the masthead again, for the :credited line — decide in the browser proof whether both lines are one too many and drop one), year, published_at (ISO 8601), description, source_url (publisher URL), content_type: "news", provider: "Bing News".
The card. The shared text card shows title, creator and a year line. A News card wants the date. If the year line can carry published_at formatted as 15 Sep 2026 when the metadata has it, that is one edit in Culture shared by every text-first type and is the only shared-component change this issue expects. If it needs more than that, stop and say so in the PR rather than growing the card.
One shared change the kit needs: a non-JSON body
Discovery.Transport.request/4 accepts a 200 only when Req decoded the body to a map or a list. Bing answers XML. This is the first provider that does, so the transport grows one capability, kept small:
- a provider may declare
body: :xmlincapabilities/0(absent means JSON, as today); - for such a provider the transport accepts a binary body and hands it to the provider's
parse_body/1, which returns{:ok, map}or:error; the map is whatretrieve/4'srequest_funyields, so the provider's ownparse/1never sees XML twice; - parse with
Floki(already a dependency, used by Johnson's TEI-XML) — note that it lowercases tags, soNews:Sourceis found asnews\\:source, or walk the tree; prove it on the captured fixture; - an unparseable body is
"malformed_response", the same code a bad JSON envelope gets today.
A conformance case that stubs an XML body and one that stubs garbage. Everything else in the pipeline is untouched.
Acceptance
-
mix dd.provider.new bing-news --archetype discovery --content-type news --transport get --pagination offsetruns (after the row exists), then the scaffold is replaced by the real envelope and a fixture captured from the real feed forbestiality(the 2026-09 response, with dates pinned in the fixture so the freshness gate is deterministic under test — inject the clock). - Conformance green, including: the dated locator renders as Uses “bestiality” in a headline, Wired, 15 September 2026.; an item whose title and description lack the whole word is dropped; an item older than the window is dropped; an XML body parses; garbage is
malformed_response; the publisher URL and never theapiclicklink is the item'ssource_urland identity. -
conformance_coverage_test.exsandcontent_types_test.exsgreen;mix precommitgreen on your ownMIX_TEST_PARTITION. - Browser proof at 1280 and 375 px:
/define/bestiality(a shelf, dated, credited, linked out),/define/logomachy(no shelf, one request spent,no_results, reload spends none),/define/war(the shelf sits after Texts and before GIFs; no horizontal scroll at 375). Screenshots indocs/discovery/issue-135-…. - The source row exists on the database the server is on (
ensure_source/1inserts it; check tier and name). -
docs/integrations/bing-news.mdholds the ledger, the four decisions, the measured rate, and the freshness decision. -
docs/discovery/README.md: the:newsrow moves from rows that do not exist yet to the sources table (:news| Bing News | — | the Guardian, keyed | attestation, dated); promise 3's dated locator is the next is updated; and the transport'sbody: :xmlcapability is one sentence under the pipeline section. - Shared files touched, listed in the PR body. Expected:
content_types.ex(the row),transport.ex(the body capability),culture.ex(the date line, if taken),config/config.exs,config/test.exs,config/runtime.exs(the refresh default). Anything else is a finding.
Not in this issue
The Guardian (needs a key; #63, and #134 Phase 1). GDELT (dropped; see #134's comment). The trending signal (#134 Phase 0). Bing's thumbnails (a hotlinked Bing-hosted image with no licence statement; leave thumbnail_keys empty). The D2 lever for search-only shelves. Urban Dictionary (#136).
Runs beside #136 (Urban Dictionary)
Another session is implementing Urban Dictionary at the same time. The two touch no common module except config/config.exs (each adds one stanza; merge is trivial). One dev server per database: before your browser proof, run the checklist's lsof / pg_stat_activity recipe; if the other session's server is on devils_dictionary_v2, coordinate rather than start a second Oban node on it — its node would take your queued run and discard it as module is not a worker.
Activity
holden commentedon Sep 21, 2026
Implemented in #140 on
codex/135-bing-news.Where the code disagreed with this issue
The most valuable part of the report, so it is first.
1. The feed publishes its own terms. This issue's table says Terms: none published for the feed. Every response carries a
<copyright>element, measured 2026-09-21:A public word page is neither. I built the issue as written and recorded the restriction rather than smoothing it —
source_attrs/0'slicensequotes it, anddocs/integrations/bing-news.mdopens with it — but the ship/no-ship decision is not a provider's to make and is not made in the PR.2.
mktis inferred from the caller's address, and it is not cosmetic. Unpinned, from this desk the market waspl-pl: the channel came back titled BingWiadomości, the<copyright>was in Polish, and/define/waranswered with six Polish games-site and Steam-store pages about War Thunder and Total War, two from 2006 and 2013. Withmkt=en-US,waranswers with the Iran war from Newsweek, UPI, the NYT and the Atlantic, andbestialitygoes from 7 items to 12, picking up Wired, the Chicago Tribune, CBS, Rolling Stone and Al Jazeera. This issue's field notes were taken from a desk the feed read differently.mkt=en-USis now a request parameter, and it also removes the need for Open Library's language gate.3. Do not quote the query — not even a multi-word lemma. The issue says a multi-word lemma needs it. Measured, the opposite:
q=red herringanswered 6 items, 4 using the phrase and 2 from the last three days;q="red herring"answered 3, the freshest from 2024. For one word,q="bestiality"returned a different seven with no more of them usable. The whole-word gate enforces the phrase anyway, so quoting only narrows the candidates.4. The project's own User-Agent works. Tested first, as the issue asked:
200,application/xml, and the same seven items a Chrome UA got. The fallback is not needed.5.
MatchReasoncannot render in a headline. The issue asks for Uses “bestiality” in a headline, Wired, 15 September 2026. and also saysMatchReasonneeds no change. Those cannot both hold: the attestation clause is"Uses " <> quoted(term) <> at(locator)andat/1is" at " <> locator, so the preposition is the renderer's. The locator is exactly the string this issue specified; it renders asA one-line change in
match_reason.exfixes it, and this issue put that file out of scope, so it is reported rather than done. Acceptance box not ticked.6. The card did show the masthead twice, as this issue suspected. Dropped: the masthead goes to
author/artistonly. The row stays:creditedrather than:none, because:nonemeans the shelf byline is the whole credit and the shelf byline is Bing.7. Four shared files beyond the expected list, each explained in the PR body:
provider.ex(the optionalparse_body/1callback the transport now calls),runtime.exsbeyond the refresh default (the env prefix builtBING-NEWS_…, which no shell can export —open-library's too), and two tests that were counting content types rather than checking a rule.Ledger
Ceiling 150, spent 38. Running totals, batch by batch, in
docs/integrations/bing-news.md.bestiality— both200, 7 itemspl-plmarketmkt=en-US: 7 → 12 items forbestiality200at 1 req/s, no429, noRetry-After, 238/262/409 ms min/median/max.referendumanswered200with zero itemsdiscovery_request_attemptsreads 3 and the true spend is 4Measured rate: paced at 1 req/s as a courtesy, not because anything forced it.
min_retry_interval_ms2 s is a posture, since nothing failed.Conformance
Baseline on fresh
mainbefore any change: 1407 tests, 0 failures —manifest_test.exsincluded, because it passes oncedata/is symlinked from the main checkout. So this branch had no environmental failure to excuse.One flake, seen once and not reproducible:
artwork_live_test.exs:326failed insideArtworks.artsy_suggestions/1on one full run, then passed alone and on a full re-run. Unrelated code; named rather than hidden.The fixture is a real capture of
?q=bestiality&format=rss&mkt=en-USfrom 2026-09-21,<copyright>and all, so the clock is injected (config/test.exspinsnow:to the day of the capture). Three of the five:resultsrows are there to be refused and the expected ids leave them out: the real Daily Telegraph item from December 2023, a second copy of the Wired story withutm_*/fbclid/#comments(folds by identity), and one constructed#Bestialitygaterow.Browser proof
CDP with
Emulation.setDeviceMetricsOverride.devils_dictionary_v2already had three Oban nodes on it (main 4007, #131's worktree 4017, #136's worktree 4027 — thelsof/pg_stat_activityrecipe confirmed all three), so rather than add a fourth this took port 4037 and its own database,devils_dictionary_bing135, cloned from the idledevils_dictionary_74_verifyand migrated to head.scrollWidth/define/bestiality/define/logomachy/define/warScreenshots:
docs/discovery/issue-135-{bestiality,logomachy,war}-{1280,375}-2026-09-21.jpg./define/bestiality— headed News, byline Bing News · dated headlines, twelve cards, each a headline, 15 Sep 2026 · News, the masthead once, Source ↗ to the publisher. No card links tobing.com, asserted in the DOM./define/war— the DOM readsculture-shelf-film,culture-shelf-text,culture-shelf-news,culture-shelf-image,giphy-d2Fy. News after Texts, before GIFs. Images falls between because a shelf nothing identified is demoted (D1 of Shelf appearance: one chrome per shelf, one-line credits, and what a search-only shelf is allowed to be #126) and the GIPHY shelf is a separate component rendered last. Only 2 of the feed's 11 items reached the shelf — the freshness gate doing exactly what it is for. The GIF shelf needed itssourcesrow inserted by hand: GIPHY istransport: :browser, so the pipeline never callsensure_source/1for it./define/logomachy— one request,no_results, 0 results, and two reloads afterwards spent none. Not ticking the "no shelf" box as written: there is no rail and no cards, but the shelf region renders the shared empty-state line In news · No matching news for this term yet., which is what every content type renders for an empty run (In film and In text are on the same screenshot) and is not this provider's to suppress.bing-news | plebs | Bing News | via Bing News. No hand correction needed — this database had no row to be stale.devils_dictionary_v2has nobing-newsrow yet and will take whateversource_attrs/0says on its first run there.Left out, named as left out
The Guardian, GDELT, the trending signal, Bing's thumbnails (
thumbnail_keys: []), the D2 lever, Urban Dictionary (#136) — all out of scope here and untouched. The two unticked acceptance boxes are #5 and thelogomachywording above.holden commentedon Sep 21, 2026
Shipped in #140 (merged 2026-09-21), reviewed with the CodeRabbit CLI and audited against this issue's acceptance list; 1475 tests, 0 failures on the merged tree.
One thing to know before turning it on. The feed's own
<copyright>element restricts its results to "rendering Bing results within an RSS aggregator for your personal, non-commercial use" and reserves any other use to Microsoft's written permission (quoted in full indocs/integrations/bing-news.md). So the provider is merged disabled:BING_NEWS_ENABLED=trueturns the News shelf's first source on, and that is the owner's call. The:newsrow, the dated locator, thebody: :xmltransport capability and the 30-day freshness gate are all in place for the Guardian (#63) or any other source to use.Two acceptance boxes the PR did not tick, both recorded there: the reason renders at a headline rather than in a headline (a one-word change in
match_reason.ex, which this issue kept out of scope), and an empty run shows the shared No matching news for this term yet line that every shelf shows rather than nothing at all.holden commentedon Sep 21, 2026
Turned on in #141 (2026-09-21) by the owner's decision, knowing the feed's
<copyright>.BING_NEWS_ENABLED=falseturns it off again.Spotify joins the world: the Music shelf, its first source (#143)