Skip to content

News shelf: the :news row and Bing's news RSS feed as its first, keyless provider (from #134) #135

Closed

Description

@holden

Implementation issue, one PR. Adds the News shelf — the :news content-type row — with Bing's news RSS feed as its first provider. Keyless, so nothing to register and nothing to wait for. Split out of #134, whose live shootout (2026-09-21) chose it: for bestiality the feed returned 7 items from 7 mastheads (Wired, the Guardian, the Telegraph, NBC…), each with the masthead's name, the date, a one-line description, a thumbnail and the publisher's real URL. Google's feed found more and cannot be linked from; GDELT missed the story.

Refs #134 (the research and the shootout), #116 (many sources, one shelf — the Guardian joins this shelf later, once a key exists), #109 (the kit this is built with), #63 (the Guardian, which follows this), #111/#131 (the word page: this is one more shelf template).

Follow docs/discovery/adding-a-provider.md in order. Probe ceiling for this issue: 150 requests, recorded as a running ledger in docs/integrations/bing-news.md.


What the reader gets

On /define/bestiality, beneath the Texts shelf, a shelf headed News: text-first cards, each a headline, its masthead, its date, and a reason that reads

Uses “bestiality” in a headline, Wired, 15 September 2026.

Every card links out to the publisher's page, not to Bing. On /define/logomachy, no News shelf: the feed's only two items are Word of the day pieces from March 2026, outside the freshness window, and the run records no_results.


The feed, as measured

Fact Measured 2026-09-21
Endpoint GET https://www.bing.com/news/search?q=<term>&format=rss
Auth none; needs a browser-like User-Agent (the project's own UA string was not tested — test it first, and fall back to a generic Mozilla UA only if the project's is refused)
Body application/xml; charset=utf-8, RSS 2.0 with a News: namespace
Per item title, description (one sentence), pubDate, News:Source (the masthead: Wired, The Guardian, HuffPost on MSN), News:Image (a Bing-hosted thumbnail), link
The link http://www.bing.com/news/apiclick.aspx?…&url=<percent-encoded publisher URL>&…. Decode the url= parameter; that is the item's URL and its identity. Never link to the apiclick redirect
Size 7 items for bestiality, 12 with a second word, 9 for nepotism, 2 for logomachy. count= is ignored. qft=interval="7" (freshness) returned fewer than the plain query and dropped the 15 September items — do not use it; filter by pubDate yourself
Quoting q="bestiality" (a quoted phrase) returned the same 7 as bare. Decide in the probe whether to quote; a multi-word lemma needs it
Substring noise not seen for this word (Openverse's Restaurante Bestial problem did not appear), but gate on it anyway
Staleness nepotism returned items from 2025; logomachy from March 2026. The feed is a search, not a wire — a freshness window is what makes it a News shelf
Rate not measured beyond ~10 requests; assume a search engine's tolerance for a polite crawler and pace at 1 request/s with a long retry gap. Measure it in the probe
Terms none published for the feed. It is undocumented and unsupported, like the rest of the keyless feeds in #100. What is kept is metadata (headline, masthead, date, URL, a sentence), which is the posture #100 records for GDELT and NYT

The spec (the four decisions)

Decision Value Why
archetype discovery live answers to a page; nothing to seed
content type news — a new row, see below a headline is not a Text (no line to cite) and not an image
transport get —
pagination offset the feed does not page; the provider never returns a cursor
identifier the publisher URL, normalised (scheme + host lowercased, tracking parameters stripped, no fragment), hashed as the external_id in namespace news_article it is the only stable identity the feed offers, and a later Guardian provider can carry the same namespace for the same URL so the shelf's identity dedup merges them
evidence attestation the headline uses the word. match_details is %{"kind" => "attestation", "query" => term, "lines" => [%{"locator" => "a headline, <Masthead>, <D Month YYYY>", "text" => headline}]} — MatchReason.attestations/1 already reads a "locator" string, so the dated locator is data, not a new clause
the gate whole-word match of the term in title (Unicode boundary, as Poetrydb.first_attestation/2 does), else in description; and pubDate within the freshness window the search proposes, the text disposes
freshness window config :devils_dictionary, :bing_news, max_age_days: 30, overridable 30 days is "in the news"; a 2025 article on nepotism is not. Older items are dropped at retrieval, so a word with only old coverage is an honest empty
persistence :persistent, with BING_NEWS_DISCOVERY_POSITIVE_REFRESH_SECONDS=86400 as the shipped default in config/runtime.exs's provider-prefixed overrides the shipped 30-day positive refresh would freeze this week's shelf for a month; a day is what "current" costs. Empty refresh stays 24 h
covers?/1 default true any word can be searched; the empty is cached negative
identity no identity_record/1 (as Openverse) — every result persists as :insufficient_evidence a headline proposes no encyclopedia identity
tier :plebs an aggregator of mastheads, the tier #100 gives the Guardian and Hacker News
source row slug bing-news, name Bing News, attribution: "via Bing News", license: "Undocumented public RSS feed; metadata only", homepage https://www.bing.com/news, url_template: nil the shelf byline; the card credits the masthead, not Bing

The :news row

One entry in DevilsDictionary.Discovery.ContentTypes, plus its place in @known (after :text, before :gif):

news: %{
  heading: "News",
  label: "news",
  badge: "News",
  aspect: nil,                       # text-first: no poster frame (README's sketch of this row)
  icon: "hero-newspaper",
  column: "w-44 sm:w-52",            # the text card's width; measure at 375 px
  title_clamp: "line-clamp-3",
  thumbnail_keys: [],                # News:Image is Bing-hosted; not shown in this phase
  attribution: :credited,            # the masthead is the credit, shown beside the creator line
  evidence: [:attestation]
}

preview_metadata per item: title (headline), author and artist (the masthead, so the creator line reads it), attribution (the masthead again, for the :credited line — decide in the browser proof whether both lines are one too many and drop one), year, published_at (ISO 8601), description, source_url (publisher URL), content_type: "news", provider: "Bing News".

The card. The shared text card shows title, creator and a year line. A News card wants the date. If the year line can carry published_at formatted as 15 Sep 2026 when the metadata has it, that is one edit in Culture shared by every text-first type and is the only shared-component change this issue expects. If it needs more than that, stop and say so in the PR rather than growing the card.


One shared change the kit needs: a non-JSON body

Discovery.Transport.request/4 accepts a 200 only when Req decoded the body to a map or a list. Bing answers XML. This is the first provider that does, so the transport grows one capability, kept small:

  • a provider may declare body: :xml in capabilities/0 (absent means JSON, as today);
  • for such a provider the transport accepts a binary body and hands it to the provider's parse_body/1, which returns {:ok, map} or :error; the map is what retrieve/4's request_fun yields, so the provider's own parse/1 never sees XML twice;
  • parse with Floki (already a dependency, used by Johnson's TEI-XML) — note that it lowercases tags, so News:Source is found as news\\:source, or walk the tree; prove it on the captured fixture;
  • an unparseable body is "malformed_response", the same code a bad JSON envelope gets today.

A conformance case that stubs an XML body and one that stubs garbage. Everything else in the pipeline is untouched.


Acceptance

  • mix dd.provider.new bing-news --archetype discovery --content-type news --transport get --pagination offset runs (after the row exists), then the scaffold is replaced by the real envelope and a fixture captured from the real feed for bestiality (the 2026-09 response, with dates pinned in the fixture so the freshness gate is deterministic under test — inject the clock).
  • Conformance green, including: the dated locator renders as Uses “bestiality” in a headline, Wired, 15 September 2026.; an item whose title and description lack the whole word is dropped; an item older than the window is dropped; an XML body parses; garbage is malformed_response; the publisher URL and never the apiclick link is the item's source_url and identity.
  • conformance_coverage_test.exs and content_types_test.exs green; mix precommit green on your own MIX_TEST_PARTITION.
  • Browser proof at 1280 and 375 px: /define/bestiality (a shelf, dated, credited, linked out), /define/logomachy (no shelf, one request spent, no_results, reload spends none), /define/war (the shelf sits after Texts and before GIFs; no horizontal scroll at 375). Screenshots in docs/discovery/issue-135-….
  • The source row exists on the database the server is on (ensure_source/1 inserts it; check tier and name).
  • docs/integrations/bing-news.md holds the ledger, the four decisions, the measured rate, and the freshness decision.
  • docs/discovery/README.md: the :news row moves from rows that do not exist yet to the sources table (:news | Bing News | — | the Guardian, keyed | attestation, dated); promise 3's dated locator is the next is updated; and the transport's body: :xml capability is one sentence under the pipeline section.
  • Shared files touched, listed in the PR body. Expected: content_types.ex (the row), transport.ex (the body capability), culture.ex (the date line, if taken), config/config.exs, config/test.exs, config/runtime.exs (the refresh default). Anything else is a finding.

Not in this issue

The Guardian (needs a key; #63, and #134 Phase 1). GDELT (dropped; see #134's comment). The trending signal (#134 Phase 0). Bing's thumbnails (a hotlinked Bing-hosted image with no licence statement; leave thumbnail_keys empty). The D2 lever for search-only shelves. Urban Dictionary (#136).

Runs beside #136 (Urban Dictionary)

Another session is implementing Urban Dictionary at the same time. The two touch no common module except config/config.exs (each adds one stanza; merge is trivial). One dev server per database: before your browser proof, run the checklist's lsof / pg_stat_activity recipe; if the other session's server is on devils_dictionary_v2, coordinate rather than start a second Oban node on it — its node would take your queued run and discard it as module is not a worker.

Activity

  1. holden commented on Sep 21, 2026

    @holden
    MemberAuthor

    Implemented in #140 on codex/135-bing-news.

    Where the code disagreed with this issue

    The most valuable part of the report, so it is first.

    1. The feed publishes its own terms. This issue's table says Terms: none published for the feed. Every response carries a <copyright> element, measured 2026-09-21:

    These XML results may not be used, reproduced or transmitted in any manner or for any purpose other than rendering Bing results within an RSS aggregator for your personal, non-commercial use. Any other use of these results requires express written permission from Microsoft Corporation.

    A public word page is neither. I built the issue as written and recorded the restriction rather than smoothing it — source_attrs/0's license quotes it, and docs/integrations/bing-news.md opens with it — but the ship/no-ship decision is not a provider's to make and is not made in the PR.

    2. mkt is inferred from the caller's address, and it is not cosmetic. Unpinned, from this desk the market was pl-pl: the channel came back titled BingWiadomości, the <copyright> was in Polish, and /define/war answered with six Polish games-site and Steam-store pages about War Thunder and Total War, two from 2006 and 2013. With mkt=en-US, war answers with the Iran war from Newsweek, UPI, the NYT and the Atlantic, and bestiality goes from 7 items to 12, picking up Wired, the Chicago Tribune, CBS, Rolling Stone and Al Jazeera. This issue's field notes were taken from a desk the feed read differently. mkt=en-US is now a request parameter, and it also removes the need for Open Library's language gate.

    3. Do not quote the query — not even a multi-word lemma. The issue says a multi-word lemma needs it. Measured, the opposite: q=red herring answered 6 items, 4 using the phrase and 2 from the last three days; q="red herring" answered 3, the freshest from 2024. For one word, q="bestiality" returned a different seven with no more of them usable. The whole-word gate enforces the phrase anyway, so quoting only narrows the candidates.

    4. The project's own User-Agent works. Tested first, as the issue asked: 200, application/xml, and the same seven items a Chrome UA got. The fallback is not needed.

    5. MatchReason cannot render in a headline. The issue asks for Uses “bestiality” in a headline, Wired, 15 September 2026. and also says MatchReason needs no change. Those cannot both hold: the attestation clause is "Uses " <> quoted(term) <> at(locator) and at/1 is " at " <> locator, so the preposition is the renderer's. The locator is exactly the string this issue specified; it renders as

    Uses “bestiality” at a headline, Wired, 15 September 2026.

    A one-line change in match_reason.ex fixes it, and this issue put that file out of scope, so it is reported rather than done. Acceptance box not ticked.

    6. The card did show the masthead twice, as this issue suspected. Dropped: the masthead goes to author/artist only. The row stays :credited rather than :none, because :none means the shelf byline is the whole credit and the shelf byline is Bing.

    7. Four shared files beyond the expected list, each explained in the PR body: provider.ex (the optional parse_body/1 callback the transport now calls), runtime.exs beyond the refresh default (the env prefix built BING-NEWS_…, which no shell can export — open-library's too), and two tests that were counting content types rather than checking a rule.

    Ledger

    Ceiling 150, spent 38. Running totals, batch by batch, in docs/integrations/bing-news.md.

    # Requests Running Result
    P1 2 2 Project UA vs Chrome UA on bestiality — both 200, 7 items
    P2 7 9 Quoting and shape; found the pl-pl market
    P3 5 14 mkt=en-US: 7 → 12 items for bestiality
    P4 20 34 20/20 200 at 1 req/s, no 429, no Retry-After, 238/262/409 ms min/median/max. referendum answered 200 with zero items
    B1–B4 4 38 The browser proof, one request per word. B1's run was deleted to re-measure the card, cascading its ledger row — counted here, so discovery_request_attempts reads 3 and the true spend is 4

    Measured rate: paced at 1 req/s as a courtesy, not because anything forced it. min_retry_interval_ms 2 s is a posture, since nothing failed.

    Conformance

    $ MIX_TEST_PARTITION=p135 mix test test/devils_dictionary/discovery/conformance/bing_news_conformance_test.exs
    Running ExUnit with seed: 807651, max_cases: 32
    ...................
    Finished in 1.5 seconds (0.00s async, 1.5s sync)
    19 tests, 0 failures
    
    $ MIX_TEST_PARTITION=p135 mix test test/devils_dictionary/discovery/providers/bing_news_test.exs
    ................................
    32 tests, 0 failures
    
    $ MIX_TEST_PARTITION=p135 mix test test/devils_dictionary/discovery/ test/devils_dictionary/artworks/corpus/conformance
    423 tests, 0 failures
    
    $ MIX_TEST_PARTITION=p135 mix precommit
    Finished in 74.4 seconds (32.5s async, 41.8s sync)
    1458 tests, 0 failures
    

    Baseline on fresh main before any change: 1407 tests, 0 failures — manifest_test.exs included, because it passes once data/ is symlinked from the main checkout. So this branch had no environmental failure to excuse.

    One flake, seen once and not reproducible: artwork_live_test.exs:326 failed inside Artworks.artsy_suggestions/1 on one full run, then passed alone and on a full re-run. Unrelated code; named rather than hidden.

    The fixture is a real capture of ?q=bestiality&format=rss&mkt=en-US from 2026-09-21, <copyright> and all, so the clock is injected (config/test.exs pins now: to the day of the capture). Three of the five :results rows are there to be refused and the expected ids leave them out: the real Daily Telegraph item from December 2023, a second copy of the Wired story with utm_*/fbclid/#comments (folds by identity), and one constructed #Bestialitygate row.

    Browser proof

    CDP with Emulation.setDeviceMetricsOverride. devils_dictionary_v2 already had three Oban nodes on it (main 4007, #131's worktree 4017, #136's worktree 4027 — the lsof/pg_stat_activity recipe confirmed all three), so rather than add a fourth this took port 4037 and its own database, devils_dictionary_bing135, cloned from the idle devils_dictionary_74_verify and migrated to head.

    Page Width scrollWidth News items
    /define/bestiality 1280 / 375 1280 / 375 12
    /define/logomachy 1280 / 375 1280 / 375 0
    /define/war 1280 / 375 1280 / 375 2

    Screenshots: docs/discovery/issue-135-{bestiality,logomachy,war}-{1280,375}-2026-09-21.jpg.

    • /define/bestiality — headed News, byline Bing News · dated headlines, twelve cards, each a headline, 15 Sep 2026 · News, the masthead once, Source ↗ to the publisher. No card links to bing.com, asserted in the DOM.
    • /define/war — the DOM reads culture-shelf-film, culture-shelf-text, culture-shelf-news, culture-shelf-image, giphy-d2Fy. News after Texts, before GIFs. Images falls between because a shelf nothing identified is demoted (D1 of Shelf appearance: one chrome per shelf, one-line credits, and what a search-only shelf is allowed to be #126) and the GIPHY shelf is a separate component rendered last. Only 2 of the feed's 11 items reached the shelf — the freshness gate doing exactly what it is for. The GIF shelf needed its sources row inserted by hand: GIPHY is transport: :browser, so the pipeline never calls ensure_source/1 for it.
    • /define/logomachy — one request, no_results, 0 results, and two reloads afterwards spent none. Not ticking the "no shelf" box as written: there is no rail and no cards, but the shelf region renders the shared empty-state line In news · No matching news for this term yet., which is what every content type renders for an empty run (In film and In text are on the same screenshot) and is not this provider's to suppress.
    • The source row on the database the server was on: bing-news | plebs | Bing News | via Bing News. No hand correction needed — this database had no row to be stale. devils_dictionary_v2 has no bing-news row yet and will take whatever source_attrs/0 says on its first run there.

    Left out, named as left out

    The Guardian, GDELT, the trending signal, Bing's thumbnails (thumbnail_keys: []), the D2 lever, Urban Dictionary (#136) — all out of scope here and untouched. The two unticked acceptance boxes are #5 and the logomachy wording above.

  2. holden commented on Sep 21, 2026

    @holden
    MemberAuthor

    Shipped in #140 (merged 2026-09-21), reviewed with the CodeRabbit CLI and audited against this issue's acceptance list; 1475 tests, 0 failures on the merged tree.

    One thing to know before turning it on. The feed's own <copyright> element restricts its results to "rendering Bing results within an RSS aggregator for your personal, non-commercial use" and reserves any other use to Microsoft's written permission (quoted in full in docs/integrations/bing-news.md). So the provider is merged disabled: BING_NEWS_ENABLED=true turns the News shelf's first source on, and that is the owner's call. The :news row, the dated locator, the body: :xml transport capability and the 30-day freshness gate are all in place for the Guardian (#63) or any other source to use.

    Two acceptance boxes the PR did not tick, both recorded there: the reason renders at a headline rather than in a headline (a one-word change in match_reason.ex, which this issue kept out of scope), and an empty run shows the shared No matching news for this term yet line that every shelf shows rather than nothing at all.

  3. holden commented on Sep 21, 2026

    @holden
    MemberAuthor

    Turned on in #141 (2026-09-21) by the owner's decision, knowing the feed's <copyright>. BING_NEWS_ENABLED=false turns it off again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions