Skip to content

The News shelf: the :news row and Bing's news RSS feed as its first, keyless provider - #140

Merged
holden merged 3 commits into
mainfrom
codex/135-bing-news
Sep 21, 2026
Merged

holden merged 3 commits into
mainfrom
codex/135-bing-news

Conversation

@holden

@holden holden commented Sep 21, 2026

Copy link
Copy Markdown
Member

Closes #135. The News shelf — the :news content-type row and Bing's keyless news RSS feed as its first provider.

Read this first: the feed publishes its own terms

#135 records Terms: none published for the feed. That is not what the feed says. Every response carries a <copyright> element, measured 2026-09-21:

These XML results may not be used, reproduced or transmitted in any manner or for any purpose other than rendering Bing results within an RSS aggregator for your personal, non-commercial use. Any other use of these results requires express written permission from Microsoft Corporation.

A public word page is not an RSS aggregator and is not personal non-commercial use. I built the issue as specified and recorded the restriction honestly rather than smoothing it: source_attrs/0's license quotes it and points at the doc, and docs/integrations/bing-news.md opens with it. Whether this ships is your call, not mine — everything else here is done and green, and the shelf is designed to hold more than one source, so the Guardian (#63) is the keyed source that does publish terms.

The four decisions

Decision Value
archetype discovery
content type news — a new ContentTypes row, @known after :text and before :gif
transport get
pagination offset — the feed does not page, so the offset is into the gated list the provider holds, as PoetryDB's is; one request per page

Plus: identity = the publisher URL normalised and hashed in namespace news_article; evidence = attestation with a dated locator; tier :plebs; no identity_record/1; covers?/1 default true; source row slug bing-news, name Bing News, attribution via Bing News.

Shared files touched

The six #135 expected:

File Change
lib/devils_dictionary/discovery/content_types.ex the :news row, and :news in @known
lib/devils_dictionary/discovery/transport.ex the body: :xml capability — one branch and one reader, body_format/1
lib/devils_dictionary_web/components/culture.ex year/1 prefers published_at as 15 Sep 2026; year stays the fallback, so nothing without published_at changed
config/config.exs the :bing_news stanza (endpoint, market, max_age_days) and "bing-news" => [positive_refresh_seconds: 24 h]
config/test.exs the test stanza, including the pinned clock
config/runtime.exs see the finding below

Four beyond that list, each a finding rather than a convenience:

  1. lib/devils_dictionary/discovery/provider.ex — the optional parse_body/1 callback. The shared transport now calls this function; leaving it undeclared would mean the kit's contract module does not mention a function shared code depends on. It is the same optional-capability shape as retryable_status?/1, and body_format/1 still guards on function_exported?, so nothing breaks without it.
  2. config/runtime.exs, beyond the refresh default — the provider env prefix was String.upcase(slug), which for a hyphenated slug builds BING-NEWS_DISCOVERY_POSITIVE_REFRESH_SECONDS. bash and zsh both refuse an env name with a hyphen, so that override was unsettable by any deployment — open-library's too. Now String.replace("-", "_"), which makes News shelf: the :news row and Bing's news RSS feed as its first, keyless provider (from #134) #135's documented BING_NEWS_… real. Nothing referenced the hyphenated form.
  3. test/devils_dictionary/discovery/content_types_test.exs — asserted :text was the only type with no image slot, and looped known() -- [:text]. :news is text-first by design, so the premise was the count and not the rule. Now it derives the text-first class from aspect == nil and holds both members to the wide column and three lines. Per the checklist I read it rather than bending it to my provider.
  4. test/devils_dictionary/tasks/dd_provider_new_test.exs — asserted the literal list film, artwork, image, text, gif in the generator's refusal message. Same shape; now derived from ContentTypes.known(), which also checks the refusal names the table's own order.

New files: the provider, its conformance fixture and 4-line suite, a 32-case provider test, the integration doc, six screenshots.

Ledger

Ceiling 150, spent 38 — running totals are in docs/integrations/bing-news.md, written batch by batch.

# Requests Running Result
P1 2 2 The project's own UA is accepted (tested first, as asked): 200, XML, same 7 items a Chrome UA got. The fallback #135 allowed is not needed
P2 7 9 Quoting and shape, unpinned market. Discovered the market is geo-inferred and was pl-pl
P3 5 14 mkt=en-US: bestiality 7 → 12 items; war went from Polish games sites to Newsweek/UPI/NYT/the Atlantic
P4 20 34 Rate: 20/20 200 at 1 req/s, no 429, no Retry-After, latency 238/262/409 ms min/median/max. referendum answered 200 with zero items
B1–B4 4 38 The browser proof, one request per word. B1's run was deleted to re-measure the card after the fix, which cascaded its ledger row — counted here, so discovery_request_attempts reads 3 and the true spend is 4

The freshness decision

max_age_days: 30, configurable, applied at retrieval. It is what makes this a News shelf rather than a search shelf, and the proof is /define/war: the feed returned 11 items and 2 reached the shelf, the other nine being from 2024 and earlier. Positive refresh is 24 h (shipped in config.exs's source_policies, overridable by the provider-prefixed env) — the shipped 30 days would freeze this week's shelf for a month. Empty refresh stays the shipped 24 h.

The clock is injected (BingNews.now/0 reads config()[:now]) because the fixture is a real capture with real September dates; config/test.exs pins it to the day of the capture, so the gate is deterministic in October.

The sources row change

None needed by hand. devils_dictionary_v2 already had three Oban nodes on it — main on 4007, the #131 page-mockup worktree on 4017 and #136's Urban Dictionary worktree on 4027, all confirmed with the checklist's lsof / pg_stat_activity recipe — so rather than add a fourth I took port 4037 and a database of my own, devils_dictionary_bing135, cloned from the idle devils_dictionary_74_verify and migrated to head. It had no bing-news row to be stale, so ensure_source/1 inserted mine:

   slug    | tier  |   name    |  attribution
-----------+-------+-----------+---------------
 bing-news | plebs | Bing News | via Bing News

devils_dictionary_v2 has no bing-news row and will get whatever source_attrs/0 says on its first run there. The 4 GB clone is still on disk if you want the evidence; dropdb devils_dictionary_bing135 when you don't.

Browser proof

CDP with Emulation.setDeviceMetricsOverride, not --window-size.

Page 1280 375 News items
/define/bestiality issue-135-bestiality-1280-2026-09-21.jpg issue-135-bestiality-375-2026-09-21.jpg 12
/define/logomachy issue-135-logomachy-1280-2026-09-21.jpg issue-135-logomachy-375-2026-09-21.jpg 0
/define/war issue-135-war-1280-2026-09-21.jpg issue-135-war-375-2026-09-21.jpg 2

document.documentElement.scrollWidth === 375 on all three at 375. No horizontal page scroll.

  • Dated, credited, linked out. Twelve cards on /define/bestiality, each a headline, 15 Sep 2026 · News, the masthead, Source ↗ to the publisher. No card links to bing.com — asserted in the DOM, not eyeballed.
  • Order. On /define/war the DOM reads culture-shelf-film, culture-shelf-text, culture-shelf-news, culture-shelf-image, giphy-d2Fy: News after Texts, before GIFs. Images falls between because a shelf nothing identified is demoted (D1 of Shelf appearance: one chrome per shelf, one-line credits, and what a search-only shelf is allowed to be #126) and the GIPHY shelf is a separate component rendered last. The GIF shelf needed its sources row inserted by hand on this database — GIPHY is transport: :browser, so the pipeline never calls ensure_source/1 for it.
  • The card showed the masthead twice, exactly as News shelf: the :news row and Bing's news RSS feed as its first, keyless provider (from #134) #135 suspected: creator line and :credited credit beneath it. Decided in the browser and dropped: the masthead goes to author/artist only (the text-sm unclamped line a fact belongs on), and nothing is written to "attribution". The row stays :credited rather than :none, because :none means the shelf byline is the whole credit and the shelf byline is Bing — the one thing a News shelf must not say about someone else's journalism.
  • The empty. /define/logomachy: one request, no_results, and two reloads afterwards spent none — the negative cache answered.

Two acceptance boxes I am not ticking

1. The reason reads at a headline, not in a headline. #135 asks for Uses “bestiality” in a headline, Wired, 15 September 2026. and also says MatchReason needs no change. Those cannot both hold: the attestation clause is "Uses " <> quoted(term) <> at(locator) and at/1 is " at " <> locator, so the preposition belongs to the renderer, not to the locator. The locator is exactly the string #135 specified; rendered, it is

Uses “bestiality” at a headline, Wired, 15 September 2026.

Fixing it is a one-line change in match_reason.ex, which #135 put out of scope, so I reported it rather than did it. The test asserts what it renders, with the reason in a comment.

2. /define/logomachy shows the shared empty-state line, not nothing at all. #135 says no News shelf. There is no rail and no cards, but the shelf region renders In news · No matching news for this term yet. — which is what every content type renders for an empty run (In film and In text are on the same screenshot) and is not this provider's to suppress. Visible in issue-135-logomachy-1280-2026-09-21.jpg.

Two other measured corrections to the issue

Tests

MIX_TEST_PARTITION=p135 mix precommit
1458 tests, 0 failures

Baseline on fresh main before any change was 1407 tests, 0 failures — including manifest_test.exs, which passes once data/ is symlinked from the main checkout, so this branch had no environmental failure to excuse. The bing-news conformance suite is 19/19 and the provider suite 32/32.

One flake seen once and not reproducible: artwork_live_test.exs:326 failed inside Artworks.artsy_suggestions/1 on one full run, then passed alone and on a full re-run. Unrelated code; flagging it rather than hiding it.

Not in scope, not touched

The Guardian, GDELT, the trending signal, Bing's thumbnails (thumbnail_keys: []), the D2 lever, Urban Dictionary (#136). No conflict with #136 beyond config/config.exs, where each adds one stanza.

🤖 Generated with Claude Code

Adds the `:news` content-type row and Bing's keyless news RSS feed as its
first provider (#135, from #134's live shootout). A headline is neither a
Text nor an image: text-first card, no poster frame, a date where the year
goes, the masthead as the credit, and evidence that is attestation with a
dated locator — the third locator shape after PoetryDB's line and Open
Library's page, and data rather than a new clause in `MatchReason`.

The search proposes and the text disposes, twice over. The feed is a search
and not a wire: it answers `nepotism` with 2025 and `logomachy` with two
Word-of-the-day pieces from March, so a 30-day `pubDate` window is what
makes this a News shelf at all. And the word has to be in the text, at a
Unicode boundary, in the headline or else in Bing's one sentence — the same
gate PoetryDB applies to a poem's lines.

Two things the probe found that the issue did not. `mkt` is inferred from
the caller's address, and from this desk it was Poland: `/define/war` came
back as Polish games-site pages about War Thunder, two of them Steam
listings from 2006. Pinned to `en-US` the same word answers with Newsweek,
UPI and the New York Times, and `bestiality` goes from 7 items to 12. And
the feed publishes its own terms in `<copyright>`, restricting these
results to an RSS aggregator for personal non-commercial use — which the
issue records as "none published". `source_attrs/0` quotes the restriction
instead of claiming none, and `docs/integrations/bing-news.md` opens with
it, because it is not a decision a provider can make.

One capability in shared code, kept to one branch: a provider may declare
`body: :xml` and the transport hands it the binary 200 for its own
`parse_body/1` to decode, with `:error` reported as the same
`"malformed_response"` a bad JSON envelope gets. The format a source speaks
is the provider's knowledge.

The publisher URL decoded out of the `apiclick` redirect is the item — its
link and its identity, normalised and hashed into the `news_article`
namespace so a keyed Guardian provider can fold into the same card later.
The redirect carries a per-response `tid`, so hashing it would make the
same article new on every fetch, and following it would send the reader to
Bing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 15 billable files and costs up to $3.75.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 20 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available. Your 59 included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Essentials

Run ID: 5ad1eea6-d9b3-479a-88ce-9d7d98807b0f

📥 Commits

Reviewing files that changed from the base of the PR and between 3c43fa7 and a99329d.

⛔ Files ignored due to path filters (6)
  • docs/discovery/issue-135-bestiality-1280-2026-09-21.jpg is excluded by !**/*.jpg
  • docs/discovery/issue-135-bestiality-375-2026-09-21.jpg is excluded by !**/*.jpg
  • docs/discovery/issue-135-logomachy-1280-2026-09-21.jpg is excluded by !**/*.jpg
  • docs/discovery/issue-135-logomachy-375-2026-09-21.jpg is excluded by !**/*.jpg
  • docs/discovery/issue-135-war-1280-2026-09-21.jpg is excluded by !**/*.jpg
  • docs/discovery/issue-135-war-375-2026-09-21.jpg is excluded by !**/*.jpg
📒 Files selected for processing (15)
  • config/config.exs
  • config/runtime.exs
  • config/test.exs
  • docs/discovery/README.md
  • docs/integrations/bing-news.md
  • lib/devils_dictionary/discovery/content_types.ex
  • lib/devils_dictionary/discovery/provider.ex
  • lib/devils_dictionary/discovery/providers/bing_news.ex
  • lib/devils_dictionary/discovery/transport.ex
  • lib/devils_dictionary_web/components/culture.ex
  • test/devils_dictionary/discovery/conformance/bing_news_conformance_test.exs
  • test/devils_dictionary/discovery/content_types_test.exs
  • test/devils_dictionary/discovery/providers/bing_news_test.exs
  • test/devils_dictionary/tasks/dd_provider_new_test.exs
  • test/support/discovery/conformance/bing_news_fixture.ex

Comment @coderabbitai help to get the list of available commands.

…at turns it on

CodeRabbit's review of #140 asked for `enabled: false` by default, for the
reason the PR itself raised: the feed's own <copyright> element restricts its
results to a personal RSS aggregator and reserves anything else to Microsoft's
written permission. The provider, the row and the tests are unchanged; the
owner turns the shelf on with one environment variable once that question is
settled. Also the two markdown lint findings in the integration doc.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@holden

holden commented Sep 21, 2026

Copy link
Copy Markdown
Member Author

CodeRabbit round (CLI, cr review --agent --base main, 2026-09-21)

The bot's own run on this PR hit its review limit and reviewed nothing, so the review was run from the CLI. Three findings, two actionable:

Severity Finding Done
major config/config.exs: ship Bing News with enabled: false, given the feed's <copyright> this PR raised Yes, in a30c965. Plus BING_NEWS_ENABLED=true in runtime.exs as the one switch that turns it on, so the owner's decision is an environment variable and not a code change. Tests unchanged (config/test.exs already enables it)
minor docs/integrations/bing-news.md: two markdown lint items (a paragraph starting #135 reads as a heading; an untagged fence) Yes, same commit

The provider, the row, the transport capability and the tests are as reviewed above. Full suite on the merge of this branch with #139: see the merge commit.

# Conflicts:
#	config/runtime.exs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

News shelf: the :news row and Bing's news RSS feed as its first, keyless provider (from #134)

1 participant