Repository navigation
The News shelf: the :news row and Bing's news RSS feed as its first, keyless provider - #140
Conversation
Adds the `:news` content-type row and Bing's keyless news RSS feed as its first provider (#135, from #134's live shootout). A headline is neither a Text nor an image: text-first card, no poster frame, a date where the year goes, the masthead as the credit, and evidence that is attestation with a dated locator — the third locator shape after PoetryDB's line and Open Library's page, and data rather than a new clause in `MatchReason`. The search proposes and the text disposes, twice over. The feed is a search and not a wire: it answers `nepotism` with 2025 and `logomachy` with two Word-of-the-day pieces from March, so a 30-day `pubDate` window is what makes this a News shelf at all. And the word has to be in the text, at a Unicode boundary, in the headline or else in Bing's one sentence — the same gate PoetryDB applies to a poem's lines. Two things the probe found that the issue did not. `mkt` is inferred from the caller's address, and from this desk it was Poland: `/define/war` came back as Polish games-site pages about War Thunder, two of them Steam listings from 2006. Pinned to `en-US` the same word answers with Newsweek, UPI and the New York Times, and `bestiality` goes from 7 items to 12. And the feed publishes its own terms in `<copyright>`, restricting these results to an RSS aggregator for personal non-commercial use — which the issue records as "none published". `source_attrs/0` quotes the restriction instead of claiming none, and `docs/integrations/bing-news.md` opens with it, because it is not a decision a provider can make. One capability in shared code, kept to one branch: a provider may declare `body: :xml` and the transport hands it the binary 200 for its own `parse_body/1` to decode, with `:error` reported as the same `"malformed_response"` a bad JSON envelope gets. The format a source speaks is the provider's knowledge. The publisher URL decoded out of the `apiclick` redirect is the item — its link and its identity, normalised and hashed into the `news_article` namespace so a keyed Guardian provider can fold into the same card later. The redirect carries a per-response `tid`, so hashing it would make the same article new on every fetch, and following it would send the reader to Bing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reached
This review includes 15 billable files and costs up to $3.75.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 20 minutes for your next included review. View limit detailsLimit details: You’ve used all 2 included reviews currently available. Your 59 included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Essentials Run ID: ⛔ Files ignored due to path filters (6)
📒 Files selected for processing (15)
Comment |
…at turns it on CodeRabbit's review of #140 asked for `enabled: false` by default, for the reason the PR itself raised: the feed's own <copyright> element restricts its results to a personal RSS aggregator and reserves anything else to Microsoft's written permission. The provider, the row and the tests are unchanged; the owner turns the shelf on with one environment variable once that question is settled. Also the two markdown lint findings in the integration doc. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
CodeRabbit round (CLI,
|
| Severity | Finding | Done |
|---|---|---|
| major | config/config.exs: ship Bing News with enabled: false, given the feed's <copyright> this PR raised |
Yes, in a30c965. Plus BING_NEWS_ENABLED=true in runtime.exs as the one switch that turns it on, so the owner's decision is an environment variable and not a code change. Tests unchanged (config/test.exs already enables it) |
| minor | docs/integrations/bing-news.md: two markdown lint items (a paragraph starting #135 reads as a heading; an untagged fence) |
Yes, same commit |
The provider, the row, the transport capability and the tests are as reviewed above. Full suite on the merge of this branch with #139: see the merge commit.
# Conflicts: # config/runtime.exs
Closes #135. The News shelf — the
:newscontent-type row and Bing's keyless news RSS feed as its first provider.Read this first: the feed publishes its own terms
#135 records Terms: none published for the feed. That is not what the feed says. Every response carries a
<copyright>element, measured 2026-09-21:A public word page is not an RSS aggregator and is not personal non-commercial use. I built the issue as specified and recorded the restriction honestly rather than smoothing it:
source_attrs/0'slicensequotes it and points at the doc, anddocs/integrations/bing-news.mdopens with it. Whether this ships is your call, not mine — everything else here is done and green, and the shelf is designed to hold more than one source, so the Guardian (#63) is the keyed source that does publish terms.The four decisions
discoverynews— a newContentTypesrow,@knownafter:textand before:gifgetoffset— the feed does not page, so the offset is into the gated list the provider holds, as PoetryDB's is; one request per pagePlus: identity = the publisher URL normalised and hashed in namespace
news_article; evidence = attestation with a dated locator; tier:plebs; noidentity_record/1;covers?/1defaulttrue; source row slugbing-news, name Bing News, attribution via Bing News.Shared files touched
The six #135 expected:
lib/devils_dictionary/discovery/content_types.ex:newsrow, and:newsin@knownlib/devils_dictionary/discovery/transport.exbody: :xmlcapability — one branch and one reader,body_format/1lib/devils_dictionary_web/components/culture.exyear/1preferspublished_atas 15 Sep 2026;yearstays the fallback, so nothing withoutpublished_atchangedconfig/config.exs:bing_newsstanza (endpoint,market,max_age_days) and"bing-news" => [positive_refresh_seconds: 24 h]config/test.exsconfig/runtime.exsFour beyond that list, each a finding rather than a convenience:
lib/devils_dictionary/discovery/provider.ex— the optionalparse_body/1callback. The shared transport now calls this function; leaving it undeclared would mean the kit's contract module does not mention a function shared code depends on. It is the same optional-capability shape asretryable_status?/1, andbody_format/1still guards onfunction_exported?, so nothing breaks without it.config/runtime.exs, beyond the refresh default — the provider env prefix wasString.upcase(slug), which for a hyphenated slug buildsBING-NEWS_DISCOVERY_POSITIVE_REFRESH_SECONDS.bashandzshboth refuse an env name with a hyphen, so that override was unsettable by any deployment —open-library's too. NowString.replace("-", "_"), which makes News shelf: the:newsrow and Bing's news RSS feed as its first, keyless provider (from #134) #135's documentedBING_NEWS_…real. Nothing referenced the hyphenated form.test/devils_dictionary/discovery/content_types_test.exs— asserted:textwas the only type with no image slot, and loopedknown() -- [:text].:newsis text-first by design, so the premise was the count and not the rule. Now it derives the text-first class fromaspect == niland holds both members to the wide column and three lines. Per the checklist I read it rather than bending it to my provider.test/devils_dictionary/tasks/dd_provider_new_test.exs— asserted the literal listfilm, artwork, image, text, gifin the generator's refusal message. Same shape; now derived fromContentTypes.known(), which also checks the refusal names the table's own order.New files: the provider, its conformance fixture and 4-line suite, a 32-case provider test, the integration doc, six screenshots.
Ledger
Ceiling 150, spent 38 — running totals are in
docs/integrations/bing-news.md, written batch by batch.200, XML, same 7 items a Chrome UA got. The fallback #135 allowed is not neededpl-plmkt=en-US:bestiality7 → 12 items;warwent from Polish games sites to Newsweek/UPI/NYT/the Atlantic200at 1 req/s, no429, noRetry-After, latency 238/262/409 ms min/median/max.referendumanswered200with zero itemsdiscovery_request_attemptsreads 3 and the true spend is 4The freshness decision
max_age_days: 30, configurable, applied at retrieval. It is what makes this a News shelf rather than a search shelf, and the proof is/define/war: the feed returned 11 items and 2 reached the shelf, the other nine being from 2024 and earlier. Positive refresh is 24 h (shipped inconfig.exs'ssource_policies, overridable by the provider-prefixed env) — the shipped 30 days would freeze this week's shelf for a month. Empty refresh stays the shipped 24 h.The clock is injected (
BingNews.now/0readsconfig()[:now]) because the fixture is a real capture with real September dates;config/test.exspins it to the day of the capture, so the gate is deterministic in October.The sources row change
None needed by hand.
devils_dictionary_v2already had three Oban nodes on it — main on 4007, the #131 page-mockup worktree on 4017 and #136's Urban Dictionary worktree on 4027, all confirmed with the checklist'slsof/pg_stat_activityrecipe — so rather than add a fourth I took port 4037 and a database of my own,devils_dictionary_bing135, cloned from the idledevils_dictionary_74_verifyand migrated to head. It had nobing-newsrow to be stale, soensure_source/1inserted mine:devils_dictionary_v2has nobing-newsrow and will get whateversource_attrs/0says on its first run there. The 4 GB clone is still on disk if you want the evidence;dropdb devils_dictionary_bing135when you don't.Browser proof
CDP with
Emulation.setDeviceMetricsOverride, not--window-size./define/bestialityissue-135-bestiality-1280-2026-09-21.jpgissue-135-bestiality-375-2026-09-21.jpg/define/logomachyissue-135-logomachy-1280-2026-09-21.jpgissue-135-logomachy-375-2026-09-21.jpg/define/warissue-135-war-1280-2026-09-21.jpgissue-135-war-375-2026-09-21.jpgdocument.documentElement.scrollWidth === 375on all three at 375. No horizontal page scroll./define/bestiality, each a headline, 15 Sep 2026 · News, the masthead, Source ↗ to the publisher. No card links tobing.com— asserted in the DOM, not eyeballed./define/warthe DOM readsculture-shelf-film,culture-shelf-text,culture-shelf-news,culture-shelf-image,giphy-d2Fy: News after Texts, before GIFs. Images falls between because a shelf nothing identified is demoted (D1 of Shelf appearance: one chrome per shelf, one-line credits, and what a search-only shelf is allowed to be #126) and the GIPHY shelf is a separate component rendered last. The GIF shelf needed itssourcesrow inserted by hand on this database — GIPHY istransport: :browser, so the pipeline never callsensure_source/1for it.:newsrow and Bing's news RSS feed as its first, keyless provider (from #134) #135 suspected: creator line and:creditedcredit beneath it. Decided in the browser and dropped: the masthead goes toauthor/artistonly (thetext-smunclamped line a fact belongs on), and nothing is written to"attribution". The row stays:creditedrather than:none, because:nonemeans the shelf byline is the whole credit and the shelf byline is Bing — the one thing a News shelf must not say about someone else's journalism./define/logomachy: one request,no_results, and two reloads afterwards spent none — the negative cache answered.Two acceptance boxes I am not ticking
1. The reason reads at a headline, not in a headline. #135 asks for Uses “bestiality” in a headline, Wired, 15 September 2026. and also says
MatchReasonneeds no change. Those cannot both hold: the attestation clause is"Uses " <> quoted(term) <> at(locator)andat/1is" at " <> locator, so the preposition belongs to the renderer, not to the locator. The locator is exactly the string #135 specified; rendered, it isFixing it is a one-line change in
match_reason.ex, which #135 put out of scope, so I reported it rather than did it. The test asserts what it renders, with the reason in a comment.2.
/define/logomachyshows the shared empty-state line, not nothing at all. #135 says no News shelf. There is no rail and no cards, but the shelf region renders In news · No matching news for this term yet. — which is what every content type renders for an empty run (In film and In text are on the same screenshot) and is not this provider's to suppress. Visible inissue-135-logomachy-1280-2026-09-21.jpg.Two other measured corrections to the issue
:newsrow and Bing's news RSS feed as its first, keyless provider (from #134) #135 guessed a multi-word lemma needs quoting. Measured, it is the opposite:q=red herringanswered 6 items, 4 using the phrase and 2 from the last three days;q="red herring"answered 3, freshest 2024. The whole-word gate enforces the phrase anyway, so quoting only narrows the candidates.mkt=en-USis a request parameter, not a preference. Details in the doc; this is the single biggest quality difference measured in the probe and News shelf: the:newsrow and Bing's news RSS feed as its first, keyless provider (from #134) #135 does not mention it, because its numbers were taken from a desk the feed read differently.Tests
Baseline on fresh
mainbefore any change was 1407 tests, 0 failures — includingmanifest_test.exs, which passes oncedata/is symlinked from the main checkout, so this branch had no environmental failure to excuse. The bing-news conformance suite is 19/19 and the provider suite 32/32.One flake seen once and not reproducible:
artwork_live_test.exs:326failed insideArtworks.artsy_suggestions/1on one full run, then passed alone and on a full re-run. Unrelated code; flagging it rather than hiding it.Not in scope, not touched
The Guardian, GDELT, the trending signal, Bing's thumbnails (
thumbnail_keys: []), the D2 lever, Urban Dictionary (#136). No conflict with #136 beyondconfig/config.exs, where each adds one stanza.🤖 Generated with Claude Code