Repository navigation
Description
Purpose
Openverse's second page for war is its first page. Every Load more on a shelf Openverse is on spends a request on it and puts nothing new on the rail, and the shelf's turns silently stop taking a turn for it. Found while proving #126 Phase 1's single Load more; it is a provider-file fix, and #126 is an appearance issue that fences provider changes to Openverse's credit line, so it is here rather than there.
Schema impact
Measured
main @ 79fd344, devils_dictionary_v2, 2026-09-19, /define/war, one click of the shelf's Load more. discovery_runs:
| page | request_parameters |
scanned | kept | next_cursor |
|---|---|---|---|---|
| 0 | {"after": null, "first": 12, "offset": "0"} |
12 | 6 | 12 |
| 1 | {"after": "12", "first": 12, "offset": "12"} |
12 | 6 | 24 |
The two pages are the same six items — same ids, same creators, in the same order:
0a8fdfa0-b192-45a8-a0a8-cb6e1be4e649 Gorod - SKY
23d512a8-a2a1-45b5-a561-7c8c022aa638 Alexey Kljatov
c80387bb-2dd9-489e-a748-75a1be138f5a zbigphotography (1M+ views)
ce2b6d3d-9188-4e32-9d0c-15bc9b36df9a tonynetone
ec14e586-fa81-45d5-901c-f6067e8a9401 Eva Rinaldi Celebrity Photographer
fc0441be-ef06-454d-a452-ac9579d3e2a1 Xristoforos aka Shooting Dog
select count(distinct res.external_id) from discovery_runs r
join discovery_results res on res.run_id = r.id
join discovery_mappings m on m.id = r.mapping_id
join sources s on s.id = m.source_id
where s.slug = 'openverse' and m.parameters->>'term' = 'war';
-- 6, across two pages of 6Shelf.dedup/2 folds them at read time, correctly, so the rail's Openverse count stayed at 6 while Commons went 12 → 24, Unsplash 12 → 24 and Pexels 24 → 34 on the same click. One live request, zero new cards, and the shelf quietly one source shorter every page from here on.
What is already ruled out
Read from the code and the persisted runs, so the next session does not re-walk it:
- The cursor is right.
next_cursor/4wrote12, the pipeline handed it back asafter, andoffset/1parsed it to12.request_parametersrecords both. - The page number is right.
retrieve/4computesdiv(12, 12) + 1 = 2andrequest_options/1sendspage=2&page_size=12alongside the sameq,license,matureandfilter_deadit sent for page 1. - It is not the
upload/1fold. That folds a creator's repeated title within one response (12 scanned → 6 kept, both pages). It cannot turn twelve different rows into six rows with page 1's ids. - It is not read-time dedup hiding good items. The six persisted on page 1 are byte-identical ids to page 0's, so the duplication happened before persistence.
So the twelve rows Openverse returned for page=2 were the twelve it returned for page=1.
The probe, and the ceiling
Ten requests, ledgered in docs/integrations/openverse.md as they are spent. Two words, war and one with a larger result set, and for each:
page=1&page_size=12andpage=2&page_size=12with the provider's exact parameter set — do the ids differ?- The same two pages with
filter_deaddropped, then with the bare (unquoted)q— which parameter, if either, makes page 2 repeat page 1? - Record
result_countandpage_countoff each body.next_cursor/4already readspage_countand would have returnednilif it were1, so whatever it reports is not the whole story.
The answer decides the fix, and the fix is one of:
- Openverse honours a different paging parameter for this query shape → send it.
- The anonymous walk is narrower than
@max_results(240) for a filtered query →next_cursor/4learns the real ceiling from the body and stops, so the shelf stops asking and no request is wasted. - Openverse repeats regardless → the provider is honestly single-page for this shape and
next_cursor/4returnsnilafter page 0. A source that cannot page is not a defect in the shelf; a source that says it can and does not, is.
Whichever it is, the guard is the same and belongs in the provider's conformance suite: a second page must not be the first page. The fixture gets a second page with distinct ids, and the suite asserts that page 1's ids are disjoint from page 0's — the check openverse_conformance_test.exs would have failed on this.
Not doing
- No change to
Shelf.dedup/2: folding the repeat is correct and is what kept the rail honest. - No change to any other provider. Commons, Unsplash and Pexels each advanced a full page on the same click; this is Openverse's.
- Not Shelf appearance: one chrome per shelf, one-line credits, and what a search-only shelf is allowed to be #126's business, beyond the cross-reference. Its Phase 2 opens
openverse.exfor D2's credit line, so a session may reasonably do both — but the credit line is not gated on this and this is not gated on the credit line.
Checklist
- The probe, with its ledger
- The fix, whichever of the three it turns out to be
- The conformance guard: page 1's ids are disjoint from page 0's
- A browser check on
/define/war: Load more moves Openverse's count, or Openverse stops offering one
Activity
holden commentedon Sep 19, 2026
Fix lands with #126 Phase 2, which opens openverse.ex for the credit line (D2) in the same session; that PR closes this issue. Recorded on #126's Phase 2 list.
holden commentedon Sep 19, 2026
Fixed by #129 (filter_dead on the first page only; the fixture models the API and the shared pagination suite fails a regression). The ledger observation stands as recorded.