Skip to content

Openverse: the second page is the first page #128

Closed

Description

@holden

Purpose

Openverse's second page for war is its first page. Every Load more on a shelf Openverse is on spends a request on it and puts nothing new on the rail, and the shelf's turns silently stop taking a turn for it. Found while proving #126 Phase 1's single Load more; it is a provider-file fix, and #126 is an appearance issue that fences provider changes to Openverse's credit line, so it is here rather than there.

Schema impact

Measured

main @ 79fd344, devils_dictionary_v2, 2026-09-19, /define/war, one click of the shelf's Load more. discovery_runs:

page request_parameters scanned kept next_cursor
0 {"after": null, "first": 12, "offset": "0"} 12 6 12
1 {"after": "12", "first": 12, "offset": "12"} 12 6 24

The two pages are the same six items — same ids, same creators, in the same order:

0a8fdfa0-b192-45a8-a0a8-cb6e1be4e649  Gorod - SKY
23d512a8-a2a1-45b5-a561-7c8c022aa638  Alexey Kljatov
c80387bb-2dd9-489e-a748-75a1be138f5a  zbigphotography (1M+ views)
ce2b6d3d-9188-4e32-9d0c-15bc9b36df9a  tonynetone
ec14e586-fa81-45d5-901c-f6067e8a9401  Eva Rinaldi Celebrity Photographer
fc0441be-ef06-454d-a452-ac9579d3e2a1  Xristoforos aka Shooting Dog
select count(distinct res.external_id) from discovery_runs r
  join discovery_results res on res.run_id = r.id
  join discovery_mappings m on m.id = r.mapping_id
  join sources s on s.id = m.source_id
 where s.slug = 'openverse' and m.parameters->>'term' = 'war';
-- 6, across two pages of 6

Shelf.dedup/2 folds them at read time, correctly, so the rail's Openverse count stayed at 6 while Commons went 12 → 24, Unsplash 12 → 24 and Pexels 24 → 34 on the same click. One live request, zero new cards, and the shelf quietly one source shorter every page from here on.

What is already ruled out

Read from the code and the persisted runs, so the next session does not re-walk it:

  • The cursor is right. next_cursor/4 wrote 12, the pipeline handed it back as after, and offset/1 parsed it to 12. request_parameters records both.
  • The page number is right. retrieve/4 computes div(12, 12) + 1 = 2 and request_options/1 sends page=2&page_size=12 alongside the same q, license, mature and filter_dead it sent for page 1.
  • It is not the upload/1 fold. That folds a creator's repeated title within one response (12 scanned → 6 kept, both pages). It cannot turn twelve different rows into six rows with page 1's ids.
  • It is not read-time dedup hiding good items. The six persisted on page 1 are byte-identical ids to page 0's, so the duplication happened before persistence.

So the twelve rows Openverse returned for page=2 were the twelve it returned for page=1.

The probe, and the ceiling

Ten requests, ledgered in docs/integrations/openverse.md as they are spent. Two words, war and one with a larger result set, and for each:

  1. page=1&page_size=12 and page=2&page_size=12 with the provider's exact parameter set — do the ids differ?
  2. The same two pages with filter_dead dropped, then with the bare (unquoted) q — which parameter, if either, makes page 2 repeat page 1?
  3. Record result_count and page_count off each body. next_cursor/4 already reads page_count and would have returned nil if it were 1, so whatever it reports is not the whole story.

The answer decides the fix, and the fix is one of:

  • Openverse honours a different paging parameter for this query shape → send it.
  • The anonymous walk is narrower than @max_results (240) for a filtered query → next_cursor/4 learns the real ceiling from the body and stops, so the shelf stops asking and no request is wasted.
  • Openverse repeats regardless → the provider is honestly single-page for this shape and next_cursor/4 returns nil after page 0. A source that cannot page is not a defect in the shelf; a source that says it can and does not, is.

Whichever it is, the guard is the same and belongs in the provider's conformance suite: a second page must not be the first page. The fixture gets a second page with distinct ids, and the suite asserts that page 1's ids are disjoint from page 0's — the check openverse_conformance_test.exs would have failed on this.

Not doing

  • No change to Shelf.dedup/2: folding the repeat is correct and is what kept the rail honest.
  • No change to any other provider. Commons, Unsplash and Pexels each advanced a full page on the same click; this is Openverse's.
  • Not Shelf appearance: one chrome per shelf, one-line credits, and what a search-only shelf is allowed to be #126's business, beyond the cross-reference. Its Phase 2 opens openverse.ex for D2's credit line, so a session may reasonably do both — but the credit line is not gated on this and this is not gated on the credit line.

Checklist

  • The probe, with its ledger
  • The fix, whichever of the three it turns out to be
  • The conformance guard: page 1's ids are disjoint from page 0's
  • A browser check on /define/war: Load more moves Openverse's count, or Openverse stops offering one

Activity

  1. holden commented on Sep 19, 2026

    @holden
    MemberAuthor

    Fix lands with #126 Phase 2, which opens openverse.ex for the credit line (D2) in the same session; that PR closes this issue. Recorded on #126's Phase 2 list.

  2. holden commented on Sep 19, 2026

    @holden
    MemberAuthor

    Fixed by #129 (filter_dead on the first page only; the fixture models the API and the shared pagination suite fails a regression). The ledger observation stands as recorded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions