From 056a56fd72c3c9ddb160e038af443277a799c6c8 Mon Sep 17 00:00:00 2001 From: Holden Thomas Date: Thu, 24 Sep 2026 14:14:47 +0200 Subject: [PATCH] Build 6: DevilsDictionary.Corpus.Seeder, and the quotations corpus seeded through the entry path (#174) The seeder moves to DevilsDictionary.Corpus.Seeder now that it holds two kinds of content; DevilsDictionary.Artworks.Corpus.Seeder delegates to it for one release, and the artworks corpora's existing suites still call the old name, unchanged and green. A wikiquote-pd row is a content/quotation SourceIdentity.Entry in the live Wikiquote provider's shape, credited authored_by (verified) by QID, so a corpus line and a live line with one wording are one subject. Its stable identifier is the fingerprint, not wikiquote_item: that id is a position on the page at a revision, and a pinned corpus proposing it would claim whichever line sits there today. Quotations.Corpus.seed_row/3 writes a source record per line (the manifest row), the evidence the build found on the corpus's credit, and the Verified badge when the item has none. Authors are prepared from the manifest's own Wikidata facts, so seeding makes no request. Source row wikiquote-pd-v1: aristocracy, corpus, dump, CC BY-SA 4.0, the manifest checksum in its config. Records are never swept. Co-Authored-By: Claude Fable 5.1 --- docs/discovery/README.md | 21 +- .../artworks/corpus/seeder.ex | 433 +-------------- lib/devils_dictionary/corpus/seeder.ex | 516 ++++++++++++++++++ lib/devils_dictionary/quotations/corpus.ex | 271 +++++++++ lib/devils_dictionary/sources/catalog.ex | 3 +- lib/mix/tasks/dd.artworks.seed.ex | 11 +- lib/mix/tasks/dd.provider.new.ex | 6 +- .../corpus/seeder_quotes_test.exs | 173 ++++++ 8 files changed, 1002 insertions(+), 432 deletions(-) create mode 100644 lib/devils_dictionary/corpus/seeder.ex create mode 100644 lib/devils_dictionary/quotations/corpus.ex create mode 100644 test/devils_dictionary/corpus/seeder_quotes_test.exs diff --git a/docs/discovery/README.md b/docs/discovery/README.md index 045a4c5d..9b4fdf27 100644 --- a/docs/discovery/README.md +++ b/docs/discovery/README.md @@ -90,9 +90,22 @@ them names a provider. ### A corpus -A committed, checksummed **selection** in `priv/artworks/manifests/`, built once -and seeded onto the registry by `DevilsDictionary.Artworks.Corpus.Seeder`. It -makes no provider call at seed time and none at read time. The reason it is a +A committed, checksummed **selection**, built once and seeded onto the registry +by `DevilsDictionary.Corpus.Seeder` (`DevilsDictionary.Artworks.Corpus.Seeder` +until #174; the old name delegates for one release). It makes no provider call +at seed time and none at read time. It holds **two kinds of content**: + +- **Works**, under `priv/artworks/manifests/`: artworks, poems and books, each + an `entities` row with its `work_details`. These are `met-highlights-v1`, + `wikidata-famous-v1`, `poetrydb-v1` and `open-library-v1`. +- **Quotations**, under `priv/quotes/manifests/`: `wikiquote-pd-v1` (#174), + public-domain Wikiquote lines that build 5's checks call *Verified* against + a pre-1931 Gutenberg text by their credited author. Each line is a + `content_items` quotation in the live Wikiquote provider's entry shape, + identified by its fingerprint, so a corpus line and a live line with one + wording are one subject. Built by `mix dd.quotes.corpus.build`, which + refuses to overwrite a set that has drifted + ([`docs/integrations/wikiquote.md`](../integrations/wikiquote.md#corpus)). The reason it is a file rather than a query: the #99 P0 probe measured the Met's own search totals as parameter-order-sensitive — the same highlight query answered 2,310, 2,299 and 62 depending on the order the parameters were written in — so a corpus that @@ -544,7 +557,7 @@ declares the first. | `:news` | Bing News, The Guardian | — | Chronicling America's bulk OCR; GDELT (#134) | attestation, **dated** | | `:music` | Spotify | — | MusicBrainz (#116), identity via `P921`/`P2207` | a labelled search, gated on the track title | | `:gif` | GIPHY, browser-only — **inside** this chrome since #144 Phase 3, through `Culture.browser_shelf/1` | — | Tenor, after K10 | a labelled search | -| `:quote` | Wikiquote (#158 build 4) | — | the public-domain Wikiquote corpus (build 6); Wiktionary as a *shelf* source only if a decision asks for it — its absorbed quotations already render **under the sense** on the word page (build 1, `Word.quotations/1`, the same `Quotation.card/1` as this shelf), not on this rail | identity: the sense's QID → the theme page's `enwikiquote` sitelink; one line from two sources folds on its `quotation_fingerprint` (ADR 0003) | +| `:quote` | Wikiquote (#158 build 4) | `wikiquote-pd-v1` (#174), held locally | Wiktionary as a *shelf* source only if a decision asks for it — its absorbed quotations already render **under the sense** on the word page (build 1, `Word.quotations/1`, the same `Quotation.card/1` as this shelf), not on this rail | identity: the sense's QID → the theme page's `enwikiquote` sitelink; one line from two sources folds on its `quotation_fingerprint` (ADR 0003) | The `:text` decision is #109's second residual, settled here: the two text corpora are seeded `evidence: :none` and stay that way. Serving attestation diff --git a/lib/devils_dictionary/artworks/corpus/seeder.ex b/lib/devils_dictionary/artworks/corpus/seeder.ex index 56956567..4928aaa8 100644 --- a/lib/devils_dictionary/artworks/corpus/seeder.ex +++ b/lib/devils_dictionary/artworks/corpus/seeder.ex @@ -1,430 +1,23 @@ defmodule DevilsDictionary.Artworks.Corpus.Seeder do @moduledoc """ - Seeds a committed corpus manifest into the existing encyclopedia structures. + The seeder's old name, kept for one release (#174). - Zero new tables. An artwork is what #86 made it: an `entities` row with - `work_details.work_kind == "artwork"` and its identity in - `external_identifiers` — namespace `met_object_id` for a Met object, - `wikidata` for a QID (both, when the Met publishes the object's QID). That is - the same shape the Artsy pilot and the Phase 2a discovery provider resolve to, - which is why the three read as one catalog. + The seeder moved to `DevilsDictionary.Corpus.Seeder` when it gained its + second kind of content, the public-domain Wikiquote corpus. This module + delegates every public function to it, unchanged, so a caller or a script + written against the old name keeps working until the alias is removed. - ## Why the depicted QIDs are metadata - - Every Met row carries its tag QIDs and every Wikidata row its `P180` - depictions, so a word page can ask *which artworks depict Q198* without a live - search. Those QIDs are not identity — an artwork is not the thing it depicts — - and the depicted concept usually has no local entity to point a claim at, so - they are stored on the artwork's `metadata` as `depicts` (term + QID) and - `depicts_qids` (the flat list the lookup filters on). A claim would need an - object; this needs only a match key. - - ## Idempotence - - The manifest is the whole input and `SourceIdentity.resolve/1` is the whole - write: a second run of the same manifest matches every row on its exact - identifier and updates nothing that is already filled, so the counts come back - identical. Nothing here consults a title. - """ - - alias DevilsDictionary.Artworks.Corpus.Manifest - alias DevilsDictionary.Registry - alias DevilsDictionary.SourceIdentity - alias DevilsDictionary.SourceIdentity.Entry - - @met_collection "The Metropolitan Museum of Art" - - @doc """ - Seeds one loaded manifest and returns a summary. - - Options: `:dry_run` (resolve nothing, just count what would be seeded), - `:limit` (seed only the first N rows) and `:refresh`. - - ## `:refresh` - - `SourceIdentity.resolve/1` never replaces a field an identity already holds — - an absorbed record must not be overwritten by a manifest-shaped summary. That - is right for identity and description, and wrong for the handful of display - facts a committed manifest *is* the source of: the image URL and the credit - that travels with it, and the depicted QIDs the lookup matches on. Correcting - a manifest would otherwise reach the database only by deleting rows. - - So `refresh: true` rewrites exactly those keys, and only on an identity this - manifest already owns — one whose metadata carries this manifest's `corpus` - version. It mints nothing and touches no other row. + Not marked `@deprecated`: the artworks corpus's existing tests call this + name on purpose, as the proof that the move changed nothing, and the gate + compiles with warnings as errors. Remove this file, and point those tests at + the new name, in the release after #174. """ - def run(manifest, opts \\ []) when is_map(manifest) do - rows = manifest["rows"] |> List.wrap() |> take(opts[:limit]) - kind = manifest["kind"] - version = manifest["manifest"] - - summary = - Enum.reduce(rows, empty_summary(kind, version, length(rows)), fn row, summary -> - case entry(kind, version, row) do - {:ok, entry} -> - if opts[:dry_run] do - tally(summary, :would_seed, row) - else - # Ownership is read before the resolve, not after: resolving fills - # this manifest's missing keys onto whatever identity it matched, - # so afterwards an absorbed record carries the `corpus` marker too - # and would look owned when it is not. - owned? = opts[:refresh] && corpus_owned?(entry, version) - resolution = SourceIdentity.resolve(entry) - refreshed? = owned? && refresh(resolution, entry) - - summary - |> tally(resolution.state, row) - |> Map.update!(:refreshed, &if(refreshed?, do: &1 + 1, else: &1)) - end - - {:error, reason} -> - summary - |> tally(:invalid, row) - |> Map.update!( - :invalid_reasons, - &Map.update(&1, to_string(reason), 1, fn n -> n + 1 end) - ) - end - end) - - {:ok, Map.delete(summary, :depicted_qid_set)} - end - - @doc "Loads a committed manifest from disk and seeds it." - def run_file(path, opts \\ []) do - path |> Manifest.load!() |> run(opts) - end - - defp take(rows, nil), do: rows - defp take(rows, limit) when is_integer(limit), do: Enum.take(rows, limit) - - defp empty_summary(kind, version, count) do - %{ - kind: kind, - manifest: version, - rows: count, - newly_created: 0, - matched: 0, - would_seed: 0, - insufficient_evidence: 0, - conflicting_identifiers: 0, - invalid: 0, - invalid_reasons: %{}, - refreshed: 0, - with_image: 0, - with_depicts: 0, - depicted_qids: 0, - depicted_qid_set: MapSet.new() - } - end - - defp tally(summary, state, row) do - depicts = depicts(row) - - summary - |> Map.update!(state, &(&1 + 1)) - |> Map.update!(:with_image, &if(present?(row["image_url"]), do: &1 + 1, else: &1)) - |> Map.update!(:with_depicts, &if(depicts != [], do: &1 + 1, else: &1)) - |> Map.update!(:depicted_qid_set, fn set -> - Enum.reduce(depicts, set, &MapSet.put(&2, &1["qid"])) - end) - |> then(&Map.put(&1, :depicted_qids, MapSet.size(&1.depicted_qid_set))) - end - - @owned_display_keys ~w(image_url image_attribution credit_line depicts depicts_qids sitelinks) - - # A row this manifest has not seeded yet is its own to write; a row it seeded - # before carries its version. Anything else belongs to whoever put it there. - defp corpus_owned?(entry, version) do - identifier = entry.stable_identifier - - case Registry.by_external_id(identifier.namespace, identifier.external_id) do - nil -> - true - - object_id -> - DevilsDictionary.Repo.get!(Registry.Entity, object_id).metadata["corpus"] == version - end - end - - defp refresh(%{object_id: object_id}, entry) when is_integer(object_id) do - entity = DevilsDictionary.Repo.get!(Registry.Entity, object_id) - owned = Map.take(entry.metadata, @owned_display_keys) - - if Map.take(entity.metadata, Map.keys(owned)) == owned do - false - else - entity - |> Registry.Entity.changeset(%{metadata: Map.merge(entity.metadata, owned)}) - |> DevilsDictionary.Repo.update!() - - true - end - end - - defp refresh(_resolution, _entry), do: false - - @doc "The provider-neutral identity proposal for one manifest row." - def entry("met-highlights", version, row) do - with {:ok, met_id} <- required(row["met_object_id"]), - {:ok, title} <- required(row["title"]) do - Entry.new(%{ - source_slug: "met", - object_kind: :entity, - entity_kind: :work, - work_kind: "artwork", - stable_identifier: %{namespace: "met_object_id", external_id: met_id}, - identifiers: - [%{namespace: "met_object_id", external_id: met_id}] ++ - qid_identifier(row["qid"]), - label: label(title), - year: begin_year(row["date"]), - metadata: - base_metadata(row, version, "met") - |> put_present("artist_display_name", row["artist"]) - |> put_present("object_date", row["date"]) - |> put_present("medium", row["medium"]) - |> put_present("department", row["department"]) - |> Map.put("collection", @met_collection), - eligibility: :eligible, - retention: :durable - }) - end - end - - def entry("wikidata-famous", version, row) do - with {:ok, qid} <- required(row["qid"]), - {:ok, title} <- required(row["title"]) do - Entry.new(%{ - source_slug: "wikidata", - object_kind: :entity, - entity_kind: :work, - work_kind: "artwork", - stable_identifier: %{namespace: "wikidata", external_id: qid}, - identifiers: [%{namespace: "wikidata", external_id: qid}], - label: label(title), - year: begin_year(row["date"]), - metadata: - base_metadata(row, version, "wikidata") - |> put_present("artist_display_name", creator_names(row)) - |> put_present("object_date", row["date"]) - |> put_present("commons_file", row["commons_file"]) - |> put_present("creator_qids", creator_qids(row)) - |> put_sitelinks(row["sitelinks"]), - eligibility: :eligible, - retention: :durable - }) - end - end - - @doc false - def entry("poetrydb", version, row) do - with {:ok, poem_id} <- required(row["poem_id"]), - {:ok, title} <- required(row["title"]), - {:ok, author} <- required(row["author"]) do - Entry.new(%{ - source_slug: "poetrydb", - object_kind: :entity, - entity_kind: :work, - work_kind: "poem", - stable_identifier: %{namespace: "poetrydb_poem", external_id: poem_id}, - # The poet's QID is **not** among these. A poem is not the person who - # wrote it, and registering `wikidata: Q82083` on *Ode to a Nightingale* - # would claim it is Keats — and collide with the entity that actually is. - # The crosswalk is recorded as what it is: a fact about the author. - identifiers: [%{namespace: "poetrydb_poem", external_id: poem_id}], - label: label(title), - metadata: - %{ - "content_type" => "text", - "catalog_source" => "poetrydb", - "corpus" => version, - "author_display_name" => author - } - |> put_present("author_qid", row["author_qid"]) - |> put_present("lines_sha256", row["lines_sha256"]) - |> put_present("source_url", row["source_url"]) - |> put_line_count(row["line_count"]), - eligibility: :eligible, - retention: :durable - }) - end - end - - @doc false - def entry("open-library", version, row) do - with {:ok, olid} <- required(row["olid"]), - {:ok, title} <- required(row["title"]) do - Entry.new(%{ - source_slug: "open-library", - object_kind: :entity, - entity_kind: :work, - work_kind: "book", - stable_identifier: %{namespace: "olid", external_id: olid}, - # The work's own Wikidata QID belongs here — unlike PoetryDB's - # `author_qid`, this one is about *this work* and not about the person - # who wrote it, so registering it collides with nothing. The authors' - # QIDs stay in metadata, where they are facts about the authors. - identifiers: [%{namespace: "olid", external_id: olid}] ++ wikidata_identifier(row["qid"]), - label: label(title), - year: row["year"], - metadata: - %{ - "content_type" => "text", - "catalog_source" => "open-library", - "corpus" => version - } - |> put_present("author_display_name", author_names(row)) - |> put_present("author_qids", author_qids(row)) - |> put_present("public_domain_basis", row["pd_basis"]) - |> put_present("source_url", row["source_url"]) - |> put_present("wikidata_url", row["wikidata_url"]) - |> put_sitelinks(row["sitelinks"]), - eligibility: :eligible, - retention: :durable - }) - end - end - - def entry(kind, _version, _row), do: {:error, "unsupported_manifest_kind_#{kind}"} - - defp wikidata_identifier(qid) do - if is_binary(qid) and valid_qid?(qid), - do: [%{namespace: "wikidata", external_id: qid, metadata: %{"field" => "P648_subject"}}], - else: [] - end - - defp author_names(%{"authors" => authors}) when is_list(authors) do - authors |> Enum.map(& &1["name"]) |> Enum.filter(&present?/1) |> Enum.join(", ") |> presence() - end - - defp author_names(_row), do: nil - - defp author_qids(%{"authors" => authors}) when is_list(authors) do - case authors |> Enum.map(& &1["qid"]) |> Enum.filter(&(is_binary(&1) and valid_qid?(&1))) do - [] -> nil - qids -> qids - end - end - - defp author_qids(_row), do: nil - - # `entities.preferred_label` is varchar(255) and Postgres counts characters, - # not bytes, so this is the column's own width. Met highlight titles reach it: - # "Rebel Cassion Destroyed by Federal Shells. At Fredericksburgh, May 3, 1863. - # Eight Horses Killed." is one of the shorter long ones. - defp label(title), do: String.slice(title, 0, 255) - - defp base_metadata(row, version, catalog_source) do - depicts = depicts(row) - - %{ - "content_type" => "artwork", - "catalog_source" => catalog_source, - "corpus" => version, - "depicts" => depicts, - "depicts_qids" => Enum.map(depicts, & &1["qid"]) - } - |> put_present("image_url", row["image_url"]) - |> put_present("image_attribution", row["credit_line"]) - |> put_present("credit_line", row["credit_line"]) - |> put_present("source_url", row["source_url"]) - end - - # A Met tag and a Wikidata `P180` statement are the same assertion in two - # vocabularies — *this picture shows that concept* — so they normalize to one - # shape and one lookup key. - defp depicts(row) do - (row["tags"] || row["depicts"] || []) - |> Enum.flat_map(fn - %{"qid" => qid} = entry when is_binary(qid) -> - if valid_qid?(qid), do: [%{"qid" => qid, "term" => presence(entry["term"])}], else: [] - - _ -> - [] - end) - |> Enum.uniq_by(& &1["qid"]) - end - - defp qid_identifier(qid) do - if is_binary(qid) and valid_qid?(qid), - do: [ - %{namespace: "wikidata", external_id: qid, metadata: %{"field" => "objectWikidata_URL"}} - ], - else: [] - end - - defp creator_names(row) do - (row["creators"] || []) - |> Enum.map(& &1["term"]) - |> Enum.filter(&present?/1) - |> Enum.join(", ") - |> presence() - end - - defp creator_qids(row) do - (row["creators"] || []) - |> Enum.map(& &1["qid"]) - |> Enum.filter(&(is_binary(&1) and valid_qid?(&1))) - |> case do - [] -> nil - qids -> qids - end - end - - defp put_sitelinks(metadata, count) when is_integer(count), - do: Map.put(metadata, "sitelinks", count) - - defp put_sitelinks(metadata, _count), do: metadata - - # `objectDate` and an inception year are free text — "ca. 1780", "1863–65". - # The first four-digit run is the only part of them that is a year. - defp begin_year(value) when is_binary(value) do - case Regex.run(~r/\d{4}/, value) do - [year] -> String.to_integer(year) - _ -> nil - end - end - - defp begin_year(_value), do: nil - - defp required(value) do - if present?(value), do: {:ok, String.trim(value)}, else: {:error, :missing_required_field} - end - - defp put_line_count(map, count) when is_integer(count), do: Map.put(map, "line_count", count) - defp put_line_count(map, _count), do: map - - defp put_present(map, _key, nil), do: map - defp put_present(map, _key, ""), do: map - defp put_present(map, key, value), do: Map.put(map, key, value) - - defp present?(value), do: is_binary(value) and String.trim(value) != "" - defp valid_qid?(value), do: Regex.match?(~r/\AQ[1-9]\d*\z/, value) - defp presence(value) when is_binary(value) do - case String.trim(value) do - "" -> nil - trimmed -> trimmed - end - end + defdelegate run(manifest, opts \\ []), to: DevilsDictionary.Corpus.Seeder - defp presence(_value), do: nil + defdelegate run_file(path, opts \\ []), to: DevilsDictionary.Corpus.Seeder - @doc "How many artworks the catalog currently holds, by catalog source." - def catalog_counts do - import Ecto.Query + defdelegate entry(kind, version, row), to: DevilsDictionary.Corpus.Seeder - DevilsDictionary.Repo.all( - from entity in Registry.Entity, - join: details in Registry.WorkDetails, - on: details.entity_id == entity.object_id and details.work_kind == "artwork", - group_by: fragment("COALESCE(?->>'catalog_source', 'unlabelled')", entity.metadata), - select: - {fragment("COALESCE(?->>'catalog_source', 'unlabelled')", entity.metadata), - count(entity.object_id)} - ) - |> Map.new() - end + defdelegate catalog_counts(), to: DevilsDictionary.Corpus.Seeder end diff --git a/lib/devils_dictionary/corpus/seeder.ex b/lib/devils_dictionary/corpus/seeder.ex new file mode 100644 index 00000000..48415126 --- /dev/null +++ b/lib/devils_dictionary/corpus/seeder.ex @@ -0,0 +1,516 @@ +defmodule DevilsDictionary.Corpus.Seeder do + @moduledoc """ + Seeds a committed corpus manifest into the existing encyclopedia structures. + + Two kinds of content (#174): **works** — artworks, poems, books, the + corpora under `priv/artworks/manifests/` — and **quotations**, the + public-domain Wikiquote corpus under `priv/quotes/manifests/`. It was + `DevilsDictionary.Artworks.Corpus.Seeder` until the second kind arrived; + that name delegates here for one release. + + ## Works + + Zero new tables. An artwork is what #86 made it: an `entities` row with + `work_details.work_kind == "artwork"` and its identity in + `external_identifiers` — namespace `met_object_id` for a Met object, + `wikidata` for a QID (both, when the Met publishes the object's QID). That is + the same shape the Artsy pilot and the Phase 2a discovery provider resolve to, + which is why the three read as one catalog. + + ## Why the depicted QIDs are metadata + + Every Met row carries its tag QIDs and every Wikidata row its `P180` + depictions, so a word page can ask *which artworks depict Q198* without a live + search. Those QIDs are not identity — an artwork is not the thing it depicts — + and the depicted concept usually has no local entity to point a claim at, so + they are stored on the artwork's `metadata` as `depicts` (term + QID) and + `depicts_qids` (the flat list the lookup filters on). A claim would need an + object; this needs only a match key. + + ## Idempotence + + The manifest is the whole input and `SourceIdentity.resolve/1` is the whole + write: a second run of the same manifest matches every row on its exact + identifier and updates nothing that is already filled, so the counts come back + identical. Nothing here consults a title. + + ## Quotations + + A `wikiquote-pd` row is a `content/quotation` entry in the shape the live + Wikiquote provider writes, identified by its fingerprint and credited + `authored_by` by QID, so a corpus line and a live line with one wording are + one subject. Its source record, its evidence and its badge are + `Quotations.Corpus.seed_row/3`'s. The authors are prepared from the + manifest's own Wikidata facts, so seeding still makes no request. + """ + + alias DevilsDictionary.Artworks.Corpus.Manifest + alias DevilsDictionary.Quotations + alias DevilsDictionary.Quotations.Fingerprint + alias DevilsDictionary.Registry + alias DevilsDictionary.SourceIdentity + alias DevilsDictionary.SourceIdentity.Entry + + @met_collection "The Metropolitan Museum of Art" + + @doc """ + Seeds one loaded manifest and returns a summary. + + Options: `:dry_run` (resolve nothing, just count what would be seeded), + `:limit` (seed only the first N rows) and `:refresh`. + + ## `:refresh` + + `SourceIdentity.resolve/1` never replaces a field an identity already holds — + an absorbed record must not be overwritten by a manifest-shaped summary. That + is right for identity and description, and wrong for the handful of display + facts a committed manifest *is* the source of: the image URL and the credit + that travels with it, and the depicted QIDs the lookup matches on. Correcting + a manifest would otherwise reach the database only by deleting rows. + + So `refresh: true` rewrites exactly those keys, and only on an identity this + manifest already owns — one whose metadata carries this manifest's `corpus` + version. It mints nothing and touches no other row. + """ + def run(manifest, opts \\ []) when is_map(manifest) do + rows = manifest["rows"] |> List.wrap() |> take(opts[:limit]) + kind = manifest["kind"] + version = manifest["manifest"] + context = if opts[:dry_run], do: nil, else: context(kind, manifest) + + summary = + Enum.reduce(rows, empty_summary(kind, version, length(rows)), fn row, summary -> + case entry(kind, version, row) do + {:ok, entry} -> + if opts[:dry_run] do + tally(summary, :would_seed, row) + else + # Ownership is read before the resolve, not after: resolving fills + # this manifest's missing keys onto whatever identity it matched, + # so afterwards an absorbed record carries the `corpus` marker too + # and would look owned when it is not. + # Only a work's display facts are the manifest's to rewrite; a + # quotation's text is a revision and is never refreshed. + owned? = + opts[:refresh] && entry.object_kind == :entity && corpus_owned?(entry, version) + + resolution = resolve(kind, entry, row, context) + refreshed? = owned? && refresh(resolution, entry) + + summary + |> tally(resolution.state, row) + |> Map.update!(:refreshed, &if(refreshed?, do: &1 + 1, else: &1)) + end + + {:error, reason} -> + summary + |> tally(:invalid, row) + |> Map.update!( + :invalid_reasons, + &Map.update(&1, to_string(reason), 1, fn n -> n + 1 end) + ) + end + end) + + {:ok, Map.delete(summary, :depicted_qid_set)} + end + + # What a kind's rows share. Works share nothing: the entry is the whole write. + defp context("wikiquote-pd", manifest), do: Quotations.Corpus.seed_context(manifest) + defp context(_kind, _manifest), do: nil + + defp resolve("wikiquote-pd", entry, row, context), + do: Quotations.Corpus.seed_row(entry, row, context) + + defp resolve(_kind, entry, _row, _context), do: SourceIdentity.resolve(entry) + + @doc "Loads a committed manifest from disk and seeds it." + def run_file(path, opts \\ []) do + path |> Manifest.load!() |> run(opts) + end + + defp take(rows, nil), do: rows + defp take(rows, limit) when is_integer(limit), do: Enum.take(rows, limit) + + defp empty_summary(kind, version, count) do + %{ + kind: kind, + manifest: version, + rows: count, + newly_created: 0, + matched: 0, + would_seed: 0, + insufficient_evidence: 0, + conflicting_identifiers: 0, + invalid: 0, + invalid_reasons: %{}, + refreshed: 0, + with_image: 0, + with_depicts: 0, + depicted_qids: 0, + depicted_qid_set: MapSet.new() + } + end + + defp tally(summary, state, row) do + depicts = depicts(row) + + summary + |> Map.update!(state, &(&1 + 1)) + |> Map.update!(:with_image, &if(present?(row["image_url"]), do: &1 + 1, else: &1)) + |> Map.update!(:with_depicts, &if(depicts != [], do: &1 + 1, else: &1)) + |> Map.update!(:depicted_qid_set, fn set -> + Enum.reduce(depicts, set, &MapSet.put(&2, &1["qid"])) + end) + |> then(&Map.put(&1, :depicted_qids, MapSet.size(&1.depicted_qid_set))) + end + + @owned_display_keys ~w(image_url image_attribution credit_line depicts depicts_qids sitelinks) + + # A row this manifest has not seeded yet is its own to write; a row it seeded + # before carries its version. Anything else belongs to whoever put it there. + defp corpus_owned?(entry, version) do + identifier = entry.stable_identifier + + case Registry.by_external_id(identifier.namespace, identifier.external_id) do + nil -> + true + + object_id -> + DevilsDictionary.Repo.get!(Registry.Entity, object_id).metadata["corpus"] == version + end + end + + defp refresh(%{object_id: object_id}, entry) when is_integer(object_id) do + entity = DevilsDictionary.Repo.get!(Registry.Entity, object_id) + owned = Map.take(entry.metadata, @owned_display_keys) + + if Map.take(entity.metadata, Map.keys(owned)) == owned do + false + else + entity + |> Registry.Entity.changeset(%{metadata: Map.merge(entity.metadata, owned)}) + |> DevilsDictionary.Repo.update!() + + true + end + end + + defp refresh(_resolution, _entry), do: false + + @doc "The provider-neutral identity proposal for one manifest row." + def entry("met-highlights", version, row) do + with {:ok, met_id} <- required(row["met_object_id"]), + {:ok, title} <- required(row["title"]) do + Entry.new(%{ + source_slug: "met", + object_kind: :entity, + entity_kind: :work, + work_kind: "artwork", + stable_identifier: %{namespace: "met_object_id", external_id: met_id}, + identifiers: + [%{namespace: "met_object_id", external_id: met_id}] ++ + qid_identifier(row["qid"]), + label: label(title), + year: begin_year(row["date"]), + metadata: + base_metadata(row, version, "met") + |> put_present("artist_display_name", row["artist"]) + |> put_present("object_date", row["date"]) + |> put_present("medium", row["medium"]) + |> put_present("department", row["department"]) + |> Map.put("collection", @met_collection), + eligibility: :eligible, + retention: :durable + }) + end + end + + def entry("wikidata-famous", version, row) do + with {:ok, qid} <- required(row["qid"]), + {:ok, title} <- required(row["title"]) do + Entry.new(%{ + source_slug: "wikidata", + object_kind: :entity, + entity_kind: :work, + work_kind: "artwork", + stable_identifier: %{namespace: "wikidata", external_id: qid}, + identifiers: [%{namespace: "wikidata", external_id: qid}], + label: label(title), + year: begin_year(row["date"]), + metadata: + base_metadata(row, version, "wikidata") + |> put_present("artist_display_name", creator_names(row)) + |> put_present("object_date", row["date"]) + |> put_present("commons_file", row["commons_file"]) + |> put_present("creator_qids", creator_qids(row)) + |> put_sitelinks(row["sitelinks"]), + eligibility: :eligible, + retention: :durable + }) + end + end + + @doc false + def entry("poetrydb", version, row) do + with {:ok, poem_id} <- required(row["poem_id"]), + {:ok, title} <- required(row["title"]), + {:ok, author} <- required(row["author"]) do + Entry.new(%{ + source_slug: "poetrydb", + object_kind: :entity, + entity_kind: :work, + work_kind: "poem", + stable_identifier: %{namespace: "poetrydb_poem", external_id: poem_id}, + # The poet's QID is **not** among these. A poem is not the person who + # wrote it, and registering `wikidata: Q82083` on *Ode to a Nightingale* + # would claim it is Keats — and collide with the entity that actually is. + # The crosswalk is recorded as what it is: a fact about the author. + identifiers: [%{namespace: "poetrydb_poem", external_id: poem_id}], + label: label(title), + metadata: + %{ + "content_type" => "text", + "catalog_source" => "poetrydb", + "corpus" => version, + "author_display_name" => author + } + |> put_present("author_qid", row["author_qid"]) + |> put_present("lines_sha256", row["lines_sha256"]) + |> put_present("source_url", row["source_url"]) + |> put_line_count(row["line_count"]), + eligibility: :eligible, + retention: :durable + }) + end + end + + @doc false + def entry("open-library", version, row) do + with {:ok, olid} <- required(row["olid"]), + {:ok, title} <- required(row["title"]) do + Entry.new(%{ + source_slug: "open-library", + object_kind: :entity, + entity_kind: :work, + work_kind: "book", + stable_identifier: %{namespace: "olid", external_id: olid}, + # The work's own Wikidata QID belongs here — unlike PoetryDB's + # `author_qid`, this one is about *this work* and not about the person + # who wrote it, so registering it collides with nothing. The authors' + # QIDs stay in metadata, where they are facts about the authors. + identifiers: [%{namespace: "olid", external_id: olid}] ++ wikidata_identifier(row["qid"]), + label: label(title), + year: row["year"], + metadata: + %{ + "content_type" => "text", + "catalog_source" => "open-library", + "corpus" => version + } + |> put_present("author_display_name", author_names(row)) + |> put_present("author_qids", author_qids(row)) + |> put_present("public_domain_basis", row["pd_basis"]) + |> put_present("source_url", row["source_url"]) + |> put_present("wikidata_url", row["wikidata_url"]) + |> put_sitelinks(row["sitelinks"]), + eligibility: :eligible, + retention: :durable + }) + end + end + + # A quotation (#174): the live provider's entry shape (`Wikiquote.identity_record/1`) + # with the fingerprint as the stable identifier — see `Quotations.Corpus` + # for why not the positional `wikiquote_item` — and the credit at + # `:verified`, because the build kept the line only when a text its author + # wrote contains it. + @doc false + def entry("wikiquote-pd", version, row) do + with {:ok, text} <- required(row["text"]), + {:ok, fingerprint} <- required(row["fingerprint"]), + {:ok, author} <- required(row["author_qid"]), + true <- Fingerprint.fingerprint(text) == fingerprint || {:error, :fingerprint_mismatch}, + true <- valid_qid?(author) || {:error, :invalid_author_qid} do + Entry.new(%{ + source_slug: Quotations.Corpus.slug(), + object_kind: :content, + content_kind: :quotation, + stable_identifier: %{namespace: "quotation_fingerprint", external_id: fingerprint}, + identifiers: [Fingerprint.identifier(text)], + label: label(text), + year: row["work_year"], + content: %{ + body: text, + canonical_url: row["source_url"], + year: row["work_year"], + rights_metadata: %{ + "license" => "CC BY-SA 4.0", + "license_url" => "https://creativecommons.org/licenses/by-sa/4.0/", + "attribution" => "Wikiquote contributors", + "revision_id" => row["revision_id"], + "page" => row["page"], + "manifest" => version + } + }, + metadata: %{ + "author_display_name" => row["author_label"], + "locator" => row["locator"], + "corpus" => version + }, + relationships: [ + %{ + role: "authored_by", + certainty: :verified, + target_identifiers: [ + %{namespace: "wikidata", external_id: author, metadata: %{"field" => "P50"}} + ] + } + ], + eligibility: :eligible, + retention: :durable + }) + end + end + + def entry(kind, _version, _row), do: {:error, "unsupported_manifest_kind_#{kind}"} + + defp wikidata_identifier(qid) do + if is_binary(qid) and valid_qid?(qid), + do: [%{namespace: "wikidata", external_id: qid, metadata: %{"field" => "P648_subject"}}], + else: [] + end + + defp author_names(%{"authors" => authors}) when is_list(authors) do + authors |> Enum.map(& &1["name"]) |> Enum.filter(&present?/1) |> Enum.join(", ") |> presence() + end + + defp author_names(_row), do: nil + + defp author_qids(%{"authors" => authors}) when is_list(authors) do + case authors |> Enum.map(& &1["qid"]) |> Enum.filter(&(is_binary(&1) and valid_qid?(&1))) do + [] -> nil + qids -> qids + end + end + + defp author_qids(_row), do: nil + + # `entities.preferred_label` is varchar(255) and Postgres counts characters, + # not bytes, so this is the column's own width. Met highlight titles reach it: + # "Rebel Cassion Destroyed by Federal Shells. At Fredericksburgh, May 3, 1863. + # Eight Horses Killed." is one of the shorter long ones. + defp label(title), do: String.slice(title, 0, 255) + + defp base_metadata(row, version, catalog_source) do + depicts = depicts(row) + + %{ + "content_type" => "artwork", + "catalog_source" => catalog_source, + "corpus" => version, + "depicts" => depicts, + "depicts_qids" => Enum.map(depicts, & &1["qid"]) + } + |> put_present("image_url", row["image_url"]) + |> put_present("image_attribution", row["credit_line"]) + |> put_present("credit_line", row["credit_line"]) + |> put_present("source_url", row["source_url"]) + end + + # A Met tag and a Wikidata `P180` statement are the same assertion in two + # vocabularies — *this picture shows that concept* — so they normalize to one + # shape and one lookup key. + defp depicts(row) do + (row["tags"] || row["depicts"] || []) + |> Enum.flat_map(fn + %{"qid" => qid} = entry when is_binary(qid) -> + if valid_qid?(qid), do: [%{"qid" => qid, "term" => presence(entry["term"])}], else: [] + + _ -> + [] + end) + |> Enum.uniq_by(& &1["qid"]) + end + + defp qid_identifier(qid) do + if is_binary(qid) and valid_qid?(qid), + do: [ + %{namespace: "wikidata", external_id: qid, metadata: %{"field" => "objectWikidata_URL"}} + ], + else: [] + end + + defp creator_names(row) do + (row["creators"] || []) + |> Enum.map(& &1["term"]) + |> Enum.filter(&present?/1) + |> Enum.join(", ") + |> presence() + end + + defp creator_qids(row) do + (row["creators"] || []) + |> Enum.map(& &1["qid"]) + |> Enum.filter(&(is_binary(&1) and valid_qid?(&1))) + |> case do + [] -> nil + qids -> qids + end + end + + defp put_sitelinks(metadata, count) when is_integer(count), + do: Map.put(metadata, "sitelinks", count) + + defp put_sitelinks(metadata, _count), do: metadata + + # `objectDate` and an inception year are free text — "ca. 1780", "1863–65". + # The first four-digit run is the only part of them that is a year. + defp begin_year(value) when is_binary(value) do + case Regex.run(~r/\d{4}/, value) do + [year] -> String.to_integer(year) + _ -> nil + end + end + + defp begin_year(_value), do: nil + + defp required(value) do + if present?(value), do: {:ok, String.trim(value)}, else: {:error, :missing_required_field} + end + + defp put_line_count(map, count) when is_integer(count), do: Map.put(map, "line_count", count) + defp put_line_count(map, _count), do: map + + defp put_present(map, _key, nil), do: map + defp put_present(map, _key, ""), do: map + defp put_present(map, key, value), do: Map.put(map, key, value) + + defp present?(value), do: is_binary(value) and String.trim(value) != "" + defp valid_qid?(value), do: Regex.match?(~r/\AQ[1-9]\d*\z/, value) + + defp presence(value) when is_binary(value) do + case String.trim(value) do + "" -> nil + trimmed -> trimmed + end + end + + defp presence(_value), do: nil + + @doc "How many artworks the catalog currently holds, by catalog source." + def catalog_counts do + import Ecto.Query + + DevilsDictionary.Repo.all( + from entity in Registry.Entity, + join: details in Registry.WorkDetails, + on: details.entity_id == entity.object_id and details.work_kind == "artwork", + group_by: fragment("COALESCE(?->>'catalog_source', 'unlabelled')", entity.metadata), + select: + {fragment("COALESCE(?->>'catalog_source', 'unlabelled')", entity.metadata), + count(entity.object_id)} + ) + |> Map.new() + end +end diff --git a/lib/devils_dictionary/quotations/corpus.ex b/lib/devils_dictionary/quotations/corpus.ex new file mode 100644 index 00000000..185ee50a --- /dev/null +++ b/lib/devils_dictionary/quotations/corpus.ex @@ -0,0 +1,271 @@ +defmodule DevilsDictionary.Quotations.Corpus do + @moduledoc """ + `wikiquote-pd-v1`, the public-domain Wikiquote corpus (#174, build 6 of #158), + once it is built: its source row, what seeding one line writes beyond the + identity, and (on the Quotes shelf) how a page finds its lines. + + The manifest is built by `Quotations.Corpus.Build` and committed at + `priv/quotes/manifests/wikiquote-pd-v1.json`. `Corpus.Seeder` seeds it the + way it seeds the artworks corpora: each row is a `SourceIdentity.Entry` in + the shape the live Wikiquote provider writes, `content/quotation` credited + `authored_by` by QID. The one difference is the stable identifier. A live + line's is `wikiquote_item`, a hash of the page, section and position, and + that is a position **on today's page**. The corpus pins a revision, so a + position it proposed could name a different line on the page a visit reads. + The corpus's stable identifier is therefore the line's + `quotation_fingerprint` (ADR 0003), the identifier a second source's copy + folds on, and a corpus line and a live line with one wording are one + subject whichever arrived first. + + ## What seeding a line writes (`seed_row/3`) + + * **a source record** under `wikiquote-pd-v1`, keyed by the fingerprint, + whose payload is the manifest row. It is the entry's provenance, and + what the evidence cites. + * **the identity**, through `SourceIdentity.resolve/2`, with the author + prepared from the manifest's own Wikidata facts (`Creators.classify/2` + over `selection.authors`). No request is made at seed time. + * **evidence** on the corpus's `authored_by`: one `:supports` row per + check the build recorded (the Gutenberg text at its line, and the + author's own Wikiquote page where it lists the line), each citing the + record above. Added once. + * **the badge**, on `content_items.metadata["provenance"]`, only when the + item has none. An item another source already holds may have been + through the verifier, whose pass is the item's and now counts this + credit too. Its clock recomputes the badge; the corpus does not + overwrite it. + + Nothing is refreshed and nothing is swept. The source record is referenced + by evidence and by the item's revision, and cleanup never takes a record + something durable stands on (`Discovery.purge_results/1`). + """ + + import Ecto.Query + + alias DevilsDictionary.Claims + alias DevilsDictionary.Claims.{Assertion, AssertionEvidence, AssertionRevision, Predicate} + alias DevilsDictionary.Corpus.SourceRecordRevision + alias DevilsDictionary.Quotations.Badge + alias DevilsDictionary.Registry.ContentItem + alias DevilsDictionary.Repo + alias DevilsDictionary.SourceIdentity + alias DevilsDictionary.SourceIdentity.Creators + alias DevilsDictionary.Sources + alias DevilsDictionary.Sources.SourceRecord + + @slug "wikiquote-pd-v1" + @manifest_path "priv/quotes/manifests/wikiquote-pd-v1.json" + @dump_url "https://dumps.wikimedia.org/enwikiquote/20260901/" + @license "CC BY-SA 4.0" + @license_url "https://creativecommons.org/licenses/by-sa/4.0/" + + @external_resource @manifest_path + + # The committed manifest's own checksums, read at compile time: the source + # row names the file it is, and recompiles when the file changes. + @checksums (case File.read(@manifest_path) do + {:ok, body} -> + manifest = Jason.decode!(body) + + %{ + "manifest_checksum" => manifest["checksum"], + "set_checksum" => manifest["set_checksum"], + "generated_at" => manifest["generated_at"], + "row_count" => manifest["row_count"] + } + + {:error, _} -> + %{} + end) + + @doc "The corpus's source slug." + def slug, do: @slug + + @doc "Where the committed manifest lives." + def manifest_path, do: @manifest_path + + @doc """ + The source row (#174 design 2): 👑 tier, `access: :dump`, CC BY-SA 4.0, and + the manifest checksum in its config. Appended to `Sources.Catalog.sources/0`. + """ + def source_attrs do + %{ + slug: @slug, + name: "Wikiquote, public domain", + tier: :aristocracy, + kind: :corpus, + access: :dump, + license: @license, + license_url: @license_url, + homepage: "https://en.wikiquote.org/", + logo: "/images/sources/wikiquote.png", + url_template: "https://en.wikiquote.org/wiki/{title}", + attribution: "Wikiquote contributors (CC BY-SA 4.0)", + active: true, + config: + Map.merge( + %{ + "mode" => "committed corpus; no request is made at seed or read time", + "manifest" => @manifest_path, + "dump_url" => @dump_url, + "built_by" => "mix dd.quotes.corpus.build", + "kept" => "Verified by build 5's checks against a pre-1931 Gutenberg text", + "retention" => "durable; never refreshed, never swept" + }, + @checksums + ) + } + end + + # ── seeding ─────────────────────────────────────────────────────────────── + + @doc """ + What every row of one manifest shares: the source row, and each author's + prepared outcome for `Creators`, read from the manifest's own facts. + """ + def seed_context(manifest) do + authors = get_in(manifest, ["selection", "authors"]) || %{} + + %{ + source: Sources.get_source_by_slug!(@slug), + prepared: + Map.new(authors, fn {qid, entity} -> {qid, Creators.classify(qid, %{qid => entity})} end), + generated_at: manifest["generated_at"], + manifest: manifest["manifest"] + } + end + + @doc """ + Seeds one row whose entry `Corpus.Seeder.entry/3` built. Returns the + resolution. + """ + def seed_row(entry, row, context) do + {:ok, resolution} = + Repo.transaction(fn -> + {:ok, record} = + Sources.upsert_record(context.source, %{ + external_id: row["fingerprint"], + url: row["source_url"], + raw: row + }) + + resolution = + entry + |> Map.merge(%{ + source_id: context.source.id, + source_record_id: record.id, + source_record_revision_id: record.current_revision.id + }) + |> SourceIdentity.resolve(prepared: context.prepared) + + if resolution.state in [:matched, :newly_created] and is_integer(resolution.object_id) do + evidence!(resolution.object_id, row, record.current_revision.id, context.source.id) + provenance!(resolution.object_id, row, context) + end + + resolution + end) + + resolution + end + + defp evidence!(content_id, row, record_revision_id, source_id) do + revision = + Repo.one( + from r in AssertionRevision, + join: a in Assertion, + on: a.id == r.assertion_id and a.source_id == ^source_id, + join: p in Predicate, + on: p.id == r.predicate_id and p.key == "authored_by", + where: r.subject_object_id == ^content_id and r.is_current, + order_by: [desc: r.id], + limit: 1 + ) + + if revision do + for check <- row["checks"] || [], check["role"] == "supports" do + locator = String.slice(to_string(check["locator"]), 0, 255) + + held? = + Repo.exists?( + from e in AssertionEvidence, + where: + e.assertion_revision_id == ^revision.id and + e.source_record_revision_id == ^record_revision_id and e.locator == ^locator + ) + + unless held? do + {:ok, _} = + Claims.add_evidence(revision.id, %{ + evidence_role: :supports, + source_record_revision_id: record_revision_id, + locator: locator, + attribution_text: attribution(check, row) + }) + end + end + end + end + + defp attribution(%{"kind" => "primary"}, row), + do: + "Project Gutenberg ##{row["gutenberg"]["ebook"]}, #{row["work_label"]} (#{row["work_year"]})" + + defp attribution(%{"source" => "wikiquote"}, row), + do: "Wikiquote, #{row["author_label"]}'s own page" + + defp attribution(check, _row), do: check["source"] + + defp provenance!(content_id, row, context) do + item = Repo.get!(ContentItem, content_id) + + if is_nil((item.metadata || %{})["provenance"]) do + provenance = %{ + "badge" => row["badge"], + "score" => row["score"], + "agreements" => row["agreements"], + "sources" => row["sources"], + "computed_at" => context.generated_at, + "verifier_version" => Badge.version(), + "computed_by" => context.manifest + } + + item + |> ContentItem.changeset(%{ + metadata: Map.put(item.metadata || %{}, "provenance", provenance) + }) + |> Repo.update!() + end + end + + # ── reading ─────────────────────────────────────────────────────────────── + + @doc """ + The seeded rows whose concepts are among `qids`, as the manifest wrote them, + with each row's `object_id` (the content item its fingerprint resolves to). + Read from the corpus's own source records, so a page asks the registry for + what was seeded and never reads the file. + """ + def rows_for_concepts([]), do: [] + + def rows_for_concepts(qids) when is_list(qids) do + Repo.all( + from r in SourceRecord, + join: s in assoc(r, :source), + join: rev in SourceRecordRevision, + on: rev.source_record_id == r.id and rev.revision_key == r.content_hash, + where: + s.slug == @slug and s.active and + fragment("? \\?| ?", rev.payload["concept_qids"], type(^qids, {:array, :string})), + order_by: [asc: r.external_id], + select: {rev.payload, r.id} + ) + |> Enum.map(fn {row, _record_id} -> + Map.put( + row, + "object_id", + DevilsDictionary.Registry.by_external_id("quotation_fingerprint", row["fingerprint"]) + ) + end) + end +end diff --git a/lib/devils_dictionary/sources/catalog.ex b/lib/devils_dictionary/sources/catalog.ex index 95e56839..8acc8786 100644 --- a/lib/devils_dictionary/sources/catalog.ex +++ b/lib/devils_dictionary/sources/catalog.ex @@ -202,7 +202,8 @@ defmodule DevilsDictionary.Sources.Catalog do ] ++ [artsy()] ++ DevilsDictionary.Discovery.Providers.source_catalog() ++ - DevilsDictionary.Quotations.Checkers.source_catalog() + DevilsDictionary.Quotations.Checkers.source_catalog() ++ + [DevilsDictionary.Quotations.Corpus.source_attrs()] end # Artsy is a **corpus**, not a provider (#144 Phase 3, and §4 of #144). diff --git a/lib/mix/tasks/dd.artworks.seed.ex b/lib/mix/tasks/dd.artworks.seed.ex index 4d1b827e..02bf66bb 100644 --- a/lib/mix/tasks/dd.artworks.seed.ex +++ b/lib/mix/tasks/dd.artworks.seed.ex @@ -3,11 +3,13 @@ defmodule Mix.Tasks.Dd.Artworks.Seed do @moduledoc """ Seeds the committed **corpus** manifests built by `mix dd.artworks.manifest` - — `met-highlights`, `wikidata-famous`, `poetrydb` — which are recognised by - their contents rather than by a flag: + — `met-highlights`, `wikidata-famous`, `poetrydb`, `open-library` — and the + quotations one built by `mix dd.quotes.corpus.build` (`wikiquote-pd`, #174), + which are recognised by their contents rather than by a flag: mix dd.artworks.seed --manifest priv/artworks/manifests/met-highlights-v1.json mix dd.artworks.seed --manifest priv/artworks/manifests/wikidata-famous-v1.json --dry-run + mix dd.artworks.seed --manifest priv/quotes/manifests/wikiquote-pd-v1.json A corpus manifest is the whole input: nothing is searched, fetched or resumed, and rerunning it matches every row on its exact identifier, so the counts come @@ -36,6 +38,7 @@ defmodule Mix.Tasks.Dd.Artworks.Seed do alias DevilsDictionary.Artworks alias DevilsDictionary.Artworks.Corpus alias DevilsDictionary.Artworks.{Manifest, WikidataCandidates} + alias DevilsDictionary.Corpus.Seeder @switches [ manifest: :string, @@ -83,7 +86,7 @@ defmodule Mix.Tasks.Dd.Artworks.Seed do manifest = Corpus.Manifest.load!(path) {:ok, summary} = - Corpus.Seeder.run(manifest, + Seeder.run(manifest, dry_run: opts[:dry_run] || false, limit: opts[:record_limit], refresh: opts[:refresh] || false @@ -96,7 +99,7 @@ defmodule Mix.Tasks.Dd.Artworks.Seed do Mix.shell().info("Corpus seed summary: " <> inspect(summary, pretty: true)) Mix.shell().info( - "Artwork catalog by source: " <> inspect(Corpus.Seeder.catalog_counts(), pretty: true) + "Artwork catalog by source: " <> inspect(Seeder.catalog_counts(), pretty: true) ) rescue error in [ArgumentError, File.Error, Jason.DecodeError] -> Mix.raise(Exception.message(error)) diff --git a/lib/mix/tasks/dd.provider.new.ex b/lib/mix/tasks/dd.provider.new.ex index 1b040fbb..2c6ec95f 100644 --- a/lib/mix/tasks/dd.provider.new.ex +++ b/lib/mix/tasks/dd.provider.new.ex @@ -61,7 +61,7 @@ defmodule Mix.Tasks.Dd.Provider.New do identity field, the `external_identifiers` namespace that field is written to, the `work_kind` it is seeded as and the `evidence` it carries — and the `entry/3` clause in - `DevilsDictionary.Artworks.Corpus.Seeder` that maps a row onto a + `DevilsDictionary.Corpus.Seeder` that maps a row onto a `SourceIdentity.Entry`. Only the person who has read the source's rows can write the second one, so the task prints both rather than guessing at them. This is a real gap between #109's K7 and the code, recorded here rather than @@ -471,7 +471,7 @@ defmodule Mix.Tasks.Dd.Provider.New do back or Corpus.Conformance asks this corpus for a contract it does not have. * add an `entry("#{assigns.slug}", version, row)` clause to - lib/devils_dictionary/artworks/corpus/seeder.ex that maps a row onto a + lib/devils_dictionary/corpus/seeder.ex that maps a row onto a SourceIdentity.Entry Both are pattern matches in shared modules, and only a reader of the @@ -1035,7 +1035,7 @@ defmodule Mix.Tasks.Dd.Provider.New do or identity and display facts only (`:none`). The suite asks a corpus only for the contract its `evidence` declares, and holds it to that declaration. - 2. `DevilsDictionary.Artworks.Corpus.Seeder.entry/3` needs a clause for + 2. `DevilsDictionary.Corpus.Seeder.entry/3` needs a clause for `"<%= @slug %>"` mapping one row onto a `DevilsDictionary.SourceIdentity.Entry` — the identity namespace, the label, the year, and the metadata the shelf reads. diff --git a/test/devils_dictionary/corpus/seeder_quotes_test.exs b/test/devils_dictionary/corpus/seeder_quotes_test.exs new file mode 100644 index 00000000..e3a03ba6 --- /dev/null +++ b/test/devils_dictionary/corpus/seeder_quotes_test.exs @@ -0,0 +1,173 @@ +defmodule DevilsDictionary.Corpus.SeederQuotesTest do + @moduledoc """ + The seeder's second kind of content (#174): the public-domain Wikiquote + corpus, seeded from a manifest the real build produced over captured answers + (`QuotesCorpusFixtures`). The artworks corpora keep their own suites, + unchanged and still calling the seeder by its old name. + """ + + use DevilsDictionary.DataCase, async: false + + import Ecto.Query + + alias DevilsDictionary.Claims.{Assertion, AssertionEvidence, AssertionRevision, Predicate} + alias DevilsDictionary.Corpus.Seeder + alias DevilsDictionary.Discovery.Providers.Wikiquote + alias DevilsDictionary.Quotations + alias DevilsDictionary.Quotations.Fingerprint + alias DevilsDictionary.QuotesCorpusFixtures + alias DevilsDictionary.Registry + alias DevilsDictionary.Registry.ContentItem + alias DevilsDictionary.SourceIdentity + alias DevilsDictionary.SourceIdentity.Creators + alias DevilsDictionary.Sources + alias DevilsDictionary.Sources.{Source, SourceRecord} + + @manifest QuotesCorpusFixtures.manifest() + + setup do + DevilsDictionary.Fixtures.seed_catalog!() + :ok + end + + defp credits(content_id) do + Repo.all( + from r in AssertionRevision, + join: p in Predicate, + on: p.id == r.predicate_id and p.key == "authored_by", + join: a in Assertion, + on: a.id == r.assertion_id, + join: s in Source, + on: s.id == a.source_id, + where: r.subject_object_id == ^content_id and r.is_current, + order_by: s.slug, + select: %{source: s.slug, revision_id: r.id, person: r.object_object_id} + ) + end + + test "the source row is the issue's: 👑, a dump, CC BY-SA 4.0" do + source = Sources.get_source_by_slug!("wikiquote-pd-v1") + + assert %{tier: :aristocracy, access: :dump, kind: :corpus, license: "CC BY-SA 4.0"} = source + assert source.config["dump_url"] =~ "dumps.wikimedia.org/enwikiquote/" + assert source.config["retention"] =~ "never swept" + end + + test "each line is a quotation, credited to its author by QID, with its evidence and badge" do + assert @manifest["row_count"] > 0 + assert {:ok, summary} = Seeder.run(@manifest) + assert summary.newly_created == @manifest["row_count"] + assert summary.invalid == 0 + + for row <- @manifest["rows"] do + content_id = Registry.by_external_id("quotation_fingerprint", row["fingerprint"]) + item = Repo.get!(ContentItem, content_id) + + assert item.content_kind == :quotation + assert item.metadata["provenance"]["badge"] == "verified" + assert item.metadata["provenance"]["computed_by"] == "wikiquote-pd-v1" + + # Minted from the manifest's own Wikidata facts: no request was made. + assert [%{source: "wikiquote-pd-v1", revision_id: revision_id, person: person}] = + credits(content_id) + + assert Registry.by_external_id("wikidata", "Q9068") == person + + evidence = + Repo.all(from e in AssertionEvidence, where: e.assertion_revision_id == ^revision_id) + + assert Enum.any?(evidence, &(&1.locator =~ "Candide (Gutenberg #19942), line")) + assert Enum.all?(evidence, &(&1.evidence_role == :supports)) + assert Enum.all?(evidence, &is_integer(&1.source_record_revision_id)) + end + end + + test "a second seed matches every line and writes nothing new" do + {:ok, _} = Seeder.run(@manifest) + evidence = Repo.aggregate(AssertionEvidence, :count) + items = Repo.aggregate(ContentItem, :count) + + assert {:ok, again} = Seeder.run(@manifest) + assert again.matched == @manifest["row_count"] + assert again.newly_created == 0 + assert Repo.aggregate(AssertionEvidence, :count) == evidence + assert Repo.aggregate(ContentItem, :count) == items + end + + test "a corpus line and a live line with one wording are one subject" do + row = hd(@manifest["rows"]) + # The live provider's copy of the same words, one character different in + # transcription: a curly apostrophe and no final full stop. + live_text = row["text"] |> String.replace("'", "’") |> String.trim_trailing(".") + assert Fingerprint.fingerprint(live_text) == row["fingerprint"] + + item = %{ + external_namespace: "wikiquote_item", + external_id: "livelineid", + identifiers: [ + %{namespace: "wikiquote_item", external_id: "livelineid", metadata: %{}}, + Fingerprint.identifier(live_text) + ], + preview_metadata: %{ + "title" => live_text, + "author_qid" => "Q9068", + "certainty" => "candidate", + "source_url" => "https://en.wikiquote.org/wiki/Voltaire" + } + } + + wikiquote = Sources.get_source_by_slug!("wikiquote") + {:ok, record} = Sources.upsert_record(wikiquote, %{external_id: "livelineid", raw: %{}}) + {:ok, entry} = Wikiquote.identity_record(item) + prepared = Quotations.Corpus.seed_context(@manifest).prepared + + live = + entry + |> Map.merge(%{ + source_id: wikiquote.id, + source_record_id: record.id, + source_record_revision_id: record.current_revision.id + }) + |> SourceIdentity.resolve(prepared: prepared) + + assert live.state == :newly_created + + {:ok, _} = Seeder.run(@manifest) + + assert Registry.by_external_id("quotation_fingerprint", row["fingerprint"]) == + live.object_id + + assert [%{source: "wikiquote"}, %{source: "wikiquote-pd-v1"}] = credits(live.object_id) + end + + test "a corpus record is never swept" do + {:ok, _} = Seeder.run(@manifest) + source = Sources.get_source_by_slug!("wikiquote-pd-v1") + held = Repo.aggregate(from(r in SourceRecord, where: r.source_id == ^source.id), :count) + assert held == @manifest["row_count"] + + DevilsDictionary.Discovery.cleanup() + + assert Repo.aggregate(from(r in SourceRecord, where: r.source_id == ^source.id), :count) == + held + end + + test "a row whose fingerprint is not its text's is refused, and nothing is written" do + [row | _] = @manifest["rows"] + tampered = %{@manifest | "rows" => [Map.put(row, "text", row["text"] <> " And more.")]} + + assert {:ok, summary} = Seeder.run(tampered) + assert summary.invalid == 1 + assert summary.invalid_reasons == %{"fingerprint_mismatch" => 1} + assert Registry.by_external_id("quotation_fingerprint", row["fingerprint"]) == nil + end + + test "the author is prepared from the manifest, never fetched" do + prepared = Quotations.Corpus.seed_context(@manifest).prepared + + assert {:ok, %{kind: :person, label: "Voltaire", birth_year: 1694}} = prepared["Q9068"] + + assert prepared["Q9068"] == + Creators.classify("Q9068", %{"Q9068" => @manifest["selection"]["authors"]["Q9068"]}) + end +end