diff --git a/docs/discovery/README.md b/docs/discovery/README.md index 045a4c5d..9b4fdf27 100644 --- a/docs/discovery/README.md +++ b/docs/discovery/README.md @@ -90,9 +90,22 @@ them names a provider. ### A corpus -A committed, checksummed **selection** in `priv/artworks/manifests/`, built once -and seeded onto the registry by `DevilsDictionary.Artworks.Corpus.Seeder`. It -makes no provider call at seed time and none at read time. The reason it is a +A committed, checksummed **selection**, built once and seeded onto the registry +by `DevilsDictionary.Corpus.Seeder` (`DevilsDictionary.Artworks.Corpus.Seeder` +until #174; the old name delegates for one release). It makes no provider call +at seed time and none at read time. It holds **two kinds of content**: + +- **Works**, under `priv/artworks/manifests/`: artworks, poems and books, each + an `entities` row with its `work_details`. These are `met-highlights-v1`, + `wikidata-famous-v1`, `poetrydb-v1` and `open-library-v1`. +- **Quotations**, under `priv/quotes/manifests/`: `wikiquote-pd-v1` (#174), + public-domain Wikiquote lines that build 5's checks call *Verified* against + a pre-1931 Gutenberg text by their credited author. Each line is a + `content_items` quotation in the live Wikiquote provider's entry shape, + identified by its fingerprint, so a corpus line and a live line with one + wording are one subject. Built by `mix dd.quotes.corpus.build`, which + refuses to overwrite a set that has drifted + ([`docs/integrations/wikiquote.md`](../integrations/wikiquote.md#corpus)). The reason it is a file rather than a query: the #99 P0 probe measured the Met's own search totals as parameter-order-sensitive — the same highlight query answered 2,310, 2,299 and 62 depending on the order the parameters were written in — so a corpus that @@ -544,7 +557,7 @@ declares the first. | `:news` | Bing News, The Guardian | — | Chronicling America's bulk OCR; GDELT (#134) | attestation, **dated** | | `:music` | Spotify | — | MusicBrainz (#116), identity via `P921`/`P2207` | a labelled search, gated on the track title | | `:gif` | GIPHY, browser-only — **inside** this chrome since #144 Phase 3, through `Culture.browser_shelf/1` | — | Tenor, after K10 | a labelled search | -| `:quote` | Wikiquote (#158 build 4) | — | the public-domain Wikiquote corpus (build 6); Wiktionary as a *shelf* source only if a decision asks for it — its absorbed quotations already render **under the sense** on the word page (build 1, `Word.quotations/1`, the same `Quotation.card/1` as this shelf), not on this rail | identity: the sense's QID → the theme page's `enwikiquote` sitelink; one line from two sources folds on its `quotation_fingerprint` (ADR 0003) | +| `:quote` | Wikiquote (#158 build 4) | `wikiquote-pd-v1` (#174), held locally | Wiktionary as a *shelf* source only if a decision asks for it — its absorbed quotations already render **under the sense** on the word page (build 1, `Word.quotations/1`, the same `Quotation.card/1` as this shelf), not on this rail | identity: the sense's QID → the theme page's `enwikiquote` sitelink; one line from two sources folds on its `quotation_fingerprint` (ADR 0003) | The `:text` decision is #109's second residual, settled here: the two text corpora are seeded `evidence: :none` and stay that way. Serving attestation diff --git a/lib/devils_dictionary/artworks/corpus/seeder.ex b/lib/devils_dictionary/artworks/corpus/seeder.ex index 56956567..4928aaa8 100644 --- a/lib/devils_dictionary/artworks/corpus/seeder.ex +++ b/lib/devils_dictionary/artworks/corpus/seeder.ex @@ -1,430 +1,23 @@ defmodule DevilsDictionary.Artworks.Corpus.Seeder do @moduledoc """ - Seeds a committed corpus manifest into the existing encyclopedia structures. + The seeder's old name, kept for one release (#174). - Zero new tables. An artwork is what #86 made it: an `entities` row with - `work_details.work_kind == "artwork"` and its identity in - `external_identifiers` — namespace `met_object_id` for a Met object, - `wikidata` for a QID (both, when the Met publishes the object's QID). That is - the same shape the Artsy pilot and the Phase 2a discovery provider resolve to, - which is why the three read as one catalog. + The seeder moved to `DevilsDictionary.Corpus.Seeder` when it gained its + second kind of content, the public-domain Wikiquote corpus. This module + delegates every public function to it, unchanged, so a caller or a script + written against the old name keeps working until the alias is removed. - ## Why the depicted QIDs are metadata - - Every Met row carries its tag QIDs and every Wikidata row its `P180` - depictions, so a word page can ask *which artworks depict Q198* without a live - search. Those QIDs are not identity — an artwork is not the thing it depicts — - and the depicted concept usually has no local entity to point a claim at, so - they are stored on the artwork's `metadata` as `depicts` (term + QID) and - `depicts_qids` (the flat list the lookup filters on). A claim would need an - object; this needs only a match key. - - ## Idempotence - - The manifest is the whole input and `SourceIdentity.resolve/1` is the whole - write: a second run of the same manifest matches every row on its exact - identifier and updates nothing that is already filled, so the counts come back - identical. Nothing here consults a title. - """ - - alias DevilsDictionary.Artworks.Corpus.Manifest - alias DevilsDictionary.Registry - alias DevilsDictionary.SourceIdentity - alias DevilsDictionary.SourceIdentity.Entry - - @met_collection "The Metropolitan Museum of Art" - - @doc """ - Seeds one loaded manifest and returns a summary. - - Options: `:dry_run` (resolve nothing, just count what would be seeded), - `:limit` (seed only the first N rows) and `:refresh`. - - ## `:refresh` - - `SourceIdentity.resolve/1` never replaces a field an identity already holds — - an absorbed record must not be overwritten by a manifest-shaped summary. That - is right for identity and description, and wrong for the handful of display - facts a committed manifest *is* the source of: the image URL and the credit - that travels with it, and the depicted QIDs the lookup matches on. Correcting - a manifest would otherwise reach the database only by deleting rows. - - So `refresh: true` rewrites exactly those keys, and only on an identity this - manifest already owns — one whose metadata carries this manifest's `corpus` - version. It mints nothing and touches no other row. + Not marked `@deprecated`: the artworks corpus's existing tests call this + name on purpose, as the proof that the move changed nothing, and the gate + compiles with warnings as errors. Remove this file, and point those tests at + the new name, in the release after #174. """ - def run(manifest, opts \\ []) when is_map(manifest) do - rows = manifest["rows"] |> List.wrap() |> take(opts[:limit]) - kind = manifest["kind"] - version = manifest["manifest"] - - summary = - Enum.reduce(rows, empty_summary(kind, version, length(rows)), fn row, summary -> - case entry(kind, version, row) do - {:ok, entry} -> - if opts[:dry_run] do - tally(summary, :would_seed, row) - else - # Ownership is read before the resolve, not after: resolving fills - # this manifest's missing keys onto whatever identity it matched, - # so afterwards an absorbed record carries the `corpus` marker too - # and would look owned when it is not. - owned? = opts[:refresh] && corpus_owned?(entry, version) - resolution = SourceIdentity.resolve(entry) - refreshed? = owned? && refresh(resolution, entry) - - summary - |> tally(resolution.state, row) - |> Map.update!(:refreshed, &if(refreshed?, do: &1 + 1, else: &1)) - end - - {:error, reason} -> - summary - |> tally(:invalid, row) - |> Map.update!( - :invalid_reasons, - &Map.update(&1, to_string(reason), 1, fn n -> n + 1 end) - ) - end - end) - - {:ok, Map.delete(summary, :depicted_qid_set)} - end - - @doc "Loads a committed manifest from disk and seeds it." - def run_file(path, opts \\ []) do - path |> Manifest.load!() |> run(opts) - end - - defp take(rows, nil), do: rows - defp take(rows, limit) when is_integer(limit), do: Enum.take(rows, limit) - - defp empty_summary(kind, version, count) do - %{ - kind: kind, - manifest: version, - rows: count, - newly_created: 0, - matched: 0, - would_seed: 0, - insufficient_evidence: 0, - conflicting_identifiers: 0, - invalid: 0, - invalid_reasons: %{}, - refreshed: 0, - with_image: 0, - with_depicts: 0, - depicted_qids: 0, - depicted_qid_set: MapSet.new() - } - end - - defp tally(summary, state, row) do - depicts = depicts(row) - - summary - |> Map.update!(state, &(&1 + 1)) - |> Map.update!(:with_image, &if(present?(row["image_url"]), do: &1 + 1, else: &1)) - |> Map.update!(:with_depicts, &if(depicts != [], do: &1 + 1, else: &1)) - |> Map.update!(:depicted_qid_set, fn set -> - Enum.reduce(depicts, set, &MapSet.put(&2, &1["qid"])) - end) - |> then(&Map.put(&1, :depicted_qids, MapSet.size(&1.depicted_qid_set))) - end - - @owned_display_keys ~w(image_url image_attribution credit_line depicts depicts_qids sitelinks) - - # A row this manifest has not seeded yet is its own to write; a row it seeded - # before carries its version. Anything else belongs to whoever put it there. - defp corpus_owned?(entry, version) do - identifier = entry.stable_identifier - - case Registry.by_external_id(identifier.namespace, identifier.external_id) do - nil -> - true - - object_id -> - DevilsDictionary.Repo.get!(Registry.Entity, object_id).metadata["corpus"] == version - end - end - - defp refresh(%{object_id: object_id}, entry) when is_integer(object_id) do - entity = DevilsDictionary.Repo.get!(Registry.Entity, object_id) - owned = Map.take(entry.metadata, @owned_display_keys) - - if Map.take(entity.metadata, Map.keys(owned)) == owned do - false - else - entity - |> Registry.Entity.changeset(%{metadata: Map.merge(entity.metadata, owned)}) - |> DevilsDictionary.Repo.update!() - - true - end - end - - defp refresh(_resolution, _entry), do: false - - @doc "The provider-neutral identity proposal for one manifest row." - def entry("met-highlights", version, row) do - with {:ok, met_id} <- required(row["met_object_id"]), - {:ok, title} <- required(row["title"]) do - Entry.new(%{ - source_slug: "met", - object_kind: :entity, - entity_kind: :work, - work_kind: "artwork", - stable_identifier: %{namespace: "met_object_id", external_id: met_id}, - identifiers: - [%{namespace: "met_object_id", external_id: met_id}] ++ - qid_identifier(row["qid"]), - label: label(title), - year: begin_year(row["date"]), - metadata: - base_metadata(row, version, "met") - |> put_present("artist_display_name", row["artist"]) - |> put_present("object_date", row["date"]) - |> put_present("medium", row["medium"]) - |> put_present("department", row["department"]) - |> Map.put("collection", @met_collection), - eligibility: :eligible, - retention: :durable - }) - end - end - - def entry("wikidata-famous", version, row) do - with {:ok, qid} <- required(row["qid"]), - {:ok, title} <- required(row["title"]) do - Entry.new(%{ - source_slug: "wikidata", - object_kind: :entity, - entity_kind: :work, - work_kind: "artwork", - stable_identifier: %{namespace: "wikidata", external_id: qid}, - identifiers: [%{namespace: "wikidata", external_id: qid}], - label: label(title), - year: begin_year(row["date"]), - metadata: - base_metadata(row, version, "wikidata") - |> put_present("artist_display_name", creator_names(row)) - |> put_present("object_date", row["date"]) - |> put_present("commons_file", row["commons_file"]) - |> put_present("creator_qids", creator_qids(row)) - |> put_sitelinks(row["sitelinks"]), - eligibility: :eligible, - retention: :durable - }) - end - end - - @doc false - def entry("poetrydb", version, row) do - with {:ok, poem_id} <- required(row["poem_id"]), - {:ok, title} <- required(row["title"]), - {:ok, author} <- required(row["author"]) do - Entry.new(%{ - source_slug: "poetrydb", - object_kind: :entity, - entity_kind: :work, - work_kind: "poem", - stable_identifier: %{namespace: "poetrydb_poem", external_id: poem_id}, - # The poet's QID is **not** among these. A poem is not the person who - # wrote it, and registering `wikidata: Q82083` on *Ode to a Nightingale* - # would claim it is Keats — and collide with the entity that actually is. - # The crosswalk is recorded as what it is: a fact about the author. - identifiers: [%{namespace: "poetrydb_poem", external_id: poem_id}], - label: label(title), - metadata: - %{ - "content_type" => "text", - "catalog_source" => "poetrydb", - "corpus" => version, - "author_display_name" => author - } - |> put_present("author_qid", row["author_qid"]) - |> put_present("lines_sha256", row["lines_sha256"]) - |> put_present("source_url", row["source_url"]) - |> put_line_count(row["line_count"]), - eligibility: :eligible, - retention: :durable - }) - end - end - - @doc false - def entry("open-library", version, row) do - with {:ok, olid} <- required(row["olid"]), - {:ok, title} <- required(row["title"]) do - Entry.new(%{ - source_slug: "open-library", - object_kind: :entity, - entity_kind: :work, - work_kind: "book", - stable_identifier: %{namespace: "olid", external_id: olid}, - # The work's own Wikidata QID belongs here — unlike PoetryDB's - # `author_qid`, this one is about *this work* and not about the person - # who wrote it, so registering it collides with nothing. The authors' - # QIDs stay in metadata, where they are facts about the authors. - identifiers: [%{namespace: "olid", external_id: olid}] ++ wikidata_identifier(row["qid"]), - label: label(title), - year: row["year"], - metadata: - %{ - "content_type" => "text", - "catalog_source" => "open-library", - "corpus" => version - } - |> put_present("author_display_name", author_names(row)) - |> put_present("author_qids", author_qids(row)) - |> put_present("public_domain_basis", row["pd_basis"]) - |> put_present("source_url", row["source_url"]) - |> put_present("wikidata_url", row["wikidata_url"]) - |> put_sitelinks(row["sitelinks"]), - eligibility: :eligible, - retention: :durable - }) - end - end - - def entry(kind, _version, _row), do: {:error, "unsupported_manifest_kind_#{kind}"} - - defp wikidata_identifier(qid) do - if is_binary(qid) and valid_qid?(qid), - do: [%{namespace: "wikidata", external_id: qid, metadata: %{"field" => "P648_subject"}}], - else: [] - end - - defp author_names(%{"authors" => authors}) when is_list(authors) do - authors |> Enum.map(& &1["name"]) |> Enum.filter(&present?/1) |> Enum.join(", ") |> presence() - end - - defp author_names(_row), do: nil - - defp author_qids(%{"authors" => authors}) when is_list(authors) do - case authors |> Enum.map(& &1["qid"]) |> Enum.filter(&(is_binary(&1) and valid_qid?(&1))) do - [] -> nil - qids -> qids - end - end - - defp author_qids(_row), do: nil - - # `entities.preferred_label` is varchar(255) and Postgres counts characters, - # not bytes, so this is the column's own width. Met highlight titles reach it: - # "Rebel Cassion Destroyed by Federal Shells. At Fredericksburgh, May 3, 1863. - # Eight Horses Killed." is one of the shorter long ones. - defp label(title), do: String.slice(title, 0, 255) - - defp base_metadata(row, version, catalog_source) do - depicts = depicts(row) - - %{ - "content_type" => "artwork", - "catalog_source" => catalog_source, - "corpus" => version, - "depicts" => depicts, - "depicts_qids" => Enum.map(depicts, & &1["qid"]) - } - |> put_present("image_url", row["image_url"]) - |> put_present("image_attribution", row["credit_line"]) - |> put_present("credit_line", row["credit_line"]) - |> put_present("source_url", row["source_url"]) - end - - # A Met tag and a Wikidata `P180` statement are the same assertion in two - # vocabularies — *this picture shows that concept* — so they normalize to one - # shape and one lookup key. - defp depicts(row) do - (row["tags"] || row["depicts"] || []) - |> Enum.flat_map(fn - %{"qid" => qid} = entry when is_binary(qid) -> - if valid_qid?(qid), do: [%{"qid" => qid, "term" => presence(entry["term"])}], else: [] - - _ -> - [] - end) - |> Enum.uniq_by(& &1["qid"]) - end - - defp qid_identifier(qid) do - if is_binary(qid) and valid_qid?(qid), - do: [ - %{namespace: "wikidata", external_id: qid, metadata: %{"field" => "objectWikidata_URL"}} - ], - else: [] - end - - defp creator_names(row) do - (row["creators"] || []) - |> Enum.map(& &1["term"]) - |> Enum.filter(&present?/1) - |> Enum.join(", ") - |> presence() - end - - defp creator_qids(row) do - (row["creators"] || []) - |> Enum.map(& &1["qid"]) - |> Enum.filter(&(is_binary(&1) and valid_qid?(&1))) - |> case do - [] -> nil - qids -> qids - end - end - - defp put_sitelinks(metadata, count) when is_integer(count), - do: Map.put(metadata, "sitelinks", count) - - defp put_sitelinks(metadata, _count), do: metadata - - # `objectDate` and an inception year are free text — "ca. 1780", "1863–65". - # The first four-digit run is the only part of them that is a year. - defp begin_year(value) when is_binary(value) do - case Regex.run(~r/\d{4}/, value) do - [year] -> String.to_integer(year) - _ -> nil - end - end - - defp begin_year(_value), do: nil - - defp required(value) do - if present?(value), do: {:ok, String.trim(value)}, else: {:error, :missing_required_field} - end - - defp put_line_count(map, count) when is_integer(count), do: Map.put(map, "line_count", count) - defp put_line_count(map, _count), do: map - - defp put_present(map, _key, nil), do: map - defp put_present(map, _key, ""), do: map - defp put_present(map, key, value), do: Map.put(map, key, value) - - defp present?(value), do: is_binary(value) and String.trim(value) != "" - defp valid_qid?(value), do: Regex.match?(~r/\AQ[1-9]\d*\z/, value) - defp presence(value) when is_binary(value) do - case String.trim(value) do - "" -> nil - trimmed -> trimmed - end - end + defdelegate run(manifest, opts \\ []), to: DevilsDictionary.Corpus.Seeder - defp presence(_value), do: nil + defdelegate run_file(path, opts \\ []), to: DevilsDictionary.Corpus.Seeder - @doc "How many artworks the catalog currently holds, by catalog source." - def catalog_counts do - import Ecto.Query + defdelegate entry(kind, version, row), to: DevilsDictionary.Corpus.Seeder - DevilsDictionary.Repo.all( - from entity in Registry.Entity, - join: details in Registry.WorkDetails, - on: details.entity_id == entity.object_id and details.work_kind == "artwork", - group_by: fragment("COALESCE(?->>'catalog_source', 'unlabelled')", entity.metadata), - select: - {fragment("COALESCE(?->>'catalog_source', 'unlabelled')", entity.metadata), - count(entity.object_id)} - ) - |> Map.new() - end + defdelegate catalog_counts(), to: DevilsDictionary.Corpus.Seeder end diff --git a/lib/devils_dictionary/corpus/seeder.ex b/lib/devils_dictionary/corpus/seeder.ex new file mode 100644 index 00000000..48415126 --- /dev/null +++ b/lib/devils_dictionary/corpus/seeder.ex @@ -0,0 +1,516 @@ +defmodule DevilsDictionary.Corpus.Seeder do + @moduledoc """ + Seeds a committed corpus manifest into the existing encyclopedia structures. + + Two kinds of content (#174): **works** — artworks, poems, books, the + corpora under `priv/artworks/manifests/` — and **quotations**, the + public-domain Wikiquote corpus under `priv/quotes/manifests/`. It was + `DevilsDictionary.Artworks.Corpus.Seeder` until the second kind arrived; + that name delegates here for one release. + + ## Works + + Zero new tables. An artwork is what #86 made it: an `entities` row with + `work_details.work_kind == "artwork"` and its identity in + `external_identifiers` — namespace `met_object_id` for a Met object, + `wikidata` for a QID (both, when the Met publishes the object's QID). That is + the same shape the Artsy pilot and the Phase 2a discovery provider resolve to, + which is why the three read as one catalog. + + ## Why the depicted QIDs are metadata + + Every Met row carries its tag QIDs and every Wikidata row its `P180` + depictions, so a word page can ask *which artworks depict Q198* without a live + search. Those QIDs are not identity — an artwork is not the thing it depicts — + and the depicted concept usually has no local entity to point a claim at, so + they are stored on the artwork's `metadata` as `depicts` (term + QID) and + `depicts_qids` (the flat list the lookup filters on). A claim would need an + object; this needs only a match key. + + ## Idempotence + + The manifest is the whole input and `SourceIdentity.resolve/1` is the whole + write: a second run of the same manifest matches every row on its exact + identifier and updates nothing that is already filled, so the counts come back + identical. Nothing here consults a title. + + ## Quotations + + A `wikiquote-pd` row is a `content/quotation` entry in the shape the live + Wikiquote provider writes, identified by its fingerprint and credited + `authored_by` by QID, so a corpus line and a live line with one wording are + one subject. Its source record, its evidence and its badge are + `Quotations.Corpus.seed_row/3`'s. The authors are prepared from the + manifest's own Wikidata facts, so seeding still makes no request. + """ + + alias DevilsDictionary.Artworks.Corpus.Manifest + alias DevilsDictionary.Quotations + alias DevilsDictionary.Quotations.Fingerprint + alias DevilsDictionary.Registry + alias DevilsDictionary.SourceIdentity + alias DevilsDictionary.SourceIdentity.Entry + + @met_collection "The Metropolitan Museum of Art" + + @doc """ + Seeds one loaded manifest and returns a summary. + + Options: `:dry_run` (resolve nothing, just count what would be seeded), + `:limit` (seed only the first N rows) and `:refresh`. + + ## `:refresh` + + `SourceIdentity.resolve/1` never replaces a field an identity already holds — + an absorbed record must not be overwritten by a manifest-shaped summary. That + is right for identity and description, and wrong for the handful of display + facts a committed manifest *is* the source of: the image URL and the credit + that travels with it, and the depicted QIDs the lookup matches on. Correcting + a manifest would otherwise reach the database only by deleting rows. + + So `refresh: true` rewrites exactly those keys, and only on an identity this + manifest already owns — one whose metadata carries this manifest's `corpus` + version. It mints nothing and touches no other row. + """ + def run(manifest, opts \\ []) when is_map(manifest) do + rows = manifest["rows"] |> List.wrap() |> take(opts[:limit]) + kind = manifest["kind"] + version = manifest["manifest"] + context = if opts[:dry_run], do: nil, else: context(kind, manifest) + + summary = + Enum.reduce(rows, empty_summary(kind, version, length(rows)), fn row, summary -> + case entry(kind, version, row) do + {:ok, entry} -> + if opts[:dry_run] do + tally(summary, :would_seed, row) + else + # Ownership is read before the resolve, not after: resolving fills + # this manifest's missing keys onto whatever identity it matched, + # so afterwards an absorbed record carries the `corpus` marker too + # and would look owned when it is not. + # Only a work's display facts are the manifest's to rewrite; a + # quotation's text is a revision and is never refreshed. + owned? = + opts[:refresh] && entry.object_kind == :entity && corpus_owned?(entry, version) + + resolution = resolve(kind, entry, row, context) + refreshed? = owned? && refresh(resolution, entry) + + summary + |> tally(resolution.state, row) + |> Map.update!(:refreshed, &if(refreshed?, do: &1 + 1, else: &1)) + end + + {:error, reason} -> + summary + |> tally(:invalid, row) + |> Map.update!( + :invalid_reasons, + &Map.update(&1, to_string(reason), 1, fn n -> n + 1 end) + ) + end + end) + + {:ok, Map.delete(summary, :depicted_qid_set)} + end + + # What a kind's rows share. Works share nothing: the entry is the whole write. + defp context("wikiquote-pd", manifest), do: Quotations.Corpus.seed_context(manifest) + defp context(_kind, _manifest), do: nil + + defp resolve("wikiquote-pd", entry, row, context), + do: Quotations.Corpus.seed_row(entry, row, context) + + defp resolve(_kind, entry, _row, _context), do: SourceIdentity.resolve(entry) + + @doc "Loads a committed manifest from disk and seeds it." + def run_file(path, opts \\ []) do + path |> Manifest.load!() |> run(opts) + end + + defp take(rows, nil), do: rows + defp take(rows, limit) when is_integer(limit), do: Enum.take(rows, limit) + + defp empty_summary(kind, version, count) do + %{ + kind: kind, + manifest: version, + rows: count, + newly_created: 0, + matched: 0, + would_seed: 0, + insufficient_evidence: 0, + conflicting_identifiers: 0, + invalid: 0, + invalid_reasons: %{}, + refreshed: 0, + with_image: 0, + with_depicts: 0, + depicted_qids: 0, + depicted_qid_set: MapSet.new() + } + end + + defp tally(summary, state, row) do + depicts = depicts(row) + + summary + |> Map.update!(state, &(&1 + 1)) + |> Map.update!(:with_image, &if(present?(row["image_url"]), do: &1 + 1, else: &1)) + |> Map.update!(:with_depicts, &if(depicts != [], do: &1 + 1, else: &1)) + |> Map.update!(:depicted_qid_set, fn set -> + Enum.reduce(depicts, set, &MapSet.put(&2, &1["qid"])) + end) + |> then(&Map.put(&1, :depicted_qids, MapSet.size(&1.depicted_qid_set))) + end + + @owned_display_keys ~w(image_url image_attribution credit_line depicts depicts_qids sitelinks) + + # A row this manifest has not seeded yet is its own to write; a row it seeded + # before carries its version. Anything else belongs to whoever put it there. + defp corpus_owned?(entry, version) do + identifier = entry.stable_identifier + + case Registry.by_external_id(identifier.namespace, identifier.external_id) do + nil -> + true + + object_id -> + DevilsDictionary.Repo.get!(Registry.Entity, object_id).metadata["corpus"] == version + end + end + + defp refresh(%{object_id: object_id}, entry) when is_integer(object_id) do + entity = DevilsDictionary.Repo.get!(Registry.Entity, object_id) + owned = Map.take(entry.metadata, @owned_display_keys) + + if Map.take(entity.metadata, Map.keys(owned)) == owned do + false + else + entity + |> Registry.Entity.changeset(%{metadata: Map.merge(entity.metadata, owned)}) + |> DevilsDictionary.Repo.update!() + + true + end + end + + defp refresh(_resolution, _entry), do: false + + @doc "The provider-neutral identity proposal for one manifest row." + def entry("met-highlights", version, row) do + with {:ok, met_id} <- required(row["met_object_id"]), + {:ok, title} <- required(row["title"]) do + Entry.new(%{ + source_slug: "met", + object_kind: :entity, + entity_kind: :work, + work_kind: "artwork", + stable_identifier: %{namespace: "met_object_id", external_id: met_id}, + identifiers: + [%{namespace: "met_object_id", external_id: met_id}] ++ + qid_identifier(row["qid"]), + label: label(title), + year: begin_year(row["date"]), + metadata: + base_metadata(row, version, "met") + |> put_present("artist_display_name", row["artist"]) + |> put_present("object_date", row["date"]) + |> put_present("medium", row["medium"]) + |> put_present("department", row["department"]) + |> Map.put("collection", @met_collection), + eligibility: :eligible, + retention: :durable + }) + end + end + + def entry("wikidata-famous", version, row) do + with {:ok, qid} <- required(row["qid"]), + {:ok, title} <- required(row["title"]) do + Entry.new(%{ + source_slug: "wikidata", + object_kind: :entity, + entity_kind: :work, + work_kind: "artwork", + stable_identifier: %{namespace: "wikidata", external_id: qid}, + identifiers: [%{namespace: "wikidata", external_id: qid}], + label: label(title), + year: begin_year(row["date"]), + metadata: + base_metadata(row, version, "wikidata") + |> put_present("artist_display_name", creator_names(row)) + |> put_present("object_date", row["date"]) + |> put_present("commons_file", row["commons_file"]) + |> put_present("creator_qids", creator_qids(row)) + |> put_sitelinks(row["sitelinks"]), + eligibility: :eligible, + retention: :durable + }) + end + end + + @doc false + def entry("poetrydb", version, row) do + with {:ok, poem_id} <- required(row["poem_id"]), + {:ok, title} <- required(row["title"]), + {:ok, author} <- required(row["author"]) do + Entry.new(%{ + source_slug: "poetrydb", + object_kind: :entity, + entity_kind: :work, + work_kind: "poem", + stable_identifier: %{namespace: "poetrydb_poem", external_id: poem_id}, + # The poet's QID is **not** among these. A poem is not the person who + # wrote it, and registering `wikidata: Q82083` on *Ode to a Nightingale* + # would claim it is Keats — and collide with the entity that actually is. + # The crosswalk is recorded as what it is: a fact about the author. + identifiers: [%{namespace: "poetrydb_poem", external_id: poem_id}], + label: label(title), + metadata: + %{ + "content_type" => "text", + "catalog_source" => "poetrydb", + "corpus" => version, + "author_display_name" => author + } + |> put_present("author_qid", row["author_qid"]) + |> put_present("lines_sha256", row["lines_sha256"]) + |> put_present("source_url", row["source_url"]) + |> put_line_count(row["line_count"]), + eligibility: :eligible, + retention: :durable + }) + end + end + + @doc false + def entry("open-library", version, row) do + with {:ok, olid} <- required(row["olid"]), + {:ok, title} <- required(row["title"]) do + Entry.new(%{ + source_slug: "open-library", + object_kind: :entity, + entity_kind: :work, + work_kind: "book", + stable_identifier: %{namespace: "olid", external_id: olid}, + # The work's own Wikidata QID belongs here — unlike PoetryDB's + # `author_qid`, this one is about *this work* and not about the person + # who wrote it, so registering it collides with nothing. The authors' + # QIDs stay in metadata, where they are facts about the authors. + identifiers: [%{namespace: "olid", external_id: olid}] ++ wikidata_identifier(row["qid"]), + label: label(title), + year: row["year"], + metadata: + %{ + "content_type" => "text", + "catalog_source" => "open-library", + "corpus" => version + } + |> put_present("author_display_name", author_names(row)) + |> put_present("author_qids", author_qids(row)) + |> put_present("public_domain_basis", row["pd_basis"]) + |> put_present("source_url", row["source_url"]) + |> put_present("wikidata_url", row["wikidata_url"]) + |> put_sitelinks(row["sitelinks"]), + eligibility: :eligible, + retention: :durable + }) + end + end + + # A quotation (#174): the live provider's entry shape (`Wikiquote.identity_record/1`) + # with the fingerprint as the stable identifier — see `Quotations.Corpus` + # for why not the positional `wikiquote_item` — and the credit at + # `:verified`, because the build kept the line only when a text its author + # wrote contains it. + @doc false + def entry("wikiquote-pd", version, row) do + with {:ok, text} <- required(row["text"]), + {:ok, fingerprint} <- required(row["fingerprint"]), + {:ok, author} <- required(row["author_qid"]), + true <- Fingerprint.fingerprint(text) == fingerprint || {:error, :fingerprint_mismatch}, + true <- valid_qid?(author) || {:error, :invalid_author_qid} do + Entry.new(%{ + source_slug: Quotations.Corpus.slug(), + object_kind: :content, + content_kind: :quotation, + stable_identifier: %{namespace: "quotation_fingerprint", external_id: fingerprint}, + identifiers: [Fingerprint.identifier(text)], + label: label(text), + year: row["work_year"], + content: %{ + body: text, + canonical_url: row["source_url"], + year: row["work_year"], + rights_metadata: %{ + "license" => "CC BY-SA 4.0", + "license_url" => "https://creativecommons.org/licenses/by-sa/4.0/", + "attribution" => "Wikiquote contributors", + "revision_id" => row["revision_id"], + "page" => row["page"], + "manifest" => version + } + }, + metadata: %{ + "author_display_name" => row["author_label"], + "locator" => row["locator"], + "corpus" => version + }, + relationships: [ + %{ + role: "authored_by", + certainty: :verified, + target_identifiers: [ + %{namespace: "wikidata", external_id: author, metadata: %{"field" => "P50"}} + ] + } + ], + eligibility: :eligible, + retention: :durable + }) + end + end + + def entry(kind, _version, _row), do: {:error, "unsupported_manifest_kind_#{kind}"} + + defp wikidata_identifier(qid) do + if is_binary(qid) and valid_qid?(qid), + do: [%{namespace: "wikidata", external_id: qid, metadata: %{"field" => "P648_subject"}}], + else: [] + end + + defp author_names(%{"authors" => authors}) when is_list(authors) do + authors |> Enum.map(& &1["name"]) |> Enum.filter(&present?/1) |> Enum.join(", ") |> presence() + end + + defp author_names(_row), do: nil + + defp author_qids(%{"authors" => authors}) when is_list(authors) do + case authors |> Enum.map(& &1["qid"]) |> Enum.filter(&(is_binary(&1) and valid_qid?(&1))) do + [] -> nil + qids -> qids + end + end + + defp author_qids(_row), do: nil + + # `entities.preferred_label` is varchar(255) and Postgres counts characters, + # not bytes, so this is the column's own width. Met highlight titles reach it: + # "Rebel Cassion Destroyed by Federal Shells. At Fredericksburgh, May 3, 1863. + # Eight Horses Killed." is one of the shorter long ones. + defp label(title), do: String.slice(title, 0, 255) + + defp base_metadata(row, version, catalog_source) do + depicts = depicts(row) + + %{ + "content_type" => "artwork", + "catalog_source" => catalog_source, + "corpus" => version, + "depicts" => depicts, + "depicts_qids" => Enum.map(depicts, & &1["qid"]) + } + |> put_present("image_url", row["image_url"]) + |> put_present("image_attribution", row["credit_line"]) + |> put_present("credit_line", row["credit_line"]) + |> put_present("source_url", row["source_url"]) + end + + # A Met tag and a Wikidata `P180` statement are the same assertion in two + # vocabularies — *this picture shows that concept* — so they normalize to one + # shape and one lookup key. + defp depicts(row) do + (row["tags"] || row["depicts"] || []) + |> Enum.flat_map(fn + %{"qid" => qid} = entry when is_binary(qid) -> + if valid_qid?(qid), do: [%{"qid" => qid, "term" => presence(entry["term"])}], else: [] + + _ -> + [] + end) + |> Enum.uniq_by(& &1["qid"]) + end + + defp qid_identifier(qid) do + if is_binary(qid) and valid_qid?(qid), + do: [ + %{namespace: "wikidata", external_id: qid, metadata: %{"field" => "objectWikidata_URL"}} + ], + else: [] + end + + defp creator_names(row) do + (row["creators"] || []) + |> Enum.map(& &1["term"]) + |> Enum.filter(&present?/1) + |> Enum.join(", ") + |> presence() + end + + defp creator_qids(row) do + (row["creators"] || []) + |> Enum.map(& &1["qid"]) + |> Enum.filter(&(is_binary(&1) and valid_qid?(&1))) + |> case do + [] -> nil + qids -> qids + end + end + + defp put_sitelinks(metadata, count) when is_integer(count), + do: Map.put(metadata, "sitelinks", count) + + defp put_sitelinks(metadata, _count), do: metadata + + # `objectDate` and an inception year are free text — "ca. 1780", "1863–65". + # The first four-digit run is the only part of them that is a year. + defp begin_year(value) when is_binary(value) do + case Regex.run(~r/\d{4}/, value) do + [year] -> String.to_integer(year) + _ -> nil + end + end + + defp begin_year(_value), do: nil + + defp required(value) do + if present?(value), do: {:ok, String.trim(value)}, else: {:error, :missing_required_field} + end + + defp put_line_count(map, count) when is_integer(count), do: Map.put(map, "line_count", count) + defp put_line_count(map, _count), do: map + + defp put_present(map, _key, nil), do: map + defp put_present(map, _key, ""), do: map + defp put_present(map, key, value), do: Map.put(map, key, value) + + defp present?(value), do: is_binary(value) and String.trim(value) != "" + defp valid_qid?(value), do: Regex.match?(~r/\AQ[1-9]\d*\z/, value) + + defp presence(value) when is_binary(value) do + case String.trim(value) do + "" -> nil + trimmed -> trimmed + end + end + + defp presence(_value), do: nil + + @doc "How many artworks the catalog currently holds, by catalog source." + def catalog_counts do + import Ecto.Query + + DevilsDictionary.Repo.all( + from entity in Registry.Entity, + join: details in Registry.WorkDetails, + on: details.entity_id == entity.object_id and details.work_kind == "artwork", + group_by: fragment("COALESCE(?->>'catalog_source', 'unlabelled')", entity.metadata), + select: + {fragment("COALESCE(?->>'catalog_source', 'unlabelled')", entity.metadata), + count(entity.object_id)} + ) + |> Map.new() + end +end diff --git a/lib/devils_dictionary/quotations/corpus.ex b/lib/devils_dictionary/quotations/corpus.ex new file mode 100644 index 00000000..185ee50a --- /dev/null +++ b/lib/devils_dictionary/quotations/corpus.ex @@ -0,0 +1,271 @@ +defmodule DevilsDictionary.Quotations.Corpus do + @moduledoc """ + `wikiquote-pd-v1`, the public-domain Wikiquote corpus (#174, build 6 of #158), + once it is built: its source row, what seeding one line writes beyond the + identity, and (on the Quotes shelf) how a page finds its lines. + + The manifest is built by `Quotations.Corpus.Build` and committed at + `priv/quotes/manifests/wikiquote-pd-v1.json`. `Corpus.Seeder` seeds it the + way it seeds the artworks corpora: each row is a `SourceIdentity.Entry` in + the shape the live Wikiquote provider writes, `content/quotation` credited + `authored_by` by QID. The one difference is the stable identifier. A live + line's is `wikiquote_item`, a hash of the page, section and position, and + that is a position **on today's page**. The corpus pins a revision, so a + position it proposed could name a different line on the page a visit reads. + The corpus's stable identifier is therefore the line's + `quotation_fingerprint` (ADR 0003), the identifier a second source's copy + folds on, and a corpus line and a live line with one wording are one + subject whichever arrived first. + + ## What seeding a line writes (`seed_row/3`) + + * **a source record** under `wikiquote-pd-v1`, keyed by the fingerprint, + whose payload is the manifest row. It is the entry's provenance, and + what the evidence cites. + * **the identity**, through `SourceIdentity.resolve/2`, with the author + prepared from the manifest's own Wikidata facts (`Creators.classify/2` + over `selection.authors`). No request is made at seed time. + * **evidence** on the corpus's `authored_by`: one `:supports` row per + check the build recorded (the Gutenberg text at its line, and the + author's own Wikiquote page where it lists the line), each citing the + record above. Added once. + * **the badge**, on `content_items.metadata["provenance"]`, only when the + item has none. An item another source already holds may have been + through the verifier, whose pass is the item's and now counts this + credit too. Its clock recomputes the badge; the corpus does not + overwrite it. + + Nothing is refreshed and nothing is swept. The source record is referenced + by evidence and by the item's revision, and cleanup never takes a record + something durable stands on (`Discovery.purge_results/1`). + """ + + import Ecto.Query + + alias DevilsDictionary.Claims + alias DevilsDictionary.Claims.{Assertion, AssertionEvidence, AssertionRevision, Predicate} + alias DevilsDictionary.Corpus.SourceRecordRevision + alias DevilsDictionary.Quotations.Badge + alias DevilsDictionary.Registry.ContentItem + alias DevilsDictionary.Repo + alias DevilsDictionary.SourceIdentity + alias DevilsDictionary.SourceIdentity.Creators + alias DevilsDictionary.Sources + alias DevilsDictionary.Sources.SourceRecord + + @slug "wikiquote-pd-v1" + @manifest_path "priv/quotes/manifests/wikiquote-pd-v1.json" + @dump_url "https://dumps.wikimedia.org/enwikiquote/20260901/" + @license "CC BY-SA 4.0" + @license_url "https://creativecommons.org/licenses/by-sa/4.0/" + + @external_resource @manifest_path + + # The committed manifest's own checksums, read at compile time: the source + # row names the file it is, and recompiles when the file changes. + @checksums (case File.read(@manifest_path) do + {:ok, body} -> + manifest = Jason.decode!(body) + + %{ + "manifest_checksum" => manifest["checksum"], + "set_checksum" => manifest["set_checksum"], + "generated_at" => manifest["generated_at"], + "row_count" => manifest["row_count"] + } + + {:error, _} -> + %{} + end) + + @doc "The corpus's source slug." + def slug, do: @slug + + @doc "Where the committed manifest lives." + def manifest_path, do: @manifest_path + + @doc """ + The source row (#174 design 2): 👑 tier, `access: :dump`, CC BY-SA 4.0, and + the manifest checksum in its config. Appended to `Sources.Catalog.sources/0`. + """ + def source_attrs do + %{ + slug: @slug, + name: "Wikiquote, public domain", + tier: :aristocracy, + kind: :corpus, + access: :dump, + license: @license, + license_url: @license_url, + homepage: "https://en.wikiquote.org/", + logo: "/images/sources/wikiquote.png", + url_template: "https://en.wikiquote.org/wiki/{title}", + attribution: "Wikiquote contributors (CC BY-SA 4.0)", + active: true, + config: + Map.merge( + %{ + "mode" => "committed corpus; no request is made at seed or read time", + "manifest" => @manifest_path, + "dump_url" => @dump_url, + "built_by" => "mix dd.quotes.corpus.build", + "kept" => "Verified by build 5's checks against a pre-1931 Gutenberg text", + "retention" => "durable; never refreshed, never swept" + }, + @checksums + ) + } + end + + # ── seeding ─────────────────────────────────────────────────────────────── + + @doc """ + What every row of one manifest shares: the source row, and each author's + prepared outcome for `Creators`, read from the manifest's own facts. + """ + def seed_context(manifest) do + authors = get_in(manifest, ["selection", "authors"]) || %{} + + %{ + source: Sources.get_source_by_slug!(@slug), + prepared: + Map.new(authors, fn {qid, entity} -> {qid, Creators.classify(qid, %{qid => entity})} end), + generated_at: manifest["generated_at"], + manifest: manifest["manifest"] + } + end + + @doc """ + Seeds one row whose entry `Corpus.Seeder.entry/3` built. Returns the + resolution. + """ + def seed_row(entry, row, context) do + {:ok, resolution} = + Repo.transaction(fn -> + {:ok, record} = + Sources.upsert_record(context.source, %{ + external_id: row["fingerprint"], + url: row["source_url"], + raw: row + }) + + resolution = + entry + |> Map.merge(%{ + source_id: context.source.id, + source_record_id: record.id, + source_record_revision_id: record.current_revision.id + }) + |> SourceIdentity.resolve(prepared: context.prepared) + + if resolution.state in [:matched, :newly_created] and is_integer(resolution.object_id) do + evidence!(resolution.object_id, row, record.current_revision.id, context.source.id) + provenance!(resolution.object_id, row, context) + end + + resolution + end) + + resolution + end + + defp evidence!(content_id, row, record_revision_id, source_id) do + revision = + Repo.one( + from r in AssertionRevision, + join: a in Assertion, + on: a.id == r.assertion_id and a.source_id == ^source_id, + join: p in Predicate, + on: p.id == r.predicate_id and p.key == "authored_by", + where: r.subject_object_id == ^content_id and r.is_current, + order_by: [desc: r.id], + limit: 1 + ) + + if revision do + for check <- row["checks"] || [], check["role"] == "supports" do + locator = String.slice(to_string(check["locator"]), 0, 255) + + held? = + Repo.exists?( + from e in AssertionEvidence, + where: + e.assertion_revision_id == ^revision.id and + e.source_record_revision_id == ^record_revision_id and e.locator == ^locator + ) + + unless held? do + {:ok, _} = + Claims.add_evidence(revision.id, %{ + evidence_role: :supports, + source_record_revision_id: record_revision_id, + locator: locator, + attribution_text: attribution(check, row) + }) + end + end + end + end + + defp attribution(%{"kind" => "primary"}, row), + do: + "Project Gutenberg ##{row["gutenberg"]["ebook"]}, #{row["work_label"]} (#{row["work_year"]})" + + defp attribution(%{"source" => "wikiquote"}, row), + do: "Wikiquote, #{row["author_label"]}'s own page" + + defp attribution(check, _row), do: check["source"] + + defp provenance!(content_id, row, context) do + item = Repo.get!(ContentItem, content_id) + + if is_nil((item.metadata || %{})["provenance"]) do + provenance = %{ + "badge" => row["badge"], + "score" => row["score"], + "agreements" => row["agreements"], + "sources" => row["sources"], + "computed_at" => context.generated_at, + "verifier_version" => Badge.version(), + "computed_by" => context.manifest + } + + item + |> ContentItem.changeset(%{ + metadata: Map.put(item.metadata || %{}, "provenance", provenance) + }) + |> Repo.update!() + end + end + + # ── reading ─────────────────────────────────────────────────────────────── + + @doc """ + The seeded rows whose concepts are among `qids`, as the manifest wrote them, + with each row's `object_id` (the content item its fingerprint resolves to). + Read from the corpus's own source records, so a page asks the registry for + what was seeded and never reads the file. + """ + def rows_for_concepts([]), do: [] + + def rows_for_concepts(qids) when is_list(qids) do + Repo.all( + from r in SourceRecord, + join: s in assoc(r, :source), + join: rev in SourceRecordRevision, + on: rev.source_record_id == r.id and rev.revision_key == r.content_hash, + where: + s.slug == @slug and s.active and + fragment("? \\?| ?", rev.payload["concept_qids"], type(^qids, {:array, :string})), + order_by: [asc: r.external_id], + select: {rev.payload, r.id} + ) + |> Enum.map(fn {row, _record_id} -> + Map.put( + row, + "object_id", + DevilsDictionary.Registry.by_external_id("quotation_fingerprint", row["fingerprint"]) + ) + end) + end +end diff --git a/lib/devils_dictionary/sources/catalog.ex b/lib/devils_dictionary/sources/catalog.ex index 95e56839..8acc8786 100644 --- a/lib/devils_dictionary/sources/catalog.ex +++ b/lib/devils_dictionary/sources/catalog.ex @@ -202,7 +202,8 @@ defmodule DevilsDictionary.Sources.Catalog do ] ++ [artsy()] ++ DevilsDictionary.Discovery.Providers.source_catalog() ++ - DevilsDictionary.Quotations.Checkers.source_catalog() + DevilsDictionary.Quotations.Checkers.source_catalog() ++ + [DevilsDictionary.Quotations.Corpus.source_attrs()] end # Artsy is a **corpus**, not a provider (#144 Phase 3, and §4 of #144). diff --git a/lib/mix/tasks/dd.artworks.seed.ex b/lib/mix/tasks/dd.artworks.seed.ex index 4d1b827e..02bf66bb 100644 --- a/lib/mix/tasks/dd.artworks.seed.ex +++ b/lib/mix/tasks/dd.artworks.seed.ex @@ -3,11 +3,13 @@ defmodule Mix.Tasks.Dd.Artworks.Seed do @moduledoc """ Seeds the committed **corpus** manifests built by `mix dd.artworks.manifest` - — `met-highlights`, `wikidata-famous`, `poetrydb` — which are recognised by - their contents rather than by a flag: + — `met-highlights`, `wikidata-famous`, `poetrydb`, `open-library` — and the + quotations one built by `mix dd.quotes.corpus.build` (`wikiquote-pd`, #174), + which are recognised by their contents rather than by a flag: mix dd.artworks.seed --manifest priv/artworks/manifests/met-highlights-v1.json mix dd.artworks.seed --manifest priv/artworks/manifests/wikidata-famous-v1.json --dry-run + mix dd.artworks.seed --manifest priv/quotes/manifests/wikiquote-pd-v1.json A corpus manifest is the whole input: nothing is searched, fetched or resumed, and rerunning it matches every row on its exact identifier, so the counts come @@ -36,6 +38,7 @@ defmodule Mix.Tasks.Dd.Artworks.Seed do alias DevilsDictionary.Artworks alias DevilsDictionary.Artworks.Corpus alias DevilsDictionary.Artworks.{Manifest, WikidataCandidates} + alias DevilsDictionary.Corpus.Seeder @switches [ manifest: :string, @@ -83,7 +86,7 @@ defmodule Mix.Tasks.Dd.Artworks.Seed do manifest = Corpus.Manifest.load!(path) {:ok, summary} = - Corpus.Seeder.run(manifest, + Seeder.run(manifest, dry_run: opts[:dry_run] || false, limit: opts[:record_limit], refresh: opts[:refresh] || false @@ -96,7 +99,7 @@ defmodule Mix.Tasks.Dd.Artworks.Seed do Mix.shell().info("Corpus seed summary: " <> inspect(summary, pretty: true)) Mix.shell().info( - "Artwork catalog by source: " <> inspect(Corpus.Seeder.catalog_counts(), pretty: true) + "Artwork catalog by source: " <> inspect(Seeder.catalog_counts(), pretty: true) ) rescue error in [ArgumentError, File.Error, Jason.DecodeError] -> Mix.raise(Exception.message(error)) diff --git a/lib/mix/tasks/dd.provider.new.ex b/lib/mix/tasks/dd.provider.new.ex index 1b040fbb..2c6ec95f 100644 --- a/lib/mix/tasks/dd.provider.new.ex +++ b/lib/mix/tasks/dd.provider.new.ex @@ -61,7 +61,7 @@ defmodule Mix.Tasks.Dd.Provider.New do identity field, the `external_identifiers` namespace that field is written to, the `work_kind` it is seeded as and the `evidence` it carries — and the `entry/3` clause in - `DevilsDictionary.Artworks.Corpus.Seeder` that maps a row onto a + `DevilsDictionary.Corpus.Seeder` that maps a row onto a `SourceIdentity.Entry`. Only the person who has read the source's rows can write the second one, so the task prints both rather than guessing at them. This is a real gap between #109's K7 and the code, recorded here rather than @@ -471,7 +471,7 @@ defmodule Mix.Tasks.Dd.Provider.New do back or Corpus.Conformance asks this corpus for a contract it does not have. * add an `entry("#{assigns.slug}", version, row)` clause to - lib/devils_dictionary/artworks/corpus/seeder.ex that maps a row onto a + lib/devils_dictionary/corpus/seeder.ex that maps a row onto a SourceIdentity.Entry Both are pattern matches in shared modules, and only a reader of the @@ -1035,7 +1035,7 @@ defmodule Mix.Tasks.Dd.Provider.New do or identity and display facts only (`:none`). The suite asks a corpus only for the contract its `evidence` declares, and holds it to that declaration. - 2. `DevilsDictionary.Artworks.Corpus.Seeder.entry/3` needs a clause for + 2. `DevilsDictionary.Corpus.Seeder.entry/3` needs a clause for `"<%= @slug %>"` mapping one row onto a `DevilsDictionary.SourceIdentity.Entry` — the identity namespace, the label, the year, and the metadata the shelf reads. diff --git a/test/devils_dictionary/corpus/seeder_quotes_test.exs b/test/devils_dictionary/corpus/seeder_quotes_test.exs new file mode 100644 index 00000000..e3a03ba6 --- /dev/null +++ b/test/devils_dictionary/corpus/seeder_quotes_test.exs @@ -0,0 +1,173 @@ +defmodule DevilsDictionary.Corpus.SeederQuotesTest do + @moduledoc """ + The seeder's second kind of content (#174): the public-domain Wikiquote + corpus, seeded from a manifest the real build produced over captured answers + (`QuotesCorpusFixtures`). The artworks corpora keep their own suites, + unchanged and still calling the seeder by its old name. + """ + + use DevilsDictionary.DataCase, async: false + + import Ecto.Query + + alias DevilsDictionary.Claims.{Assertion, AssertionEvidence, AssertionRevision, Predicate} + alias DevilsDictionary.Corpus.Seeder + alias DevilsDictionary.Discovery.Providers.Wikiquote + alias DevilsDictionary.Quotations + alias DevilsDictionary.Quotations.Fingerprint + alias DevilsDictionary.QuotesCorpusFixtures + alias DevilsDictionary.Registry + alias DevilsDictionary.Registry.ContentItem + alias DevilsDictionary.SourceIdentity + alias DevilsDictionary.SourceIdentity.Creators + alias DevilsDictionary.Sources + alias DevilsDictionary.Sources.{Source, SourceRecord} + + @manifest QuotesCorpusFixtures.manifest() + + setup do + DevilsDictionary.Fixtures.seed_catalog!() + :ok + end + + defp credits(content_id) do + Repo.all( + from r in AssertionRevision, + join: p in Predicate, + on: p.id == r.predicate_id and p.key == "authored_by", + join: a in Assertion, + on: a.id == r.assertion_id, + join: s in Source, + on: s.id == a.source_id, + where: r.subject_object_id == ^content_id and r.is_current, + order_by: s.slug, + select: %{source: s.slug, revision_id: r.id, person: r.object_object_id} + ) + end + + test "the source row is the issue's: 👑, a dump, CC BY-SA 4.0" do + source = Sources.get_source_by_slug!("wikiquote-pd-v1") + + assert %{tier: :aristocracy, access: :dump, kind: :corpus, license: "CC BY-SA 4.0"} = source + assert source.config["dump_url"] =~ "dumps.wikimedia.org/enwikiquote/" + assert source.config["retention"] =~ "never swept" + end + + test "each line is a quotation, credited to its author by QID, with its evidence and badge" do + assert @manifest["row_count"] > 0 + assert {:ok, summary} = Seeder.run(@manifest) + assert summary.newly_created == @manifest["row_count"] + assert summary.invalid == 0 + + for row <- @manifest["rows"] do + content_id = Registry.by_external_id("quotation_fingerprint", row["fingerprint"]) + item = Repo.get!(ContentItem, content_id) + + assert item.content_kind == :quotation + assert item.metadata["provenance"]["badge"] == "verified" + assert item.metadata["provenance"]["computed_by"] == "wikiquote-pd-v1" + + # Minted from the manifest's own Wikidata facts: no request was made. + assert [%{source: "wikiquote-pd-v1", revision_id: revision_id, person: person}] = + credits(content_id) + + assert Registry.by_external_id("wikidata", "Q9068") == person + + evidence = + Repo.all(from e in AssertionEvidence, where: e.assertion_revision_id == ^revision_id) + + assert Enum.any?(evidence, &(&1.locator =~ "Candide (Gutenberg #19942), line")) + assert Enum.all?(evidence, &(&1.evidence_role == :supports)) + assert Enum.all?(evidence, &is_integer(&1.source_record_revision_id)) + end + end + + test "a second seed matches every line and writes nothing new" do + {:ok, _} = Seeder.run(@manifest) + evidence = Repo.aggregate(AssertionEvidence, :count) + items = Repo.aggregate(ContentItem, :count) + + assert {:ok, again} = Seeder.run(@manifest) + assert again.matched == @manifest["row_count"] + assert again.newly_created == 0 + assert Repo.aggregate(AssertionEvidence, :count) == evidence + assert Repo.aggregate(ContentItem, :count) == items + end + + test "a corpus line and a live line with one wording are one subject" do + row = hd(@manifest["rows"]) + # The live provider's copy of the same words, one character different in + # transcription: a curly apostrophe and no final full stop. + live_text = row["text"] |> String.replace("'", "’") |> String.trim_trailing(".") + assert Fingerprint.fingerprint(live_text) == row["fingerprint"] + + item = %{ + external_namespace: "wikiquote_item", + external_id: "livelineid", + identifiers: [ + %{namespace: "wikiquote_item", external_id: "livelineid", metadata: %{}}, + Fingerprint.identifier(live_text) + ], + preview_metadata: %{ + "title" => live_text, + "author_qid" => "Q9068", + "certainty" => "candidate", + "source_url" => "https://en.wikiquote.org/wiki/Voltaire" + } + } + + wikiquote = Sources.get_source_by_slug!("wikiquote") + {:ok, record} = Sources.upsert_record(wikiquote, %{external_id: "livelineid", raw: %{}}) + {:ok, entry} = Wikiquote.identity_record(item) + prepared = Quotations.Corpus.seed_context(@manifest).prepared + + live = + entry + |> Map.merge(%{ + source_id: wikiquote.id, + source_record_id: record.id, + source_record_revision_id: record.current_revision.id + }) + |> SourceIdentity.resolve(prepared: prepared) + + assert live.state == :newly_created + + {:ok, _} = Seeder.run(@manifest) + + assert Registry.by_external_id("quotation_fingerprint", row["fingerprint"]) == + live.object_id + + assert [%{source: "wikiquote"}, %{source: "wikiquote-pd-v1"}] = credits(live.object_id) + end + + test "a corpus record is never swept" do + {:ok, _} = Seeder.run(@manifest) + source = Sources.get_source_by_slug!("wikiquote-pd-v1") + held = Repo.aggregate(from(r in SourceRecord, where: r.source_id == ^source.id), :count) + assert held == @manifest["row_count"] + + DevilsDictionary.Discovery.cleanup() + + assert Repo.aggregate(from(r in SourceRecord, where: r.source_id == ^source.id), :count) == + held + end + + test "a row whose fingerprint is not its text's is refused, and nothing is written" do + [row | _] = @manifest["rows"] + tampered = %{@manifest | "rows" => [Map.put(row, "text", row["text"] <> " And more.")]} + + assert {:ok, summary} = Seeder.run(tampered) + assert summary.invalid == 1 + assert summary.invalid_reasons == %{"fingerprint_mismatch" => 1} + assert Registry.by_external_id("quotation_fingerprint", row["fingerprint"]) == nil + end + + test "the author is prepared from the manifest, never fetched" do + prepared = Quotations.Corpus.seed_context(@manifest).prepared + + assert {:ok, %{kind: :person, label: "Voltaire", birth_year: 1694}} = prepared["Q9068"] + + assert prepared["Q9068"] == + Creators.classify("Q9068", %{"Q9068" => @manifest["selection"]["authors"]["Q9068"]}) + end +end