diff --git a/README.md b/README.md index f6449f0b..c76100d1 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Internal name: `devils_dictionary` (the Phoenix app, the repo). Product name is This file is the **living record** of the project: what it is, what it is not, the decisions we've made, and what we plan to build next. Update it whenever a decision changes. The encyclopedia foundation is specified in [issue #74](https://github.com/razrfly/dictionary/issues/74); [issue #69](https://github.com/razrfly/dictionary/issues/69) defines the original source-coverage scorecard. -Automatic cultural discovery and the direct-browser GIPHY integration are documented in [`docs/integrations/giphy.md`](docs/integrations/giphy.md). +Automatic cultural discovery and the direct-browser GIPHY integration are documented in [`docs/integrations/giphy.md`](docs/integrations/giphy.md). Durable cross-provider film identity and its adapter/backfill contract are documented in [`docs/integrations/source-identity.md`](docs/integrations/source-identity.md). ## Current delivery — 11 September 2026 diff --git a/docs/integrations/source-identity.md b/docs/integrations/source-identity.md new file mode 100644 index 00000000..7f517656 --- /dev/null +++ b/docs/integrations/source-identity.md @@ -0,0 +1,92 @@ +# Source identity adapters + +Dictionary keeps source ingestion, durable identity, discovery relevance and +editorial interpretation as separate layers. An adapter may propose a durable +object through `DevilsDictionary.SourceIdentity.Adapter`; the shared resolver, +not the adapter, decides whether that proposal matches or creates an object. + +## Required proposal fields + +An eligible durable proposal declares: + +- the source slug and source-record provenance; +- `object_kind` and the registry subtype (`entity_kind`, plus `work_kind` for + works); +- one stable, namespaced source identifier; +- every exact cross-source identifier supported by that source record; +- a display label and optional description, year, language and metadata; +- explicit eligibility and retention decisions; and +- optional relationships to independently identified objects, such as an + author or an original artwork. + +The resolver only matches verified exact identifiers. Titles, years, keyword +matches and image similarity are never identity evidence. A source record whose +identifiers point to different objects, assert two exclusive values in one +namespace, or contradict the held object subtype produces an open +`external_identifier_conflict` reconciliation case and no automatic link. + +Identifier values must already reflect the source adapter's rank and validity +policy. For Wikidata that means preferred statements replace normal statements, +deprecated statements are ignored, TMDb movie identifiers are positive decimal +IDs and IMDb title identifiers have the `tt` title form. + +## Current implementation + +CineGraph discovery results create or reuse film works from TMDb movie IDs and, +when supplied, IMDb title IDs. Wikidata film records add the Wikidata QID and +their exact P4947/P345 crosswalks. Either source may arrive first; both paths +converge on the same registry object and keep the original object ID and local +URL. + +The same proposal shape represents artwork, reproductions, quotations and +creator relationships. Issue #93 intentionally implements live ingestion only +for films. Artwork and quotation fixtures verify that future adapters can reuse +the contract without treating a reproduction as its original or uncredited +wording as a durable work. + +Discovery results remain provider cache rows. Resolution never turns a keyword +match into an `illustrates` claim. A film page presents automatic discovery +appearances separately from attributed, reviewable meaning connections. + +## Backfilling retained CineGraph rows + +The backfill is local, bounded and resumable; it does not call CineGraph and +does not merge or delete objects: + +```console +mix dd.films.backfill --limit 100 +mix dd.films.backfill --limit 100 --after 4200 +``` + +The report separates matched, newly created, insufficient-evidence and +conflicting-identifier outcomes. Use the printed checkpoint for the next batch. +Rerunning a batch is safe because the same exact-identifier resolver handles +both live ingestion and backfill. + +## Upgrade existing Wikidata entries first + +Run `mix dd.films.backfill --wikidata --limit 100` and continue with the printed +`--after` checkpoint. This scans a bounded batch of allowed Wikidata source records. +Legacy film records queue deduplicated jobs through the existing enrichment worker; +these jobs fetch the fields older projections discarded, then materialize them onto +the existing Wikidata identity. Current projections are replayed locally. Counts +separate queued, current, skipped and failed records; a malformed retained record is +reported without stopping the batch, and queued does not mean completed. + +Wait for enrichment jobs to succeed, inspect failures/conflicts, then run the +CineGraph backfill above. Rerunning the Wikidata phase is safe. Completed refreshes +carry a projection-version marker so a successful negative crosswalk does not cause +endless refreshes. Requests use the existing Wikimedia timeout, retry and rate-limit +handling. The version participates in the source revision hash so an unchanged +upstream payload can still upgrade an older local projection without overwriting its +historical revision. Source records not classified as films are skipped; this is a +film migration, not a full Wikidata crawl or automatic title search. + +If separate objects already exist, contradictory IDs enter reconciliation rather +than being silently merged. Missing crosswalks remain valid source-backed local films. + +Film image projections record their source-record evidence. Public display checks +source activation, record visibility and output retirement. Legacy posters without +that marker must be supported by an allowed record with the exact same URL. Withdrawn +posters disappear without deleting the film or its connections. Connected meanings +use the existing cursor pagination so all public relationships remain reachable. diff --git a/lib/devils_dictionary/absorb/materializer.ex b/lib/devils_dictionary/absorb/materializer.ex index 47a901a4..8c52a8e4 100644 --- a/lib/devils_dictionary/absorb/materializer.ex +++ b/lib/devils_dictionary/absorb/materializer.ex @@ -74,6 +74,7 @@ defmodule DevilsDictionary.Absorb.Materializer do alias DevilsDictionary.Claims alias DevilsDictionary.Repo alias DevilsDictionary.Registry.Lexeme + alias DevilsDictionary.SourceIdentity alias DevilsDictionary.Sources.SourceRecord # Postgres caps a statement at 65,535 bind parameters; the widest row here is @@ -134,7 +135,9 @@ defmodule DevilsDictionary.Absorb.Materializer do |> Ecto.Multi.run(:forms, fn _repo, changes -> {:ok, upsert_forms(merged.lexemes, changes.lexemes, revisions, now)} end) - |> Ecto.Multi.run(:concepts, fn _repo, _ -> {:ok, upsert_entities(merged.concepts, now)} end) + |> Ecto.Multi.run(:concepts, fn _repo, _ -> + {:ok, upsert_entities(merged.concepts, now, module, revisions, run_id)} + end) |> Ecto.Multi.run(:senses, fn _repo, changes -> {:ok, upsert_senses(merged.senses, changes.lexemes, records, revisions, run_id, now, module)} @@ -551,9 +554,52 @@ defmodule DevilsDictionary.Absorb.Materializer do defp initial_projection(metadata), do: metadata - defp upsert_entities([], _now), do: %{} + defp upsert_entities([], _now, _module, _revisions, _run_id), do: %{} + + defp upsert_entities(rows, now, module, revisions, run_id) do + {identity_rows, ordinary_rows} = identity_rows(rows, module) + + ordinary_ids = upsert_ordinary_entities(ordinary_rows, now) + + identity_ids = + identity_rows + |> Enum.uniq_by(fn {row, _entry} -> row.key end) + |> Enum.flat_map(fn {row, entry} -> + entry = %{ + entry + | source_id: row[:source_id], + source_record_id: row[:source_record_id], + source_record_revision_id: revisions[row[:source_record_id]], + import_run_id: run_id + } + + case SourceIdentity.resolve(entry) do + %{object_id: object_id} when is_integer(object_id) -> [{row.key, object_id}] + _resolution -> [] + end + end) + |> Map.new() + + Map.merge(ordinary_ids, identity_ids) + end + + defp identity_rows(rows, module) do + if function_exported?(module, :identity_record, 1) do + Enum.reduce(rows, {[], []}, fn row, {identity, ordinary} -> + case module.identity_record(row) do + {:ok, entry} -> {[{row, entry} | identity], ordinary} + :ignore -> {identity, [row | ordinary]} + {:error, reason} -> throw({:materialize_failed, row.source_id, row.key, reason}) + end + end) + else + {[], rows} + end + end + + defp upsert_ordinary_entities([], _now), do: %{} - defp upsert_entities(rows, now) do + defp upsert_ordinary_entities(rows, now) do qids = Enum.map(rows, & &1.key) existing = diff --git a/lib/devils_dictionary/absorb/sources/wikidata.ex b/lib/devils_dictionary/absorb/sources/wikidata.ex index d77fb690..f7a4bc72 100644 --- a/lib/devils_dictionary/absorb/sources/wikidata.ex +++ b/lib/devils_dictionary/absorb/sources/wikidata.ex @@ -25,6 +25,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do """ @behaviour DevilsDictionary.Absorb.Source + @behaviour DevilsDictionary.SourceIdentity.Adapter import Ecto.Query @@ -34,6 +35,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do alias DevilsDictionary.Lexicon.ScopeMember alias DevilsDictionary.Registry.{Entity, ExternalIdentifier, Sense, SenseRevision} alias DevilsDictionary.Repo + alias DevilsDictionary.SourceIdentity.Entry alias DevilsDictionary.Sources alias DevilsDictionary.Sources.{Source, SourceRecord} @@ -44,7 +46,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do # The fifteen properties the pipeline reads. Everything else is dropped before # storage; a 130 KB entity becomes a couple of KB. - @keep_properties ~w(P18 P31 P279 P171 P105 P225 P1843 P5063 P8814 P13176 P1420 P910 P373 P846 P685) + @keep_properties ~w(P18 P31 P279 P171 P105 P225 P1843 P5063 P8814 P13176 P1420 P910 P373 P846 P685 P345 P4947 P577) # Parent properties walked to closure. **Only these two**: P171 is the # taxonomy proper and terminates at Animalia, and P13176 is the one hop from @@ -82,6 +84,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do @impl true def trim(raw) do %{ + "_film_identity_version" => 1, "id" => raw["id"], "labels" => take_lang(raw["labels"]), "descriptions" => take_lang(raw["descriptions"]), @@ -486,7 +489,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do external_id: qid, url: entity_url(qid), raw: trim(entity), - content_hash: SourceRecord.content_hash(entity) + content_hash: SourceRecord.content_hash(%{"entity" => entity, "projection_version" => 1}) } end @@ -548,6 +551,9 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do label: clamp(label(raw, scientific_name)), description: get_in(raw, ["descriptions", "en", "value"]), kind: entity_kind(raw, scientific_name), + work_kind: work_kind(raw), + first_published_year: publication_year(raw), + external_identifiers: external_identifiers(qid, raw), taxon_concept: taxon_concept(raw, scientific_name), metadata: raw @@ -562,6 +568,27 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do {:ok, %{concepts: [concept], concept_relations: relations(record, qid, raw)}} end + @impl DevilsDictionary.SourceIdentity.Adapter + def identity_record(%{kind: :work, work_kind: work_kind} = row) + when work_kind in ["film", "artwork"] do + Entry.new(%{ + source_slug: slug(), + object_kind: :entity, + entity_kind: :work, + work_kind: work_kind, + stable_identifier: %{namespace: "wikidata", external_id: row.qid}, + identifiers: row.external_identifiers, + label: row.label, + description: row.description, + year: row.first_published_year, + metadata: row.metadata, + eligibility: :eligible, + retention: :durable + }) + end + + def identity_record(_row), do: :ignore + # English first, then the multilingual label a taxon carries instead, then the # scientific name itself. A concept with no name at all is not worth a card. defp label(raw, scientific_name) do @@ -603,6 +630,9 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do end end + @film_classes ~w(Q11424 Q202866 Q506240) + @artwork_classes ~w(Q3305213) + @kind_classes %{ person: ~w(Q5), organization: ~w(Q43229 Q4830453 Q163740 Q783794), @@ -616,8 +646,82 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do defp entity_kind(raw, _scientific_name) do classes = MapSet.new(Client.entity_ids(raw, "P31")) - Enum.find_value(@kind_classes, :concept, fn {kind, recognized} -> - if Enum.any?(recognized, &MapSet.member?(classes, &1)), do: kind + cond do + Enum.any?(@film_classes, &MapSet.member?(classes, &1)) -> + :work + + Enum.any?(@artwork_classes, &MapSet.member?(classes, &1)) -> + :work + + true -> + Enum.find_value(@kind_classes, :concept, fn {kind, recognized} -> + if Enum.any?(recognized, &MapSet.member?(classes, &1)), do: kind + end) + end + end + + defp work_kind(raw) do + classes = MapSet.new(Client.entity_ids(raw, "P31")) + + cond do + Enum.any?(@film_classes, &MapSet.member?(classes, &1)) -> "film" + Enum.any?(@artwork_classes, &MapSet.member?(classes, &1)) -> "artwork" + true -> nil + end + end + + defp external_identifiers(qid, raw) do + [ + %{ + namespace: "wikidata", + external_id: qid, + metadata: %{"provider_field" => "id"} + } + ] ++ + external_identifier_values(raw, "P4947", "tmdb_movie", &valid_tmdb_movie_id?/1) ++ + external_identifier_values(raw, "P345", "imdb_title", &valid_imdb_title_id?/1) + end + + defp external_identifier_values(raw, property, namespace, valid?) do + raw + |> Client.strings(property) + |> Enum.filter(valid?) + |> Enum.uniq() + |> Enum.map(fn external_id -> + %{ + namespace: namespace, + external_id: external_id, + metadata: %{ + "wikidata_property" => property, + "statement_policy" => "preferred_else_normal" + } + } + end) + end + + defp valid_tmdb_movie_id?(value) when is_binary(value), + do: Regex.match?(~r/\A[1-9]\d*\z/, value) + + defp valid_tmdb_movie_id?(_value), do: false + + defp valid_imdb_title_id?(value) when is_binary(value), + do: Regex.match?(~r/\Att\d{7,10}\z/, value) + + defp valid_imdb_title_id?(_value), do: false + + defp publication_year(raw) do + raw + |> Client.statements("P577") + |> Enum.find_value(fn + %{value: %{"time" => <>}} + when sign in ["+", "-"] -> + case Integer.parse(sign <> year) do + {value, ""} -> value + _ -> nil + end + + _ -> + nil end) end diff --git a/lib/devils_dictionary/claims/connection.ex b/lib/devils_dictionary/claims/connection.ex index 0356d41c..afb82d2e 100644 --- a/lib/devils_dictionary/claims/connection.ex +++ b/lib/devils_dictionary/claims/connection.ex @@ -346,6 +346,7 @@ defmodule DevilsDictionary.Claims.Connection do {s.object_id, %{ label: l.lemma, + detail: r.gloss, path: fragment("'/words/' || ? || '/' || ?", l.object_id, l.slug) }} ) diff --git a/lib/devils_dictionary/discovery.ex b/lib/devils_dictionary/discovery.ex index ca8d589e..f5c63233 100644 --- a/lib/devils_dictionary/discovery.ex +++ b/lib/devils_dictionary/discovery.ex @@ -4,8 +4,10 @@ defmodule DevilsDictionary.Discovery do Definitions never wait for this context. A valid page creates an automatic term recipe lazily, admits at most one database-coordinated refresh, and lets - an Oban worker perform provider I/O. Discovery results remain search cache: - they never create `illustrates` claims or permanent objects. + an Oban worker perform provider I/O. Discovery appearances remain search + cache and never create `illustrates` claims. Approved adapters may resolve an + eligible result to a durable registry identity, whose lifecycle is separate + from this cache. """ import Ecto.Query @@ -13,8 +15,10 @@ defmodule DevilsDictionary.Discovery do alias DevilsDictionary.Claims.AssertionRevision alias DevilsDictionary.Discovery.{Mapping, Policy, Providers, Result, Run} alias DevilsDictionary.Discovery.RunWorker - alias DevilsDictionary.Registry.{ExternalIdentifier, Lexeme, Object, Sense} + alias DevilsDictionary.Registry.{Lexeme, Object, Sense} alias DevilsDictionary.Repo + alias DevilsDictionary.SourceIdentity + alias DevilsDictionary.SourceIdentity.Resolution alias DevilsDictionary.Sources alias DevilsDictionary.Sources.{Actor, Source, SourceRecord} @@ -836,7 +840,7 @@ defmodule DevilsDictionary.Discovery do result = Repo.transaction(fn -> - if persistent?, do: persist_results(run, items) + if persistent?, do: persist_results(run, provider, items) run |> Run.lifecycle_changeset(%{ @@ -904,51 +908,88 @@ defmodule DevilsDictionary.Discovery do {:notify, deferred, [], {:snooze, seconds}} end - defp persist_results(run, items) do - identity_map = existing_object_ids(items) + defp persist_results(run, provider, items) do source = run.mapping.source - Enum.each(items, fn item -> + proposed = Enum.map(items, &{&1, identity_proposal(provider, &1)}) + + proposed + |> Enum.flat_map(fn + {_item, {:ok, entry}} -> [entry] + _ -> [] + end) + |> SourceIdentity.lock_entries() + + Enum.each(proposed, fn {item, proposal} -> {:ok, record} = Sources.upsert_record(source, %{ external_id: "#{item.external_namespace}:#{item.external_id}", url: item.preview_metadata["source_url"], - raw: %{ - "external_namespace" => item.external_namespace, - "external_id" => item.external_id - } + raw: source_payload(item) }) + resolution = resolve_identity(proposal, source, record) + attrs = item |> Map.put(:run_id, run.id) - |> Map.put(:object_id, identity_map[{item.external_namespace, item.external_id}]) + |> Map.put(:object_id, resolution.object_id) |> Map.put(:source_record_id, record.id) + |> Map.put(:resolution_state, resolution.state) %Result{} |> Result.changeset(attrs) |> Repo.insert!() end) end - defp existing_object_ids(items) do - identities = Enum.map(items, &{&1.external_namespace, &1.external_id}) - namespaces = identities |> Enum.map(&elem(&1, 0)) |> Enum.uniq() - external_ids = identities |> Enum.map(&elem(&1, 1)) |> Enum.uniq() + defp source_payload(item) do + %{ + "external_namespace" => item.external_namespace, + "external_id" => item.external_id, + "identifiers" => Enum.map(item[:identifiers] || [], &stringify_identifier/1), + "preview_metadata" => item.preview_metadata, + "match_details" => item.match_details + } + end - Repo.all( - from identifier in ExternalIdentifier, - where: - identifier.status == :verified and identifier.namespace in ^namespaces and - identifier.external_id in ^external_ids, - select: {identifier.namespace, identifier.external_id, identifier.object_id} - ) - |> Enum.filter(fn {namespace, external_id, _object_id} -> - {namespace, external_id} in identities - end) - |> Map.new(fn {namespace, external_id, object_id} -> - {{namespace, external_id}, object_id} - end) + defp stringify_identifier(identifier) do + %{ + "namespace" => identifier.namespace, + "external_id" => identifier.external_id, + "metadata" => identifier[:metadata] || %{}, + "exclusive" => Map.get(identifier, :exclusive, true) + } + end + + defp identity_proposal(provider, item) do + if function_exported?(provider, :identity_record, 1) do + case provider.identity_record(item) do + {:ok, entry} -> + {:ok, entry} + + :ignore -> + {:resolution, %Resolution{state: :insufficient_evidence, reason: "adapter_ignored"}} + + {:error, reason} -> + {:resolution, %Resolution{state: :insufficient_evidence, reason: to_string(reason)}} + end + else + {:resolution, + %Resolution{state: :insufficient_evidence, reason: "adapter_has_no_identity_contract"}} + end end + defp resolve_identity({:ok, entry}, source, record) do + entry + |> Map.merge(%{ + source_id: source.id, + source_record_id: record.id, + source_record_revision_id: record.current_revision.id + }) + |> SourceIdentity.resolve() + end + + defp resolve_identity({:resolution, resolution}, _source, _record), do: resolution + defp start_run(run_id) do Repo.transaction(fn -> run = Repo.get!(Run, run_id) |> Repo.preload(mapping: :source) diff --git a/lib/devils_dictionary/discovery/providers/cine_graph.ex b/lib/devils_dictionary/discovery/providers/cine_graph.ex index a07f68e5..a66242e1 100644 --- a/lib/devils_dictionary/discovery/providers/cine_graph.ex +++ b/lib/devils_dictionary/discovery/providers/cine_graph.ex @@ -2,8 +2,11 @@ defmodule DevilsDictionary.Discovery.Providers.CineGraph do @moduledoc "CineGraph keyword discovery normalized into provider-neutral culture cards." @behaviour DevilsDictionary.Discovery.Provider + @behaviour DevilsDictionary.SourceIdentity.Adapter - @adapter_version "cinegraph.graphql.v1" + alias DevilsDictionary.SourceIdentity.Entry + + @adapter_version "cinegraph.graphql.v2" @operation "keyword_discovery" @keyword_query """ @@ -53,6 +56,40 @@ defmodule DevilsDictionary.Discovery.Providers.CineGraph do } end + @impl DevilsDictionary.SourceIdentity.Adapter + def identity_record( + %{ + external_namespace: "tmdb_movie", + external_id: tmdb_id, + preview_metadata: metadata + } = item + ) do + identifiers = + item + |> Map.get(:identifiers, [%{namespace: "tmdb_movie", external_id: tmdb_id}]) + + Entry.new(%{ + source_slug: slug(), + object_kind: :entity, + entity_kind: :work, + work_kind: "film", + stable_identifier: %{namespace: "tmdb_movie", external_id: tmdb_id}, + identifiers: identifiers, + label: metadata["title"], + year: integer_year(metadata["year"]), + metadata: %{ + "image_url" => metadata["poster_url"], + "image_attribution" => "Poster delivered by TMDb through CineGraph", + "release_date" => metadata["release_date"], + "content_type" => "film" + }, + eligibility: :eligible, + retention: :durable + }) + end + + def identity_record(_item), do: {:error, :unsupported_cinegraph_identity} + @impl true def enabled? do config = config() @@ -251,11 +288,22 @@ defmodule DevilsDictionary.Discovery.Providers.CineGraph do is_list(keywords) and is_list(genres) do with {:ok, keywords} <- normalize_tags(keywords), - {:ok, genres} <- normalize_tags(genres) do + {:ok, genres} <- normalize_tags(genres), + {:ok, imdb_id} <- normalize_imdb_id(movie["imdbId"]) do + tmdb_id = Integer.to_string(tmdb_id) + {:ok, %{ external_namespace: "tmdb_movie", - external_id: Integer.to_string(tmdb_id), + external_id: tmdb_id, + identifiers: + [ + %{ + namespace: "tmdb_movie", + external_id: tmdb_id, + metadata: %{"provider_field" => "tmdbId"} + } + ] ++ imdb_identifier(imdb_id), position: position, match_details: %{ "kind" => "keyword", @@ -266,6 +314,7 @@ defmodule DevilsDictionary.Discovery.Providers.CineGraph do preview_metadata: %{ "title" => title, "year" => year(movie["releaseDate"]), + "release_date" => movie["releaseDate"], "source_url" => source_url(movie["cinegraphUrl"]), "poster_url" => poster_url(movie["posterPath"]), "content_type" => "film", @@ -310,6 +359,28 @@ defmodule DevilsDictionary.Discovery.Providers.CineGraph do end end + defp normalize_imdb_id(nil), do: {:ok, nil} + + defp normalize_imdb_id(imdb_id) when is_binary(imdb_id) do + if Regex.match?(~r/\Att\d{7,10}\z/, imdb_id), + do: {:ok, imdb_id}, + else: {:error, "malformed_response"} + end + + defp normalize_imdb_id(_imdb_id), do: {:error, "malformed_response"} + + defp imdb_identifier(nil), do: [] + + defp imdb_identifier(imdb_id) do + [ + %{ + namespace: "imdb_title", + external_id: imdb_id, + metadata: %{"provider_field" => "imdbId"} + } + ] + end + defp next_cursor(%{"hasNextPage" => true, "endCursor" => cursor}) when is_binary(cursor) and byte_size(cursor) <= 2_048, do: cursor @@ -319,6 +390,15 @@ defmodule DevilsDictionary.Discovery.Providers.CineGraph do defp year(<>), do: year defp year(_), do: nil + defp integer_year(<>) do + case Integer.parse(year) do + {value, ""} -> value + _ -> nil + end + end + + defp integer_year(_year), do: nil + defp poster_url(path) when is_binary(path) and path != "" do case config()[:image_base_url] do base when is_binary(base) and base != "" -> diff --git a/lib/devils_dictionary/discovery/result.ex b/lib/devils_dictionary/discovery/result.ex index aa3095fe..3378f6ef 100644 --- a/lib/devils_dictionary/discovery/result.ex +++ b/lib/devils_dictionary/discovery/result.ex @@ -20,6 +20,10 @@ defmodule DevilsDictionary.Discovery.Result do field :preview_metadata, :map, default: %{} field :display_allowed, :boolean, default: true + field :resolution_state, Ecto.Enum, + values: [:matched, :newly_created, :insufficient_evidence, :conflicting_identifiers], + default: :insufficient_evidence + timestamps(type: :utc_datetime_usec) end @@ -34,7 +38,8 @@ defmodule DevilsDictionary.Discovery.Result do :position, :match_details, :preview_metadata, - :display_allowed + :display_allowed, + :resolution_state ]) |> validate_required([ :run_id, @@ -43,7 +48,8 @@ defmodule DevilsDictionary.Discovery.Result do :position, :match_details, :preview_metadata, - :display_allowed + :display_allowed, + :resolution_state ]) |> validate_number(:position, greater_than_or_equal_to: 0) |> unique_constraint([:run_id, :external_namespace, :external_id]) @@ -51,5 +57,6 @@ defmodule DevilsDictionary.Discovery.Result do |> foreign_key_constraint(:run_id) |> foreign_key_constraint(:object_id) |> foreign_key_constraint(:source_record_id) + |> check_constraint(:resolution_state, name: :discovery_results_resolution_state) end end diff --git a/lib/devils_dictionary/encyclopedia.ex b/lib/devils_dictionary/encyclopedia.ex index 4f24cfe4..75b82bf0 100644 --- a/lib/devils_dictionary/encyclopedia.ex +++ b/lib/devils_dictionary/encyclopedia.ex @@ -77,14 +77,18 @@ defmodule DevilsDictionary.Encyclopedia do """ def view(nil), do: nil - def view(%Entity{} = entity) do + def view(%Entity{} = entity), + do: view(entity, DevilsDictionary.SourceIdentity.Display.preload([entity])) + + @doc "Projects an entity using already-loaded source visibility evidence." + def view(%Entity{} = entity, image_evidence) when is_map(image_evidence) do %{ object_id: entity.object_id, qid: qid(entity.object_id), label: entity.preferred_label, description: entity.description, kind: entity.entity_kind, - image_url: entity.metadata["image_url"], + image_url: DevilsDictionary.SourceIdentity.Display.image_url(entity, image_evidence), image_attribution: entity.metadata["image_attribution"], wikipedia_title: entity.metadata["wikipedia_title"], taxon: entity.metadata["taxon"] diff --git a/lib/devils_dictionary/encyclopedia/entity_page.ex b/lib/devils_dictionary/encyclopedia/entity_page.ex index 6aac70ed..e85cc27a 100644 --- a/lib/devils_dictionary/encyclopedia/entity_page.ex +++ b/lib/devils_dictionary/encyclopedia/entity_page.ex @@ -41,10 +41,13 @@ defmodule DevilsDictionary.Encyclopedia.EntityPage do alias DevilsDictionary.Claims alias DevilsDictionary.Claims.{Connection, Visibility} + alias DevilsDictionary.Discovery.{Mapping, Result, Run} alias DevilsDictionary.Encyclopedia alias DevilsDictionary.Registry alias DevilsDictionary.Registry.{ContentItem, ContentRevision, Entity, Lexeme, Sense} alias DevilsDictionary.Repo + alias DevilsDictionary.SourceIdentity.Display + alias DevilsDictionary.Sources.{MaterializedOutput, Source, SourceRecord} defstruct entity: nil, identity: %{state: :active, requested_id: nil, canonical_id: nil, outputs: []}, @@ -54,6 +57,9 @@ defmodule DevilsDictionary.Encyclopedia.EntityPage do definitions: [], editions: [], contents: [], + meaning_connections: [], + discovery_appearances: [], + sources: [], connections: %{incoming: [], outgoing: []}, pagination: %{} @@ -116,6 +122,12 @@ defmodule DevilsDictionary.Encyclopedia.EntityPage do {contents, contents_page} = contents_of(entity, id, opts[:contents_after]) + {meaning_connections, meaning_connections_page} = + meaning_connections(id, opts[:meaning_connections_after]) + + {discovery_appearances, discovery_appearances_page} = + discovery_appearances(id, opts[:discovery_appearances_after]) + {connections_in, connections_in_page} = other_connections(:incoming, id, opts[:connections_in_after]) @@ -136,6 +148,9 @@ defmodule DevilsDictionary.Encyclopedia.EntityPage do definitions: definition_views(Enum.map(definitions, & &1.subject_object_id)), editions: entity_views(Enum.map(editions, & &1.subject_object_id)), contents: definition_views(Enum.map(contents, & &1.subject_object_id)), + meaning_connections: meaning_connections, + discovery_appearances: discovery_appearances, + sources: source_views(id), connections: %{incoming: connections_in, outgoing: connections_out}, pagination: %{ biography: biography_page, @@ -143,6 +158,8 @@ defmodule DevilsDictionary.Encyclopedia.EntityPage do definitions: definitions_page, editions: editions_page, contents: contents_page, + meaning_connections: meaning_connections_page, + discovery_appearances: discovery_appearances_page, connections_in: connections_in_page, connections_out: connections_out_page } @@ -159,7 +176,7 @@ defmodule DevilsDictionary.Encyclopedia.EntityPage do filters = case direction do :incoming -> [exclude_predicates: @presented_predicates] - :outgoing -> [] + :outgoing -> [exclude_predicates: ["illustrates"]] end opts = filters ++ [after: after_cursor, limit: @section_cap + 1] @@ -209,6 +226,147 @@ defmodule DevilsDictionary.Encyclopedia.EntityPage do {connection_rows, page} end + defp meaning_connections(object_id, after_cursor) do + all_rows = + Claims.outgoing(object_id, + predicate: "illustrates", + after: after_cursor, + limit: @section_cap + 1 + ) + + rows = Enum.take(all_rows, @section_cap) + endpoints = Connection.endpoint_summaries(Enum.map(rows, & &1.object_object_id)) + review_states = Claims.display_review_states(Enum.map(rows, & &1.id)) + + views = + Enum.map(rows, fn row -> + row + |> Map.from_struct() + |> Map.merge(Map.fetch!(endpoints, row.object_object_id)) + |> Map.put(:review_state, Map.fetch!(review_states, row.id)) + end) + + {views, + %{ + count: Claims.count_outgoing(object_id, predicate: "illustrates"), + next: if(length(all_rows) > @section_cap, do: Claims.next_cursor(rows)) + }} + end + + defp discovery_appearances(object_id, after_cursor) do + family = Registry.canonical_family(object_id) + + base = discovery_appearance_query(family) + + count = + base + |> distinct([_result, _run, mapping, source, _record], [ + mapping.target_object_id, + source.slug + ]) + |> select([_result, _run, mapping, source, _record], %{ + target_object_id: mapping.target_object_id, + provider_slug: source.slug + }) + |> subquery() + |> Repo.aggregate(:count) + + all_rows = + base + |> after_discovery_appearance(after_cursor) + |> distinct([_result, _run, mapping, source, _record], [ + mapping.target_object_id, + source.slug + ]) + |> order_by([_result, run, mapping, source, _record], + asc: mapping.target_object_id, + asc: source.slug, + desc: run.completed_at, + desc: run.id + ) + |> limit(^(@section_cap + 1)) + |> select([_result, _run, mapping, source, _record], %{ + target_object_id: mapping.target_object_id, + provider: source.name, + provider_slug: source.slug, + term: mapping.parameters["term"] + }) + |> Repo.all() + + rows = Enum.take(all_rows, @section_cap) + + endpoints = Connection.endpoint_summaries(Enum.map(rows, & &1.target_object_id)) + + views = + Enum.map(rows, fn row -> + Map.merge(row, Map.fetch!(endpoints, row.target_object_id)) + end) + + next = + if length(all_rows) > @section_cap do + row = List.last(rows) + "#{row.target_object_id}:#{row.provider_slug}" + end + + {views, %{count: count, next: next}} + end + + defp discovery_appearance_query(family) do + from result in Result, + join: run in Run, + on: run.id == result.run_id, + join: mapping in Mapping, + on: mapping.id == run.mapping_id, + join: source in Source, + on: source.id == mapping.source_id, + left_join: record in SourceRecord, + on: record.id == result.source_record_id, + where: + result.object_id in ^family and result.display_allowed and run.display_allowed and + run.status == :succeeded and mapping.enabled and source.active and + (is_nil(record.id) or record.display_allowed) + end + + defp after_discovery_appearance(query, nil), do: query + + defp after_discovery_appearance(query, cursor) when is_binary(cursor) do + with [target, source_slug] <- String.split(cursor, ":", parts: 2), + {target_object_id, ""} when target_object_id > 0 <- Integer.parse(target), + true <- source_slug != "" do + where( + query, + [_result, _run, mapping, source, _record], + mapping.target_object_id > ^target_object_id or + (mapping.target_object_id == ^target_object_id and source.slug > ^source_slug) + ) + else + _ -> query + end + end + + defp after_discovery_appearance(query, _cursor), do: query + + defp source_views(object_id) do + family = Registry.canonical_family(object_id) + + Repo.all( + from output in MaterializedOutput, + join: record in assoc(output, :source_record), + join: source in assoc(record, :source), + where: + output.output_object_id in ^family and is_nil(output.retired_at) and + record.display_allowed and source.active, + distinct: [source.slug, record.url], + order_by: [asc: source.slug, asc: record.url], + select: %{ + slug: source.slug, + name: source.name, + attribution: source.attribution, + url: record.url + } + ) + end + defp incoming_page(object_id, predicate, subject_kind, after_cursor) do filters = [predicate: predicate, subject_kind: subject_kind] @@ -231,31 +389,43 @@ defmodule DevilsDictionary.Encyclopedia.EntityPage do # A person's subtype row, a work's, an edition's — whichever this entity has. # One query, and nil for a kind that has no detail table. defp details(%Entity{entity_kind: :person, object_id: id}), - do: Repo.get(Registry.PersonDetails, id) |> Map.take([:birth_date, :death_date]) + do: detail_fields(Repo.get(Registry.PersonDetails, id), [:birth_date, :death_date]) defp details(%Entity{entity_kind: :work, object_id: id}), do: - Repo.get(Registry.WorkDetails, id) - |> Map.take([:work_kind, :original_language, :first_published_year]) + detail_fields(Repo.get(Registry.WorkDetails, id), [ + :work_kind, + :original_language, + :first_published_year + ]) defp details(%Entity{entity_kind: :edition, object_id: id}), do: - Repo.get(Registry.EditionDetails, id) - |> Map.take([:work_id, :edition_label, :publication_year, :language_tag]) + detail_fields(Repo.get(Registry.EditionDetails, id), [ + :work_id, + :edition_label, + :publication_year, + :language_tag + ]) defp details(%Entity{}), do: %{} + defp detail_fields(nil, _fields), do: %{} + defp detail_fields(details, fields), do: Map.take(details, fields) + defp entity_views([]), do: [] defp entity_views(ids) do canonical = Registry.canonical_ids(ids) canonical_ids = Enum.map(ids, &Map.fetch!(canonical, &1)) - by_id = + entities = Entity |> where([e], e.object_id in ^canonical_ids) |> Repo.all() - |> Map.new(&{&1.object_id, Encyclopedia.view(&1)}) + + image_evidence = Display.preload(entities) + by_id = Map.new(entities, &{&1.object_id, Encyclopedia.view(&1, image_evidence)}) canonical_ids |> Enum.map(&Map.get(by_id, &1)) |> Enum.reject(&is_nil/1) end diff --git a/lib/devils_dictionary/source_identity.ex b/lib/devils_dictionary/source_identity.ex new file mode 100644 index 00000000..1c4e9480 --- /dev/null +++ b/lib/devils_dictionary/source_identity.ex @@ -0,0 +1,456 @@ +defmodule DevilsDictionary.SourceIdentity do + @moduledoc """ + Resolves approved source records into the existing object registry. + + Exact verified identifiers are the only automatic merge evidence. All usable + identifiers are locked and resolved before a new object is minted, so + retries and concurrent imports converge. If identifiers point at different + objects, disagree with an object's exclusive identifier, or imply an + incompatible subtype, the resolver opens an existing reconciliation case and + does not pick a winner. + + The first implementation creates entity records (especially film works). + The adapter contract also describes content, creators/authors and retention + so future artwork and quotation providers do not need a second identity + system. + """ + + import Ecto.Query + + alias DevilsDictionary.Registry + + alias DevilsDictionary.Registry.{ + Entity, + ExternalIdentifier, + Object, + PersonDetails, + WorkDetails + } + + alias DevilsDictionary.Repo + alias DevilsDictionary.SourceIdentity.Entry + alias DevilsDictionary.SourceIdentity.Resolution + alias DevilsDictionary.Sources.{MaterializedOutput, ReconciliationCase} + + @conflict_kind "external_identifier_conflict" + + @doc "Resolves an adapter entry without consulting labels for identity." + def resolve(%Entry{eligibility: eligibility} = entry) when eligibility != :eligible do + %Resolution{ + state: :insufficient_evidence, + reason: entry.eligibility_reason || to_string(eligibility), + identifiers: entry.identifiers + } + end + + def resolve(%Entry{retention: retention} = entry) when retention != :durable do + %Resolution{ + state: :insufficient_evidence, + reason: "transient_record", + identifiers: entry.identifiers + } + end + + def resolve(%Entry{object_kind: object_kind} = entry) when object_kind != :entity do + %Resolution{ + state: :insufficient_evidence, + reason: "object_kind_not_implemented", + identifiers: entry.identifiers + } + end + + def resolve(%Entry{} = entry) do + {:ok, resolution} = + Repo.transaction(fn -> + lock_entries([entry]) + matches = identifier_matches(entry.identifiers) + object_ids = matches |> Map.values() |> Enum.map(&Registry.canonical_id/1) |> Enum.uniq() + proposal_conflicts = exclusive_proposal_conflicts(entry.identifiers) + + cond do + proposal_conflicts != [] -> + conflict( + entry, + matches, + "source_asserts_multiple_exclusive_identifiers", + proposal_conflicts + ) + + length(object_ids) > 1 -> + conflict(entry, matches, "identifiers_resolve_to_different_objects") + + length(object_ids) == 1 -> + object_id = hd(object_ids) + + case compatibility_conflicts(entry, object_id) do + [] -> + {:ok, entity} = enrich_entity(object_id, entry) + ensure_identifiers(entity.object_id, entry) + attach_source_record(entity.object_id, entry) + matched(entry, entity.object_id) + + conflicts -> + conflict(entry, matches, "identifier_or_subtype_contradiction", conflicts) + end + + true -> + {:ok, entity} = create_entity(entry) + ensure_identifiers(entity.object_id, entry) + attach_source_record(entity.object_id, entry) + created(entry, entity.object_id) + end + end) + + resolution + end + + @doc "Acquires one deterministic transaction-lock set for adapter entries." + def lock_entries(entries) when is_list(entries) do + entries + |> Enum.flat_map(&lock_keys/1) + |> Enum.uniq() + |> Enum.sort() + |> Enum.each(fn key -> + Repo.query!("SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", [key]) + end) + + :ok + end + + defp lock_keys(%Entry{} = entry) do + Enum.map(entry.identifiers, &"#{&1.namespace}:#{&1.external_id}") ++ + if(entry.source_record_id, do: ["source-record:#{entry.source_record_id}"], else: []) + end + + defp identifier_matches([]), do: %{} + + defp identifier_matches(identifiers) do + namespaces = identifiers |> Enum.map(& &1.namespace) |> Enum.uniq() + external_ids = identifiers |> Enum.map(& &1.external_id) |> Enum.uniq() + wanted = MapSet.new(identifiers, &{&1.namespace, &1.external_id}) + + ExternalIdentifier + |> where( + [identifier], + identifier.status == :verified and identifier.namespace in ^namespaces and + identifier.external_id in ^external_ids + ) + |> select([identifier], { + identifier.namespace, + identifier.external_id, + identifier.object_id + }) + |> Repo.all() + |> Enum.filter(fn {namespace, external_id, _object_id} -> + MapSet.member?(wanted, {namespace, external_id}) + end) + |> Map.new(fn {namespace, external_id, object_id} -> + {{namespace, external_id}, object_id} + end) + end + + defp exclusive_proposal_conflicts(identifiers) do + identifiers + |> Enum.filter(& &1.exclusive) + |> Enum.group_by(& &1.namespace, & &1.external_id) + |> Enum.flat_map(fn {namespace, external_ids} -> + case Enum.uniq(external_ids) do + [_one] -> [] + many -> ["#{namespace}:#{Enum.join(Enum.sort(many), ",")}"] + end + end) + end + + defp compatibility_conflicts(entry, object_id) do + object = Repo.one(from object in Object, where: object.id == ^object_id, lock: "FOR UPDATE") + + entity = + Repo.one(from entity in Entity, where: entity.object_id == ^object_id, lock: "FOR UPDATE") + + kind_conflicts = + cond do + is_nil(object) or object.kind != :entity or is_nil(entity) -> ["object_kind"] + entity.entity_kind == entry.entity_kind -> [] + entity.entity_kind == :concept -> [] + true -> ["entity_kind:#{entity.entity_kind}:#{entry.entity_kind}"] + end + + subtype_conflicts = + case {entry.entity_kind, Repo.get(WorkDetails, object_id)} do + {:work, %WorkDetails{work_kind: held}} + when is_binary(held) and is_binary(entry.work_kind) and held != entry.work_kind -> + ["work_kind:#{held}:#{entry.work_kind}"] + + _ -> + [] + end + + exclusive = Enum.filter(entry.identifiers, & &1.exclusive) + namespaces = exclusive |> Enum.map(& &1.namespace) |> Enum.uniq() + + held = + ExternalIdentifier + |> where( + [identifier], + identifier.object_id == ^object_id and identifier.status == :verified and + identifier.namespace in ^namespaces + ) + |> select([identifier], {identifier.namespace, identifier.external_id}) + |> Repo.all() + + expected = MapSet.new(exclusive, &{&1.namespace, &1.external_id}) + + identifier_conflicts = + for {namespace, external_id} <- held, + not MapSet.member?(expected, {namespace, external_id}) do + "#{namespace}:#{external_id}" + end + + kind_conflicts ++ subtype_conflicts ++ identifier_conflicts + end + + defp create_entity(%Entry{entity_kind: :work} = entry) do + Registry.create_work(%{ + preferred_label: entry.label, + description: entry.description, + metadata: evidence_metadata(entry.metadata, entry), + work_kind: entry.work_kind, + original_language: entry.original_language, + first_published_year: entry.year + }) + end + + defp create_entity(%Entry{entity_kind: :person} = entry) do + Registry.create_person(%{ + preferred_label: entry.label, + description: entry.description, + metadata: entry.metadata + }) + end + + defp create_entity(%Entry{} = entry) do + Registry.create_entity(%{ + entity_kind: entry.entity_kind, + preferred_label: entry.label, + description: entry.description, + metadata: entry.metadata + }) + end + + defp enrich_entity(object_id, entry) do + entity = Repo.get!(Entity, object_id) + + target_kind = + if(entity.entity_kind == :concept, do: entry.entity_kind, else: entity.entity_kind) + + # A concept label is a provisional encyclopedia title (often carrying a + # disambiguation suffix). When exact identifiers sharpen that object into a + # typed work, the typed adapter's label becomes the display title. The + # object id remains unchanged, so the former slug still redirects safely. + preferred_label = + if entity.entity_kind == :concept and target_kind != :concept, + do: first_present(entry.label, entity.preferred_label), + else: first_present(entity.preferred_label, entry.label) + + entity = + entity + |> Entity.changeset(%{ + entity_kind: target_kind, + preferred_label: preferred_label, + description: first_present(entity.description, entry.description), + metadata: evidence_metadata(fill_missing(entity.metadata, entry.metadata), entry) + }) + |> Repo.update!() + + case target_kind do + :work -> ensure_work_details(object_id, entry) + :person -> ensure_person_details(object_id) + _ -> :ok + end + + {:ok, entity} + end + + defp ensure_work_details(object_id, entry) do + attrs = %{ + entity_id: object_id, + work_kind: entry.work_kind, + original_language: entry.original_language, + first_published_year: entry.year + } + + case Repo.get(WorkDetails, object_id) do + nil -> + %WorkDetails{} |> WorkDetails.changeset(attrs) |> Repo.insert!() + + details -> + details + |> WorkDetails.changeset(%{ + work_kind: first_present(details.work_kind, entry.work_kind), + original_language: first_present(details.original_language, entry.original_language), + first_published_year: details.first_published_year || entry.year + }) + |> Repo.update!() + end + end + + defp ensure_person_details(object_id) do + if is_nil(Repo.get(PersonDetails, object_id)) do + %PersonDetails{} + |> PersonDetails.changeset(%{entity_id: object_id}) + |> Repo.insert!() + end + end + + defp ensure_identifiers(object_id, entry) do + Enum.each(entry.identifiers, fn identifier -> + case Repo.get_by(ExternalIdentifier, + namespace: identifier.namespace, + external_id: identifier.external_id, + status: :verified + ) do + nil -> + {:ok, _identifier} = + Registry.add_external_id(object_id, identifier.namespace, identifier.external_id, %{ + metadata: + Map.merge(identifier.metadata, %{ + "asserted_by" => entry.source_slug, + "identity_evidence" => "exact_source_identifier" + }), + source_record_revision_id: entry.source_record_revision_id + }) + + %ExternalIdentifier{object_id: ^object_id} = held -> + held + |> ExternalIdentifier.changeset(%{ + metadata: Map.merge(held.metadata, identifier.metadata), + source_record_revision_id: + held.source_record_revision_id || entry.source_record_revision_id + }) + |> Repo.update!() + end + end) + end + + defp attach_source_record(_object_id, %Entry{source_record_id: nil}), do: :ok + + defp attach_source_record(object_id, entry) do + stable = entry.stable_identifier + now = DateTime.utc_now() + + attrs = %{ + source_record_id: entry.source_record_id, + output_role: "entity", + output_key: "source_identity:#{stable.namespace}:#{stable.external_id}", + output_object_id: object_id, + last_seen_run_id: entry.import_run_id, + retired_at: nil, + inserted_at: now, + updated_at: now + } + + %MaterializedOutput{} + |> MaterializedOutput.changeset(attrs) + |> Repo.insert!( + on_conflict: {:replace, [:output_object_id, :last_seen_run_id, :retired_at, :updated_at]}, + conflict_target: [:source_record_id, :output_role, :output_key] + ) + end + + defp conflict(entry, matches, reason, details \\ []) do + payload = %{ + "reason" => reason, + "details" => details, + "identifiers" => + Enum.map( + entry.identifiers, + &%{"namespace" => &1.namespace, "external_id" => &1.external_id} + ), + "matched_objects" => + Enum.map(matches, fn {{namespace, external_id}, object_id} -> + %{"namespace" => namespace, "external_id" => external_id, "object_id" => object_id} + end) + } + + object_id = matches |> Map.values() |> Enum.uniq() |> List.first() + now = DateTime.utc_now() + + attrs = %{ + source_id: entry.source_id, + source_record_id: entry.source_record_id, + kind: @conflict_kind, + object_id: object_id, + payload: payload, + status: :open + } + + case open_conflict(entry.source_record_id) do + nil -> + %ReconciliationCase{} + |> ReconciliationCase.changeset(attrs) + |> Repo.insert!() + + existing -> + existing + |> ReconciliationCase.changeset(%{object_id: object_id, payload: payload}) + |> Ecto.Changeset.change(updated_at: now) + |> Repo.update!() + end + |> then(fn kase -> + %Resolution{ + state: :conflicting_identifiers, + conflict_id: kase.id, + reason: reason, + identifiers: entry.identifiers + } + end) + end + + defp open_conflict(nil), do: nil + + defp open_conflict(source_record_id) do + Repo.one( + from kase in ReconciliationCase, + where: + kase.source_record_id == ^source_record_id and kase.kind == ^@conflict_kind and + kase.status == :open, + lock: "FOR UPDATE" + ) + end + + defp matched(entry, object_id) do + %Resolution{state: :matched, object_id: object_id, identifiers: entry.identifiers} + end + + defp created(entry, object_id) do + %Resolution{state: :newly_created, object_id: object_id, identifiers: entry.identifiers} + end + + defp evidence_metadata(metadata, entry) do + if entry.source_record_id do + evidence = metadata["source_identity_evidence"] || %{} + + evidence = + Enum.reduce(["image_url"], evidence, fn field, acc -> + if present?(entry.metadata[field]) and metadata[field] == entry.metadata[field], + do: Map.put(acc, field, entry.source_record_id), + else: acc + end) + + Map.put(metadata, "source_identity_evidence", evidence) + else + metadata + end + end + + defp fill_missing(held, offered) do + Map.merge(offered || %{}, held || %{}, fn _key, offered_value, held_value -> + if present?(held_value), do: held_value, else: offered_value + end) + end + + defp first_present(held, offered), do: if(present?(held), do: held, else: offered) + defp present?(value) when is_binary(value), do: String.trim(value) != "" + defp present?(nil), do: false + defp present?(value), do: value != %{} and value != [] +end diff --git a/lib/devils_dictionary/source_identity/adapter.ex b/lib/devils_dictionary/source_identity/adapter.ex new file mode 100644 index 00000000..44527636 --- /dev/null +++ b/lib/devils_dictionary/source_identity/adapter.ex @@ -0,0 +1,20 @@ +defmodule DevilsDictionary.SourceIdentity.Adapter do + @moduledoc """ + Contract for turning a provider record into a durable identity proposal. + + Adapters keep provider parsing at the edge and return a validated + `DevilsDictionary.SourceIdentity.Entry`. The entry names the registry kind + and subtype, a stable source identifier, any exact cross-source identifiers, + source-supported display facts, creator/author references, eligibility and + retention policy. The resolver owns matching and creation; adapters never + merge by title. + + Film adapters are supported first. Artwork and quotation adapters use this + same shape, but provider ingestion for those media is deliberately outside + issue #93. + """ + + alias DevilsDictionary.SourceIdentity.Entry + + @callback identity_record(map()) :: {:ok, Entry.t()} | :ignore | {:error, term()} +end diff --git a/lib/devils_dictionary/source_identity/backfill.ex b/lib/devils_dictionary/source_identity/backfill.ex new file mode 100644 index 00000000..26626fc7 --- /dev/null +++ b/lib/devils_dictionary/source_identity/backfill.ex @@ -0,0 +1,212 @@ +defmodule DevilsDictionary.SourceIdentity.Backfill do + @moduledoc """ + Bounded, resumable reconciliation for durable CineGraph film identities. + + Old discovery rows are rebuilt from their retained preview and source record. + The same `SourceIdentity` resolver used during live ingestion does the work, + so a rerun is idempotent and conflicts enter the normal reconciliation queue. + This backfill never calls a remote API and never deletes or merges objects. + """ + + import Ecto.Query + + alias DevilsDictionary.Corpus.SourceRecordRevision + alias DevilsDictionary.Discovery.Result + alias DevilsDictionary.Discovery.Providers.CineGraph + alias DevilsDictionary.Repo + alias DevilsDictionary.SourceIdentity + alias DevilsDictionary.SourceIdentity.Resolution + alias DevilsDictionary.Sources + + @default_limit 100 + @max_limit 1_000 + + @doc "Reconciles one deterministic batch and returns per-state counts plus a checkpoint." + def run(opts \\ []) do + limit = opts |> Keyword.get(:limit, @default_limit) |> bounded_limit!() + after_id = Keyword.get(opts, :after_id) + + results = + Result + |> join(:inner, [result], run in assoc(result, :run)) + |> join(:inner, [_result, run], mapping in assoc(run, :mapping)) + |> join(:inner, [_result, _run, mapping], source in assoc(mapping, :source)) + |> where([_result, _run, _mapping, source], source.slug == "cinegraph") + |> after_result(after_id) + |> order_by([result], asc: result.id) + |> limit(^limit) + |> preload([_result, run, mapping, _source], run: {run, mapping: {mapping, :source}}) + |> preload(:source_record) + |> Repo.all() + + summary = + Enum.reduce(results, empty_summary(), fn result, summary -> + resolution = reconcile_result(result) + + summary + |> Map.update!(:scanned, &(&1 + 1)) + |> Map.update!(resolution.state, &(&1 + 1)) + end) + + Map.put(summary, :next_after, if(length(results) == limit, do: List.last(results).id)) + end + + @doc "Refreshes legacy Wikidata film records through bounded, deduplicated Oban jobs." + def wikidata(opts \\ []) do + limit = opts |> Keyword.get(:limit, @default_limit) |> bounded_limit!() + + records = + DevilsDictionary.Sources.SourceRecord + |> join(:inner, [record], source in assoc(record, :source)) + |> where( + [record, source], + source.slug == "wikidata" and source.active and record.display_allowed + ) + |> after_result(Keyword.get(opts, :after_id)) + |> order_by([record], asc: record.id) + |> limit(^limit) + |> Repo.all() + + summary = + Enum.reduce( + records, + %{scanned: 0, queued: 0, current: 0, skipped: 0, failed: 0}, + fn record, acc -> + state = wikidata_record_state(record) + + acc |> Map.update!(:scanned, &(&1 + 1)) |> Map.update!(state, &(&1 + 1)) + end + ) + + Map.put(summary, :next_after, if(length(records) == limit, do: List.last(records).id)) + end + + defp reconcile_result(result) do + source = result.run.mapping.source + record = ensure_record(result, source) + raw = Sources.raw(record) || %{} + + item = %{ + external_namespace: result.external_namespace, + external_id: result.external_id, + identifiers: + raw["identifiers"] || + [%{namespace: result.external_namespace, external_id: result.external_id}], + preview_metadata: result.preview_metadata, + match_details: result.match_details + } + + resolution = + case CineGraph.identity_record(item) do + {:ok, entry} -> + SourceIdentity.resolve(%{ + entry + | source_id: source.id, + source_record_id: record.id, + source_record_revision_id: current_revision_id(record) + }) + + {:error, reason} -> + %Resolution{state: :insufficient_evidence, reason: to_string(reason)} + end + + result + |> Result.changeset(%{ + object_id: resolution.object_id || result.object_id, + source_record_id: record.id, + resolution_state: resolution.state + }) + |> Repo.update!() + + resolution + end + + defp wikidata_record_state(record) do + raw = Sources.raw(record) || %{} + adapter = DevilsDictionary.Absorb.Sources.Wikidata + + with {:ok, materialized} <- adapter.materialize(%{record | raw: raw}) do + concept = materialized |> Map.get(:concepts, []) |> List.first() || %{} + + cond do + concept[:work_kind] != "film" -> + :skipped + + raw["_film_identity_version"] == 1 -> + # Replay even an already-materialized record to register its crosswalks. + case DevilsDictionary.Absorb.Materializer.run(%{record | raw: raw}, adapter) do + {:ok, _} -> :current + _ -> :failed + end + + true -> + %{"source" => "wikidata", "target" => record.external_id} + |> DevilsDictionary.Workers.EnrichWorker.new( + unique: [ + period: :infinity, + fields: [:worker, :args], + states: [:available, :scheduled, :executing, :retryable] + ] + ) + |> Oban.insert!() + + :queued + end + else + _ -> :failed + end + rescue + _error -> :failed + end + + defp ensure_record(%Result{source_record: nil} = result, source) do + {:ok, record} = + Sources.upsert_record(source, %{ + external_id: "#{result.external_namespace}:#{result.external_id}", + url: result.preview_metadata["source_url"], + raw: %{ + "external_namespace" => result.external_namespace, + "external_id" => result.external_id, + "preview_metadata" => result.preview_metadata, + "match_details" => result.match_details + } + }) + + record + end + + defp ensure_record(%Result{source_record: record}, _source), do: record + + defp current_revision_id(record) do + Repo.one!( + from revision in SourceRecordRevision, + where: + revision.source_record_id == ^record.id and + revision.revision_key == ^record.content_hash, + select: revision.id + ) + end + + defp after_result(query, nil), do: query + + defp after_result(query, id) when is_integer(id) and id > 0, + do: where(query, [result], result.id > ^id) + + defp after_result(_query, _id), do: raise(ArgumentError, "after_id must be a positive integer") + + defp bounded_limit!(limit) when is_integer(limit) and limit in 1..@max_limit, do: limit + + defp bounded_limit!(_limit), + do: raise(ArgumentError, "limit must be between 1 and #{@max_limit}") + + defp empty_summary do + %{ + scanned: 0, + matched: 0, + newly_created: 0, + insufficient_evidence: 0, + conflicting_identifiers: 0, + next_after: nil + } + end +end diff --git a/lib/devils_dictionary/source_identity/display.ex b/lib/devils_dictionary/source_identity/display.ex new file mode 100644 index 00000000..62e4f3d0 --- /dev/null +++ b/lib/devils_dictionary/source_identity/display.ex @@ -0,0 +1,79 @@ +defmodule DevilsDictionary.SourceIdentity.Display do + @moduledoc "Applies current source visibility to images copied onto durable identities." + import Ecto.Query + alias DevilsDictionary.Repo + alias DevilsDictionary.Registry.Entity + alias DevilsDictionary.Corpus.SourceRecordRevision + alias DevilsDictionary.Sources.{MaterializedOutput, SourceRecord} + + @doc "Loads image visibility evidence for a collection in one query." + def preload(entities) when is_list(entities) do + object_ids = + entities + |> Enum.filter(&image_entity?/1) + |> Enum.map(& &1.object_id) + |> Enum.uniq() + + if object_ids == [] do + %{} + else + MaterializedOutput + |> join(:inner, [output], record in SourceRecord, on: record.id == output.source_record_id) + |> join(:inner, [_output, record], source in assoc(record, :source)) + |> join(:left, [_output, record, _source], revision in SourceRecordRevision, + on: + revision.source_record_id == record.id and + revision.revision_key == record.content_hash + ) + |> where( + [output], + output.output_object_id in ^object_ids and + like(output.output_key, "source_identity:%") + ) + |> select([output, record, source, revision], %{ + object_id: output.output_object_id, + record_id: record.id, + allowed: record.display_allowed, + active: source.active, + retired_at: output.retired_at, + raw: revision.payload + }) + |> Repo.all() + |> Enum.group_by(& &1.object_id) + end + end + + @doc "Returns an image only while its retained source evidence remains displayable." + def image_url(%Entity{} = entity), do: image_url(entity, preload([entity])) + def image_url(_entity), do: nil + + def image_url(%{metadata: %{"image_url" => image}} = entity, evidence) + when is_binary(image) and is_map(evidence) do + record_id = get_in(entity.metadata, ["source_identity_evidence", "image_url"]) + outputs = Map.get(evidence, entity.object_id, []) + + cond do + is_integer(record_id) -> + if Enum.any?(outputs, &(&1.record_id == record_id and allowed?(&1))), do: image + + outputs == [] -> + image + + true -> + # Legacy projections have no field provenance. Only retain a poster + # when an allowed record independently supplies that exact URL. + if Enum.any?(outputs, fn output -> + allowed?(output) and + get_in(output.raw || %{}, ["preview_metadata", "poster_url"]) == image + end), + do: image + end + end + + def image_url(_entity, _evidence), do: nil + + defp allowed?(output), do: output.allowed and output.active and is_nil(output.retired_at) + + defp image_entity?(%{metadata: %{"image_url" => image}}), do: is_binary(image) + defp image_entity?(_entity), do: false +end diff --git a/lib/devils_dictionary/source_identity/entry.ex b/lib/devils_dictionary/source_identity/entry.ex new file mode 100644 index 00000000..b244ece0 --- /dev/null +++ b/lib/devils_dictionary/source_identity/entry.ex @@ -0,0 +1,188 @@ +defmodule DevilsDictionary.SourceIdentity.Entry do + @moduledoc """ + A provider-neutral proposal for one encyclopedia identity. + + `stable_identifier` is the provider's durable key and `identifiers` contains + exact, namespaced crosswalks. Titles and dates are display facts only: they + are never consulted for identity resolution. An eligible durable entity must + have a credible source, stable identifier, supported subtype and label. + + Relationships are references, not embedded identities. A quotation can + therefore remain ineligible while its independently identified author is + resolved, and a reproduction can point at an artwork without becoming the + artwork. + """ + + @entity_kinds DevilsDictionary.Registry.Entity.kinds() + + @type external_identifier :: %{ + required(:namespace) => String.t(), + required(:external_id) => String.t(), + optional(:metadata) => map(), + optional(:exclusive) => boolean() + } + + @type t :: %__MODULE__{} + + defstruct source_slug: nil, + source_id: nil, + source_record_id: nil, + source_record_revision_id: nil, + import_run_id: nil, + object_kind: :entity, + entity_kind: nil, + work_kind: nil, + content_kind: nil, + stable_identifier: nil, + identifiers: [], + label: nil, + description: nil, + year: nil, + original_language: nil, + metadata: %{}, + eligibility: :eligible, + eligibility_reason: nil, + retention: :durable, + relationships: [] + + @doc "Builds and validates one adapter proposal." + def new(attrs) when is_map(attrs) or is_list(attrs) do + entry = struct(__MODULE__, attrs) + + with {:ok, stable} <- normalize_optional_identifier(entry.stable_identifier), + {:ok, identifiers} <- normalize_identifiers(entry.identifiers, stable), + :ok <- validate_shape(entry, stable, identifiers), + :ok <- validate_relationships(entry.relationships) do + {:ok, %{entry | stable_identifier: stable, identifiers: identifiers}} + end + end + + def new(_attrs), do: {:error, :invalid_entry} + + defp validate_shape(entry, stable, identifiers) do + cond do + entry.object_kind not in [:entity, :content] -> + {:error, :unsupported_object_kind} + + entry.object_kind == :entity and entry.entity_kind not in @entity_kinds -> + {:error, :unsupported_entity_kind} + + entry.object_kind == :entity and entry.entity_kind == :work and + not present?(entry.work_kind) -> + {:error, :work_kind_required} + + entry.eligibility not in [:eligible, :insufficient_evidence, :restricted] -> + {:error, :invalid_eligibility} + + entry.retention not in [:durable, :transient] -> + {:error, :invalid_retention} + + entry.eligibility == :eligible and entry.retention != :durable -> + {:error, :eligible_entry_must_be_durable} + + entry.eligibility == :eligible and not present?(entry.source_slug) -> + {:error, :source_required} + + entry.eligibility == :eligible and is_nil(stable) -> + {:error, :stable_identifier_required} + + entry.eligibility == :eligible and not present?(entry.label) -> + {:error, :label_required} + + stable && stable not in identifiers -> + {:error, :stable_identifier_not_supported} + + true -> + :ok + end + end + + defp normalize_identifiers(identifiers, stable) when is_list(identifiers) do + identifiers = if stable, do: [stable | identifiers], else: identifiers + + identifiers + |> Enum.reduce_while({:ok, []}, fn identifier, {:ok, acc} -> + case normalize_identifier(identifier) do + {:ok, normalized} -> {:cont, {:ok, [normalized | acc]}} + error -> {:halt, error} + end + end) + |> case do + {:ok, normalized} -> + {:ok, + normalized + |> Enum.reverse() + |> Enum.uniq_by(&{&1.namespace, &1.external_id})} + + error -> + error + end + end + + defp normalize_identifiers(_, _stable), do: {:error, :invalid_identifiers} + + defp normalize_optional_identifier(nil), do: {:ok, nil} + defp normalize_optional_identifier(identifier), do: normalize_identifier(identifier) + + defp normalize_identifier({namespace, external_id}), + do: normalize_identifier(%{namespace: namespace, external_id: external_id}) + + defp normalize_identifier(%{} = identifier) do + namespace = identifier[:namespace] || identifier["namespace"] + external_id = identifier[:external_id] || identifier["external_id"] + metadata = identifier[:metadata] || identifier["metadata"] || %{} + exclusive = Map.get(identifier, :exclusive, Map.get(identifier, "exclusive", true)) + + cond do + not present?(namespace) or not Regex.match?(~r/\A[a-z0-9_]{2,80}\z/, namespace) -> + {:error, :invalid_identifier_namespace} + + not present?(external_id) or byte_size(external_id) > 255 -> + {:error, :invalid_external_identifier} + + not is_map(metadata) or not is_boolean(exclusive) -> + {:error, :invalid_identifier_metadata} + + true -> + {:ok, + %{ + namespace: namespace, + external_id: external_id, + metadata: metadata, + exclusive: exclusive + }} + end + end + + defp normalize_identifier(_identifier), do: {:error, :invalid_identifier} + + defp validate_relationships(relationships) when is_list(relationships) do + if Enum.all?(relationships, &valid_relationship?/1), + do: :ok, + else: {:error, :invalid_relationship} + end + + defp validate_relationships(_relationships), do: {:error, :invalid_relationships} + + defp valid_relationship?(%{} = relationship) do + role = relationship[:role] || relationship["role"] + target_object_id = relationship[:target_object_id] || relationship["target_object_id"] + target_identifiers = relationship[:target_identifiers] || relationship["target_identifiers"] + certainty = relationship[:certainty] || relationship["certainty"] + + present?(to_string(role || "")) and + certainty in [:verified, :candidate, "verified", "candidate"] and + (is_integer(target_object_id) or valid_target_identifiers?(target_identifiers)) + end + + defp valid_relationship?(_relationship), do: false + + defp valid_target_identifiers?(identifiers) when is_list(identifiers) and identifiers != [] do + Enum.all?(identifiers, fn identifier -> match?({:ok, _}, normalize_identifier(identifier)) end) + end + + defp valid_target_identifiers?(_identifiers), do: false + + defp present?(value) when is_binary(value), do: String.trim(value) != "" + defp present?(_value), do: false +end diff --git a/lib/devils_dictionary/source_identity/resolution.ex b/lib/devils_dictionary/source_identity/resolution.ex new file mode 100644 index 00000000..708badf7 --- /dev/null +++ b/lib/devils_dictionary/source_identity/resolution.ex @@ -0,0 +1,12 @@ +defmodule DevilsDictionary.SourceIdentity.Resolution do + @moduledoc "The durable outcome of resolving one source identity proposal." + + @type state :: :matched | :newly_created | :insufficient_evidence | :conflicting_identifiers + @type t :: %__MODULE__{} + + defstruct state: :insufficient_evidence, + object_id: nil, + conflict_id: nil, + reason: nil, + identifiers: [] +end diff --git a/lib/devils_dictionary_web/components/culture.ex b/lib/devils_dictionary_web/components/culture.ex index e1695e59..dbd6251f 100644 --- a/lib/devils_dictionary_web/components/culture.ex +++ b/lib/devils_dictionary_web/components/culture.ex @@ -3,6 +3,7 @@ defmodule DevilsDictionaryWeb.Culture do use DevilsDictionaryWeb, :html attr :states, :map, required: true + attr :return_path, :string, default: nil def section(assigns) do assigns = @@ -13,11 +14,16 @@ defmodule DevilsDictionaryWeb.Culture do ) ~H""" - <.compact_section :if={@provider_states != []} states={@provider_states} /> + <.compact_section + :if={@provider_states != []} + states={@provider_states} + return_path={@return_path} + /> """ end attr :states, :list, required: true + attr :return_path, :string, default: nil defp compact_section(assigns) do assigns = assign(assigns, :provider_count, length(assigns.states)) @@ -53,7 +59,11 @@ defmodule DevilsDictionaryWeb.Culture do id={"culture-result-#{item.external_namespace}-#{item.external_id}"} class="w-24 shrink-0 snap-start sm:w-28" > - <.culture_thumbnail item={item} type={content_type(state)} /> + <.culture_thumbnail + item={item} + type={content_type(state)} + return_path={@return_path} + />
@@ -88,54 +98,111 @@ defmodule DevilsDictionaryWeb.Culture do attr :item, :map, required: true attr :type, :atom, required: true + attr :return_path, :string, default: nil defp culture_thumbnail(assigns) do assigns = - assign(assigns, :image, thumbnail_url(assigns.item.preview_metadata)) + assigns + |> assign(:image, thumbnail_url(assigns.item.preview_metadata)) + |> assign(:entry_path, entry_path(assigns.item, assigns.return_path)) ~H""" - -
- -
- <.icon name={if(@type == :film, do: "hero-film", else: "hero-photo")} class="size-5" /> -
-
+
+ <.link + :if={@entry_path} + navigate={@entry_path} + id={"culture-entry-image-#{@item.external_id}"} + aria-label={"Open #{@item.preview_metadata["title"]} in Dictionary"} + class="rounded-sm focus-visible:outline-2 focus-visible:outline-offset-2" + > + <.culture_image item={@item} image={@image} type={@type} /> + + + <.culture_image item={@item} image={@image} type={@type} /> +
-

- {@item.preview_metadata["title"]} +

+ <.link + :if={@entry_path} + navigate={@entry_path} + id={"culture-entry-title-#{@item.external_id}"} + class="rounded-sm group-hover:underline focus-visible:outline-2 focus-visible:outline-offset-2" + > + {@item.preview_metadata["title"]} + + + {@item.preview_metadata["title"]} +

-

+

{@item.preview_metadata["year"] || "Year unknown"} + · Film

+ + Source ↗ + +
+
+ """ + end + + attr :item, :map, required: true + attr :image, :string, default: nil + attr :type, :atom, required: true + + defp culture_image(assigns) do + ~H""" +
+ +
+ <.icon name={if(@type == :film, do: "hero-film", else: "hero-photo")} class="size-5" />
- +
""" end + defp entry_path(%{object_id: object_id, preview_metadata: metadata}, return_path) + when is_integer(object_id) do + slug = DevilsDictionary.Claims.Connection.slugify(metadata["title"]) + query = if return_path, do: %{from: return_path}, else: %{} + ~p"/entities/#{object_id}/#{slug}?#{query}" + end + + defp entry_path(_item, _return_path), do: nil + attr :state, :map, required: true defp compact_note(assigns) do diff --git a/lib/devils_dictionary_web/live/entity_live.ex b/lib/devils_dictionary_web/live/entity_live.ex index dc3dd17b..18d3efae 100644 --- a/lib/devils_dictionary_web/live/entity_live.ex +++ b/lib/devils_dictionary_web/live/entity_live.ex @@ -26,7 +26,7 @@ defmodule DevilsDictionaryWeb.EntityLive do @impl true def mount(_params, _session, socket), - do: {:ok, assign(socket, page: nil, id: nil, cursors: %{}, entity_slug: nil)} + do: {:ok, assign(socket, page: nil, id: nil, cursors: %{}, entity_slug: nil, back_path: nil)} @impl true def handle_params(%{"id" => id, "slug" => slug} = params, _uri, socket) do @@ -38,6 +38,7 @@ defmodule DevilsDictionaryWeb.EntityLive do defp load(socket, object_id, slug, params) do cursors = cursor_params(params) + back_path = safe_back_path(params["from"]) case EntityPage.build(object_id, page_opts(cursors)) do nil -> @@ -53,11 +54,14 @@ defmodule DevilsDictionaryWeb.EntityLive do |> assign(:id, object_id) |> assign(:cursors, cursors) |> assign(:entity_slug, canonical) + |> assign(:back_path, back_path) |> assign(:page_title, page.entity.label)} else # The slug is cosmetic, so a wrong one is not an error — it is a # redirect to the readable form of the identity that was asked for. - {:noreply, push_navigate(socket, to: ~p"/entities/#{object_id}/#{canonical}")} + query = if back_path, do: %{from: back_path}, else: %{} + + {:noreply, push_navigate(socket, to: ~p"/entities/#{object_id}/#{canonical}?#{query}")} end end end @@ -84,6 +88,15 @@ defmodule DevilsDictionaryWeb.EntityLive do <.a navigate={~p"/"} class="mt-6">Start somewhere else
<% else %> + <.a + :if={@back_path} + id="entity-back-link" + navigate={@back_path} + class="mb-7 inline-flex items-center gap-2 text-sm text-mist-500 transition-colors hover:text-mist-950 dark:hover:text-white" + > + <.icon name="hero-arrow-left" class="size-4 stroke-current" /> Back to definition + +
-
- <.eyebrow>{@page.entity.kind} - <.heading>{@page.entity.label} - <.text :if={@page.entity.description} class="mt-2"> - {@page.entity.description} - -

+

- {detail_line(@page.details)} -

-

- <.a href={"https://www.wikidata.org/wiki/#{@page.entity.qid}"}> - {@page.entity.qid} ↗ - -

+ {"Poster +
+ <.icon name="hero-film" class="size-7 stroke-current" /> +
+
+
+ <.eyebrow>{@page.entity.kind} + <.heading>{@page.entity.label} +

+ {detail_line(@page.details)} +

+ <.text :if={@page.entity.description} class="mt-3 max-w-2xl"> + {@page.entity.description} + +
+ <.panel + :if={@page.meaning_connections != []} + id="entity-meaning-connections" + label="connected meanings" + count={@page.pagination.meaning_connections.count} + > +
    +
  • + <.a :if={claim.path} navigate={claim.path} class="font-medium">{claim.label} + {claim.label} +
    +

    + {claim.detail} +

    +

    + {review_label(claim.review_state)} · {claim.rationale} + <.a + navigate={~p"/connections/#{claim.assertion_id}"} + aria-label="Inspect meaning connection" + class="ml-1 inline-flex align-text-bottom text-mist-400 hover:text-mist-950 dark:hover:text-white" + > + <.icon name="hero-information-circle" class="size-4 stroke-current" /> + +

    +
    +
  • +
+ <.pager + :if={@page.pagination.meaning_connections.next} + id="more-meaning-connections" + label="More connected meanings" + path={ + next_path( + @id, + @entity_slug, + @cursors, + :meaning_connections, + @page.pagination.meaning_connections.next, + @back_path + ) + } + /> + + + <.panel + :if={@page.discovery_appearances != []} + id="entity-discovery-appearances" + label="also appeared in discovery for" + count={@page.pagination.discovery_appearances.count} + > +
    +
  • + <.a :if={appearance.path} navigate={appearance.path} class="font-medium"> + {appearance.label} + + {appearance.term} + + Automatic match · {appearance.provider} + +
  • +
+ <.pager + :if={@page.pagination.discovery_appearances.next} + id="discovery-appearances-next" + label="More discovery appearances" + path={ + next_path( + @id, + @entity_slug, + @cursors, + :discovery_appearances, + @page.pagination.discovery_appearances.next, + @back_path + ) + } + /> + + <.panel :if={@page.biography != []} id="entity-biography" - label="biography" + label={if(@page.entity.kind == :person, do: "biography", else: "about")} count={@page.pagination.biography.count} >
@@ -195,7 +321,16 @@ defmodule DevilsDictionaryWeb.EntityLive do <.pager :if={@page.pagination.works.next} id="works-next" - path={next_path(@id, @entity_slug, @cursors, :works, @page.pagination.works.next)} + path={ + next_path( + @id, + @entity_slug, + @cursors, + :works, + @page.pagination.works.next, + @back_path + ) + } label="More works" /> @@ -247,7 +382,8 @@ defmodule DevilsDictionaryWeb.EntityLive do @entity_slug, @cursors, :definitions, - @page.pagination.definitions.next + @page.pagination.definitions.next, + @back_path ) } label="More definitions" @@ -270,7 +406,16 @@ defmodule DevilsDictionaryWeb.EntityLive do <.pager :if={@page.pagination.editions.next} id="editions-next" - path={next_path(@id, @entity_slug, @cursors, :editions, @page.pagination.editions.next)} + path={ + next_path( + @id, + @entity_slug, + @cursors, + :editions, + @page.pagination.editions.next, + @back_path + ) + } label="More editions" /> @@ -295,7 +440,16 @@ defmodule DevilsDictionaryWeb.EntityLive do <.pager :if={@page.pagination.contents.next} id="contents-next" - path={next_path(@id, @entity_slug, @cursors, :contents, @page.pagination.contents.next)} + path={ + next_path( + @id, + @entity_slug, + @cursors, + :contents, + @page.pagination.contents.next, + @back_path + ) + } label="More contents" /> @@ -350,7 +504,8 @@ defmodule DevilsDictionaryWeb.EntityLive do @entity_slug, @cursors, :connections_in, - @page.pagination.connections_in.next + @page.pagination.connections_in.next, + @back_path ) } label="More incoming connections" @@ -364,13 +519,24 @@ defmodule DevilsDictionaryWeb.EntityLive do @entity_slug, @cursors, :connections_out, - @page.pagination.connections_out.next + @page.pagination.connections_out.next, + @back_path ) } label="More outgoing connections" /> + + <.panel :if={@page.sources != []} id="entity-sources" label="sources"> +
    +
  • + <.a href={source.url} target="_blank" rel="noreferrer"> + {source.name} ↗ + +
  • +
+ <% end %> @@ -411,17 +577,13 @@ defmodule DevilsDictionaryWeb.EntityLive do """ end - @cursor_keys ~w(biography works definitions editions contents connections_in connections_out)a + @cursor_keys ~w(biography works definitions editions contents connections_in connections_out meaning_connections discovery_appearances)a defp cursor_params(params) do Map.new(@cursor_keys, fn key -> param = "#{key}_after" - value = - case Integer.parse(params[param] || "") do - {cursor, ""} when cursor > 0 -> cursor - _ -> nil - end + value = cursor_value(key, params[param]) {key, value} end) @@ -430,21 +592,66 @@ defmodule DevilsDictionaryWeb.EntityLive do defp page_opts(cursors), do: Enum.map(cursors, fn {key, value} -> {String.to_atom("#{key}_after"), value} end) - defp next_path(id, slug, cursors, section, cursor) do + defp cursor_value(:discovery_appearances, value) when is_binary(value) do + with [target, source_slug] <- String.split(value, ":", parts: 2), + {target_object_id, ""} when target_object_id > 0 <- Integer.parse(target), + true <- Regex.match?(~r/\A[a-z0-9_-]+\z/, source_slug) do + value + else + _ -> nil + end + end + + defp cursor_value(_key, value) when is_binary(value) do + case Integer.parse(value) do + {cursor, ""} when cursor > 0 -> cursor + _ -> nil + end + end + + defp cursor_value(_key, _value), do: nil + + defp next_path(id, slug, cursors, section, cursor, back_path) do query = cursors |> Map.put(section, cursor) |> Enum.reject(fn {_key, value} -> is_nil(value) end) |> Map.new(fn {key, value} -> {"#{key}_after", value} end) + |> then(fn query -> if back_path, do: Map.put(query, "from", back_path), else: query end) ~p"/entities/#{id}/#{slug}?#{query}" end + defp safe_back_path(path) when is_binary(path) do + uri = URI.parse(path) + + if is_nil(uri.scheme) and is_nil(uri.host) and + (String.starts_with?(uri.path || "", "/words/") or + String.starts_with?(uri.path || "", "/define/")), + do: path, + else: nil + end + + defp safe_back_path(_path), do: nil + + defp review_label(:accepted), do: "Reviewed connection" + defp review_label(:disputed), do: "Disputed connection" + defp review_label(:changed_since_review), do: "Changed since review" + defp review_label(_state), do: "Awaiting review" + defp detail_line(details) do - details - |> Enum.reject(fn {_k, v} -> is_nil(v) end) - |> Enum.map_join(" · ", fn {key, value} -> - "#{key |> to_string() |> String.replace("_", " ")}: #{value}" - end) + case details do + %{work_kind: work_kind} when is_binary(work_kind) -> + [String.capitalize(work_kind), details[:first_published_year]] + |> Enum.reject(&is_nil/1) + |> Enum.join(" · ") + + _ -> + details + |> Enum.reject(fn {_k, v} -> is_nil(v) end) + |> Enum.map_join(" · ", fn {key, value} -> + "#{key |> to_string() |> String.replace("_", " ")}: #{value}" + end) + end end end diff --git a/lib/devils_dictionary_web/live/word_live.ex b/lib/devils_dictionary_web/live/word_live.ex index c7eecbfd..843922af 100644 --- a/lib/devils_dictionary_web/live/word_live.ex +++ b/lib/devils_dictionary_web/live/word_live.ex @@ -344,6 +344,11 @@ defmodule DevilsDictionaryWeb.WordLive do defp title(%{headword: %{lemma: nil}}, slug), do: "#{slug} — no such word" defp title(%{headword: %{lemma: lemma}}, _slug), do: lemma + defp word_path(%{headword: %{lemma: lemma, lexemes: [lexeme | _]}}), + do: ~p"/words/#{lexeme.id}/#{DevilsDictionary.Registry.Lexeme.slug(lemma)}" + + defp word_path(_page), do: nil + @impl true def render(assigns) do ~H""" @@ -381,6 +386,7 @@ defmodule DevilsDictionaryWeb.WordLive do diff --git a/lib/mix/tasks/dd.films.backfill.ex b/lib/mix/tasks/dd.films.backfill.ex new file mode 100644 index 00000000..107eda9b --- /dev/null +++ b/lib/mix/tasks/dd.films.backfill.ex @@ -0,0 +1,46 @@ +defmodule Mix.Tasks.Dd.Films.Backfill do + @shortdoc "Backfills durable film identities from retained CineGraph results" + + @moduledoc """ + Reconciles a bounded, stable-id CineGraph batch locally. The Wikidata phase + queues legacy film refreshes through the existing enrichment worker: + + mix dd.films.backfill --wikidata --limit 100 + + + mix dd.films.backfill --limit 100 + mix dd.films.backfill --limit 100 --after 4200 + + The report separates matched, newly created, insufficient and conflicting + records. The Wikidata phase also reports malformed records as failed without + stopping its batch. Pass the reported `--after` checkpoint to continue. + Rerunning a batch is safe: exact identifiers converge through the shared resolver. + """ + + use Mix.Task + + alias DevilsDictionary.SourceIdentity.Backfill + + @switches [limit: :integer, after: :integer, wikidata: :boolean] + + @impl Mix.Task + def run(args) do + Mix.Task.run("app.start") + {opts, rest, invalid} = OptionParser.parse(args, strict: @switches) + + if rest != [] or invalid != [], + do: Mix.raise("invalid arguments; run `mix help dd.films.backfill`") + + options = [limit: opts[:limit] || 100, after_id: opts[:after]] + summary = if opts[:wikidata], do: Backfill.wikidata(options), else: Backfill.run(options) + Mix.shell().info("Film identity backfill: " <> inspect(Map.delete(summary, :next_after))) + + if summary.next_after do + Mix.shell().info("Next checkpoint: --after #{summary.next_after}") + else + Mix.shell().info("Batch complete: no continuation checkpoint.") + end + rescue + error in ArgumentError -> Mix.raise(Exception.message(error)) + end +end diff --git a/priv/repo/migrations/20260912164611_add_discovery_identity_resolution.exs b/priv/repo/migrations/20260912164611_add_discovery_identity_resolution.exs new file mode 100644 index 00000000..2c49aa93 --- /dev/null +++ b/priv/repo/migrations/20260912164611_add_discovery_identity_resolution.exs @@ -0,0 +1,25 @@ +defmodule DevilsDictionary.Repo.Migrations.AddDiscoveryIdentityResolution do + use Ecto.Migration + + def change do + alter table(:discovery_results) do + add :resolution_state, :string, null: false, default: "insufficient_evidence" + end + + create constraint(:discovery_results, :discovery_results_resolution_state, + check: + "resolution_state IN ('matched','newly_created','insufficient_evidence','conflicting_identifiers')" + ) + + create index(:discovery_results, [:resolution_state, :id]) + + # One open identity disagreement per source record is enough to put the + # complete assertion set in front of a reviewer. Re-running a discovery or + # backfill refreshes that case instead of filling the queue with copies. + create unique_index(:reconciliation_cases, [:source_record_id, :kind], + where: + "source_record_id IS NOT NULL AND kind = 'external_identifier_conflict' AND status = 'open'", + name: :reconciliation_cases_open_external_identifier_conflict_index + ) + end +end diff --git a/test/devils_dictionary/absorb/sources/wikidata_test.exs b/test/devils_dictionary/absorb/sources/wikidata_test.exs index 5fbfb971..1042977f 100644 --- a/test/devils_dictionary/absorb/sources/wikidata_test.exs +++ b/test/devils_dictionary/absorb/sources/wikidata_test.exs @@ -9,6 +9,7 @@ defmodule DevilsDictionary.Absorb.Sources.WikidataTest do defp records(lemma), do: Fixtures.raw("wikidata", lemma) defp all_records, do: Enum.flat_map(@lemmas, &records/1) defp general_records, do: Fixtures.raw("wikidata", "general_entities") + defp film_records, do: Fixtures.raw("wikidata", "films") defp entity(lemma, qid), do: Enum.find(records(lemma), &(&1["id"] == qid)) @@ -79,6 +80,60 @@ defmodule DevilsDictionary.Absorb.Sources.WikidataTest do end describe "materialize/1" do + test "film records expose exact cross-provider identifiers and release year" do + titanic = film_records() |> Enum.find(&(&1["id"] == "Q44578")) |> out() + assert [concept] = titanic.concepts + + assert concept.kind == :work + assert concept.work_kind == "film" + assert concept.first_published_year == 1997 + + assert MapSet.new(concept.external_identifiers, &{&1.namespace, &1.external_id}) == + MapSet.new([ + {"wikidata", "Q44578"}, + {"tmdb_movie", "597"}, + {"imdb_title", "tt0120338"} + ]) + end + + test "publication years preserve BCE signs" do + raw = Enum.find(film_records(), &(&1["id"] == "Q44578")) + + bce = + put_in( + raw, + ["claims", "P577", Access.at(0), "mainsnak", "datavalue", "value", "time"], + "-0500-01-01T00:00:00Z" + ) + + assert [concept] = out(bce).concepts + assert concept.first_published_year == -500 + end + + test "film crosswalks honor preferred and deprecated Wikidata ranks" do + raw = Enum.find(film_records(), &(&1["id"] == "Q44578")) + + ranked = + put_in(raw, ["claims", "P345"], [ + string_statement("tt0000001", "normal"), + string_statement("tt0120338", "preferred"), + string_statement("tt9999999", "deprecated") + ]) + + assert [concept] = out(ranked).concepts + + assert Enum.filter(concept.external_identifiers, &(&1.namespace == "imdb_title")) == [ + %{ + namespace: "imdb_title", + external_id: "tt0120338", + metadata: %{ + "statement_policy" => "preferred_else_normal", + "wikidata_property" => "P345" + } + } + ] + end + test "projects representative people, works, organizations, places and events" do expected = %{ "Q191050" => :person, @@ -213,4 +268,15 @@ defmodule DevilsDictionary.Absorb.Sources.WikidataTest do assert Wikidata.rate_limit_ms() >= 200 end end + + defp string_statement(value, rank) do + %{ + "rank" => rank, + "type" => "statement", + "mainsnak" => %{ + "snaktype" => "value", + "datavalue" => %{"type" => "string", "value" => value} + } + } + end end diff --git a/test/devils_dictionary/discovery_test.exs b/test/devils_dictionary/discovery_test.exs index d08e8ddb..768a0e46 100644 --- a/test/devils_dictionary/discovery_test.exs +++ b/test/devils_dictionary/discovery_test.exs @@ -54,7 +54,9 @@ defmodule DevilsDictionary.DiscoveryTest do assert [%Result{external_id: "10"} = item] = state.items assert item.match_details["keywords"] == [%{"id" => 273_967, "name" => "war"}] assert item.preview_metadata["title"] == "A title without the query" - assert Repo.aggregate(Object, :count) == object_count + assert item.object_id + assert item.resolution_state == :newly_created + assert Repo.aggregate(Object, :count) == object_count + 1 assert {:cached, %Run{id: cached_id}} = Discovery.request(target(word), "cinegraph") assert cached_id == run.id @@ -268,6 +270,62 @@ defmodule DevilsDictionary.DiscoveryTest do assert Repo.aggregate(Oban.Job, :count) == 1 end + @tag :unboxed + test "concurrent reversed film batches acquire one global identifier lock order", ctx do + first_word = word!(ctx, "lock-order-a", ~w(wordnet)) + second_word = word!(ctx, "lock-order-b", ~w(wordnet)) + first_movie = movie(840, "First lock film") |> Map.put("imdbId", "tt0000840") + second_movie = movie(841, "Second lock film") |> Map.put("imdbId", "tt0000841") + parent = self() + gate = make_ref() + + Req.Test.stub(CineGraph, fn conn -> + body = request_body(conn) + + if String.contains?(body["query"], "searchMovieKeywords") do + term = body["variables"]["query"] + keyword_response(conn, term, if(term == first_word.lemma, do: 840, else: 841)) + else + [keyword_id] = body["variables"]["keywords"] + send(parent, {:discovery_ready, self()}) + + receive do + {:publish, ^gate} -> + movies = + if keyword_id == 840, + do: [first_movie, second_movie], + else: [second_movie, first_movie] + + discovery_response(conn, movies, nil, "lock-order", keyword_id) + end + end + end) + + assert {:queued, first_run} = Discovery.request(target(first_word), "cinegraph") + assert {:queued, second_run} = Discovery.request(target(second_word), "cinegraph") + + tasks = + Enum.map([first_run, second_run], fn run -> + Task.async(fn -> Discovery.execute_run(run.id) end) + end) + + publishers = + for _task <- tasks do + assert_receive {:discovery_ready, publisher}, 5_000 + publisher + end + + Enum.each(publishers, &send(&1, {:publish, gate})) + assert Enum.map(tasks, &Task.await(&1, 10_000)) == [:ok, :ok] + assert Enum.all?([first_run, second_run], &(Repo.get!(Run, &1.id).status == :succeeded)) + + assert Registry.by_external_id("tmdb_movie", "840") == + Registry.by_external_id("imdb_title", "tt0000840") + + assert Registry.by_external_id("tmdb_movie", "841") == + Registry.by_external_id("imdb_title", "tt0000841") + end + test "mapping versions isolate old work and a slow old response cannot replace the new one", ctx do word = word!(ctx, "changeable", ~w(wordnet)) @@ -464,9 +522,19 @@ defmodule DevilsDictionary.DiscoveryTest do select: revision.payload ) - assert [payload] = payloads - assert Map.keys(payload) |> Enum.sort() == ["external_id", "external_namespace"] - assert payload["external_id"] == "10" + assert Enum.map(payloads, & &1["external_id"]) == ["10", "10"] + + assert Enum.map(payloads, &get_in(&1, ["preview_metadata", "title"])) |> Enum.sort() == + ["First title", "Updated title"] + + [object_id] = results |> Enum.map(& &1.object_id) |> Enum.uniq() + + assert Repo.aggregate( + from(c in DevilsDictionary.Claims.AssertionRevision, + where: c.subject_object_id == ^object_id + ), + :count + ) == 0 end test "refresh-due and legacy-expired previews remain usable while withdrawal does not", ctx do diff --git a/test/devils_dictionary/source_identity/backfill_test.exs b/test/devils_dictionary/source_identity/backfill_test.exs new file mode 100644 index 00000000..16db2da5 --- /dev/null +++ b/test/devils_dictionary/source_identity/backfill_test.exs @@ -0,0 +1,167 @@ +defmodule DevilsDictionary.SourceIdentity.BackfillTest do + use DevilsDictionary.DataCase, async: false + use Oban.Testing, repo: DevilsDictionary.Repo + + import DevilsDictionary.WordFixtures + + alias DevilsDictionary.Discovery + alias DevilsDictionary.Discovery.Result + alias DevilsDictionary.Registry.Object + alias DevilsDictionary.Repo + alias DevilsDictionary.SourceIdentity.Backfill + alias DevilsDictionary.Sources + alias DevilsDictionary.Sources.SourceRecord + + setup do + catalog = DevilsDictionary.Fixtures.seed_catalog!() + %{sources: catalog.sources, animals: catalog.scopes["animals"]} + end + + test "legacy CineGraph rows are reconciled in bounded, idempotent batches", ctx do + word = word!(ctx, "legacy-film", ~w(wordnet)) + assert {:queued, run} = Discovery.request(target(word), "cinegraph") + + legacy = + %Result{} + |> Result.changeset(%{ + run_id: run.id, + external_namespace: "tmdb_movie", + external_id: "597", + position: 0, + match_details: %{"query" => "legacy-film", "keywords" => []}, + preview_metadata: %{ + "title" => "Titanic", + "year" => "1997", + "release_date" => "1997-11-18", + "poster_url" => "https://image.tmdb.org/t/p/w342/titanic.jpg", + "source_url" => "https://cinegraph.org/movies/597", + "content_type" => "film" + } + }) + |> Repo.insert!() + + object_count = Repo.aggregate(Object, :count) + legacy_id = legacy.id + + assert %{ + scanned: 1, + newly_created: 1, + matched: 0, + conflicting_identifiers: 0, + next_after: ^legacy_id + } = Backfill.run(limit: 1) + + resolved = Repo.get!(Result, legacy.id) + assert resolved.object_id + assert resolved.resolution_state == :newly_created + assert resolved.source_record_id + assert Repo.aggregate(Object, :count) == object_count + 1 + + assert %{scanned: 1, matched: 1, newly_created: 0} = Backfill.run(limit: 1) + assert Repo.get!(Result, legacy.id).object_id == resolved.object_id + assert Repo.aggregate(Object, :count) == object_count + 1 + + record = Repo.get!(SourceRecord, resolved.source_record_id) + + {:ok, _record} = + Sources.upsert_record(ctx.sources["cinegraph"], %{ + external_id: record.external_id, + url: record.url, + raw: %{ + "identifiers" => [ + %{"namespace" => "tmdb_movie", "external_id" => "597"}, + %{"namespace" => "tmdb_movie", "external_id" => "598"} + ], + "preview_metadata" => legacy.preview_metadata, + "match_details" => legacy.match_details + } + }) + + assert %{scanned: 1, conflicting_identifiers: 1} = Backfill.run(limit: 1) + conflicted = Repo.get!(Result, legacy.id) + assert conflicted.object_id == resolved.object_id + assert conflicted.resolution_state == :conflicting_identifiers + + assert %{scanned: 0, next_after: nil} = + Backfill.run(limit: 1, after_id: legacy.id) + end + + test "invalid bounds are rejected before work begins" do + assert_raise ArgumentError, ~r/limit must be between 1 and 1000/, fn -> + Backfill.run(limit: 0) + end + + assert_raise ArgumentError, ~r/after_id must be a positive integer/, fn -> + Backfill.run(after_id: "1") + end + end + + test "legacy Wikidata films queue a deduplicated refresh and current records replay", ctx do + alias DevilsDictionary.Absorb.Sources.Wikidata + alias DevilsDictionary.Registry + raw = DevilsDictionary.Fixtures.raw("wikidata", "films") |> Enum.find(&(&1["id"] == "Q44578")) + + legacy = + raw + |> Wikidata.trim() + |> Map.delete("_film_identity_version") + |> update_in(["claims"], &Map.drop(&1, ["P345", "P4947"])) + + {:ok, record} = + Sources.upsert_record(ctx.sources["wikidata"], %{external_id: "Q44578", raw: legacy}) + + {:ok, entity} = Registry.create_entity(%{entity_kind: :concept, preferred_label: "Titanic"}) + {:ok, _} = Registry.add_external_id(entity.object_id, "wikidata", "Q44578") + assert %{queued: 1} = Backfill.wikidata(limit: 100) + assert %{queued: 1} = Backfill.wikidata(limit: 100) + assert length(all_enqueued(worker: DevilsDictionary.Workers.EnrichWorker)) == 1 + + Req.Test.stub(DevilsDictionary.Absorb.Clients, fn conn -> + conn + |> Plug.Conn.put_resp_content_type("application/json") + |> Plug.Conn.send_resp(200, Jason.encode!(%{"entities" => %{"Q44578" => raw}})) + end) + + assert :ok = + perform_job( + DevilsDictionary.Workers.EnrichWorker, + %{"source" => "wikidata", "target" => record.external_id} + ) + + assert %{current: 1, queued: 0} = Backfill.wikidata(limit: 100) + assert Registry.by_external_id("tmdb_movie", "597") == entity.object_id + assert Registry.by_external_id("imdb_title", "tt0120338") == entity.object_id + assert %{current: 1} = Backfill.wikidata(limit: 100) + end + + test "malformed Wikidata records are counted without stopping the batch", ctx do + {:ok, malformed} = + Sources.upsert_record(ctx.sources["wikidata"], %{ + external_id: "Q-malformed", + raw: %{"id" => "Q-malformed", "labels" => %{}, "claims" => []} + }) + + {:ok, valid} = + Sources.upsert_record(ctx.sources["wikidata"], %{ + external_id: "Q-empty", + raw: %{} + }) + + assert malformed.id < valid.id + + assert %{ + scanned: 2, + failed: 1, + skipped: 1, + queued: 0, + current: 0, + next_after: next_after + } = Backfill.wikidata(limit: 2) + + assert next_after == valid.id + end + + defp target(word) do + %{object_id: word.object_id, term: word.lemma, language: word.language_tag, relevance: "term"} + end +end diff --git a/test/devils_dictionary/source_identity/display_test.exs b/test/devils_dictionary/source_identity/display_test.exs new file mode 100644 index 00000000..dc8afab6 --- /dev/null +++ b/test/devils_dictionary/source_identity/display_test.exs @@ -0,0 +1,50 @@ +defmodule DevilsDictionary.SourceIdentity.DisplayTest do + use DevilsDictionary.DataCase, async: false + alias DevilsDictionary.{Sources, SourceIdentity, Repo} + alias DevilsDictionary.Registry.Entity + alias DevilsDictionary.SourceIdentity.Display + alias DevilsDictionary.SourceIdentity.Entry + alias DevilsDictionary.Encyclopedia.EntityPage + + test "withdrawn sole source does not continue serving its image" do + %{sources: sources} = DevilsDictionary.Fixtures.seed_catalog!() + source = sources["cinegraph"] + + {:ok, record} = + Sources.upsert_record(source, %{ + external_id: "audit-film", + url: "https://cinegraph.org/movies/audit", + raw: %{ + "title" => "Audit film", + "preview_metadata" => %{ + "poster_url" => "https://image.tmdb.org/t/p/w342/withdrawn.jpg" + } + } + }) + + {:ok, entry} = + Entry.new(%{ + source_slug: "cinegraph", + source_id: source.id, + source_record_id: record.id, + entity_kind: :work, + work_kind: "film", + label: "Audit film", + stable_identifier: %{namespace: "tmdb_movie", external_id: "999999999"}, + metadata: %{"image_url" => "https://image.tmdb.org/t/p/w342/withdrawn.jpg"} + }) + + resolution = SourceIdentity.resolve(entry) + assert EntityPage.build(resolution.object_id).entity.image_url + + entity = Repo.get!(Entity, resolution.object_id) + legacy = %{entity | metadata: Map.delete(entity.metadata, "source_identity_evidence")} + evidence = Display.preload([legacy]) + assert Display.image_url(legacy, evidence) == entity.metadata["image_url"] + + record |> Ecto.Changeset.change(display_allowed: false) |> Repo.update!() + page = EntityPage.build(resolution.object_id) + assert page.sources == [] + assert is_nil(page.entity.image_url) + end +end diff --git a/test/devils_dictionary/source_identity_test.exs b/test/devils_dictionary/source_identity_test.exs new file mode 100644 index 00000000..38b8ac11 --- /dev/null +++ b/test/devils_dictionary/source_identity_test.exs @@ -0,0 +1,404 @@ +defmodule DevilsDictionary.SourceIdentityTest do + use DevilsDictionary.DataCase, async: false + + alias DevilsDictionary.Absorb.Materializer + alias DevilsDictionary.Absorb.Sources.{Wikidata, Wikipedia} + alias DevilsDictionary.Claims + alias DevilsDictionary.Discovery.Providers.CineGraph + alias DevilsDictionary.Encyclopedia.EntityPage + alias DevilsDictionary.Registry + alias DevilsDictionary.Registry.{Entity, ExternalIdentifier, WorkDetails} + alias DevilsDictionary.SourceIdentity + alias DevilsDictionary.SourceIdentity.Entry + alias DevilsDictionary.Sources + alias DevilsDictionary.Sources.ReconciliationCase + + setup do + %{sources: sources} = DevilsDictionary.Fixtures.seed_catalog!() + %{sources: sources} + end + + describe "film identity resolution" do + test "CineGraph then Wikidata preserves one Titanic object and URL", ctx do + movie = cinegraph_movie(597) + wikidata = wikidata_movie("Q44578") + + first = resolve_cinegraph!(ctx.sources["cinegraph"], movie) + assert first.state == :newly_created + original_path = entity_path(first.object_id, "Titanic") + + resolve_wikidata!(ctx.sources["wikidata"], wikidata) + + wikipedia_content = + resolve_wikipedia!(ctx.sources["wikipedia"], wikipedia_movie("Q44578")) + + assert Registry.by_external_id("wikidata", "Q44578") == first.object_id + assert Registry.by_external_id("wikidata", "Q44578") == wikipedia_content.about_object_id + assert_film(first.object_id, "Titanic", 1997) + assert_identifiers(first.object_id, "597", "tt0120338", "Q44578") + assert entity_path(first.object_id, "Titanic") == original_path + + assert wikipedia_content.source_url == + "https://en.wikipedia.org/wiki/Titanic_(1997_film)" + + page = EntityPage.build(first.object_id) + assert Enum.map(page.sources, & &1.slug) == ["cinegraph", "wikidata"] + assert Enum.any?(page.biography, &(&1.url == wikipedia_content.source_url)) + end + + test "Wikidata then CineGraph preserves one Everest object and URL", ctx do + wikidata = wikidata_movie("Q15631013") + movie = cinegraph_movie(253_412) + + wikipedia_id = + resolve_wikipedia!(ctx.sources["wikipedia"], wikipedia_movie("Q15631013")).about_object_id + + first_id = resolve_wikidata!(ctx.sources["wikidata"], wikidata) + result = resolve_cinegraph!(ctx.sources["cinegraph"], movie) + + assert result.state == :matched + assert result.object_id == first_id + assert wikipedia_id == first_id + assert_film(first_id, "Everest", 2015) + assert_identifiers(first_id, "253412", "tt2719848", "Q15631013") + assert entity_path(first_id, "Everest") == "/entities/#{first_id}/everest" + end + + test "TMDb-only and IMDb-only proposals reuse supported exact identifiers", ctx do + tmdb = entry!(ctx, "tmdb-only", [%{namespace: "tmdb_movie", external_id: "700"}]) + imdb = entry!(ctx, "imdb-only", [%{namespace: "imdb_title", external_id: "tt0000700"}]) + + tmdb_created = SourceIdentity.resolve(tmdb) + imdb_created = SourceIdentity.resolve(imdb) + + assert SourceIdentity.resolve(tmdb).object_id == tmdb_created.object_id + assert SourceIdentity.resolve(imdb).object_id == imdb_created.object_id + refute tmdb_created.object_id == imdb_created.object_id + end + + test "missing Wikidata and Wikipedia still creates a useful source-backed film", ctx do + result = + resolve_cinegraph!(ctx.sources["cinegraph"], %{ + "tmdbId" => 991_001, + "imdbId" => nil, + "title" => "Fixture Without Crosswalk", + "releaseDate" => "2024-01-01", + "posterPath" => nil, + "cinegraphUrl" => "https://cinegraph.org/movies/991001" + }) + + assert result.state == :newly_created + assert_film(result.object_id, "Fixture Without Crosswalk", 2024) + assert Registry.by_external_id("wikidata", "Q991001") == nil + + page = EntityPage.build(result.object_id) + assert page.entity.label == "Fixture Without Crosswalk" + assert page.details.work_kind == "film" + assert Enum.map(page.sources, & &1.slug) == ["cinegraph"] + + assert {:error, :stable_identifier_required} = + Entry.new(%{ + source_slug: "fixture", + object_kind: :entity, + entity_kind: :work, + work_kind: "film", + label: "Film without any source identity", + eligibility: :eligible, + retention: :durable + }) + end + + test "contradictory identifiers open one rerunnable review case", ctx do + a = entry!(ctx, "conflict-a", [%{namespace: "tmdb_movie", external_id: "810"}]) + b = entry!(ctx, "conflict-b", [%{namespace: "imdb_title", external_id: "tt0000811"}]) + first = SourceIdentity.resolve(a) + second = SourceIdentity.resolve(b) + + conflict = + entry!(ctx, "conflict-record", [ + %{namespace: "tmdb_movie", external_id: "810"}, + %{namespace: "imdb_title", external_id: "tt0000811"} + ]) + + resolution = SourceIdentity.resolve(conflict) + again = SourceIdentity.resolve(conflict) + + assert resolution.state == :conflicting_identifiers + assert resolution.object_id == nil + assert resolution.conflict_id == again.conflict_id + assert Repo.aggregate(ReconciliationCase, :count) == 1 + assert first.object_id != second.object_id + end + + test "titles and years never merge remakes or a non-film namesake", ctx do + first = entry!(ctx, "same-title-a", [%{namespace: "tmdb_movie", external_id: "820"}]) + second = entry!(ctx, "same-title-b", [%{namespace: "tmdb_movie", external_id: "821"}]) + + first = SourceIdentity.resolve(%{first | label: "Titanic", year: 1953}) + second = SourceIdentity.resolve(%{second | label: "Titanic", year: 1997}) + {:ok, ship} = Registry.create_entity(%{entity_kind: :artifact, preferred_label: "Titanic"}) + + assert first.object_id != second.object_id + assert ship.object_id not in [first.object_id, second.object_id] + end + end + + @tag :unboxed + test "concurrent creation and retries converge on one object", ctx do + entry = entry!(ctx, "concurrent", [%{namespace: "tmdb_movie", external_id: "830"}]) + parent = self() + gate = make_ref() + + tasks = + for _ <- 1..2 do + Task.async(fn -> + send(parent, {:ready, self()}) + receive do: ({:go, ^gate} -> SourceIdentity.resolve(entry)) + end) + end + + pids = + for _ <- tasks do + assert_receive {:ready, pid} + pid + end + + Enum.each(pids, &send(&1, {:go, gate})) + results = Enum.map(tasks, &Task.await(&1, 5_000)) + + assert [object_id] = results |> Enum.map(& &1.object_id) |> Enum.uniq() + assert Registry.by_external_id("tmdb_movie", "830") == object_id + end + + describe "shared adapter contract" do + test "an artwork reuses Wikidata while a reproduction remains distinct", ctx do + {:ok, artwork} = + Registry.create_work(%{preferred_label: "Fixture Artwork", work_kind: "artwork"}) + + {:ok, _} = Registry.add_external_id(artwork.object_id, "wikidata", "Q900100") + + proposal = + entry!( + ctx, + "artwork", + [ + %{namespace: "museum_artwork", external_id: "A-1"}, + %{namespace: "wikidata", external_id: "Q900100"} + ], + work_kind: "artwork" + ) + + reproduction = + entry!(ctx, "reproduction", [%{namespace: "museum_object", external_id: "R-1"}], + work_kind: "reproduction", + relationships: [ + %{role: :reproduction_of, target_object_id: artwork.object_id, certainty: :verified} + ] + ) + + assert SourceIdentity.resolve(proposal).object_id == artwork.object_id + refute SourceIdentity.resolve(reproduction).object_id == artwork.object_id + end + + test "an ineligible quotation can still reference an independently resolved author", ctx do + author = + entry!(ctx, "author", [%{namespace: "wikidata", external_id: "Q900200"}], + entity_kind: :person, + work_kind: nil, + label: "Fixture Author" + ) + + author_resolution = SourceIdentity.resolve(author) + + assert {:ok, quotation} = + Entry.new(%{ + source_slug: "fixture", + object_kind: :content, + content_kind: :quotation, + entity_kind: nil, + eligibility: :insufficient_evidence, + eligibility_reason: "wording_has_no_stable_identity", + retention: :durable, + relationships: [ + %{ + role: :author, + target_object_id: author_resolution.object_id, + certainty: :verified + } + ] + }) + + assert SourceIdentity.resolve(quotation).state == :insufficient_evidence + assert hd(quotation.relationships).target_object_id == author_resolution.object_id + assert Repo.get!(Entity, author_resolution.object_id).entity_kind == :person + assert Claims.count_incoming(author_resolution.object_id) == 0 + + assert {:ok, uncertain} = + Entry.new(%{ + source_slug: "fixture", + object_kind: :content, + content_kind: :quotation, + entity_kind: nil, + eligibility: :insufficient_evidence, + eligibility_reason: "uncertain_attribution", + retention: :durable, + relationships: [ + %{ + role: :author, + target_object_id: author_resolution.object_id, + certainty: :candidate + } + ] + }) + + assert SourceIdentity.resolve(uncertain).state == :insufficient_evidence + assert Claims.count_incoming(author_resolution.object_id) == 0 + end + end + + defp resolve_cinegraph!(source, movie) do + tmdb_id = Integer.to_string(movie["tmdbId"]) + + item = %{ + external_namespace: "tmdb_movie", + external_id: tmdb_id, + identifiers: + [%{namespace: "tmdb_movie", external_id: tmdb_id}] ++ + if(movie["imdbId"], + do: [%{namespace: "imdb_title", external_id: movie["imdbId"]}], + else: [] + ), + preview_metadata: %{ + "title" => movie["title"], + "year" => String.slice(movie["releaseDate"], 0, 4), + "release_date" => movie["releaseDate"], + "poster_url" => movie["posterPath"], + "source_url" => movie["cinegraphUrl"] + } + } + + {:ok, record} = + Sources.upsert_record(source, %{ + external_id: "tmdb_movie:#{tmdb_id}", + url: movie["cinegraphUrl"], + raw: movie + }) + + {:ok, entry} = CineGraph.identity_record(item) + + SourceIdentity.resolve(%{ + entry + | source_id: source.id, + source_record_id: record.id, + source_record_revision_id: record.current_revision.id + }) + end + + defp resolve_wikidata!(source, raw) do + {:ok, record} = + Sources.upsert_record(source, %{ + external_id: raw["id"], + url: "https://www.wikidata.org/wiki/#{raw["id"]}", + raw: Wikidata.trim(raw) + }) + + assert {:ok, %{concepts: 1}} = Materializer.run(record, Wikidata) + Registry.by_external_id("wikidata", raw["id"]) + end + + defp resolve_wikipedia!(source, raw) do + {:ok, record} = + Sources.upsert_record(source, %{ + external_id: "concept:#{get_in(raw, ["pageprops", "wikibase_item"])}", + url: raw["fullurl"], + raw: Wikipedia.trim(raw) + }) + + assert {:ok, %{concepts: 1, entries: 1}} = Materializer.run(record, Wikipedia) + [about] = Claims.incoming(Registry.by_external_id("wikidata", qid(raw)), predicate: "about") + + %{ + about_object_id: about.object_object_id, + source_url: Registry.current_content_revision(about.subject_object_id).canonical_url + } + end + + defp entry!(ctx, key, identifiers, opts \\ []) do + source = ctx.sources["cinegraph"] + + {:ok, record} = + Sources.upsert_record(source, %{ + external_id: "fixture:#{key}", + url: "https://example.test/#{key}", + raw: %{"key" => key} + }) + + stable = hd(identifiers) + + {:ok, entry} = + Entry.new(%{ + source_slug: source.slug, + source_id: source.id, + source_record_id: record.id, + source_record_revision_id: record.current_revision.id, + object_kind: :entity, + entity_kind: Keyword.get(opts, :entity_kind, :work), + work_kind: Keyword.get(opts, :work_kind, "film"), + stable_identifier: stable, + identifiers: identifiers, + label: Keyword.get(opts, :label, "Fixture Film"), + year: Keyword.get(opts, :year, 2024), + metadata: %{}, + eligibility: :eligible, + retention: :durable, + relationships: Keyword.get(opts, :relationships, []) + }) + + entry + end + + defp cinegraph_movie(tmdb_id) do + DevilsDictionary.Fixtures.raw("cinegraph", "films") + |> Enum.find(&(&1["tmdbId"] == tmdb_id)) + end + + defp wikidata_movie(qid) do + DevilsDictionary.Fixtures.raw("wikidata", "films") + |> Enum.find(&(&1["id"] == qid)) + end + + defp wikipedia_movie(qid) do + DevilsDictionary.Fixtures.raw("wikipedia", "films") + |> Enum.find(&(qid(&1) == qid)) + end + + defp qid(raw), do: get_in(raw, ["pageprops", "wikibase_item"]) + + defp assert_film(object_id, label, year) do + assert %Entity{entity_kind: :work, preferred_label: ^label} = Repo.get!(Entity, object_id) + + assert %WorkDetails{work_kind: "film", first_published_year: ^year} = + Repo.get!(WorkDetails, object_id) + end + + defp assert_identifiers(object_id, tmdb_id, imdb_id, qid) do + identifiers = + Repo.all( + from identifier in ExternalIdentifier, + where: identifier.object_id == ^object_id and identifier.status == :verified, + select: {identifier.namespace, identifier.external_id} + ) + |> MapSet.new() + + assert identifiers == + MapSet.new([ + {"tmdb_movie", tmdb_id}, + {"imdb_title", imdb_id}, + {"wikidata", qid} + ]) + end + + defp entity_path(object_id, label) do + "/entities/#{object_id}/#{DevilsDictionary.Claims.Connection.slugify(label)}" + end +end diff --git a/test/devils_dictionary_web/live/culture_discovery_live_test.exs b/test/devils_dictionary_web/live/culture_discovery_live_test.exs index ac6c9839..a8773453 100644 --- a/test/devils_dictionary_web/live/culture_discovery_live_test.exs +++ b/test/devils_dictionary_web/live/culture_discovery_live_test.exs @@ -51,6 +51,8 @@ defmodule DevilsDictionaryWeb.CultureDiscoveryLiveTest do assert has_element?(live, "#culture-results") assert has_element?(live, "#culture-result-tmdb_movie-301") + assert has_element?(live, "#culture-entry-image-301[href^='/entities/']") + assert has_element?(live, "#culture-entry-title-301[href^='/entities/']") assert has_element?(live, "#culture-source-301[href^='https://cinegraph.org/']") assert has_element?(live, "#culture-about-cinegraph") refute has_element?(live, "#culture-failed") diff --git a/test/devils_dictionary_web/live/film_identity_flow_test.exs b/test/devils_dictionary_web/live/film_identity_flow_test.exs new file mode 100644 index 00000000..9cc230c1 --- /dev/null +++ b/test/devils_dictionary_web/live/film_identity_flow_test.exs @@ -0,0 +1,290 @@ +defmodule DevilsDictionaryWeb.FilmIdentityFlowTest do + use DevilsDictionaryWeb.ConnCase, async: false + use Oban.Testing, repo: DevilsDictionary.Repo + + import DevilsDictionary.WordFixtures + import Phoenix.LiveViewTest + + alias DevilsDictionary.Claims + alias DevilsDictionary.Discovery + alias DevilsDictionary.Discovery.{Result, Run} + alias DevilsDictionary.Discovery.Providers.CineGraph + alias DevilsDictionary.Encyclopedia.EntityPage + alias DevilsDictionary.Registry + alias DevilsDictionary.Repo + + setup %{conn: conn} do + catalog = DevilsDictionary.Fixtures.seed_catalog!() + Map.merge(catalog, %{conn: conn, animals: catalog.scopes["animals"]}) + end + + test "definition → film card → local film → connected definition is a reversible flow", ctx do + mountain = word!(ctx, "mountain", ~w(wordnet)) + mountain_sense = sense!(ctx, mountain, "wordnet", gloss: "a very high natural elevation") + grief = word!(ctx, "grief", ~w(wiktionary)) + grief_sense = sense!(ctx, grief, "wiktionary", gloss: "deep and lasting sorrow") + rejected = word!(ctx, "ship", ~w(wordnet)) + ship_sense = sense!(ctx, rejected, "wordnet", gloss: "a large seagoing vessel") + + stub_titanic("mountain") + {:ok, definition_live, _html} = live(ctx.conn, ~p"/define/mountain") + assert :ok = Discovery.execute_run(Repo.one!(Run).id) + _ = render(definition_live) + + result = Repo.one!(Result) + assert result.resolution_state == :newly_created + assert result.object_id + assert Registry.by_external_id("tmdb_movie", "597") == result.object_id + assert Registry.by_external_id("imdb_title", "tt0120338") == result.object_id + + assert {:error, {:live_redirect, %{to: stable_path}}} = + live(ctx.conn, "/entities/#{result.object_id}/an-old-title") + + assert stable_path == "/entities/#{result.object_id}/titanic" + + accepted = illustrates!(result.object_id, mountain_sense, "Its scale evokes this meaning.") + accepted_revision = Claims.current_revision(accepted.id) + + {:ok, review_context} = + Claims.open_review_context( + accepted_revision.id, + Claims.current_context_items(accepted_revision) + ) + + {:ok, _review} = + Claims.review(accepted_revision.id, :accepted, review_context_id: review_context.id) + + pending = illustrates!(result.object_id, grief_sense, "A proposed thematic connection.") + hidden = illustrates!(result.object_id, ship_sense, "A rejected title-only reading.") + {:ok, _review} = Claims.review(Claims.current_revision(hidden.id).id, :rejected) + + local_path = + "/entities/#{result.object_id}/titanic?from=%2Fwords%2F#{mountain.object_id}%2Fmountain" + + assert has_element?( + definition_live, + "#culture-entry-image-597[href='#{local_path}']" + ) + + assert has_element?( + definition_live, + "#culture-entry-title-597[href='#{local_path}']", + "Titanic" + ) + + assert has_element?( + definition_live, + "#culture-source-597[href='https://cinegraph.org/movies/597']" + ) + + {:ok, film_live, film_html} = + definition_live + |> element("#culture-entry-title-597") + |> render_click() + |> follow_redirect(ctx.conn) + + assert film_html =~ "Titanic" + assert has_element?(film_live, "#entity-header") + assert has_element?(film_live, "#entity-artwork img") + + assert has_element?( + film_live, + "#entity-back-link[href='/words/#{mountain.object_id}/mountain']" + ) + + assert has_element?(film_live, "#entity-source-cinegraph") + assert has_element?(film_live, "#entity-discovery-appearances", "Automatic match") + assert has_element?(film_live, "#entity-meaning-connections", "2 total") + assert has_element?(film_live, "#connection-out-#{accepted.id}", "Reviewed connection") + assert has_element?(film_live, "#connection-out-#{accepted.id}", mountain_sense_gloss()) + assert has_element?(film_live, "#connection-out-#{pending.id}", "Awaiting review") + assert has_element?(film_live, "#connection-out-#{pending.id}", "deep and lasting sorrow") + refute has_element?(film_live, "#connection-out-#{hidden.id}") + + {:ok, returned_live, _html} = + film_live + |> element("#connection-out-#{accepted.id} a[href^='/words/']") + |> render_click() + |> follow_redirect(ctx.conn) + + assert has_element?(returned_live, "#headword", "mountain") + + assert {:ok, _withdrawn} = Claims.withdraw(pending.id, reason: "connection removed") + {:ok, refreshed, _html} = live(ctx.conn, "/entities/#{result.object_id}/titanic") + assert has_element?(refreshed, "#connection-out-#{accepted.id}") + refute has_element?(refreshed, "#connection-out-#{pending.id}") + assert Repo.get!(Registry.Entity, result.object_id).preferred_label == "Titanic" + + assert {:ok, %{active: false}} = Discovery.set_provider_active("cinegraph", false) + {:ok, withdrawn_source, _html} = live(ctx.conn, "/entities/#{result.object_id}/titanic") + assert has_element?(withdrawn_source, "#connection-out-#{accepted.id}") + refute has_element?(withdrawn_source, "#entity-source-cinegraph") + refute has_element?(withdrawn_source, "#entity-discovery-appearances") + assert Repo.get!(Registry.Entity, result.object_id).preferred_label == "Titanic" + end + + defp mountain_sense_gloss, do: "a very high natural elevation" + + test "all connected meanings are reachable through pagination", ctx do + {:ok, film} = Registry.create_work(%{preferred_label: "Pagination film", work_kind: "film"}) + + for n <- 1..26 do + word = word!(ctx, "paging-#{n}", ~w(wordnet)) + sense = sense!(ctx, word, "wordnet", gloss: "meaning #{n}") + illustrates!(film.object_id, sense, "Supported connection #{n}") + end + + {:ok, view, _} = + live( + ctx.conn, + "/entities/#{film.object_id}/pagination-film?from=%2Fdefine%2Fpaging" + ) + + first = EntityPage.build(film.object_id) + assert length(first.meaning_connections) == 24 + assert first.pagination.meaning_connections.count == 26 + assert has_element?(view, "#more-meaning-connections") + view |> element("#more-meaning-connections") |> render_click() + refute has_element?(view, "#more-meaning-connections") + assert has_element?(view, "#entity-back-link[href='/define/paging']") + + second = + EntityPage.build(film.object_id, + meaning_connections_after: first.pagination.meaning_connections.next + ) + + assert length(second.meaning_connections) == 2 + + assert MapSet.disjoint?( + MapSet.new(first.meaning_connections, & &1.assertion_id), + MapSet.new(second.meaning_connections, & &1.assertion_id) + ) + end + + test "discovery appearances are bounded, reachable and retain the definition return path", + ctx do + {:ok, film} = Registry.create_work(%{preferred_label: "Discovery pages", work_kind: "film"}) + now = DateTime.utc_now() + + for n <- 1..26 do + word = word!(ctx, "appearance-#{n}", ~w(wordnet)) + assert {:queued, run} = Discovery.request(target_for(word), "cinegraph") + + run + |> Run.lifecycle_changeset(%{ + status: :succeeded, + started_at: now, + completed_at: DateTime.add(now, n, :second), + refresh_after: now, + expires_at: now, + completion_reason: :results, + result_count: 1 + }) + |> Repo.update!() + + %Result{} + |> Result.changeset(%{ + run_id: run.id, + external_namespace: "tmdb_movie", + external_id: "appearance-#{n}", + object_id: film.object_id, + position: 0, + match_details: %{}, + preview_metadata: %{}, + resolution_state: :matched + }) + |> Repo.insert!() + end + + first = EntityPage.build(film.object_id) + assert length(first.discovery_appearances) == 24 + assert first.pagination.discovery_appearances.count == 26 + assert first.pagination.discovery_appearances.next + + second = + EntityPage.build(film.object_id, + discovery_appearances_after: first.pagination.discovery_appearances.next + ) + + assert length(second.discovery_appearances) == 2 + assert second.pagination.discovery_appearances.next == nil + + assert MapSet.disjoint?( + MapSet.new(first.discovery_appearances, & &1.target_object_id), + MapSet.new(second.discovery_appearances, & &1.target_object_id) + ) + + {:ok, view, _html} = + live( + ctx.conn, + "/entities/#{film.object_id}/discovery-pages?from=%2Fdefine%2Fappearance-1" + ) + + assert has_element?(view, "#entity-discovery-appearances", "26 total") + assert has_element?(view, "#discovery-appearances-next") + view |> element("#discovery-appearances-next") |> render_click() + refute has_element?(view, "#discovery-appearances-next") + assert has_element?(view, "#entity-back-link[href='/define/appearance-1']") + end + + defp illustrates!(film_id, sense, rationale) do + {:ok, assertion} = + Claims.assert(film_id, "illustrates", sense.object_id, %{ + method: "curated", + rationale: rationale + }) + + assertion + end + + defp target_for(word) do + %{object_id: word.object_id, term: word.lemma, language: word.language_tag, relevance: "term"} + end + + defp stub_titanic(term) do + movie = + DevilsDictionary.Fixtures.raw("cinegraph", "films") + |> Enum.find(&(&1["tmdbId"] == 597)) + + Req.Test.stub(CineGraph, fn conn -> + body = request_body(conn) + + if String.contains?(body["query"], "searchMovieKeywords") do + json(conn, %{ + "data" => %{ + "searchMovieKeywords" => [%{"tmdbId" => 1, "name" => term, "movieCount" => 1}] + } + }) + else + json(conn, %{ + "data" => %{ + "discoverMovies" => %{ + "edges" => [ + %{ + "cursor" => "titanic", + "node" => %{ + "movie" => movie, + "matchedKeywords" => [%{"tmdbId" => 1, "name" => term}], + "matchedGenres" => [] + } + } + ], + "pageInfo" => %{"endCursor" => nil, "hasNextPage" => false} + } + } + }) + end + end) + end + + defp request_body(conn) do + {:ok, body, _conn} = Plug.Conn.read_body(conn) + Jason.decode!(body) + end + + defp json(conn, body) do + conn + |> Plug.Conn.put_resp_content_type("application/json") + |> Plug.Conn.send_resp(200, Jason.encode!(body)) + end +end diff --git a/test/support/fixtures/cinegraph/films.json b/test/support/fixtures/cinegraph/films.json new file mode 100644 index 00000000..99da6bd6 --- /dev/null +++ b/test/support/fixtures/cinegraph/films.json @@ -0,0 +1,20 @@ +{ + "records": [ + { + "tmdbId": 597, + "imdbId": "tt0120338", + "title": "Titanic", + "releaseDate": "1997-11-18", + "posterPath": "/titanic.jpg", + "cinegraphUrl": "https://cinegraph.org/movies/597" + }, + { + "tmdbId": 253412, + "imdbId": "tt2719848", + "title": "Everest", + "releaseDate": "2015-09-10", + "posterPath": "/everest.jpg", + "cinegraphUrl": "https://cinegraph.org/movies/253412" + } + ] +} diff --git a/test/support/fixtures/wikidata/films.json b/test/support/fixtures/wikidata/films.json new file mode 100644 index 00000000..6411a726 --- /dev/null +++ b/test/support/fixtures/wikidata/films.json @@ -0,0 +1,30 @@ +{ + "records": [ + { + "id": "Q44578", + "labels": {"en": {"language": "en", "value": "Titanic"}}, + "descriptions": {"en": {"language": "en", "value": "1997 film directed by James Cameron"}}, + "aliases": {"en": []}, + "sitelinks": {"enwiki": {"site": "enwiki", "title": "Titanic (1997 film)"}}, + "claims": { + "P31": [{"rank": "normal", "type": "statement", "mainsnak": {"snaktype": "value", "datavalue": {"type": "wikibase-entityid", "value": {"id": "Q11424"}}}}], + "P4947": [{"rank": "normal", "type": "statement", "mainsnak": {"snaktype": "value", "datavalue": {"type": "string", "value": "597"}}}], + "P345": [{"rank": "normal", "type": "statement", "mainsnak": {"snaktype": "value", "datavalue": {"type": "string", "value": "tt0120338"}}}], + "P577": [{"rank": "normal", "type": "statement", "mainsnak": {"snaktype": "value", "datavalue": {"type": "time", "value": {"time": "+1997-11-18T00:00:00Z", "precision": 11}}}}] + } + }, + { + "id": "Q15631013", + "labels": {"en": {"language": "en", "value": "Everest"}}, + "descriptions": {"en": {"language": "en", "value": "2015 film directed by Baltasar Kormákur"}}, + "aliases": {"en": []}, + "sitelinks": {"enwiki": {"site": "enwiki", "title": "Everest (2015 film)"}}, + "claims": { + "P31": [{"rank": "normal", "type": "statement", "mainsnak": {"snaktype": "value", "datavalue": {"type": "wikibase-entityid", "value": {"id": "Q11424"}}}}], + "P4947": [{"rank": "normal", "type": "statement", "mainsnak": {"snaktype": "value", "datavalue": {"type": "string", "value": "253412"}}}], + "P345": [{"rank": "normal", "type": "statement", "mainsnak": {"snaktype": "value", "datavalue": {"type": "string", "value": "tt2719848"}}}], + "P577": [{"rank": "normal", "type": "statement", "mainsnak": {"snaktype": "value", "datavalue": {"type": "time", "value": {"time": "+2015-09-10T00:00:00Z", "precision": 11}}}}] + } + } + ] +} diff --git a/test/support/fixtures/wikipedia/films.json b/test/support/fixtures/wikipedia/films.json new file mode 100644 index 00000000..d45c92ec --- /dev/null +++ b/test/support/fixtures/wikipedia/films.json @@ -0,0 +1,26 @@ +{ + "records": [ + { + "title": "Titanic (1997 film)", + "pageid": 26964606, + "ns": 0, + "description": "1997 film by James Cameron", + "extract": "Titanic is a 1997 epic romance and disaster film directed by James Cameron.", + "thumbnail": {"source": "https://upload.wikimedia.org/wikipedia/commons/thumb/f/fd/RMS_Titanic_3.jpg/500px-RMS_Titanic_3.jpg"}, + "pageprops": {"wikibase_item": "Q44578"}, + "fullurl": "https://en.wikipedia.org/wiki/Titanic_(1997_film)", + "_probe": {"qid": "Q44578", "title": "Titanic (1997 film)"} + }, + { + "title": "Everest (2015 film)", + "pageid": 41931150, + "ns": 0, + "description": "2015 film by Baltasar Kormákur", + "extract": "Everest is a 2015 biographical survival adventure film directed by Baltasar Kormákur.", + "thumbnail": {"source": "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4b/Mount_Everest_as_seen_from_Drukair2_PLW_edit.jpg/500px-Mount_Everest_as_seen_from_Drukair2_PLW_edit.jpg"}, + "pageprops": {"wikibase_item": "Q15631013"}, + "fullurl": "https://en.wikipedia.org/wiki/Everest_(2015_film)", + "_probe": {"qid": "Q15631013", "title": "Everest (2015 film)"} + } + ] +}