Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Internal name: `devils_dictionary` (the Phoenix app, the repo). Product name is

This file is the **living record** of the project: what it is, what it is not, the decisions we've made, and what we plan to build next. Update it whenever a decision changes. The encyclopedia foundation is specified in [issue #74](https://github.com/razrfly/dictionary/issues/74); [issue #69](https://github.com/razrfly/dictionary/issues/69) defines the original source-coverage scorecard.

Automatic cultural discovery and the direct-browser GIPHY integration are documented in [`docs/integrations/giphy.md`](docs/integrations/giphy.md).
Automatic cultural discovery and the direct-browser GIPHY integration are documented in [`docs/integrations/giphy.md`](docs/integrations/giphy.md). Durable cross-provider film identity and its adapter/backfill contract are documented in [`docs/integrations/source-identity.md`](docs/integrations/source-identity.md).

## Current delivery — 11 September 2026

Expand Down
92 changes: 92 additions & 0 deletions docs/integrations/source-identity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Source identity adapters

Dictionary keeps source ingestion, durable identity, discovery relevance and
editorial interpretation as separate layers. An adapter may propose a durable
object through `DevilsDictionary.SourceIdentity.Adapter`; the shared resolver,
not the adapter, decides whether that proposal matches or creates an object.

## Required proposal fields

An eligible durable proposal declares:

- the source slug and source-record provenance;
- `object_kind` and the registry subtype (`entity_kind`, plus `work_kind` for
works);
- one stable, namespaced source identifier;
- every exact cross-source identifier supported by that source record;
- a display label and optional description, year, language and metadata;
- explicit eligibility and retention decisions; and
- optional relationships to independently identified objects, such as an
author or an original artwork.

The resolver only matches verified exact identifiers. Titles, years, keyword
matches and image similarity are never identity evidence. A source record whose
identifiers point to different objects, assert two exclusive values in one
namespace, or contradict the held object subtype produces an open
`external_identifier_conflict` reconciliation case and no automatic link.

Identifier values must already reflect the source adapter's rank and validity
policy. For Wikidata that means preferred statements replace normal statements,
deprecated statements are ignored, TMDb movie identifiers are positive decimal
IDs and IMDb title identifiers have the `tt` title form.

## Current implementation

CineGraph discovery results create or reuse film works from TMDb movie IDs and,
when supplied, IMDb title IDs. Wikidata film records add the Wikidata QID and
their exact P4947/P345 crosswalks. Either source may arrive first; both paths
converge on the same registry object and keep the original object ID and local
URL.

The same proposal shape represents artwork, reproductions, quotations and
creator relationships. Issue #93 intentionally implements live ingestion only
for films. Artwork and quotation fixtures verify that future adapters can reuse
the contract without treating a reproduction as its original or uncredited
wording as a durable work.

Discovery results remain provider cache rows. Resolution never turns a keyword
match into an `illustrates` claim. A film page presents automatic discovery
appearances separately from attributed, reviewable meaning connections.

## Backfilling retained CineGraph rows

The backfill is local, bounded and resumable; it does not call CineGraph and
does not merge or delete objects:

```console
mix dd.films.backfill --limit 100
mix dd.films.backfill --limit 100 --after 4200
```

The report separates matched, newly created, insufficient-evidence and
conflicting-identifier outcomes. Use the printed checkpoint for the next batch.
Rerunning a batch is safe because the same exact-identifier resolver handles
both live ingestion and backfill.

## Upgrade existing Wikidata entries first

Run `mix dd.films.backfill --wikidata --limit 100` and continue with the printed
`--after` checkpoint. This scans a bounded batch of allowed Wikidata source records.
Legacy film records queue deduplicated jobs through the existing enrichment worker;
these jobs fetch the fields older projections discarded, then materialize them onto
the existing Wikidata identity. Current projections are replayed locally. Counts
separate queued, current, skipped and failed records; a malformed retained record is
reported without stopping the batch, and queued does not mean completed.

Wait for enrichment jobs to succeed, inspect failures/conflicts, then run the
CineGraph backfill above. Rerunning the Wikidata phase is safe. Completed refreshes
carry a projection-version marker so a successful negative crosswalk does not cause
endless refreshes. Requests use the existing Wikimedia timeout, retry and rate-limit
handling. The version participates in the source revision hash so an unchanged
upstream payload can still upgrade an older local projection without overwriting its
historical revision. Source records not classified as films are skipped; this is a
film migration, not a full Wikidata crawl or automatic title search.

If separate objects already exist, contradictory IDs enter reconciliation rather
than being silently merged. Missing crosswalks remain valid source-backed local films.

Film image projections record their source-record evidence. Public display checks
source activation, record visibility and output retirement. Legacy posters without
that marker must be supported by an allowed record with the exact same URL. Withdrawn
posters disappear without deleting the film or its connections. Connected meanings
use the existing cursor pagination so all public relationships remain reachable.
52 changes: 49 additions & 3 deletions lib/devils_dictionary/absorb/materializer.ex
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ defmodule DevilsDictionary.Absorb.Materializer do
alias DevilsDictionary.Claims
alias DevilsDictionary.Repo
alias DevilsDictionary.Registry.Lexeme
alias DevilsDictionary.SourceIdentity
alias DevilsDictionary.Sources.SourceRecord

# Postgres caps a statement at 65,535 bind parameters; the widest row here is
Expand Down Expand Up @@ -134,7 +135,9 @@ defmodule DevilsDictionary.Absorb.Materializer do
|> Ecto.Multi.run(:forms, fn _repo, changes ->
{:ok, upsert_forms(merged.lexemes, changes.lexemes, revisions, now)}
end)
|> Ecto.Multi.run(:concepts, fn _repo, _ -> {:ok, upsert_entities(merged.concepts, now)} end)
|> Ecto.Multi.run(:concepts, fn _repo, _ ->
{:ok, upsert_entities(merged.concepts, now, module, revisions, run_id)}
end)
|> Ecto.Multi.run(:senses, fn _repo, changes ->
{:ok,
upsert_senses(merged.senses, changes.lexemes, records, revisions, run_id, now, module)}
Expand Down Expand Up @@ -551,9 +554,52 @@ defmodule DevilsDictionary.Absorb.Materializer do

defp initial_projection(metadata), do: metadata

defp upsert_entities([], _now), do: %{}
defp upsert_entities([], _now, _module, _revisions, _run_id), do: %{}

defp upsert_entities(rows, now, module, revisions, run_id) do
{identity_rows, ordinary_rows} = identity_rows(rows, module)

ordinary_ids = upsert_ordinary_entities(ordinary_rows, now)

identity_ids =
identity_rows
|> Enum.uniq_by(fn {row, _entry} -> row.key end)
|> Enum.flat_map(fn {row, entry} ->
entry = %{
entry
| source_id: row[:source_id],
source_record_id: row[:source_record_id],
source_record_revision_id: revisions[row[:source_record_id]],
import_run_id: run_id
}

case SourceIdentity.resolve(entry) do
%{object_id: object_id} when is_integer(object_id) -> [{row.key, object_id}]
_resolution -> []
end
end)
|> Map.new()

Map.merge(ordinary_ids, identity_ids)
end

defp identity_rows(rows, module) do
if function_exported?(module, :identity_record, 1) do
Enum.reduce(rows, {[], []}, fn row, {identity, ordinary} ->
case module.identity_record(row) do
{:ok, entry} -> {[{row, entry} | identity], ordinary}
:ignore -> {identity, [row | ordinary]}
{:error, reason} -> throw({:materialize_failed, row.source_id, row.key, reason})
end
end)
else
{[], rows}
end
end

defp upsert_ordinary_entities([], _now), do: %{}

defp upsert_entities(rows, now) do
defp upsert_ordinary_entities(rows, now) do
qids = Enum.map(rows, & &1.key)

existing =
Expand Down
112 changes: 108 additions & 4 deletions lib/devils_dictionary/absorb/sources/wikidata.ex
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do
"""

@behaviour DevilsDictionary.Absorb.Source
@behaviour DevilsDictionary.SourceIdentity.Adapter

import Ecto.Query

Expand All @@ -34,6 +35,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do
alias DevilsDictionary.Lexicon.ScopeMember
alias DevilsDictionary.Registry.{Entity, ExternalIdentifier, Sense, SenseRevision}
alias DevilsDictionary.Repo
alias DevilsDictionary.SourceIdentity.Entry
alias DevilsDictionary.Sources
alias DevilsDictionary.Sources.{Source, SourceRecord}

Expand All @@ -44,7 +46,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do

# The fifteen properties the pipeline reads. Everything else is dropped before
# storage; a 130 KB entity becomes a couple of KB.
@keep_properties ~w(P18 P31 P279 P171 P105 P225 P1843 P5063 P8814 P13176 P1420 P910 P373 P846 P685)
@keep_properties ~w(P18 P31 P279 P171 P105 P225 P1843 P5063 P8814 P13176 P1420 P910 P373 P846 P685 P345 P4947 P577)

# Parent properties walked to closure. **Only these two**: P171 is the
# taxonomy proper and terminates at Animalia, and P13176 is the one hop from
Expand Down Expand Up @@ -82,6 +84,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do
@impl true
def trim(raw) do
%{
"_film_identity_version" => 1,
"id" => raw["id"],
"labels" => take_lang(raw["labels"]),
"descriptions" => take_lang(raw["descriptions"]),
Expand Down Expand Up @@ -486,7 +489,7 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do
external_id: qid,
url: entity_url(qid),
raw: trim(entity),
content_hash: SourceRecord.content_hash(entity)
content_hash: SourceRecord.content_hash(%{"entity" => entity, "projection_version" => 1})
}
end

Expand Down Expand Up @@ -548,6 +551,9 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do
label: clamp(label(raw, scientific_name)),
description: get_in(raw, ["descriptions", "en", "value"]),
kind: entity_kind(raw, scientific_name),
work_kind: work_kind(raw),
first_published_year: publication_year(raw),
external_identifiers: external_identifiers(qid, raw),
taxon_concept: taxon_concept(raw, scientific_name),
metadata:
raw
Expand All @@ -562,6 +568,27 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do
{:ok, %{concepts: [concept], concept_relations: relations(record, qid, raw)}}
end

@impl DevilsDictionary.SourceIdentity.Adapter
def identity_record(%{kind: :work, work_kind: work_kind} = row)
when work_kind in ["film", "artwork"] do
Entry.new(%{
source_slug: slug(),
object_kind: :entity,
entity_kind: :work,
work_kind: work_kind,
stable_identifier: %{namespace: "wikidata", external_id: row.qid},
identifiers: row.external_identifiers,
label: row.label,
description: row.description,
year: row.first_published_year,
metadata: row.metadata,
eligibility: :eligible,
retention: :durable
})
end

def identity_record(_row), do: :ignore

# English first, then the multilingual label a taxon carries instead, then the
# scientific name itself. A concept with no name at all is not worth a card.
defp label(raw, scientific_name) do
Expand Down Expand Up @@ -603,6 +630,9 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do
end
end

@film_classes ~w(Q11424 Q202866 Q506240)
@artwork_classes ~w(Q3305213)

@kind_classes %{
person: ~w(Q5),
organization: ~w(Q43229 Q4830453 Q163740 Q783794),
Expand All @@ -616,8 +646,82 @@ defmodule DevilsDictionary.Absorb.Sources.Wikidata do
defp entity_kind(raw, _scientific_name) do
classes = MapSet.new(Client.entity_ids(raw, "P31"))

Enum.find_value(@kind_classes, :concept, fn {kind, recognized} ->
if Enum.any?(recognized, &MapSet.member?(classes, &1)), do: kind
cond do
Enum.any?(@film_classes, &MapSet.member?(classes, &1)) ->
:work

Enum.any?(@artwork_classes, &MapSet.member?(classes, &1)) ->
:work

true ->
Enum.find_value(@kind_classes, :concept, fn {kind, recognized} ->
if Enum.any?(recognized, &MapSet.member?(classes, &1)), do: kind
end)
end
end

defp work_kind(raw) do
classes = MapSet.new(Client.entity_ids(raw, "P31"))

cond do
Enum.any?(@film_classes, &MapSet.member?(classes, &1)) -> "film"
Enum.any?(@artwork_classes, &MapSet.member?(classes, &1)) -> "artwork"
true -> nil
end
end

defp external_identifiers(qid, raw) do
[
%{
namespace: "wikidata",
external_id: qid,
metadata: %{"provider_field" => "id"}
}
] ++
external_identifier_values(raw, "P4947", "tmdb_movie", &valid_tmdb_movie_id?/1) ++
external_identifier_values(raw, "P345", "imdb_title", &valid_imdb_title_id?/1)
end

defp external_identifier_values(raw, property, namespace, valid?) do
raw
|> Client.strings(property)
|> Enum.filter(valid?)
|> Enum.uniq()
|> Enum.map(fn external_id ->
%{
namespace: namespace,
external_id: external_id,
metadata: %{
"wikidata_property" => property,
"statement_policy" => "preferred_else_normal"
}
}
end)
end

defp valid_tmdb_movie_id?(value) when is_binary(value),
do: Regex.match?(~r/\A[1-9]\d*\z/, value)

defp valid_tmdb_movie_id?(_value), do: false

defp valid_imdb_title_id?(value) when is_binary(value),
do: Regex.match?(~r/\Att\d{7,10}\z/, value)

defp valid_imdb_title_id?(_value), do: false

defp publication_year(raw) do
raw
|> Client.statements("P577")
|> Enum.find_value(fn
%{value: %{"time" => <<sign::binary-size(1), year::binary-size(4), _rest::binary>>}}
when sign in ["+", "-"] ->
case Integer.parse(sign <> year) do
{value, ""} -> value
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
_ -> nil
end

_ ->
nil
end)
end

Expand Down
1 change: 1 addition & 0 deletions lib/devils_dictionary/claims/connection.ex
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,7 @@ defmodule DevilsDictionary.Claims.Connection do
{s.object_id,
%{
label: l.lemma,
detail: r.gloss,
path: fragment("'/words/' || ? || '/' || ?", l.object_id, l.slug)
}}
)
Expand Down
Loading