Skip to content

Unify source identities into clickable encyclopedia entries for films, artwork and quotes #93

Closed

Description

@holden

Problem and desired outcome

Dictionary is an encyclopedia. A film, artwork, author or other suitable item returned by an integrated source should have a clickable local encyclopedia entry, and records describing the same thing should enrich that entry instead of producing disconnected provider cards or duplicate objects.

For example, CineGraph and Wikidata/Wikipedia descriptions of Titanic (1997) should resolve to one local film page. That page should show its supported information and let readers navigate to the meanings connected to it. Artwork should follow the same pattern in the next art integration. Quotes should link to their author and, where justified and supported, to an entry for the quotation itself.

This issue is the shared identity and navigation prerequisite for future source integrations. Implement movies first, then reuse the contract for art and quotes. It does not authorize replacing the existing registry or implementing every future provider at once.

Related:

Audit findings: existing foundations and missing wiring

Code inspected September 12, 2026; recheck current main before implementation because caching work is proceeding independently.

  • lib/devils_dictionary/registry.ex already provides objects/entities, typed work creation, namespaced external identifiers and transactional creation. Reuse these structures.
  • lib/devils_dictionary/absorb/materializer.ex, upsert_entities/2, resolves imported entities by verified Wikidata identifier. It does not currently resolve their TMDb/IMDb identifiers.
  • lib/devils_dictionary/absorb/sources/wikidata.ex does not project TMDb movie IDs or IMDb IDs into the shared identifier registry. Its direct-instance work classification also omits film and painting classes, so records can fall back to concept.
  • lib/devils_dictionary/absorb/sources/wikipedia.ex has a concept-target path based on existing Wikidata-linked entities. Importing Wikipedia does not imply every notable film/artwork already exists locally.
  • lib/devils_dictionary/discovery/providers/cine_graph.ex receives movie IDs including IMDb, but normalized results use tmdb_movie and do not retain IMDb in their normalized identity data.
  • lib/devils_dictionary/discovery.ex result persistence looks up existing objects by the normalized verified identifier; a result can retain a null object link. It does not create a local work on an eligible cache miss.
  • lib/devils_dictionary/discovery/result.ex already has optional object and source-record associations.
  • lib/devils_dictionary/encyclopedia/entity_page.ex and lib/devils_dictionary_web/live/entity_live.ex provide entity pages and incoming/outgoing connection foundations. Verify the actual reader flow; do not assume that cached appearances already produce durable reviewed connections.
  • Source records/revisions and identifier uniqueness already exist. Inventory them before proposing migrations. No second generic art database or parallel relationship graph.

These are code findings, not a census of production records. Mona Lisa and Titanic below are acceptance examples, not claims that those records currently exist.

1. Shared source contract and entry eligibility

Define a small reusable adapter contract for:

  • Entity kind/subtype, stable source identifier, supported cross-source identifiers.
  • Minimal source-supported display facts, provenance, creator/author references.
  • Entry eligibility and any publication/retention restrictions.
  • Resolution state: matched, newly created, insufficient evidence, or conflicting identifiers.

For approved encyclopedia sources, eligible records should automatically resolve to or create local entries. Do not require the reader to manually “promote” each film. A Wikipedia article is helpful enrichment, not a prerequisite for a local entry or the sole notability gate.

A viable entry needs a stable identity, a credible source and sufficient supported information for a useful page. Define and document this minimum per source within the common contract. Do not invent descriptions or treat a title alone as verified identity.

Ephemeral or insufficiently identified results may remain discovery-only. GIPHY is currently such a use case and need not automatically become a permanent catalog. This is an explicit capability choice, not a separate reader interaction model.

2. Resolve identity before creating another entry

  1. Normalize stable, namespaced IDs from the provider and supported crosswalks.
  2. Resolve all usable verified identifiers against the registry.
  3. If they agree on one local object, reuse it and attach the provider record.
  4. If none resolves and the record is eligible, create the local object, correct subtype, identifiers and source attachment transactionally.
  5. If identifiers resolve to different objects, record a reviewable conflict. Do not pick one silently or auto-merge by title.
  6. If identity evidence is insufficient, keep the result usable without a false local link. Retry resolution when better evidence arrives.
  7. Backfill older discovery results and imported objects when crosswalks become available.

Requirements:

  • Support both ingestion orders: Wikidata first and provider first.
  • Preserve the local object ID and URL through enrichment, classification correction and provider retirement.
  • Unique identifiers and concurrent imports must converge on one object; retries must not create duplicates.
  • Keep evidence/source and status for identifier assertions. Wikidata is a crosswalk source, not an infallible authority.
  • Title/year/creator similarity may suggest a review candidate, never independently authorize an automatic merge.
  • Inventory existing duplicates before backfill. Conflicts go to review; do not perform destructive bulk merges.
  • If a reviewed merge is supported, preserve relationships and source history and redirect old URLs through existing mechanisms.
  • Wikipedia articles and language versions describe an object; they are not separate identities for the same work.

Film implementation

  • Retain and normalize CineGraph's TMDb movie and IMDb title IDs.
  • Project supported Wikidata TMDb movie and IMDb identifiers into the same namespaces.
  • Verify identifier type: film IDs must not collide with people, TV or other namespaces.
  • Correct film/work classification without replacing existing objects or violating subtype invariants. Handle relevant subclass/type cases explicitly; avoid broad guessing.
  • Reconcile existing Wikidata-backed records and CineGraph discovery results.
  • Do not assume IMDb exists on every record. Use any supported exact crosswalk; absence of one identifier is not a fatal error.

Example identity boundaries:

  • Titanic (1997 film), Titanic (1953 film), RMS Titanic and the word “titanic” are distinct.
  • A remake is a different work. Editions, cuts and reproductions follow the existing work/edition/artifact distinctions rather than being merged indiscriminately.

3. Clickable reader experience

Preserve the compact poster shelf design and thumbnails.

  • For an eligible resolved item, its thumbnail and title link to its local entry.
  • Keep an explicit secondary external source link; the primary click must not unexpectedly leave Dictionary.
  • Render a useful source-backed entry even when Wikipedia prose is unavailable.
  • Show creators/authors as local links when resolved.
  • On an entry, show related meanings with links back to the relevant definition/sense, preserving navigation context where supported.
  • Distinguish reviewed meaning connections from automatic discovery appearances. A keyword match is not an approved interpretation.
  • Show supported source attribution without exposing internal resolution jargon to readers.
  • If unresolved, avoid broken local links: provide the existing result and clearly labeled source destination. Lack of a cross-source match alone must not prevent a useful local entry when a stable provider identity exists.
  • Respect public visibility/review rules. Private or rejected connections must not leak through backlinks.
  • Stable routes, keyboard access, responsive layout and sensible thumbnail fallback are required.

Wireframe A — definition page / compact shelf

MOUNTAIN
[definition and existing content]

In culture
Films
┌──────────┐ ┌──────────┐ ┌──────────┐
│ thumbnail│ │ thumbnail│ │ thumbnail│
└──────────┘ └──────────┘ └──────────┘
Film title ↗  Film title ↗ Film title ↗
1997 · Film   2015 · Film  2020 · Film

Thumbnail/title → local encyclopedia entry
Secondary “Source” → external provider
(Arrow here denotes a clickable title, not an external-link icon.)

Wireframe B — local film or artwork page

← Back to mountain

[poster/image] TITANIC
               Film · 1997
               [source-supported summary]

People
[Creator / director]   [Other supported people]

Connected meanings
[Meaning A]   Reviewed connection · rationale
[Meaning B]   Reviewed connection · rationale

Also appeared in discovery for
[Meaning C]   Automatic match

Sources
[Wikipedia] [CineGraph] [Other verified source]

The same page structure should serve artwork with artist/date/medium rather than film-specific fields. Do not fabricate fields solely to fill the layout.

Wireframe C — quote with independently resolved author

Quotes
“[Source-supported quotation text]”
— [Author name → local author entry]
[Quotation entry →]  [Source]

Quotation entry appears only when independently eligible.
An unresolved quotation must not block a resolved author link.
A resolved quotation must not force an uncertain author identity.

Quotation text above is a placeholder, not an attributed real quote.

4. Artwork and quote extension requirements

Artwork / upcoming Artsy work

Use the same resolution contract for artwork IDs, Wikidata IDs and creator IDs. A museum record, an Artsy record and a Wikipedia article may describe one local artwork.

  • Match with documented stable crosswalks; do not assume every provider supplies them.
  • Separate the artwork from an image file, reproduction, print/edition and physical instance where necessary.
  • Preserve source-specific provenance and avoid silently overwriting conflicting dates/titles/attributions.
  • Attach artist identities independently.
  • Prove the shared contract with fixtures and a bounded art example; full provider ingestion remains in Artsy: verify access before retirement, seed significant artworks and connect art to meanings #86 or its follow-up.
  • Artsy public API retirement is an access risk, not evidence that the underlying artworks disappear.
  • Permanent image retention is separate from identity retention. Keep only assets supported by applicable rights, with attribution and evidence.
  • MoMA and The Met are different sources. MoMA's open dataset excludes images. Open-access museum images are an alternative where durable thumbnails are needed; choosing a different primary art provider is a separate decision.

Quotes / future integration

  • Resolve author identities independently from quotation identities.
  • When a quotation has a supported stable identity or Wikipedia/Wikidata entry, reuse/create its eligible local entry.
  • Otherwise, a quote may remain attributed content attached to its source and linked author without manufacturing an encyclopedia identity.
  • Identical wording alone does not establish identical quotation identity, authorship or translation.
  • Model uncertain/disputed attribution as sourced uncertainty. Never confidently attach an author merely from a name match.
  • Inspect existing content/claim/object types before introducing a quote subtype or relation.
  • Full quote ingestion is future work; document and test the reusable contract now.

5. Persistence, caching and source withdrawal

Keep these lifecycles distinct:

  1. Discovery cache: which results were returned for a meaning/query.
  2. Durable object: the film/artwork/person/eligible quotation.
  3. Provider record/revision: source-specific observations and identifiers.
  4. Meaning connection: editorial rationale, evidence, visibility and review.
  5. Media asset: stored bytes, rights, attribution and availability.

Coordinate with #92's common automatic display and configurable refresh policy. Cache refresh, empty responses or expiry must not delete local objects or reviewed connections. Repeated cached appearances should not create duplicate links.

Provider failure should leave independently supported entries usable. Withdraw content that cannot be retained independently without cascade-deleting the identity. Review connections whose only supporting evidence was withdrawn.

Do not equate a remote image URL with a retained image. Where asset downloads are in scope, use the existing job system with retryable pending/verified/ready states, checksum and rights evidence; database creation and remote uploads are not one atomic transaction. A dedicated preservation pipeline may remain a scoped art follow-up.

6. Implementation sequence

  • Inventory registry, identifiers, subtype constraints, page routes and existing data; document the smallest changes.
  • Define the shared source identity/eligibility contract and conflict behavior.
  • Implement CineGraph ↔ Wikidata/Wikipedia crosswalks, preserving IMDb and TMDb identifiers.
  • Correct typed work classification and add a bounded, rerunnable backfill with counts for matched/created/unresolved/conflicting records.
  • Make film cards navigate to useful local pages; expose safe reverse meaning links.
  • Add art and quote/author contract fixtures and document adapter requirements.
  • Demonstrate film behavior end to end before expanding art ingestion.
  • Record remaining art asset/provider and quote ingestion work explicitly rather than claiming those integrations are complete.

Acceptance tests / definition of done

  • The same film imported through CineGraph and Wikidata in either order produces one local object and one stable page.
  • TMDb-only and IMDb-only supported matches work; missing IDs degrade safely.
  • Concurrent imports and retries do not create duplicates.
  • Conflicting IDs produce a reviewable conflict, not a silent merge.
  • Same-title films/remakes and the ship/word example remain separate.
  • Existing Wikidata-backed concepts are correctly classified without changing object identity.
  • Existing unresolved discovery results are backfilled, and rerunning the backfill is safe.
  • A film connects to two exact meanings; both are navigable from its page, respecting visibility. Removing one connection preserves the film and the other connection.
  • Search appearances are not mislabeled as reviewed semantic connections.
  • A source-backed film without Wikipedia content still has a useful clickable local page.
  • A bounded artwork fixture reuses an existing Wikidata object; a reproduction is not merged with the artwork.
  • A quotation with a resolved author and no independent quote identity links the author correctly; uncertain attribution is not asserted.
  • Cache expiration, provider errors and source withdrawal do not remove independently supported identities or leak withdrawn material.
  • Poster-shelf thumbnails/title links and entry/back navigation work in a real browser on desktop and mobile, including keyboard navigation.
  • Public documentation explains how each future adapter supplies identity, provenance, eligibility, relationships and fallback behavior.

Research references

The research recommendation is to prove identity reuse with existing films first. Open museum collections are a strong option for durable art assets; Artsy remains the planned art discussion in #86. This issue records the shared architecture and reader behavior without silently changing that provider decision.

Status at creation: research/specification only. No implementation of this issue is claimed.

Readiness verification — live examples, September 12, 2026

The identity bridge has now been checked against the configured CineGraph HTTP GraphQL API and live Wikidata entity JSON, not only schema fields.

Movie CineGraph TMDb ID CineGraph IMDb ID Wikidata entity Wikidata IDs English Wikipedia sitelink
Titanic (1997) 597 tt0120338 Q44578 TMDb 597; IMDb tt0120338 Titanic (1997 film)
Everest (2015) 253412 tt2719848 Q15631013 TMDb 253412; IMDb tt2719848 Everest (2015 film)

Both identifiers agree for both examples. CineGraph's existing movies(tmdbIds: ...) query returns the title, TMDb ID and IMDb ID with the current configured consumer credential. No API change is demonstrated necessary for this vertical slice. The local CineGraph database also contains both identifiers for these two films and for the newer mountain-shelf results Where is Ava? and Le Son du Temps. Those newer films' Wikidata coverage was not verified.

Precise readiness conclusion

Ready to implement the film-first milestone. This proves available identifiers and a working API read; it does not prove the unimplemented local resolver, local film page creation, backlinks or complete catalog match coverage. Do not claim guaranteed Wikipedia matches. No issue-93 application changes were made as part of this probe.

Required lookup strategy and fixtures

  • Start with CineGraph's stable TMDb movie ID; retain its IMDb ID when present.
  • Search the local verified identifier index first. Project Wikidata P4947 (TMDb movie) and P345 (IMDb) into compatible namespaces.
  • If additional remote resolution is needed, use a bounded exact-identifier lookup through the existing import/job infrastructure. Do not download the whole Wikidata graph or scan titles on each page render.
  • Treat zero matches, multiple matches, deprecated/ranked statements, unsupported qualifiers and contradictory identifiers explicitly. Respect existing statement/evidence policy; do not blindly verify every raw statement.
  • A remote lookup must have bounded retries/timeouts, request deduplication and configurable negative-result freshness. Do not query Wikidata repeatedly for every visitor to an unmatched film.
  • Missing Wikipedia/Wikidata linkage does not prevent an eligible source-backed local film entry. Its stable URL must survive later enrichment.
  • Check English sitelinks when available, without assuming that every language or every film has an article.
  • Capture small sanitized provider/Wikidata fixtures for the two verified examples. Test both import orders, then click a discovered film to the same local page.
  • Add a deterministic fixture for a movie with a valid CineGraph identifier but no Wikidata match, plus a conflicting-ID fixture. These should be labeled fixtures, not falsely presented as live negatives.
  • Measure actual resolution coverage in a bounded sample before broad rollout. Report matched, locally created without a crosswalk, unresolved and conflicting counts separately.

The mountain shelf remains a keyword association: this issue changes the card destination to a local film entry; it does not claim every matching film is a reviewed illustration of a particular mountain sense.

Activity

  1. holden commented on Sep 17, 2026

    @holden
    MemberAuthor

    Closing: delivered by PR #95, merged 2026-09-12.

    Verified on main @ 1380564 (2026-09-17): docs/integrations/source-identity.md, test/.../film_identity_flow_test.exs, and external_identifiers in devils_dictionary_v2 carrying 49 tmdb_movie / 15 imdb_title rows. #86 and #99 both already build on this contract, and the identity adapter is duck-typed (function_exported?(provider, :identity_record, 1) in discovery.ex), so a new provider gets it without wiring.

    The art/quote reuse of the contract is sequenced in the 2026-09-17 discovery triage issue rather than here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions