Repository navigation
Description
Goal
Use the remaining Artsy public-API window to establish a useful art discovery integration and a selected catalog of culturally significant works, with explicit retention rights and an independent-source fallback.
Readers should encounter relevant art beside a meaning; contributors should search known artists/works and propose their own connections. Artsy's categorization may accelerate discovery. We want a reusable encyclopedia collection, not an art marketplace or an indiscriminate copy of every listing.
Parent rollout: #85. Public connection/review correctness depends on #84; media presentation belongs with #66/#67. The bounded access/coverage investigation below can happen now, ahead of the film-first rollout, because the API is retiring. It must not wait for all of #84, and it must not become a broad schema rewrite.
Critical finding: separate access from durable ownership
Research date: 2026-09-11. Artsy announces public API retirement without a firm shutdown date on the documentation page. Partner integrations are described as unaffected. Authenticated live calls on 2026-09-12 verified credentials and selected endpoints; see the live readiness audit below for measured scope and remaining gaps.
The API terms, particularly §§3, 4, 6 and 18, permit reasonable caching with freshness obligations, specify non-commercial use and attribution, and require removal of API content/copies upon termination, subject to applicable separate licenses. They also address changes/unavailability within 24 hours of notification. A public endpoint is not permission to build a permanent post-retirement mirror.
Therefore deliver two explicit modes:
- Discovery/cache mode: use supported access within its applicable conditions; provider content can expire or be removed.
- Durable collection mode: retain only material covered by independently verified open licenses or a separate agreement. Document metadata, descriptions, gene assignments and images separately. A public-domain painting does not automatically license Artsy's descriptive text or classification data.
Do not promise a permanent Artsy archive before resolving this distinction. If persistent Artsy genome data requires permission, prepare a concrete request describing the selected scope and intended use; sending it is a separate owner-authorized action.
What the previous application actually did
Verified from ruby-rails-archive, not from a live old deployment:
- ArtsyService exchanged client credentials for an XAPP token and searched
/api/searchwith the topic/query text. It passedtype=artwork, which must be revalidated rather than assumed supported. - Topic page fetched artwork suggestions from the topic title.
- Picker/controller supplied search/select and a path to save an
Artreference attached to a topic; the result partial also submitted to the generic media route. Trace the actual route in any restoration. - It extracted a thumbnail/title and attempted artist/date extraction directly from search results, without a complete resource-hydration pass. It constructed website URLs by rewriting API URLs.
- No catalog harvest, gene ingestion, semantic mapping or durable offline art library was found in this path.
Restore the interaction, not these assumptions. Use actual IDs and permalinks, hydrate resource links, handle missing artists, and never copy its token-logging behavior. Implement HTTP calls with the project's Req client.
How the documented API fits this project
| Capability | Documented mechanism | Implementation consequence |
|---|---|---|
| Authentication | Client credentials → XAPP token; X-XAPP-Token header |
Server-side secrets/token lifecycle; redact logs. Authentication. |
| Search | /api/search?q=...; mixed resource types |
Filter types and hydrate results; search covers more than the retrievable artwork subset. A hit may resolve to 404. Record corrections/redirects. Search. |
| Artworks | /api/artworks, /api/artworks/{id}, artist/partner/show filters; linked artists, genes and similar works |
Follow actual resource links. Store source IDs, titles, dates, medium, available attribution/rights, timestamps and permalink. Similarity is a discovery signal. Artworks. |
| Artists | Artist resources and artwork/genre-related traversal | Preserve collective/unknown creators; identity is not just a name string. Artists. |
| “Genes” | Characteristics such as subject, style, technique or visual properties; linked from artists/artworks; gene → artwork links | Useful source vocabulary, not our dictionary meanings. Preserve gene IDs, labels and evidenced assignments distinctly. Genes. |
| Pagination | Follow _links.next; cursor/offset modes |
Resumable traversal; optional totals are not a completeness guarantee. Pagination. |
| Throttling | Documentation currently specifies 5 requests/sec per application | Start conservatively below this, share one limiter, handle 429/backoff and enforce total budgets. Rate limits. |
The artwork example includes iconicity, but the documentation does not establish a reliable global “most important art” ranking contract. Inspect its actual availability/meaning before using it, and never equate it with relevance to a word. Gene docs distinguish public-domain and published-artwork links; one example link appears inconsistent with its gene context. Test returned links rather than trusting examples blindly. No assumption that the complete Art Genome Project, all weights, or every famous work is exposed.
P0 — Immediate bounded feasibility report
Suggested first implementation session; stop with measured findings if access is unavailable rather than building a speculative importer.
- Locate configured credential availability without printing secrets; verify the intended application's access and applicable retention/display policy.
- Exercise token exchange, one artist lookup, artwork hydration, genes, gene traversal, search and pagination with a shared limit and maximum 200 requests for the initial probe. No image download is needed.
- Probe a declared sample of 10 known artists/works and 10 meanings. Include nepotism, war, grief, love, power, family, solitude, justice, bank and Pop Art; record exact queries, null matches, redirects, retrievable IDs and useful gene assignments.
- Compare direct
nepotismsearch with explicitly proposed broader queries such as family/power/patronage. Broader queries are not synonyms or evidence of nepotism; label them as exploratory. - Produce a capability matrix: documented / live verified / unavailable; count search hits separately from hydrated records and usable media.
- Record request cost, reachable coverage, rights findings and recommended collection size. Sanitize fixtures; apply retention rules to raw responses, fixtures and backups too.
Exit decision: supported Artsy discovery + permissible collection; separate permission needed for durable retention; or no viable access, proceed with the same feature backed by open museum data. Report which, not “all Artsy imported.”
P1 — Select important art intentionally
- Create a versioned seed manifest of recognized artists and specific landmark works, selected by the owner/curators and/or independently sourced museum highlights. Include selection reasons and identifiers.
- Prefer important works over indiscriminately ingesting every print or sale listing by a famous artist. Distinguish underlying work, edition and individual object; title similarity alone must not merge them.
- Initial proposed cap: 50 artist seeds, 500 accessible artwork records and their available genes. Adjust only after P0's coverage and request accounting; a cap is not a promise that 500 qualifying records exist.
- Prioritize seed membership, independently evidenced museum significance and useful subject coverage. Track period/geography/medium coverage so the collection is not accidentally only one market segment. No global ban on anonymous historical works merely because the maker is unknown.
- Make fetching resumable/idempotent with run manifests, deduplication, bounded related-record depth, per-record status and final coverage report. Do not recurse through every similar artist/work.
- Widen the selected corpus only when measured quality and retention conditions justify it. “Entire repository” is not this issue's completion criterion.
Required: repeatable Wikidata/Wikipedia-linked artwork seeding
Provide an operator-facing Mix task (or equivalent existing import entry point) to discover and import a substantial collection of known artworks with verified Artsy links. This is a required deliverable, not a manually maintained shortlist only.
- Start from both existing local Wikidata-linked artwork entries and a bounded, paginated Wikidata candidate manifest containing Artsy artwork identifiers (P11005). Wikipedia sitelinks may help prioritize recognizable works, but a Wikipedia article is not mandatory and a title is never an identity match.
- Resolve Wikipedia selections through their Wikidata identity, then the verified Artsy artwork identifier. Keep paintings as the initial scope; make supported artwork kinds and selection criteria explicit and configurable.
- Offer dry-run, configurable batch/request/record limits, resumable checkpoints, and a saved manifest. Start with the existing 500-record pilot cap; support deliberate larger runs after the pilot reports coverage and quality. Never make the default an unbounded crawl.
- Hydrate the Artsy record, verify its returned identity, reuse an existing local artwork when identifiers agree, or create an eligible source-supported entry using the shared identity contract. Resolve available creators to reusable artist entries. Do not overwrite conflicting identifiers or merge originals, reproductions, editions, or same-title works.
- Persist provenance and supported metadata through existing source records. Record image references and applicable rights separately; downloading image files requires an explicit supported retention policy. Bulk metadata seeding does not establish permission for permanent Artsy copies.
- Report candidates discovered, matched existing entries, created entries, conflicts, unavailable records, missing links, skipped records, requests consumed, and resumable position. Re-running an unchanged manifest must not duplicate works, creators, or connections.
- Do not infer definition/meaning relationships merely because an artwork was imported. Populate the reusable encyclopedia catalog first; semantic connections retain their separate evidence/review rules.
- Document copyable run commands for a small preview, pilot import, resume, and larger bounded import, including where the operator sees the resulting clickable artwork pages.
Acceptance: demonstrate a live bounded pilot containing at least one pre-existing entry and one newly created entry; show artwork and creator navigation, verify exact Wikidata/Artsy reuse, and replay the same manifest without duplicates. Automated tests cover interruption/resume, unavailable artworks, identifier conflicts, reproduction separation, quota exhaustion, and source withdrawal. Report actual attainable coverage rather than promising all famous paintings exist in Artsy.
P2 — Reusable local art catalog and automatic candidates
Use the existing registry/content/claim structure; document the smallest required additions before coding:
Artwork identity ── creator relationship ──> Person / collective
│
├── Artsy record: ID, permalink, fields, policy, revision
├── Museum record: independent identifiers, fields, policy
└── Artsy gene assignment ──> Artsy gene identity
│
reviewed vocabulary mapping
↓
Dictionary meaning
Mapping + assignment → candidate connection → review → featured example
- Keep source fields and support separately removable. Independently obtained museum evidence must actually support surviving fields; adding a museum link does not make Artsy's text/genes independently licensed.
- Separate image permissions from metadata; a missing/unusable image should produce a permissible text/link card rather than break the entry.
- Build a reviewed mapping table from exact meaning/language to provider gene or discovery query, with exact/broader/related match type, provenance and version. Never equate gene labels with word senses by string equality alone.
- A source gene assignment can automatically generate a labeled candidate. A keyword search hit supplies a weaker, separately recorded reason. Neither automatically creates an accepted “illustrates nepotism” claim.
- A direct source classification such as a movement can display as an attributed source relationship when supported; the editorial decision to feature the work remains distinct.
- Index permissible local title, artist and tag fields for future search without provider calls. Keep optional live search separate. No network fan-out on every definition view.
NEPOTISM · favoritism toward relatives
Art suggestions [Add known work]
[Artwork thumbnail or text card]
Found through: exploratory query “patronage”
Connection to nepotism: NOT YET REVIEWED
[Inspect artwork/source] [Explain why it fits] [Dismiss]
ADD KNOWN WORK
Search local catalog [title / artist ...........] [Optional live Artsy]
[Select work] → [Select exact meaning] → [Rationale + evidence]
[Submit for review]
Later contributors can attach one artwork to multiple meanings with independent explanations, without duplicating the artwork. Removing a connection must not remove the work elsewhere. Publication waits for #84's review/visibility guarantees.
P3 — Survive the provider disappearing
The Met API/Open Access dataset is a strong comparison: CC0 selected metadata, public-domain image availability, subject tags, stable object IDs and isHighlight are directly useful for a selected, durable corpus. Its current docs announce /v1.1/search pagination and retirement of the old search endpoint on October 1, 2026; use the current interface. The Art Institute of Chicago API is another candidate with searchable collection metadata and public-domain indicators.
- Compare at least 20 selected works/queries across Artsy and one open museum source for retrievability, known-work coverage, subject tagging, usable images and retention conditions.
- Choose whether Artsy is primary discovery, optional enrichment or unavailable. Prefer independently licensed museum records/images for durable storage where available.
- Implement a provider shutdown/termination operation that disables discovery and removes disallowed source payloads/projections/cache copies while preserving independently supported local objects and original editorial work where permitted. Re-review connections losing their evidence.
- Do not promise Artsy classifications survive shutdown without the necessary license. The long-term catalog may retain the work through another source while losing Artsy's description, image or genes.
Acceptance / deliverables
- P0 report and go/no-go decision published first; no fabricated live-access claims.
- Declared seed corpus, successful/skipped counts, available gene coverage, retention matrix and reproducible bounded run commands.
- Automatic suggestions show their actual match reason and exact meaning. Test wrong-sense matches, no result, inaccessible search hits, absent genes and an unsupported filter.
- Local search/select/review demonstrated on desktop/mobile; no raw credential/token logs; provider URLs/redirects validated before following.
- Tests for pagination resume, 429, token expiry, duplicates/editions, multiple creators, source withdrawal and API-off local operation;
mix precommitfor changes. - Explain how many useful reviewed art examples we gained, not merely downloaded-record counts. First target: 10 meanings with genuinely useful reviewed examples where the corpus supports them; report gaps rather than force matches.
This issue authorizes planning and a bounded implementation path, not an indiscriminate archival crawl. Its value is earlier art discovery and a reusable selected collection with an honest plan for continued use after Artsy's public API disappears.
Live readiness audit — 2026-09-12
Decision: GO for a bounded Artsy adapter and selected-seed implementation; NOT a verified permanent archive or unrestricted automatic semantic matching. Credentials work. P0 is partially completed; retain the remaining coverage, rights and museum-comparison gates. No integration code, database import, or image download was performed by this audit.
Measured API evidence
39 Artsy requests attempted across the probe runs, including authentication and one failed token request. Successful runs used an identifying User-Agent; one request without that header received 403 (correlation, not proof of cause). Credentials/tokens were never printed or written to the report.
- Token exchange, artist lookup (Vincent van Gogh), artwork hydration (Klimt's The Kiss), artist-to-artwork traversal, and successive artwork and gene pages succeeded.
- Tested these ten meaning queries, first five results each: nepotism, war, grief, love, power, family, solitude, justice, bank, Pop Art. All returned search hits; this is NOT ten useful art examples. “bank” ranked Banksy; “war” ranked Warhol and exhibitions. Exact meanings require reviewed mappings.
- Tested ten known-work query strings: Mona Lisa, The Starry Night, The Kiss, The Third of May, Guernica, The Scream, Water Lilies, The Birth of Venus, American Gothic, The Great Wave. Title search frequently ranked other works/reproductions and exhibitions. Do not infer canonical identity from titles or search rank.
- Goya's The Third of May appeared as the expected search result; Klimt's The Kiss was hydrated by known ID. Van Gogh traversal returned Self-Portrait with Bandaged Ear and subsequent works.
- “nepotism” returned Adam Ames, Nepotism (2014), but its artwork endpoint returned 404. Search visibility does not establish retrievability. Broader patronage search returned artwork hits, not verified evidence of nepotism.
type=artworkproduced artwork-only samples for war and Mona Lisa. However returned search self/next links omitted that filter. Preserve the validated filter when paginating, validate returned result types, and regression-test this case.- The Kiss's genes included Art Nouveau and further pages. Following an Art Nouveau gene artwork link returned both The Kiss and The Third of May—the same first two items as the unfiltered artwork list. Gene-filter semantics remain unverified/suspicious; do not treat this traversal as evidence of a gene assignment. Verify per-work assignments and compare control queries before enabling gene-based acquisition.
- Artwork records contain image links and rights strings; images were not fetched and usable-media counts are not established. The Kiss names a Wikimedia Commons/public-domain source; Goya names the Prado without an explicit open license. Neither permits a blanket retention assumption.
Exact identity wiring required
Reuse #93's source identity contract, which exists in the inspected checkout. Artwork classification exists, but Artsy identifier projection was not found in the inspected Wikidata adapter.
- Wikidata P11005 identifies Artsy artwork slugs, not the API's opaque record IDs.
- Wikidata P2042 identifies Artsy artist slugs.
- Live proof: the P11005 example
rembrandt-van-rijn-the-anatomy-lesson-of-dr-nicolaes-tulpsuccessfully hydrated the corresponding artwork endpoint. - Keep opaque provider IDs and validated slug aliases in distinct identifier namespaces. Match Wikidata slug properties against provider-confirmed slugs; never compare them directly to opaque API IDs. Handle changed slugs/conflicts through reconciliation.
- Preserve artist/work separation, multiple creators, originals versus reproductions, and incomplete crosswalk coverage. Missing Wikidata links must not force a title merge or prevent an otherwise eligible local entry.
- Add fixtures covering exact artwork and artist crosswalks, changed aliases, conflicting IDs, absent Wikidata coverage, and a reproduction with a canonical-looking title.
Implementation order and remaining gates
- Confirm Unify source identities into clickable encyclopedia entries for films, artwork and quotes #93 is in the chosen base; build server-side authentication, bounded requests and normalized artwork/artist records using the existing contract.
- Start with a small explicit set of known artwork IDs/slugs. Demonstrate one clickable local artwork page, its creator link, and exact Wikidata reuse without duplicates.
- Implement search hydration failures, filter-preserving pagination, and measured gene-filter checks before any larger seed import.
- Complete remaining P0 evidence: hydrated/usable counts across the sample, verified gene traversal, retention matrix and meaningful coverage. The 20-query search probe alone does not complete P0.
- Complete the planned comparison with an open museum source before choosing Artsy as the primary durable corpus. That comparison was not performed in this live audit.
- Keep source removal and independent provenance requirements. Artsy still announces retirement; working credentials today do not guarantee future availability.
Readiness grade: B+ for the full integration, with access validated and concrete implementation risks now documented. Ready to start the bounded adapter and identity slice; not ready to promise complete famous-work coverage, automatic meaning correctness, permanent Artsy retention, or issue closure.
Activity
holden commentedon Sep 12, 2026
Artsy feasibility and open-source comparison — issue #86
Research and live-probe date: 2026-09-12. The issue's earlier readiness audit is the
baseline. This report completes the remaining bounded evidence before implementation.
It is a measurement of current API behavior, not a promise of future access or a
license opinion.
Exit decision
GO for Artsy as optional, disposable discovery and exact-identifier enrichment.
Do not use Artsy as the durable primary collection. Wikidata supplies cross-source
identity. Independently licensed museum metadata and images, starting with the Met,
are preferred for durable display. Artsy descriptions, gene assignments, image URLs,
and cached responses must remain source-specific and removable.
The first pilot stays capped at 500 manifest candidates and 500 created/matched
records, with smaller operator-configured request and batch limits. Given the live
search hydration rate, start with Wikidata P11005-linked records rather than broad
title search. Widen only after the pilot reports its actual coverage.
Probe bounds and declared sample
429 responses. Combined issue evidence: 132 Artsy requests, below the 200-request
initial-probe ceiling.
/public/collection/v1.1/searchendpoint and object hydration endpoint.in process memory and never printed or written.
Guernica; The Scream; Water Lilies; The Birth of Venus; American Gothic; The Great
Wave.
Pop Art.
Measured Artsy behavior
type=artworkbehavior_links.nextwarpage 1 returned only artworks, but its next link again omittedtype=artwork; explicitly restoring the filter kept page 2 artwork-onlyrembrandt-van-rijn-the-anatomy-lesson-of-dr-nicolaes-tulpand opaque ID4eb063a5b1976400010071f0artsy_artwork_slugandartsy_artwork_idin distinct namespaces. Follow only validated redirects.vincent-van-goghredirected, then returned opaque ID4d8b92944eb68a1b2c000264gene_id=19th Centuryartwork IDs were identical to the unfiltered first 10The three hydrated search records were:
4d8b92ee4eb68a1b2c0009ab, expected known work.515bb06394714c2e38001285, a different work from the famous Botticelli title.515d45217b70570a13003fc1, meaning-search candidate only.No meaning query produced an automatically acceptable semantic connection. The probe
therefore gained zero reviewed art examples.
family,war,grief, and otherqueries may support future human-reviewed candidates;
bankand title-like queriesdemonstrate wrong-sense/creator/reproduction risk.
Met comparison over the same 20 queries
The Met v1.1 highlight-and-image searches returned 48 sampled object records. Of those,
30 exposed a public-domain image reference and 33 had subject tags. Four queries had
no highlight result: Mona Lisa, Guernica, nepotism, and solitude.
The Met covered a recognizable Great Wave object (ID 39799) and supplied strong
open candidates for grief, family, and justice, but it also returned broad or unrelated
matches. For example,
American Gothicreturned a cabinet, Jubal and Miriam, and apitcher. Open licensing solves durability, not semantic relevance or identity.
Retention matrix
isPublicDomain=true, retaining source/credit metadataArtsy's terms permit reasonable freshness-oriented caching, require changes/removal
after notice, and require API content/copies to be removed after termination. API
availability or a public-domain underlying painting does not license Artsy's text,
classification, or image rendition.
Meaning and acquisition policy
family,power, orpatronageare exploratory and are notsynonyms or evidence for
nepotism.meaning only through a versioned reviewed mapping. A search hit records the exact
query and is weaker.
illustratesclaim. Existing evidence/review rulesremain authoritative.
returned population as documented.
Sources
holden commentedon Sep 12, 2026
Independent audit: Add Wikidata-linked Artsy artwork catalog
PR: #98
Audited head: 5de3858
Date: 2026-09-12
Grade: C+. Do not merge or close issue #86 yet. Substantial implementation and useful research exist, but the completion claim exceeds the verified behavior. The stacked identity PR is not the only blocker.
Verification performed
/private/tmp/artsy_audit_test.exs: 5 tests, 5 failures. They reproduce current-record evidence loss, post-withdrawal provider calls, incomplete imports marked complete, rejected creator visibility, and empty creator data in local candidate discovery. All use sandboxed test data and stubbed HTTP; no production writes or new Artsy API requests.Merge blockers
1. P1 — Preserve existing Wikidata source evidence when seeding
lib/devils_dictionary/artworks/seeder.ex:452and:482.The seeder replaces the current Wikidata source revision for the same QID with a synthesized minimal snapshot. It drops images, aliases, other claims and original statement evidence from the current payload. Historical revisions remain, so this is not physical deletion, but the reduced record becomes authoritative for current replay/enrichment. The regression supplies a richer current record containing P18; after seeding, P18 is absent from its current payload. Local candidate labels/descriptions can also be reasserted as Wikidata evidence even though they came from a local projection.
Fix: retain actual source observations/provenance; do not replace a full record with a partial manifest projection. Reuse full retained records or acquire the supported fields with explicit partial-record semantics. Test artwork and artist records with existing rich data and cross-source/local edits.
2. P1 — Honor public review decisions for creators
lib/devils_dictionary/artworks.ex:293.Creator cards query current active revisions without the shared public visibility policy. A rejected authored-by assertion remains visible as “by [creator]”. Reproduced directly. The search query also left-joins the predicate without requiring a matching predicate, so unrelated or hidden relationships can affect creator-name search.
Fix: reuse the shared visibility-filtered relationship query in both display and search. Test rejected, withdrawn, hidden-source and unrelated-predicate relationships.
3. P1 — Shutdown must stop provider access
lib/devils_dictionary_web/live/artwork_live.ex:15,:42;lib/devils_dictionary/artsy/client.ex:35;lib/devils_dictionary/artworks/seeder.ex:28.Availability only checks credentials. The documented
enabled: falseswitch is not read, and neither live lookup nor the seeder checks the withdrawn source before requests. The audit withdrew Artsy, then seeded again: four HTTP requests still occurred. The shutdown transaction alone does not enforce shutdown across callers or in-flight work.Fix: one effective enablement gate used by all entry points, shared state checked before requests/writes, plus a test showing zero calls and no new payloads after withdrawal. Retain independent identities and editorial content.
4. P1 — Partial import must remain resumable
lib/devils_dictionary/artworks/seeder.ex:240and:272;lib/devils_dictionary/artworks/manifest.ex:55.If authentication and artwork fetch consume the remaining budget, artist/gene calls fail, but the work is still marked created/matched. Those statuses are terminal for resume, so the missing creators/genes will never be fetched by
--resume. Other transient/auth failures become terminal skipped records too. Reproduced with a two-request budget.Fix: track completion stages, distinguish temporary from permanent failures, retain successful data, and resume unfinished stages without duplicate records/links. Artist/gene pagination must also be completed or explicitly checkpointed; current collection helpers read only one page.
5. P1 — Preserve creator identity through local seeding and provider loss
lib/devils_dictionary/artworks/wikidata_candidates.ex:64;lib/devils_dictionary/artworks/seeder.ex:349,:420.Local candidate discovery emits an empty creator list, even when creator QIDs are known. It can displace the richer remote candidate during deduplication. Creator creation is then dependent on a successful Artsy artwork/artist response and a matching creator slug. Missing Artsy artworks do not acquire available Wikidata creator links. Creator assertions that are created are Artsy-owned, so withdrawal removes that relationship despite available independent P170 evidence. The live pilot visibly has four works without creator navigation.
Fix: resolve and materialize independently evidenced Wikidata creator relationships even if Artsy is unavailable; preserve that support on withdrawal. Merge candidate evidence across local/remote and page boundaries rather than keeping the first row. Test multiple creators, missing P2042, and a creator split across SPARQL pages.
Remaining issue-completion gaps
6. P2 — Shared limits and freshness are not implemented
The client sleeps per client instance; visitors and import processes do not share a limiter. The browser's 30-request counter resets on reload and is updated only after async completion. This is not the application-wide limit specified in the issue. Retry-After is capped at ten seconds even when the server asks for longer. Wikidata's internal Req retries are not counted in the reported discovery request budget. Persisted Artsy records have no enforced refresh/expiry mechanism despite being described as short-lived cache.
Fix shared provider pacing/backoff, actual attempt accounting, and an explicit freshness policy. Test concurrent users/imports and a long Retry-After without waiting in real time.
7. P2 — Mapping claims are stronger than the implementation
lib/devils_dictionary/artworks.ex:359;priv/artworks/meaning-mappings-v1.json.The ten “exact sense” mappings use lemma/language plus substring matching against glosses, and gene-name equality; none pins a stable gene ID or sense identity. They lack recorded review/provenance. The UI omits the broader/related match type and explanatory note. This is especially important for Family → nepotism. Suggestions remain unreviewed, which is correct, but that does not make the mapping exact or reviewed.
Fix stable meaning/gene identities, mapping review provenance and version, and display the actual broader/related reason. Preserve candidate-only behavior. Demonstrate useful real candidates and honestly report unsupported meanings.
8. P2 — Reader/catalog flow remains incomplete
The live five-work catalog has no thumbnails: the seed snapshot omits P18 and the view never uses the retained Artsy thumbnail. Text fallback is valid when media is unavailable, but no independently supported image path was demonstrated. The local catalog caps results at 50 with no pagination. Local search covers title/relationship target names, not retained tags. Remote available results not already local have no save/select action; the page is only an availability check. Candidate-to-composer navigation passes the work but loses the exact candidate sense/evidence.
Complete the issue's actual local search/select/review path, preserve candidate context, supply permissible images where available, and demonstrate access beyond the first catalog page. Do not hide genuine missing-image cases.
Positive findings
Recommended order
No application source was changed, no PR or issue was closed, and nothing was merged or deployed in this audit.
Owner clarification and A+ acceptance bar
The owner requires a rich seed, not bare identity cards. Missing images in the current live pilot are an implementation gap: the seeder omits Wikidata P18 and the UI does not consume the Artsy thumbnail reference it retains. Implement an explicit source-supported image/credit path with the applicable display/retention policy. Show real images on both catalog cards and artwork pages wherever permitted images are available; preserve a truthful fallback only for genuine unavailable/restricted media. Include creator, date, medium, collection, source links, and supported descriptions where the sources provide them. Preserve all richer existing observations and never relabel local edits as source facts.
Quota clarification: the audit failure was reproduced using a deliberately small local client request budget, not an exhausted real Artsy account allowance. Both local request-budget exhaustion and provider 429 responses must leave incomplete imports resumable.
A+ requires all blockers and completion gaps above addressed with regression tests, a richer bounded live seed showing existing-entry reuse and new-entry creation, visible artwork thumbnails and creator links, useful exact-meaning candidates, and the complete contributor selection/review flow. Report actual created/matched/unavailable counts separately; do not describe five created local identities as one creation just because only one Artsy record hydrated. No indiscriminate crawl or invented data is requested. Keep this issue open until independent re-audit.
holden commentedon Sep 12, 2026
Independent-audit repair complete on PR #98
Updated existing PR #98 at
19daf6b; no new PR was opened. Issue #93's shared identity work is in the base through merged PR #95. Nothing was merged/deployed and this issue remains open for independent audit.Audit findings
Live bounded results
Total Artsy campaign ceiling: 189/200 attempts; no image bytes downloaded and no credential/token printed or committed.
matched: 2,created: 0,unavailable: 0; one already rich entry reused, one existing Wikidata entry newly Artsy-hydrated, 1 creator link, 28 genes.created: 0, unavailable: 1; no provider data/image was invented.Desktop and narrow mobile-width Chrome verified the seven-work catalog, local filtering, rich Ranuccio page, new Les Âmes Déçues page, Titian/Hodler navigation, and the authenticated composer preselected with exact Cupid + WordNet
waridentities,illustrates, immutable source revision, opaque gene locator, and rationale. No claim was submitted.Checks and honest gaps
mix precommit: 934 tests, 0 failures (the two existing ignored corpus archives were temporarily linked read-only from the original Dictionary checkout, then removed).load_in_query: false(about 660 MB aggregate). The repository regression usesSources.raw/1and proves exact rich payload/P18 preservation. Re-enabling default payload loading would undo Unify source identities into clickable encyclopedia entries for films, artwork and quotes #93's memory-safety fix.Full requirement ledger, commands, retention matrix, live manifests, and browser evidence:
docs/audits/2026-09-12-issue86-repair.md.PR #98 is ready for independent re-audit. Please do not merge or close #86 based solely on this implementation report.
holden commentedon Sep 12, 2026
PR #98 CodeRabbit follow-up is complete and pushed through 2e5fbf7. The additional repair replaces browser-reset request accounting with an atomic server-wide allowance, makes malformed artwork/artist payloads and missing coordinators fail closed, reports Wikidata-only creator gaps, wraps creator assertion/evidence writes transactionally, makes Wikidata request counts physical, and separates committed portable manifests from ignored resumable checkpoints. All committed pilot manifests are pending and contain no local database IDs. Verification is now 72/72 changed-file tests and 944/944 mix precommit tests. Full details: #98 (comment). Reviewed examples remain honestly zero pending normal human review; this is not represented as completed editorial coverage. PR remains unmerged and ready for independent re-audit.
holden commentedon Sep 12, 2026
Independent repair re-audit: B — not ready to close
The basic Artsy integration works, but the current result does not satisfy the image coverage, rich seeding, and reliable discovery requirements. Keep this issue open and hold PR #98 until the following fixes and acceptance checks pass.
What was actually verified
Required repairs and acceptance criteria
lib/devils_dictionary/artworks.ex,suggestions/1requests 500 catalog entries, but the catalog clamps the result to the first 100 titles. The assignment scan also stops at 500. An independent regression reproduced this: one matched artwork produces one suggestion; adding 100 earlier-sorting artworks causes that suggestion to disappear. Query relevant assignments first and load their specific identities, with deliberate pagination or bounds rather than unrelated global cutoffs. Test matching works beyond both boundaries, including irrelevant earlier records.Architecture conclusion
The integration validates reuse of durable local identities, source records, encyclopedia pages, and relationships for both films and art. It does not yet prove bulk-scale discovery or a generic source loader. Provider retrieval, identifier matching, metadata rules, and retention remain source-specific. These findings do not justify introducing another universal entity table or rewriting the existing model.
Test evidence
Completion gate: fixes covered by regression tests; rich-data backfill and bounded repeatable seeding demonstrated; real imagery and complete navigation verified; then independent re-audit. No code changes, merge, deployment, or issue closure were performed as part of this audit.
Repair issue 86 second independent re-audit findings
holden commentedon Sep 13, 2026
Re-audit repairs pushed to PR #98 — ready for independent re-audit
Head is
839527aoncodex/issue-86-artsy-catalog. Nothing was merged, deployed, or closed, and this is not an independent re-audit.This round spent zero Artsy HTTP attempts. Every fix and every proof below uses retained provider data, local fixtures, the shared Wikidata adapter, or the browser. The cumulative campaign stays at no more than 194 attempts, under the issue's 200-request ceiling.
Each audit item, its fix, and its evidence
1. Discovery silently dropping artworks as the catalog grows.
Artworks.suggestions/1now queries the relevant Artsy gene assignments first, paginated byoutput_object_idin 500-row pages, then loads exactly those identities. The unrelated 100-title catalog clamp and the 500-row assignment cutoff are both gone.Evidence: a deterministic fixture builds 501 artworks and 501 assignments in which only the last matches, including 500 earlier-sorting irrelevant records; the suggestion is found.
2. The interactive request allowance never renewing.
RequestCoordinatorrecords a per-scope window start and resets that scope's usage once the window elapses.ArtworkLivereadsinteractive_request_limitandinteractive_request_window_msfrom config, defaulting to 30 requests per 60 seconds. Finite per-import ceilings are untouched and remain separate.Evidence: a test drives shared accounting, exhaustion and recovery through an injected clock.
3. Queued HTTP requests sent after provider disable or withdrawal.
After the paced sleep the client revalidates its reservation through the coordinator and rechecks availability and generation immediately before transmission. Callers already queued re-wait provider-wide
Retry-Afterdeferrals. The existing protection against publishing a response invalidated in flight is preserved.Evidence: the mocked regression that disabled the provider mid-wait now observes zero transmissions; a second test asserts a queued caller respects a later
Retry-After.4. Duplicate visible creator and work relationships.
Encyclopedia sections group identical semantic relationships — subject, predicate, object, context, jurisdiction, language and validity — for both display and counts, keeping one representative per group. The seeder attaches Artsy evidence to an existing
authored_byclaim instead of creating a parallel one. No assertion or evidence row is deleted.Evidence: Ranuccio's page shows 1 creator link, previously 3. Titian's page lists 3 works, each once, with a matching count of 3, where it previously showed the same two paintings as six. Underneath, 14 active
authored_byassertions over 9 distinct work/creator pairs survive and render as exactly 9 visible links.5. Backfilling existing sparse artwork records safely.
mix dd.artworks.seed --wikidata-only --refresh-wikidatahydrates only artwork identities that already exist locally, plus their linked creator QIDs, through the existing bounded Wikidata adapter. It never adds catalog entries and spends no Artsy requests. Independent Wikidata data and editorial work are preserved; unavailable records are reported rather than promised.6. Verifying imagery end to end.
An
ArtworkImagehook drives each card from the realloadanderrorevents intoloading,loaded,errororempty, showing the placeholder when decoding fails. Image URL and credit are now resolved as a pair.Evidence: see the browser table below, reporting decoded pixel dimensions rather than non-empty fields.
7. Larger, repeatable seeding.
The seed task reports candidates, local identities, images, Commons images, Artsy-thumbnail fallbacks, visible creator links and unavailable records separately. Scale past 100 artworks and 500 assignments is proven with fixtures, deliberately, so no live quota is spent to prove scale.
8. The word-page flow.
A definition page's artwork candidate now carries an internal contributor into the review composer with the exact sense,
illustrates, the immutable Artsy source-record revision, the opaque gene locator and the rationale preselected.WordLivemounts the current scope read-only;/connectstays gated on the server.Evidence: full browser walkthrough ending in a submitted claim held at
needs_review.Two defects the audit's insistence on real pages exposed
Both now carry regression tests that fail without their fix.
Meaning mappings
The enabled set went from four to seven. Before this, no retained artwork carried any enabled gene, so the catalog produced zero candidates and the word-page flow could not be demonstrated. The three added gene IDs were verified by direct observation on retained Artsy artwork records rather than by new probes, so they cost no requests: Allegory, Classical Mythology and Portrait, each bound to an exact Open English WordNet sense, each still
approved_for_candidate_generation_onlywith a note saying what the gene does not establish. The six unresolved slugs stay disabled.Bounded pilot and duplicate-free rerun
authored_byassertionsIdentity, assertion and link counts were identical before, between and after both passes.
Browser acceptance
/artworks/define/allegoryRelated Artsy gene "Allegory" · not yet reviewedwith its caution note, linking to the work and to TitianCourtesy National Gallery of Art, Washington, with no Wikimedia credit anywhere on the pageloaded→error, image hidden, placeholder shown, no broken-image glyphillustrates, sense #308977allegory, source-record revision 6984046 and the gene locator; submitting produced claim #3856927 atneeds_reviewwith its evidence attachedTwo of those states are not the default, and both were produced by reversible development-database changes to real code paths using real retained data, then restored: the independent image projection was temporarily removed from one entity to exercise the thumbnail branch, and
display_allowedwas temporarily cleared on the three Artsy source records — the same flag withdrawal sets. A destructive withdrawal was deliberately not run, because it permanently deletes retained payloads that cannot be rebuilt without provider requests this campaign no longer has; that path stays covered by the existing automated withdrawal test.Verification
mix precommit: 954 tests, 0 failures with the corpus archives present. Without them the corpus manifest test fails on two large downloaded archives that are not in the working checkout; supplying them read-only makes that file pass 11/11.deps.unlock --unusedand formatting all pass..env. No token was printed, logged, or committed.Remaining limitations, stated plainly
needs_review. Nothing was approved and no interpretation was invented.Requirement-by-requirement evidence is in
docs/audits/2026-09-13-issue86-reaudit-repair.md.Leaving #86 open and PR #98 unmerged for independent re-audit.
holden commentedon Sep 17, 2026
Closing: PR #98 merged 2026-09-13 11:55Z at head
2debb3a, three commits past the839527apush that the previous comment left "awaiting independent re-audit". The merge is the acceptance.Verified on
main@1380564(2026-09-17): 969 tests / 0 failures; 9 artworks inwork_details(work_kind: "artwork"), 3 Artsysource_records, 2artsy_artwork_id+ 1artsy_artist_ididentifiers (the 9artsy_artwork_slugrows came from Wikidata P11005, credential-free); ≤194 live Artsy requests cumulative per the two repair ledgers; the definition page reads the local catalog only (Artworks.suggestions/1is pure Ecto, no HTTP);mix dd.artsy.withdrawhides the section. Zero migrations shipped. The retirement is survivable because identity is Wikidata's.Residuals carried forward, not closed here:
Discovery.Providermodule has noretrieve/4; the catalog runs on its ownArtsy.Client+RequestCoordinator, outsideDiscovery.Budget/Discovery.Transport. Only CineGraph actually runs through the pipeline.All three go to #99 and the 2026-09-17 discovery triage issue.