Skip to content

Build 6: DevilsDictionary.Corpus.Seeder, and the quotations corpus seeded through the entry path (#174) - #177

Closed
holden wants to merge 1 commit into
claude/wikiquote-corpus-buildfrom
claude/wikiquote-corpus-seed
Closed

holden wants to merge 1 commit into
claude/wikiquote-corpus-buildfrom
claude/wikiquote-corpus-seed

Conversation

@holden

@holden holden commented Sep 24, 2026

Copy link
Copy Markdown
Member

Seeding for #174. Stacked on #176 (the build), which is stacked on #175.

The rename

DevilsDictionary.Artworks.Corpus.Seeder → DevilsDictionary.Corpus.Seeder (git mv).

  • The old name is a module that delegates to the new one (run/2, run_file/2, entry/3, catalog_counts/0) for one release.
  • The artworks corpus is unchanged, and its existing tests prove it. seeder_test.exs and the four corpus conformance suites are untouched and still call the old name. test/devils_dictionary/artworks (99 tests) is green.
  • The alias is not marked @deprecated: the gate compiles test/support with warnings as errors, and the old callers there are the proof. Remove it in the release after this.

The quotations kind

  • entry("wikiquote-pd", …) builds a content/quotation SourceIdentity.Entry in the live Wikiquote provider's shape. It is credited authored_by at :verified by the author's QID (P50 field), so a corpus line and a live line with one wording are one subject. There is a test with a live Wikiquote entry first, then the seed: one item, two credits (wikiquote, wikiquote-pd-v1).
  • Quotations.Corpus.seed_row/3 writes, per line:
    • a source record under wikiquote-pd-v1, keyed by fingerprint, whose payload is the manifest row
    • the build's checks as :supports evidence on the corpus credit (the Gutenberg locator, and the author's own page), once
    • the Verified badge on content_items.metadata["provenance"], only when the item has none. An item the verifier has already seen keeps its badge until the verifier's own clock.
  • Authors are prepared offline. They come from the manifest's selection.authors through Creators.classify/2, so seeding makes no request.
  • Source row wikiquote-pd-v1: tier: :aristocracy, kind: :corpus, access: :dump, CC BY-SA 4.0, and config holding dump_url, manifest_checksum and set_checksum (read from the committed file at compile time).
  • Durable, never swept. Discovery.cleanup/0 after a seed leaves every record, which is tested.
  • mix dd.artworks.seed --manifest priv/quotes/manifests/wikiquote-pd-v1.json seeds it. Against the test database, the whole committed manifest seeds as 1,053 created and 0 invalid.

What the code taught the issue

  • Build 6 of #158: the public-domain Wikiquote corpus — verified lines from the dump, committed as a checksummed selection, seeded as the 👑 layer of the Quotes shelf #174 asks for wikiquote_item as an identifier on each corpus line. It must not be one. It is a hash of page, section and position at a revision. A corpus pinned to revision R would propose the id of whatever line sits at that position on today's page, and resolution would either conflict or record another line as an alternate text of this one. The corpus's stable identifier is the fingerprint. wikiquote_item stays on the row as a locator.
  • The schema table's discovery_results row is wrong for this corpus, as it is for the artworks one. Neither writes discovery_results: a corpus reaches the shelf at read time (PR 4).
  • The artworks seeder's :refresh rewrites an entity's display keys. For a quotation it is skipped: a line's text is a revision and is never refreshed.

MIX_TEST_PARTITION=_corpus174 mix precommit: 16 doctests, 1,884 tests, 0 failures.

Refs #174.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Essentials

Run ID: 264872c5-6f84-4993-8493-ae250b43dd88

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

…eded through the entry path (#174)

The seeder moves to DevilsDictionary.Corpus.Seeder now that it holds two
kinds of content; DevilsDictionary.Artworks.Corpus.Seeder delegates to it
for one release, and the artworks corpora's existing suites still call the
old name, unchanged and green.

A wikiquote-pd row is a content/quotation SourceIdentity.Entry in the live
Wikiquote provider's shape, credited authored_by (verified) by QID, so a
corpus line and a live line with one wording are one subject. Its stable
identifier is the fingerprint, not wikiquote_item: that id is a position
on the page at a revision, and a pinned corpus proposing it would claim
whichever line sits there today. Quotations.Corpus.seed_row/3 writes a
source record per line (the manifest row), the evidence the build found
on the corpus's credit, and the Verified badge when the item has none.
Authors are prepared from the manifest's own Wikidata facts, so seeding
makes no request.

Source row wikiquote-pd-v1: aristocracy, corpus, dump, CC BY-SA 4.0, the
manifest checksum in its config. Records are never swept.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@holden
holden force-pushed the claude/wikiquote-corpus-seed branch from a3cbc8f to 056a56f Compare September 24, 2026 13:33
@holden

holden commented Sep 24, 2026

Copy link
Copy Markdown
Member Author

Consolidated into #178, which now targets main and carries this PR's commits unchanged, so it can be reviewed and merged as one. The review threads here are answered, and their fixes are in #178.

@holden holden closed this Sep 24, 2026
@holden
holden deleted the claude/wikiquote-corpus-seed branch November 7, 2026 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant