From c9985846fd2474a8511b1d06da5fbfb4235286bd Mon Sep 17 00:00:00 2001 From: Holden Thomas Date: Sat, 31 Oct 2026 19:27:27 -0500 Subject: [PATCH 1/2] Prepare ten-page quality retest with coverage and sense-definition leads --- docs/curation/agents-mvp.md | 32 +++ docs/curation/coverage-2026-10-31.md | 28 +++ docs/curation/scorecard-v2-2026-10-31.md | 100 +++++++++ .../curation/agents/persona.ex | 22 +- lib/devils_dictionary/curation/coverage.ex | 212 ++++++++++++++++++ .../curation/runtime/contract.ex | 4 +- .../curation/runtime/packet.ex | 10 + .../curation/runtime/section_output.ex | 8 +- lib/devils_dictionary/curation/whole_entry.ex | 5 +- lib/mix/tasks/dd.curation.coverage.ex | 70 ++++++ lib/mix/tasks/dd.curation.editor.ex | 3 +- .../agents/personas/voltaire/v7/profile.json | 51 +++++ ...mit_whole_entry_sense_lead_instruction.exs | 137 +++++++++++ .../curation/coverage_test.exs | 77 +++++++ .../curation/runtime/packet_test.exs | 17 +- .../curation/whole_entry_test.exs | 41 ++++ .../whole_entry_compatibility_test.exs | 31 +++ 17 files changed, 839 insertions(+), 9 deletions(-) create mode 100644 docs/curation/coverage-2026-10-31.md create mode 100644 docs/curation/scorecard-v2-2026-10-31.md create mode 100644 lib/devils_dictionary/curation/coverage.ex create mode 100644 lib/mix/tasks/dd.curation.coverage.ex create mode 100644 priv/curation/agents/personas/voltaire/v7/profile.json create mode 100644 priv/repo/migrations/20261101000313_admit_whole_entry_sense_lead_instruction.exs create mode 100644 test/devils_dictionary/curation/coverage_test.exs diff --git a/docs/curation/agents-mvp.md b/docs/curation/agents-mvp.md index 300756b1..599ffab8 100644 --- a/docs/curation/agents-mvp.md +++ b/docs/curation/agents-mvp.md @@ -1,3 +1,35 @@ +# Current next experiment: frozen quality preflight (31 October 2026) + +The 29 October trial remains a quality NO-GO. Read scorecard-v2-2026-10-31.md +before any new calls. The current implementation follow-up adds only a read-only +coverage report and immutable voltaire-whole-entry-v6 (same dossier, weights, +output-v4 contract, section/candidate/context/budget bounds). It clarifies that a +supplied sense_definition can lead without a content entry; Bierce priority and +honest abstention remain. v1–v5 text/hashes and live configuration7 are untouched. +No model start, enrollment beyond the ten pages or activation is part of this work. + +Coverage examples (PHX_SERVER unset, explicit database target, DD_NO_OBAN=1): + mix dd.curation.coverage --all --from 25 --to 35 + mix dd.curation.coverage --slug love --from 25 --to 35 --limit 20 + +These report existing records inside a read-only repeatable-read transaction. +Eligible pool is the historical packet counts.offered; raw available inventory is +not historically recorded and is labelled unknown. Packet omissions, selections, +published placements and current shown/withheld state have separate denominators. +The command prints no source payloads or editorial reasons, requests no discovery, +and calls no model. A later snapshot can have fewer visible items after expiry. +Do not archive provider content for evaluation. + +A later reviewed migration admits v6 while preserving deployed v5 compatibility. +Do not apply it, register/admit/bind v6, enable the editor or start the runtime until +separately authorized. Rollback refuses once v6 config/profile history exists; +disable policy and roll forward instead of deleting history. + +The old ngrok-only chat heartbeat was removed; public verification remains assigned +to #271/#252 once access is restored. Application cleanup/dispatch jobs are unchanged. +Historical operating notes below are retained as evidence, not current go-live +instructions or a replacement for the dated report in #275. + # Headless source preparation The first cohort exposed an input gap: only love had discovery candidates. The other diff --git a/docs/curation/coverage-2026-10-31.md b/docs/curation/coverage-2026-10-31.md new file mode 100644 index 00000000..b43b02b1 --- /dev/null +++ b/docs/curation/coverage-2026-10-31.md @@ -0,0 +1,28 @@ +# Existing-record coverage snapshot — 31 October 2026 + +Read-only coverage-v1 report, attempts25–35, fixed ten pilots. No new model calls. +Observed at 2026-11-01T00:24:43.465652Z. Generated using dd.curation.coverage in an isolated process: Repo read-only, Oban peer/queues/plugins disabled, no HTTP server. +Main/live code remains49dbf87. Report artifact: /Volumes/LLM Models/dictionary/run/275-quality-preflight-2026-10-31/coverage.json. +These are historical packet counts and a later current-eligibility snapshot, not a new quality score or a promise that old provider payloads are available. + +| Page | Attempt | Eligible pool then | Packet | Selected | Published versions from attempt | Current shown / withheld | +|---|---:|---:|---:|---:|---|---| +| nepotism | 30 | 71 | 24 | 8 | 2 (8 placements) | 5 / 3 | +| greed | 31 | 101 | 24 | 6 | 1 (6 placements) | 3 / 3 | +| war | 27 | 232 | 24 | 11 | none | 5 / 6 | +| war | 35 | 232 | 24 | 11 | 2 (11 placements) | 5 / 6 | +| love | 25 | 202 | 24 | 7 | 4 (7 placements) | 4 / 3 | +| coward | 26 | 116 | 24 | 12 | 1 (11 placements) | 5 / 6 | +| patience | 32 | 86 | 24 | 6 | 2 (6 placements) | 4 / 2 | +| justice | 33 | 113 | 24 | 6 | 2 (6 placements) | 4 / 2 | +| truth | 29 | 119 | 24 | 10 | 2 (10 placements) | 5 / 5 | +| hope | 28 | 177 | 24 | 11 | 2 (11 placements) | 6 / 5 | +| bestiality | 34 | 78 | 24 | 0 | none | 0 / 0 | + +Attempt35 is the separate war recovery, not a replacement in the primary25–34 quality denominator. +Current shown/withheld totals are the page snapshot and repeat on war rows: do not sum them per attempt. +Publication placements may differ from selections because duplicate-display omissions/owner precedence are retained in existing receipts. No publication is attributed to failed assembly attempt27. +Raw historical inventory was not recorded: unknown, not zero. Eligible-pool count is stored counts.offered; packet/omitted counts and selection identities come from the frozen attempt summary/result. +No titles, excerpts, provider payloads or editorial reasons are copied into this report. + +Before/after live checks: expected cluster7693849764459364596 on5434,35 total inference attempts, editor switch off. No migration/model/activation/deployment is part of this snapshot. diff --git a/docs/curation/scorecard-v2-2026-10-31.md b/docs/curation/scorecard-v2-2026-10-31.md new file mode 100644 index 00000000..4386bc49 --- /dev/null +++ b/docs/curation/scorecard-v2-2026-10-31.md @@ -0,0 +1,100 @@ +# Ten-page curator scorecard v2 — frozen 31 October 2026 + +Status: frozen before any further model calls; NOT RUN. This is a prospective +development evaluation, not routine human approval. The 29 October report remains +a quality NO-GO. Code/tests passing does not pass this scorecard. + +## Cohort and execution boundary + +Exactly: nepotism, greed, war, love, coward, patience, justice, truth, hope, +bestiality. These pages have already been observed and tuned: this is an iterative +retest, not a blind holdout or proof of generalization. No extra pages or personas. +Keep the installed Qwen3.5 9B artifact, one physical inference slot, existing +8K context/1,024 output-token/120-second call/30-minute shared daily bounds, +24 candidates/12 meanings and existing section limits. Do not fill weak slots. +No model call, model registration, admission, activation or deployment follows +from this document. A separately approved run manifest must freeze code SHA, +scorecard SHA256, model artifact/config hash, persona/instruction/configuration +version, packet-policy version, time window, primary-cohort inclusion rule and starting attempt-ID cutoff before calls. +Record the exact allocated attempt IDs/range after the window; do not predict future IDs. +New packets may reflect lawful provider changes: record those differences. +The first assessment of each page in that manifest is the primary cohort. +Failures/refusals/abstentions remain in its denominator. Repairs or later retries +are separate results, never replacements used to improve the primary score. + +## Per-choice rubric and denominators + +Evaluate every proposed choice (lead included) from each primary attempt, including +choices omitted during assembly; separately report actual published placements. +For each record classify intended-meaning alignment as supported, wrong, or +insufficient supplied evidence. Metadata-only material may justify only a modest +claim supported by its supplied fields. Do not infer unseen plots, lyrics, images, +articles or poem contents. Correct record type and creator attribution are required. +Score every reason separately: grounded, unsupported/misattributed, or policy- +withheld/unassessable. A title/name match alone does not establish topical relevance. + +Report all three meaning counts and all reason counts. Primary meaning rate = +supported / all proposed choices, with wrong AND insufficient-evidence choices in +the denominator. Also show scoring coverage = (supported + wrong) / all choices, +and the old assessable-only rate labelled secondary. Policy-withheld reasons are +unassessable, not automatically grounded. Zero choices has no percentage and cannot +pass usefulness. No sampling or dropping inconvenient, expired or failed cases. +If lawful evidence is no longer available, mark insufficient evidence; do not +archive prohibited payloads or reconstruct them from memory. Inspect before expiry +when possible, retaining only permitted judgments, identities/digests and metrics. + +## Page outcome and quality GO + +Every page is assessed as useful, justified empty, incorrect abstention, failed/ +refused, or not assessed. A useful page has an appropriate exact sourced lead and +a coherent useful core; judge each supplementary choice too, not just the lead. +Optional sections remain empty if no offered material supports them. The bestiality +regression specifically requires recognizing an eligible supplied sense definition, +not a hard-coded output. A truly empty/unsafe packet may justify abstention, but +that is not a useful curated entry. + +Quality GO requires ALL: +- Ten primary page outcomes recorded, with zero invented quotations or unauthorized + source-content writes, and zero unsupported/misattributed published explanations. +- At least 90% supported meaning across ALL proposed choices by the denominator + above; report absolute counts and uncertainty, never only assessable-subset 98%. +- Ten useful curated entries, including bestiality when suitable native definitions + are supplied. Justified empty outcomes are honest partial progress, not completion + of the owner's ten-useful-entry goal. +- Each publication has inspectable exact references, actual agent/model/persona/ + instruction/attempt/date, grounded reason/basis and authority. No fake human or + consensus attribution. Failed/abstained/unchanged results remain distinguishable. + +Report intermediate improvement even on NO-GO. Do not change thresholds after calls. +A changed rubric/instruction/cohort creates a new dated experiment and preserves +this version, its failures and the original 29 October report. + +## Lifecycle acceptance and cost + +Quality and operational verdicts are separate. The subsequent authorized multi-day +window must observe a real due assessment without a page visit, a fresh repeat +with zero inference, unchanged assessment without duplicate publication, owner hold +preventing overwrite, honest offline deferral, and expiry with permitted citation +survival. Existing news-expiry evidence may be referenced by exact IDs/dates; do not +force a fresh news choice merely to fill a demonstration. Distinguish expired +selection/cache from provider fields explicitly allowed to remain durable. + +Record every call including readiness, refused/failed/repair calls, artifact/config/ +profile/instruction hashes, prompt/output tokens, elapsed inference, API wait, +available memory/swap and shared daily budget. Record developer evaluation minutes +prospectively; if unavailable say unmeasured. Useful published choices per machine +minute is reported, not inferred from successful calls alone. Small ten-page results +justify another bounded experiment, not corpus-wide autonomy or model superiority. + +## Coverage report and evidence handling + +Use mix dd.curation.coverage with DD_NO_OBAN=1 and PHX_SERVER unset. +The report reads existing attempts and the current published receipt; it requests +neither discovery nor inference and creates no rows. Historical counts.offered is +the eligible screen pool, candidates is the bounded packet, omitted gives exclusion +counts/reasons, result gives actual selected candidate IDs. Current shown/withheld +status is a later snapshot, not historical publication-time eligibility. +Total raw API inventory and historical developer scores were not captured in every +attempt: report unknown, not zero or an invented funnel. Do not sum snapshots across +attempts or claim every available source item was considered. Reports contain only +counts, statuses, permitted local IDs/hashes and dates, never provider text/reasons. diff --git a/lib/devils_dictionary/curation/agents/persona.ex b/lib/devils_dictionary/curation/agents/persona.ex index b1f5ddaa..f593630d 100644 --- a/lib/devils_dictionary/curation/agents/persona.ex +++ b/lib/devils_dictionary/curation/agents/persona.ex @@ -77,12 +77,23 @@ defmodule DevilsDictionary.Curation.Agents.Persona do @selection_manifest, ~w(dossier source_refs deceased_evidence_refs template_version) ) + @sense_lead_path Path.expand( + "../../../../priv/curation/agents/personas/voltaire/v7/profile.json", + __DIR__ + ) + @external_resource @sense_lead_path + @sense_lead_manifest @sense_lead_path |> File.read!() |> Jason.decode!() + @sense_lead_body Map.take( + @sense_lead_manifest, + ~w(dossier source_refs deceased_evidence_refs template_version) + ) @whole_bodies %{ "voltaire-whole-entry-v1" => @whole_body, "voltaire-whole-entry-v2" => @revised_body, "voltaire-whole-entry-v3" => @sections_body, "voltaire-whole-entry-v4" => @relevance_body, - "voltaire-whole-entry-v5" => @selection_body + "voltaire-whole-entry-v5" => @selection_body, + "voltaire-whole-entry-v6" => @sense_lead_body } def manifest, do: @manifest @@ -106,6 +117,9 @@ defmodule DevilsDictionary.Curation.Agents.Persona do @sections_manifest["dossier"]["disclosure"] <> "\n" <> @sections_manifest["dossier"]["selection"] + def selection_instruction("voltaire-whole-entry-v6"), + do: selection_instruction("voltaire-whole-entry-v5") + def selection_instruction("voltaire-whole-entry-v5"), do: selection_instruction("voltaire-whole-entry-v4") @@ -150,7 +164,8 @@ defmodule DevilsDictionary.Curation.Agents.Persona do "voltaire-whole-entry-v2", "voltaire-whole-entry-v3", "voltaire-whole-entry-v4", - "voltaire-whole-entry-v5" + "voltaire-whole-entry-v5", + "voltaire-whole-entry-v6" ], :persona_instruction_mismatch ) do @@ -184,7 +199,8 @@ defmodule DevilsDictionary.Curation.Agents.Persona do "voltaire-whole-entry-v2", "voltaire-whole-entry-v3", "voltaire-whole-entry-v4", - "voltaire-whole-entry-v5" + "voltaire-whole-entry-v5", + "voltaire-whole-entry-v6" ], do: "single_editor_v1", else: "proposes only" diff --git a/lib/devils_dictionary/curation/coverage.ex b/lib/devils_dictionary/curation/coverage.ex new file mode 100644 index 00000000..689eae11 --- /dev/null +++ b/lib/devils_dictionary/curation/coverage.ex @@ -0,0 +1,212 @@ +defmodule DevilsDictionary.Curation.Coverage do + @moduledoc """ + Bounded read-only pilot coverage from existing attempts and published receipts. + Historical packet counts and current eligibility are separate snapshots. + No discovery/model admission, provider text, new persistence or inferred scores. + """ + import Ecto.Query + alias DevilsDictionary.{Repo, Lexicon} + alias DevilsDictionary.Lexicon.WordPage + + alias DevilsDictionary.Curation.{ + Pilot, + Configurations, + Compositions, + CompositionVersion, + CompositionPublication, + Receipt, + WholeEntry + } + + alias DevilsDictionary.Curation.Runtime.{Attempt, ModelConfig} + + @omissions ~w(over_quota meaning_out_of_bound meaning_unavailable no_excerpt outside_shelf_focus) + @kinds ~w(content sense_definition sense_quotation work exemplar claim shelf) + @sections ~w(definitions examples quotes images films music books poems news) + + def read(slug, opts \\ []) do + if slug in Pilot.autonomous_pages() do + read_pilot(slug, opts) + else + {:error, :outside_pilot} + end + end + + defp read_pilot(slug, opts) do + limit = Keyword.get(opts, :limit, 20) + from_id = Keyword.get(opts, :from, 1) + to_id = Keyword.get(opts, :to, 9_223_372_036_854_775_807) + + if is_integer(limit) and limit in 1..100 and is_integer(from_id) and + is_integer(to_id) and from_id > 0 and to_id >= from_id do + page = slug |> Lexicon.lookup() |> WordPage.build(viewer: :public) + lexemes = page.headword.lexemes + ids = Enum.sort(Enum.map(lexemes, & &1.id)) + + attempts = + if ids == [] do + [] + else + Repo.all( + from a in Attempt, + where: a.purpose == :request and a.id >= ^from_id and a.id <= ^to_id, + where: + fragment( + "? #> '{target,lexeme_ids}' <@ ?::jsonb AND jsonb_array_length(? #> '{target,lexeme_ids}') > 0", + a.packet_summary, + ^ids, + a.packet_summary + ), + order_by: [desc: a.id], + limit: ^(limit + 1) + ) + end + + %{ + report_version: "curation-coverage-v1", + slug: slug, + observed_at: DateTime.utc_now(), + historical_available_inventory: :not_recorded, + attempt_limit: limit, + attempts_truncated: length(attempts) > limit, + attempts: Enum.map(Enum.take(attempts, limit), &attempt/1), + current_publication: current(slug, lexemes, ids) + } + else + {:error, :invalid_bounds} + end + end + + @doc "Safe counts and local selection IDs from a stored attempt, never its reason text." + def summarize(%Attempt{} = attempt) do + summary = attempt.packet_summary || %{} + candidates = summary["candidates"] || [] + omitted = summary["omitted"] + + choices = + if attempt.outcome in [:accepted, :abstained] and is_map(attempt.result) do + List.wrap(attempt.result["lead"]) ++ (attempt.result["highlights"] || []) + end + + by_id = Map.new(candidates, &{&1["candidate_id"], &1}) + + %{ + attempt_id: attempt.id, + state: attempt.state, + outcome: attempt.outcome, + attempt_finished_at: attempt.finished_at, + model_config_id: attempt.model_config_id, + packet_hash: attempt.packet_hash, + instruction_version: summary["instruction_version"], + eligible_pool: get_in(summary, ["counts", "offered"]), + packet_candidates: length(candidates), + packet_by_section: Enum.frequencies_by(candidates, §ion/1), + omitted: if(is_list(omitted), do: length(omitted)), + omitted_by_reason: + if(is_list(omitted), + do: + Enum.frequencies_by( + omitted, + fn row -> if row["reason"] in @omissions, do: row["reason"], else: "unknown" end + ) + ), + selected: if(is_list(choices), do: length(choices)), + selected_by_section: + if(is_list(choices), + do: + Enum.frequencies_by( + choices, + fn choice -> section(by_id[choice["candidate_id"]] || %{}) end + ) + ), + selected_candidate_ids: if(is_list(choices), do: Enum.map(choices, & &1["candidate_id"])) + } + end + + defp section(candidate) do + if candidate["kind"] in @kinds do + result = WholeEntry.section(candidate) + if result in @sections, do: result, else: "unknown" + else + "unknown" + end + end + + defp attempt(row) do + config = Repo.get(ModelConfig, row.model_config_id) + + publications = + Repo.all( + from v in CompositionVersion, + join: p in CompositionPublication, + on: p.published_version_id == v.id and p.action == :publish, + where: v.attempt_id == ^row.id, + distinct: true, + order_by: v.id, + select: %{version_id: v.id, version: v.version, composition_id: v.composition_id} + ) + + summarize(row) + |> Map.put(:model_config_hash, config && config.config_hash) + |> Map.put( + :publication_versions, + Enum.map(publications, fn version -> + Map.put(version, :placed_items, length(Compositions.items(version.version_id))) + end) + ) + end + + defp current(_slug, [], _ids), do: %{status: :no_scope, items: []} + + defp current(slug, [first | _], ids) do + with {:ok, %{configuration: configuration}} <- Configurations.resolve_default(), + composition when not is_nil(composition) <- + Compositions.active( + configuration.id, + :lexical_page, + first.language, + Compositions.signature(ids) + ), + id when is_integer(id) <- composition.current_published_version_id, + version <- Repo.get!(CompositionVersion, id), + {:ok, receipt} <- Receipt.read(slug, version.version) do + items = + Enum.map(receipt.items, fn item -> + %{ + item_id: item.id, + role: item.role, + position: item.position, + kind: item.kind, + status: item.status, + status_reason: reason_code(item.status_reason), + original_attempt_id: item.chooser.attempt_id, + original_model_config_id: item.chooser.model_config_id + } + end) + + %{ + status: :published, + composition_id: composition.id, + version_id: version.id, + version: version.version, + placed: length(items), + by_status: Enum.frequencies_by(items, & &1.status), + items: items + } + else + _ -> %{status: :no_current_publication, items: []} + end + end + + defp reason_code(nil), do: nil + defp reason_code(reason) when is_atom(reason), do: Atom.to_string(reason) + + defp reason_code(reason) when is_tuple(reason), + do: + reason + |> Tuple.to_list() + |> Enum.filter(&is_atom/1) + |> Enum.map_join(":", &Atom.to_string/1) + + defp reason_code(_), do: "withheld" +end diff --git a/lib/devils_dictionary/curation/runtime/contract.ex b/lib/devils_dictionary/curation/runtime/contract.ex index 6951e1d9..1f72768d 100644 --- a/lib/devils_dictionary/curation/runtime/contract.ex +++ b/lib/devils_dictionary/curation/runtime/contract.ex @@ -33,6 +33,7 @@ defmodule DevilsDictionary.Curation.Runtime.Contract do @max_quote_chars 400 def version, do: @version + def version("voltaire-whole-entry-v6"), do: "selection-output-v4" def version("voltaire-whole-entry-v5"), do: "selection-output-v4" def version("voltaire-whole-entry-v4"), do: "selection-output-v4" def version("voltaire-whole-entry-v3"), do: "selection-output-v4" @@ -323,7 +324,8 @@ defmodule DevilsDictionary.Curation.Runtime.Contract do "voltaire-whole-entry-v2", "voltaire-whole-entry-v3", "voltaire-whole-entry-v4", - "voltaire-whole-entry-v5" + "voltaire-whole-entry-v5", + "voltaire-whole-entry-v6" ] and not is_nil(choice["quote"]), do: [[where, "quote_not_allowed"]], diff --git a/lib/devils_dictionary/curation/runtime/packet.ex b/lib/devils_dictionary/curation/runtime/packet.ex index 0a8cafa8..f38925b3 100644 --- a/lib/devils_dictionary/curation/runtime/packet.ex +++ b/lib/devils_dictionary/curation/runtime/packet.ex @@ -241,12 +241,22 @@ defmodule DevilsDictionary.Curation.Runtime.Packet do Before returning, remove every choice whose own reason describes a rejection; represent it as null/[] instead. Preserve other useful choices. """ + @whole_sense_lead @whole_selection <> + """ + Lead clarification for pages without dictionary entries: + A supplied kind sense_definition is a complete eligible lead definition, not merely background context. No kind content candidate is required. + If priority_candidates is nonempty, choose from it as before. Otherwise evaluate supplied content and sense_definition candidates by the entry's intended meaning and use an allowed meaning_id. + When a supplied eligible sense_definition clearly defines this entry, select it even if no Bierce or Johnson entry or optional material is supplied. Do not abstain solely because kind content is absent. + Do not invent a definition or candidate. Abstain when the supplied definition evidence is irrelevant, unsafe or insufficient; never force a weak lead. + """ + @instructions %{ "voltaire-whole-entry-v1" => @whole_entry, "voltaire-whole-entry-v2" => @whole_entry_v2, "voltaire-whole-entry-v3" => @whole_sections, "voltaire-whole-entry-v4" => @whole_relevance, "voltaire-whole-entry-v5" => @whole_selection, + "voltaire-whole-entry-v6" => @whole_sense_lead, "generic-editor-v1" => @generic_editor_v1, "generic-editor-v2" => @generic_editor_v2, "voltaire-selector-v1" => diff --git a/lib/devils_dictionary/curation/runtime/section_output.ex b/lib/devils_dictionary/curation/runtime/section_output.ex index 1f31d975..4a0844cf 100644 --- a/lib/devils_dictionary/curation/runtime/section_output.ex +++ b/lib/devils_dictionary/curation/runtime/section_output.ex @@ -1,7 +1,13 @@ defmodule DevilsDictionary.Curation.Runtime.SectionOutput do @moduledoc "Versioned wire shape for one-call whole-entry selections; no separate storage or approval." alias DevilsDictionary.Curation.WholeEntry - @instructions ["voltaire-whole-entry-v3", "voltaire-whole-entry-v4", "voltaire-whole-entry-v5"] + + @instructions [ + "voltaire-whole-entry-v3", + "voltaire-whole-entry-v4", + "voltaire-whole-entry-v5", + "voltaire-whole-entry-v6" + ] @top ~w(decision lead selections abstain_reason) @choice ~w(candidate_id meaning_id reason quote) def enabled?(packet), do: packet["instruction_version"] in @instructions diff --git a/lib/devils_dictionary/curation/whole_entry.ex b/lib/devils_dictionary/curation/whole_entry.ex index 5da4e13c..0a5613d1 100644 --- a/lib/devils_dictionary/curation/whole_entry.ex +++ b/lib/devils_dictionary/curation/whole_entry.ex @@ -20,10 +20,11 @@ defmodule DevilsDictionary.Curation.WholeEntry do "voltaire-whole-entry-v2", "voltaire-whole-entry-v3", "voltaire-whole-entry-v4", - "voltaire-whole-entry-v5" + "voltaire-whole-entry-v5", + "voltaire-whole-entry-v6" ] - def instruction, do: "voltaire-whole-entry-v5" + def instruction, do: "voltaire-whole-entry-v6" def enabled?(%{manifest: m}), do: m["selection_scope"] == "whole_entry_v1" def packet?(%{"packet_version" => "runtime-packet-v4"}), do: true def packet?(_), do: false diff --git a/lib/mix/tasks/dd.curation.coverage.ex b/lib/mix/tasks/dd.curation.coverage.ex new file mode 100644 index 00000000..a58a889b --- /dev/null +++ b/lib/mix/tasks/dd.curation.coverage.ex @@ -0,0 +1,70 @@ +defmodule Mix.Tasks.Dd.Curation.Coverage do + @shortdoc "Read-only ten-page coverage from stored attempts and current receipts" + @moduledoc """ + DD_NO_OBAN=1 mix dd.curation.coverage --slug love --from 25 --to 35 + DD_NO_OBAN=1 mix dd.curation.coverage --all --from 25 --to 35 + + PHX_SERVER must be unset. Starts no server, discovery or model work; uses one + repeatable-read, read-only transaction. Default latest 20 attempts per page, + --limit 1..100; reports truncation. No provider text or editorial reasons printed. + Use explicit database environment settings; never infer a target from a URL. + """ + use Mix.Task + alias DevilsDictionary.{Repo, Curation.Coverage, Curation.Pilot} + @impl Mix.Task + def run(args) do + unless System.get_env("DD_NO_OBAN") == "1" and is_nil(System.get_env("PHX_SERVER")), + do: Mix.raise("Set DD_NO_OBAN=1 and unset PHX_SERVER; coverage runs no jobs or server.") + + {opts, rest, invalid} = + OptionParser.parse(args, + strict: [slug: :string, all: :boolean, from: :integer, to: :integer, limit: :integer] + ) + + unless rest == [] and invalid == [] and + ((is_binary(opts[:slug]) and not opts[:all]) or + (opts[:all] == true and is_nil(opts[:slug]))), + do: Mix.raise("Use exactly --slug PILOT or --all, with optional --from/--to/--limit.") + + slugs = if opts[:all], do: Pilot.autonomous_pages(), else: [opts[:slug]] + + {repo, oban, endpoint} = + read_only_config( + Application.fetch_env!(:devils_dictionary, Repo), + Application.fetch_env!(:devils_dictionary, Oban), + Application.fetch_env!(:devils_dictionary, DevilsDictionaryWeb.Endpoint) + ) + + Application.put_env(:devils_dictionary, Repo, repo) + Application.put_env(:devils_dictionary, Oban, oban) + Application.put_env(:devils_dictionary, DevilsDictionaryWeb.Endpoint, endpoint) + Mix.Task.run("app.start") + + {:ok, reports} = + Repo.transaction( + fn -> + Repo.query!("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY") + + Enum.map(slugs, fn slug -> + case Coverage.read(slug, Keyword.take(opts, [:from, :to, :limit])) do + {:error, reason} -> Mix.raise("Coverage refused: #{reason}") + report -> report + end + end) + end, + timeout: 120_000 + ) + + Mix.shell().info(Jason.encode!(reports, pretty: true)) + end + + @doc false + def read_only_config(repo, oban, endpoint) do + parameters = + Keyword.put(Keyword.get(repo, :parameters, []), :default_transaction_read_only, "on") + + {Keyword.put(repo, :parameters, parameters), + Keyword.merge(oban, queues: false, plugins: false, peer: false, testing: :manual), + Keyword.put(endpoint, :server, false)} + end +end diff --git a/lib/mix/tasks/dd.curation.editor.ex b/lib/mix/tasks/dd.curation.editor.ex index de8fee87..e1f22f7e 100644 --- a/lib/mix/tasks/dd.curation.editor.ex +++ b/lib/mix/tasks/dd.curation.editor.ex @@ -119,7 +119,8 @@ defmodule Mix.Tasks.Dd.Curation.Editor do "voltaire-whole-entry-v2", "voltaire-whole-entry-v3", "voltaire-whole-entry-v4", - "voltaire-whole-entry-v5" + "voltaire-whole-entry-v5", + "voltaire-whole-entry-v6" ], do: Mix.raise("the persona requires a config pinned to voltaire-selector-v1") diff --git a/priv/curation/agents/personas/voltaire/v7/profile.json b/priv/curation/agents/personas/voltaire/v7/profile.json new file mode 100644 index 00000000..d77212c7 --- /dev/null +++ b/priv/curation/agents/personas/voltaire/v7/profile.json @@ -0,0 +1,51 @@ +{ + "slug": "voltaire", + "template_version": "voltaire-whole-entry-v6", + "dossier": { + "disclosure": "An AI selection lens inspired by Voltaire; not his voice or an assertion of his opinions about modern events.", + "selection": "Prefer sourced material that makes dogma, unjustified certainty or intolerance visible through a concrete contrast. Prefer evidence of consequences to rhetorical agreement. The lens cannot change lead policy, source eligibility, meaning constraints or exact quotation. Explain the relevance, not an imagined endorsement by Voltaire. Abstain when no supplied evidence fits. Do not prefer an item merely because it agrees with this lens.", + "principles": [ + "Test confident abstractions against particular consequences.", + "Distinguish satire from factual assertion." + ], + "blind_spots": [ + "Satire may oversimplify sincere beliefs; do not assume bad faith." + ], + "positive_example": "Select a supplied, meaning-relevant example of a claim contradicted by its documented consequences.", + "negative_example": "Reject an irrelevant news headline chosen only because it mocks a disliked belief." + }, + "source_refs": [ + { + "url": "https://www.gutenberg.org/ebooks/19942", + "work": "Candide", + "edition": "Project Gutenberg 19942, English text, released 2006-11-27; translator not identified in this dossier", + "locators": [ + "Chapter I: Pangloss and the castle", + "Chapter III: war and its consequences", + "Chapter XXX: conclusion" + ], + "rights": "Public domain in the USA; two short passages and original editorial paraphrases; check jurisdiction before reuse.", + "basis": "These chapters contrast philosophical assurances with observed consequences; the selection lens is our interpretation, not a historical quotation.", + "passages": [ + { + "locator": "Chapter I, Pangloss on spectacles", + "speaker": "Pangloss (fictional character), not a literal statement of the author’s beliefs", + "text": "the nose has been formed to bear spectacles" + }, + { + "locator": "Chapter XXX, final reply", + "speaker": "Candide (fictional character)", + "text": "but let us cultivate our garden." + } + ] + } + ], + "deceased_evidence_refs": [ + { + "url": "https://catalogue.bnf.fr/ark:/12148/cb11928669t", + "authority": "Bibliothèque nationale de France person authority", + "field": "Mort", + "value": "1778-05-30" + } + ] +} diff --git a/priv/repo/migrations/20261101000313_admit_whole_entry_sense_lead_instruction.exs b/priv/repo/migrations/20261101000313_admit_whole_entry_sense_lead_instruction.exs new file mode 100644 index 00000000..5d060e24 --- /dev/null +++ b/priv/repo/migrations/20261101000313_admit_whole_entry_sense_lead_instruction.exs @@ -0,0 +1,137 @@ +defmodule DevilsDictionary.Repo.Migrations.AdmitWholeEntrySenseLeadInstruction do + use Ecto.Migration + + def up do + execute "ALTER TABLE curation_configuration_versions DROP CONSTRAINT curation_configuration_versions_shape" + + execute """ + ALTER TABLE curation_configuration_versions ADD CONSTRAINT curation_configuration_versions_shape CHECK (version >= 1 AND max_highlights BETWEEN 0 AND 12 AND ((max_highlights <= 3 AND COALESCE(manifest->>'selection_scope','') <> 'whole_entry_v1' AND COALESCE(manifest->'editor'->>'instruction_version','') NOT IN ('voltaire-whole-entry-v1','voltaire-whole-entry-v2','voltaire-whole-entry-v3','voltaire-whole-entry-v4','voltaire-whole-entry-v5','voltaire-whole-entry-v6')) OR (max_highlights = 12 AND COALESCE(manifest->>'selection_scope','') = 'whole_entry_v1' AND COALESCE(manifest->'editor'->>'instruction_version','') IN ('voltaire-whole-entry-v1','voltaire-whole-entry-v2','voltaire-whole-entry-v3','voltaire-whole-entry-v4','voltaire-whole-entry-v5','voltaire-whole-entry-v6'))) AND lead_policy IN ('bierce_first_v1') AND btrim(change_reason) <> ''); + """ + + execute """ + CREATE OR REPLACE FUNCTION editorial_composition_versions_author_guard() RETURNS trigger + LANGUAGE plpgsql AS $$ + DECLARE + editor jsonb; + BEGIN + IF NEW.origin = 'model' THEN + SELECT v.manifest -> 'editor' INTO editor + FROM curation_configuration_versions v WHERE v.id = NEW.configuration_version_id; + + IF editor IS NULL OR jsonb_typeof(editor) <> 'object' THEN + RAISE EXCEPTION 'a model version is made under a configuration version that binds an editor' + USING ERRCODE = 'integrity_constraint_violation'; + END IF; + + IF NOT EXISTS ( + SELECT 1 FROM curator_profiles p + WHERE p.id = (editor ->> 'profile_id')::bigint + AND (p.kind = 'generic_editor' OR ( + p.kind = 'persona' AND p.slug = 'voltaire' AND EXISTS ( + SELECT 1 FROM curator_profile_versions pv, local_model_configs mc + WHERE pv.id = p.current_version_id AND mc.id = (editor ->> 'model_config_id')::bigint + AND pv.template_version IN ('voltaire-selector-v1','voltaire-whole-entry-v1','voltaire-whole-entry-v2','voltaire-whole-entry-v3','voltaire-whole-entry-v4','voltaire-whole-entry-v5','voltaire-whole-entry-v6') + AND mc.instruction_version = pv.template_version + AND pv.dossier = '{"disclosure": "An AI selection lens inspired by Voltaire; not his voice or an assertion of his opinions about modern events.", "selection": "Prefer sourced material that makes dogma, unjustified certainty or intolerance visible through a concrete contrast. Prefer evidence of consequences to rhetorical agreement. The lens cannot change lead policy, source eligibility, meaning constraints or exact quotation. Explain the relevance, not an imagined endorsement by Voltaire. Abstain when no supplied evidence fits. Do not prefer an item merely because it agrees with this lens.", "principles": ["Test confident abstractions against particular consequences.", "Distinguish satire from factual assertion."], "blind_spots": ["Satire may oversimplify sincere beliefs; do not assume bad faith."], "positive_example": "Select a supplied, meaning-relevant example of a claim contradicted by its documented consequences.", "negative_example": "Reject an irrelevant news headline chosen only because it mocks a disliked belief."}'::jsonb + AND pv.source_refs = '[{"url": "https://www.gutenberg.org/ebooks/19942", "work": "Candide", "edition": "Project Gutenberg 19942, English text, released 2006-11-27; translator not identified in this dossier", "locators": ["Chapter I: Pangloss and the castle", "Chapter III: war and its consequences", "Chapter XXX: conclusion"], "rights": "Public domain in the USA; two short passages and original editorial paraphrases; check jurisdiction before reuse.", "basis": "These chapters contrast philosophical assurances with observed consequences; the selection lens is our interpretation, not a historical quotation.", "passages": [{"locator": "Chapter I, Pangloss on spectacles", "speaker": "Pangloss (fictional character), not a literal statement of the author’s beliefs", "text": "the nose has been formed to bear spectacles"}, {"locator": "Chapter XXX, final reply", "speaker": "Candide (fictional character)", "text": "but let us cultivate our garden."}]}]'::jsonb + AND pv.deceased_evidence_refs = '[{"url": "https://catalogue.bnf.fr/ark:/12148/cb11928669t", "authority": "Bibliothèque nationale de France person authority", "field": "Mort", "value": "1778-05-30"}]'::jsonb + ))) AND p.state = 'admitted' + AND p.current_version_id = (editor ->> 'profile_version_id')::bigint + AND p.bot_actor_id = NEW.created_by_actor_id + AND (editor ->> 'bot_actor_id')::bigint = NEW.created_by_actor_id + ) THEN + RAISE EXCEPTION 'a model version is written by the bot actor of the admitted editor its configuration version binds' + USING ERRCODE = 'integrity_constraint_violation'; + END IF; + + IF NOT EXISTS ( + SELECT 1 FROM inference_attempts a + WHERE a.id = NEW.attempt_id AND a.state = 'completed' AND a.outcome = 'accepted' + AND a.requested_by_actor_id = NEW.created_by_actor_id + AND a.model_config_id = (editor ->> 'model_config_id')::bigint + ) THEN + RAISE EXCEPTION 'a model version comes from a completed, accepted attempt of its bot under the bound model config' + USING ERRCODE = 'integrity_constraint_violation'; + END IF; + ELSIF NOT curation_actor_is(NEW.created_by_actor_id, false) THEN + RAISE EXCEPTION '%.% must be a human account', TG_TABLE_NAME, 'created_by_actor_id' + USING ERRCODE = 'integrity_constraint_violation'; + END IF; + + RETURN NEW; + END; + $$ + """ + end + + def down do + execute """ + DO $$ BEGIN + IF EXISTS (SELECT 1 FROM local_model_configs WHERE instruction_version='voltaire-whole-entry-v6') OR EXISTS (SELECT 1 FROM curator_profile_versions WHERE template_version='voltaire-whole-entry-v6') THEN + RAISE EXCEPTION 'sense-lead instruction history exists; disable policy and roll forward'; + END IF; + END $$; + """ + + execute "ALTER TABLE curation_configuration_versions DROP CONSTRAINT curation_configuration_versions_shape" + + execute """ + ALTER TABLE curation_configuration_versions ADD CONSTRAINT curation_configuration_versions_shape CHECK (version >= 1 AND max_highlights BETWEEN 0 AND 12 AND ((max_highlights <= 3 AND COALESCE(manifest->>'selection_scope','') <> 'whole_entry_v1' AND COALESCE(manifest->'editor'->>'instruction_version','') NOT IN ('voltaire-whole-entry-v1','voltaire-whole-entry-v2','voltaire-whole-entry-v3','voltaire-whole-entry-v4','voltaire-whole-entry-v5')) OR (max_highlights = 12 AND COALESCE(manifest->>'selection_scope','') = 'whole_entry_v1' AND COALESCE(manifest->'editor'->>'instruction_version','') IN ('voltaire-whole-entry-v1','voltaire-whole-entry-v2','voltaire-whole-entry-v3','voltaire-whole-entry-v4','voltaire-whole-entry-v5'))) AND lead_policy IN ('bierce_first_v1') AND btrim(change_reason) <> ''); + """ + + execute """ + CREATE OR REPLACE FUNCTION editorial_composition_versions_author_guard() RETURNS trigger + LANGUAGE plpgsql AS $$ + DECLARE + editor jsonb; + BEGIN + IF NEW.origin = 'model' THEN + SELECT v.manifest -> 'editor' INTO editor + FROM curation_configuration_versions v WHERE v.id = NEW.configuration_version_id; + + IF editor IS NULL OR jsonb_typeof(editor) <> 'object' THEN + RAISE EXCEPTION 'a model version is made under a configuration version that binds an editor' + USING ERRCODE = 'integrity_constraint_violation'; + END IF; + + IF NOT EXISTS ( + SELECT 1 FROM curator_profiles p + WHERE p.id = (editor ->> 'profile_id')::bigint + AND (p.kind = 'generic_editor' OR ( + p.kind = 'persona' AND p.slug = 'voltaire' AND EXISTS ( + SELECT 1 FROM curator_profile_versions pv, local_model_configs mc + WHERE pv.id = p.current_version_id AND mc.id = (editor ->> 'model_config_id')::bigint + AND pv.template_version IN ('voltaire-selector-v1','voltaire-whole-entry-v1','voltaire-whole-entry-v2','voltaire-whole-entry-v3','voltaire-whole-entry-v4','voltaire-whole-entry-v5') + AND mc.instruction_version = pv.template_version + AND pv.dossier = '{"disclosure": "An AI selection lens inspired by Voltaire; not his voice or an assertion of his opinions about modern events.", "selection": "Prefer sourced material that makes dogma, unjustified certainty or intolerance visible through a concrete contrast. Prefer evidence of consequences to rhetorical agreement. The lens cannot change lead policy, source eligibility, meaning constraints or exact quotation. Explain the relevance, not an imagined endorsement by Voltaire. Abstain when no supplied evidence fits. Do not prefer an item merely because it agrees with this lens.", "principles": ["Test confident abstractions against particular consequences.", "Distinguish satire from factual assertion."], "blind_spots": ["Satire may oversimplify sincere beliefs; do not assume bad faith."], "positive_example": "Select a supplied, meaning-relevant example of a claim contradicted by its documented consequences.", "negative_example": "Reject an irrelevant news headline chosen only because it mocks a disliked belief."}'::jsonb + AND pv.source_refs = '[{"url": "https://www.gutenberg.org/ebooks/19942", "work": "Candide", "edition": "Project Gutenberg 19942, English text, released 2006-11-27; translator not identified in this dossier", "locators": ["Chapter I: Pangloss and the castle", "Chapter III: war and its consequences", "Chapter XXX: conclusion"], "rights": "Public domain in the USA; two short passages and original editorial paraphrases; check jurisdiction before reuse.", "basis": "These chapters contrast philosophical assurances with observed consequences; the selection lens is our interpretation, not a historical quotation.", "passages": [{"locator": "Chapter I, Pangloss on spectacles", "speaker": "Pangloss (fictional character), not a literal statement of the author’s beliefs", "text": "the nose has been formed to bear spectacles"}, {"locator": "Chapter XXX, final reply", "speaker": "Candide (fictional character)", "text": "but let us cultivate our garden."}]}]'::jsonb + AND pv.deceased_evidence_refs = '[{"url": "https://catalogue.bnf.fr/ark:/12148/cb11928669t", "authority": "Bibliothèque nationale de France person authority", "field": "Mort", "value": "1778-05-30"}]'::jsonb + ))) AND p.state = 'admitted' + AND p.current_version_id = (editor ->> 'profile_version_id')::bigint + AND p.bot_actor_id = NEW.created_by_actor_id + AND (editor ->> 'bot_actor_id')::bigint = NEW.created_by_actor_id + ) THEN + RAISE EXCEPTION 'a model version is written by the bot actor of the admitted editor its configuration version binds' + USING ERRCODE = 'integrity_constraint_violation'; + END IF; + + IF NOT EXISTS ( + SELECT 1 FROM inference_attempts a + WHERE a.id = NEW.attempt_id AND a.state = 'completed' AND a.outcome = 'accepted' + AND a.requested_by_actor_id = NEW.created_by_actor_id + AND a.model_config_id = (editor ->> 'model_config_id')::bigint + ) THEN + RAISE EXCEPTION 'a model version comes from a completed, accepted attempt of its bot under the bound model config' + USING ERRCODE = 'integrity_constraint_violation'; + END IF; + ELSIF NOT curation_actor_is(NEW.created_by_actor_id, false) THEN + RAISE EXCEPTION '%.% must be a human account', TG_TABLE_NAME, 'created_by_actor_id' + USING ERRCODE = 'integrity_constraint_violation'; + END IF; + + RETURN NEW; + END; + $$ + """ + end +end diff --git a/test/devils_dictionary/curation/coverage_test.exs b/test/devils_dictionary/curation/coverage_test.exs new file mode 100644 index 00000000..4e4c4b97 --- /dev/null +++ b/test/devils_dictionary/curation/coverage_test.exs @@ -0,0 +1,77 @@ +defmodule DevilsDictionary.Curation.CoverageTest do + use ExUnit.Case, async: true + alias DevilsDictionary.Curation.Coverage + alias DevilsDictionary.Curation.Runtime.Attempt + + test "historical pool, packet, omitted and selected counts have separate denominators" do + attempt = %Attempt{ + id: 34, + state: :completed, + outcome: :accepted, + packet_summary: %{ + "counts" => %{"offered" => 71}, + "candidates" => [ + %{"candidate_id" => "c1", "kind" => "sense_definition"}, + %{"candidate_id" => "c2", "kind" => "shelf", "shelf" => %{"shelf_kind" => "news"}} + ], + "omitted" => [%{"reason" => "over_quota"}, %{"reason" => "meaning_unavailable"}] + }, + result: %{ + "lead" => %{"candidate_id" => "c1", "reason" => "PROVIDER_SECRET"}, + "highlights" => [%{"candidate_id" => "c2", "reason" => "PROVIDER_SECRET"}] + } + } + + report = Coverage.summarize(attempt) + assert report.eligible_pool == 71 + assert report.packet_candidates == 2 + assert report.omitted == 2 + assert report.selected == 2 + assert report.packet_by_section == %{"definitions" => 1, "news" => 1} + assert report.selected_candidate_ids == ["c1", "c2"] + refute Jason.encode!(report) =~ "PROVIDER_SECRET" + end + + test "missing historic counts are unknown; refusal is not a zero-choice assessment" do + report = Coverage.summarize(%Attempt{outcome: :refused, packet_summary: %{}}) + assert is_nil(report.eligible_pool) + assert is_nil(report.omitted) + assert is_nil(report.selected) + + assert Coverage.summarize(%Attempt{ + outcome: :abstained, + result: %{"lead" => nil, "highlights" => []} + }).selected == 0 + end + + test "outside-pilot requests are refused before reading or preparing a page" do + assert Coverage.read("not-a-pilot") == {:error, :outside_pilot} + assert Coverage.read("love", limit: 101) == {:error, :invalid_bounds} + assert Coverage.read("love", from: 35, to: 25) == {:error, :invalid_bounds} + end + + test "a model call finish is not labelled as a successful editor assessment" do + time = ~U[2026-10-29 21:00:00Z] + report = Coverage.summarize(%Attempt{outcome: :refused, finished_at: time}) + assert report.attempt_finished_at == time + refute Map.has_key?(report, :assessed_at) + end + + test "report startup disables server and every Oban execution/peer path before connecting read-only" do + {repo, oban, endpoint} = + Mix.Tasks.Dd.Curation.Coverage.read_only_config( + [database: "private_test", parameters: [application_name: "coverage"]], + [queues: [curation: 1], plugins: [Oban.Plugins.Cron], peer: Oban.Peers.Database], + server: true + ) + + assert repo[:database] == "private_test" + assert repo[:parameters][:default_transaction_read_only] == "on" + assert repo[:parameters][:application_name] == "coverage" + assert oban[:queues] == false + assert oban[:plugins] == false + assert oban[:peer] == false + assert oban[:testing] == :manual + assert endpoint[:server] == false + end +end diff --git a/test/devils_dictionary/curation/runtime/packet_test.exs b/test/devils_dictionary/curation/runtime/packet_test.exs index d60468fe..aeaba0b7 100644 --- a/test/devils_dictionary/curation/runtime/packet_test.exs +++ b/test/devils_dictionary/curation/runtime/packet_test.exs @@ -31,10 +31,25 @@ defmodule DevilsDictionary.Curation.Runtime.PacketTest do "voltaire-whole-entry-v2", "voltaire-whole-entry-v3", "voltaire-whole-entry-v4", - "voltaire-whole-entry-v5" + "voltaire-whole-entry-v5", + "voltaire-whole-entry-v6" ] + sense_lead = Packet.instruction!("voltaire-whole-entry-v6") + + assert Digest.sha256(sense_lead) == + "e037baf1f599d321ea35ba740bb0d5deaf3bbfb880ec47ce6c33bdfb96e67212" + + assert sense_lead =~ "No kind content candidate is required" + assert sense_lead =~ "never force a weak lead" + assert sense_lead =~ "If priority_candidates is nonempty" selection = Packet.instruction!("voltaire-whole-entry-v5") + + assert Digest.sha256(selection) == + "29e9af8c58c6d8a6a871cf4603f31d90da614e9f1eeb86b7e66fbd7fe7ed90a5" + + assert String.starts_with?(sense_lead, selection) + assert selection =~ "every choice object is an instruction to SHOW" assert selection =~ ~s("news": null) assert selection =~ "Never invent a positive connection" diff --git a/test/devils_dictionary/curation/whole_entry_test.exs b/test/devils_dictionary/curation/whole_entry_test.exs index 564726d2..93c3228b 100644 --- a/test/devils_dictionary/curation/whole_entry_test.exs +++ b/test/devils_dictionary/curation/whole_entry_test.exs @@ -135,6 +135,47 @@ defmodule DevilsDictionary.Curation.WholeEntryTest do assert Repo.reload!(prepared.profile).bot_actor_id == prepared.bot_actor.id end + @tag instruction: "voltaire-whole-entry-v6" + test "sense-only v6 packet publishes an exact sense lead and read-only coverage reports it", + w do + for slug <- ["bierce", "johnson"] do + source = DevilsDictionary.Sources.get_source_by_slug(slug) + source |> Ecto.Changeset.change(active: false) |> Repo.update!() + end + + frozen = packet("voltaire-whole-entry-v6") + refute Enum.any?(frozen.packet["candidates"], &(&1["kind"] == "content")) + assert frozen.packet["priority_candidates"] == [] + definition = Enum.find(frozen.packet["candidates"], &(&1["kind"] == "sense_definition")) + assert definition + + wire = %{ + "decision" => "select", + "lead" => %{ + "candidate_id" => definition["candidate_id"], + "meaning_id" => hd(definition["allowed_meanings"]), + "reason" => "A supplied definition of this meaning." + }, + "selections" => + Map.new(WholeEntry.sections(), &{&1, if(&1 in ["examples", "quotes"], do: [], else: nil)}), + "abstain_reason" => nil + } + + assert {:ok, %{outcome: :accepted}} = Contract.validate(answer(wire), frozen) + stub_answer!(w, wire) + assert {:ok, %{last_outcome_reason: "published_automatically"}} = run!(w) + settle!() + before = Repo.aggregate(CompositionPublication, :count) + report = DevilsDictionary.Curation.Coverage.read("love") + assert report.current_publication.placed == 1 + assert report.current_publication.by_status == %{shown: 1} + assert [attempt] = report.attempts + assert attempt.selected == 1 + assert attempt.packet_by_section["definitions"] > 0 + assert [%{placed_items: 1}] = attempt.publication_versions + assert Repo.aggregate(CompositionPublication, :count) == before + end + @tag instruction: "voltaire-whole-entry-v5" test "relevance instruction uses the same typed contract, publishes a suitable definition with sparse extras and keeps abstention available", w do diff --git a/test/devils_dictionary/installation/whole_entry_compatibility_test.exs b/test/devils_dictionary/installation/whole_entry_compatibility_test.exs index 3054f44d..86fa8cc3 100644 --- a/test/devils_dictionary/installation/whole_entry_compatibility_test.exs +++ b/test/devils_dictionary/installation/whole_entry_compatibility_test.exs @@ -208,6 +208,37 @@ defmodule DevilsDictionary.Installation.WholeEntryCompatibilityTest do end assert Repo.get!(DevilsDictionary.Curation.Runtime.ModelConfig, selection_config.id) + sense_lead_version = 20_261_101_000_313 + + for direction <- [:up, :down, :up] do + opts = if direction == :up, do: [to: sense_lead_version], else: [step: 1] + + Ecto.Migrator.run( + Repo, + path, + direction, + opts ++ [dynamic_repo: Repo.get_dynamic_repo(), log: false] + ) + + assert {:ok, _} = Receipt.read("love", first.version.version) + assert Repo.get!(DevilsDictionary.Curation.Runtime.ModelConfig, selection_config.id) + end + + sense_config = + DevilsDictionary.RuntimeFixtures.model_config!(%{ + instruction_version: "voltaire-whole-entry-v6", + output_contract_version: "selection-output-v4" + }) + + assert_raise Postgrex.Error, ~r/sense-lead instruction history exists/, fn -> + Ecto.Migrator.run(Repo, path, :down, + step: 1, + dynamic_repo: Repo.get_dynamic_repo(), + log: false + ) + end + + assert Repo.get!(DevilsDictionary.Curation.Runtime.ModelConfig, sense_config.id) end) end end From c533c76fe156d60aa869c8e295f7e52100d3e43f Mon Sep 17 00:00:00 2001 From: Holden Thomas Date: Sat, 31 Oct 2026 19:34:12 -0500 Subject: [PATCH 2/2] Record coverage eligibility under deployed retention policy --- docs/curation/coverage-2026-10-31.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/curation/coverage-2026-10-31.md b/docs/curation/coverage-2026-10-31.md index b43b02b1..e4fcde5f 100644 --- a/docs/curation/coverage-2026-10-31.md +++ b/docs/curation/coverage-2026-10-31.md @@ -1,22 +1,23 @@ # Existing-record coverage snapshot — 31 October 2026 Read-only coverage-v1 report, attempts25–35, fixed ten pilots. No new model calls. -Observed at 2026-11-01T00:24:43.465652Z. Generated using dd.curation.coverage in an isolated process: Repo read-only, Oban peer/queues/plugins disabled, no HTTP server. +Observed at 2026-11-01T00:33:36.141655Z. Generated using dd.curation.coverage in an isolated process: Repo read-only, Oban peer/queues/plugins disabled, no HTTP server. Main/live code remains49dbf87. Report artifact: /Volumes/LLM Models/dictionary/run/275-quality-preflight-2026-10-31/coverage.json. +Current eligibility uses the deployed dev discovery-retention policy (config/config.exs with dev settings), explicitly replacing shortened test policy in this isolated report process. The first trial snapshot used test windows and was discarded; this table is the corrected snapshot. No live configuration changed. These are historical packet counts and a later current-eligibility snapshot, not a new quality score or a promise that old provider payloads are available. | Page | Attempt | Eligible pool then | Packet | Selected | Published versions from attempt | Current shown / withheld | |---|---:|---:|---:|---:|---|---| -| nepotism | 30 | 71 | 24 | 8 | 2 (8 placements) | 5 / 3 | -| greed | 31 | 101 | 24 | 6 | 1 (6 placements) | 3 / 3 | -| war | 27 | 232 | 24 | 11 | none | 5 / 6 | -| war | 35 | 232 | 24 | 11 | 2 (11 placements) | 5 / 6 | +| nepotism | 30 | 71 | 24 | 8 | 2 (8 placements) | 8 / 0 | +| greed | 31 | 101 | 24 | 6 | 1 (6 placements) | 6 / 0 | +| war | 27 | 232 | 24 | 11 | none | 10 / 1 | +| war | 35 | 232 | 24 | 11 | 2 (11 placements) | 10 / 1 | | love | 25 | 202 | 24 | 7 | 4 (7 placements) | 4 / 3 | -| coward | 26 | 116 | 24 | 12 | 1 (11 placements) | 5 / 6 | -| patience | 32 | 86 | 24 | 6 | 2 (6 placements) | 4 / 2 | -| justice | 33 | 113 | 24 | 6 | 2 (6 placements) | 4 / 2 | -| truth | 29 | 119 | 24 | 10 | 2 (10 placements) | 5 / 5 | -| hope | 28 | 177 | 24 | 11 | 2 (11 placements) | 6 / 5 | +| coward | 26 | 116 | 24 | 12 | 1 (11 placements) | 11 / 0 | +| patience | 32 | 86 | 24 | 6 | 2 (6 placements) | 6 / 0 | +| justice | 33 | 113 | 24 | 6 | 2 (6 placements) | 6 / 0 | +| truth | 29 | 119 | 24 | 10 | 2 (10 placements) | 9 / 1 | +| hope | 28 | 177 | 24 | 11 | 2 (11 placements) | 10 / 1 | | bestiality | 34 | 78 | 24 | 0 | none | 0 / 0 | Attempt35 is the separate war recovery, not a replacement in the primary25–34 quality denominator.