Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/curation/agents-mvp.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,35 @@
# Current next experiment: frozen quality preflight (31 October 2026)

The 29 October trial remains a quality NO-GO. Read scorecard-v2-2026-10-31.md
before any new calls. The current implementation follow-up adds only a read-only
coverage report and immutable voltaire-whole-entry-v6 (same dossier, weights,
output-v4 contract, section/candidate/context/budget bounds). It clarifies that a
supplied sense_definition can lead without a content entry; Bierce priority and
honest abstention remain. v1–v5 text/hashes and live configuration7 are untouched.
No model start, enrollment beyond the ten pages or activation is part of this work.

Coverage examples (PHX_SERVER unset, explicit database target, DD_NO_OBAN=1):
mix dd.curation.coverage --all --from 25 --to 35
mix dd.curation.coverage --slug love --from 25 --to 35 --limit 20

These report existing records inside a read-only repeatable-read transaction.
Eligible pool is the historical packet counts.offered; raw available inventory is
not historically recorded and is labelled unknown. Packet omissions, selections,
published placements and current shown/withheld state have separate denominators.
The command prints no source payloads or editorial reasons, requests no discovery,
and calls no model. A later snapshot can have fewer visible items after expiry.
Do not archive provider content for evaluation.

A later reviewed migration admits v6 while preserving deployed v5 compatibility.
Do not apply it, register/admit/bind v6, enable the editor or start the runtime until
separately authorized. Rollback refuses once v6 config/profile history exists;
disable policy and roll forward instead of deleting history.

The old ngrok-only chat heartbeat was removed; public verification remains assigned
to #271/#252 once access is restored. Application cleanup/dispatch jobs are unchanged.
Historical operating notes below are retained as evidence, not current go-live
instructions or a replacement for the dated report in #275.

# Headless source preparation

The first cohort exposed an input gap: only love had discovery candidates. The other
Expand Down
29 changes: 29 additions & 0 deletions docs/curation/coverage-2026-10-31.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Existing-record coverage snapshot — 31 October 2026

Read-only coverage-v1 report, attempts25–35, fixed ten pilots. No new model calls.
Observed at 2026-11-01T00:33:36.141655Z. Generated using dd.curation.coverage in an isolated process: Repo read-only, Oban peer/queues/plugins disabled, no HTTP server.
Main/live code remains49dbf87. Report artifact: /Volumes/LLM Models/dictionary/run/275-quality-preflight-2026-10-31/coverage.json.
Current eligibility uses the deployed dev discovery-retention policy (config/config.exs with dev settings), explicitly replacing shortened test policy in this isolated report process. The first trial snapshot used test windows and was discarded; this table is the corrected snapshot. No live configuration changed.
These are historical packet counts and a later current-eligibility snapshot, not a new quality score or a promise that old provider payloads are available.

| Page | Attempt | Eligible pool then | Packet | Selected | Published versions from attempt | Current shown / withheld |
|---|---:|---:|---:|---:|---|---|
| nepotism | 30 | 71 | 24 | 8 | 2 (8 placements) | 8 / 0 |
| greed | 31 | 101 | 24 | 6 | 1 (6 placements) | 6 / 0 |
| war | 27 | 232 | 24 | 11 | none | 10 / 1 |
| war | 35 | 232 | 24 | 11 | 2 (11 placements) | 10 / 1 |
| love | 25 | 202 | 24 | 7 | 4 (7 placements) | 4 / 3 |
| coward | 26 | 116 | 24 | 12 | 1 (11 placements) | 11 / 0 |
| patience | 32 | 86 | 24 | 6 | 2 (6 placements) | 6 / 0 |
| justice | 33 | 113 | 24 | 6 | 2 (6 placements) | 6 / 0 |
| truth | 29 | 119 | 24 | 10 | 2 (10 placements) | 9 / 1 |
| hope | 28 | 177 | 24 | 11 | 2 (11 placements) | 10 / 1 |
| bestiality | 34 | 78 | 24 | 0 | none | 0 / 0 |

Attempt35 is the separate war recovery, not a replacement in the primary25–34 quality denominator.
Current shown/withheld totals are the page snapshot and repeat on war rows: do not sum them per attempt.
Publication placements may differ from selections because duplicate-display omissions/owner precedence are retained in existing receipts. No publication is attributed to failed assembly attempt27.
Raw historical inventory was not recorded: unknown, not zero. Eligible-pool count is stored counts.offered; packet/omitted counts and selection identities come from the frozen attempt summary/result.
No titles, excerpts, provider payloads or editorial reasons are copied into this report.

Before/after live checks: expected cluster7693849764459364596 on5434,35 total inference attempts, editor switch off. No migration/model/activation/deployment is part of this snapshot.
100 changes: 100 additions & 0 deletions docs/curation/scorecard-v2-2026-10-31.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Ten-page curator scorecard v2 — frozen 31 October 2026

Status: frozen before any further model calls; NOT RUN. This is a prospective
development evaluation, not routine human approval. The 29 October report remains
a quality NO-GO. Code/tests passing does not pass this scorecard.

## Cohort and execution boundary

Exactly: nepotism, greed, war, love, coward, patience, justice, truth, hope,
bestiality. These pages have already been observed and tuned: this is an iterative
retest, not a blind holdout or proof of generalization. No extra pages or personas.
Keep the installed Qwen3.5 9B artifact, one physical inference slot, existing
8K context/1,024 output-token/120-second call/30-minute shared daily bounds,
24 candidates/12 meanings and existing section limits. Do not fill weak slots.
No model call, model registration, admission, activation or deployment follows
from this document. A separately approved run manifest must freeze code SHA,
scorecard SHA256, model artifact/config hash, persona/instruction/configuration
version, packet-policy version, time window, primary-cohort inclusion rule and starting attempt-ID cutoff before calls.
Record the exact allocated attempt IDs/range after the window; do not predict future IDs.
New packets may reflect lawful provider changes: record those differences.
The first assessment of each page in that manifest is the primary cohort.
Failures/refusals/abstentions remain in its denominator. Repairs or later retries
are separate results, never replacements used to improve the primary score.

## Per-choice rubric and denominators

Evaluate every proposed choice (lead included) from each primary attempt, including
choices omitted during assembly; separately report actual published placements.
For each record classify intended-meaning alignment as supported, wrong, or
insufficient supplied evidence. Metadata-only material may justify only a modest
claim supported by its supplied fields. Do not infer unseen plots, lyrics, images,
articles or poem contents. Correct record type and creator attribution are required.
Score every reason separately: grounded, unsupported/misattributed, or policy-
withheld/unassessable. A title/name match alone does not establish topical relevance.

Report all three meaning counts and all reason counts. Primary meaning rate =
supported / all proposed choices, with wrong AND insufficient-evidence choices in
the denominator. Also show scoring coverage = (supported + wrong) / all choices,
and the old assessable-only rate labelled secondary. Policy-withheld reasons are
unassessable, not automatically grounded. Zero choices has no percentage and cannot
pass usefulness. No sampling or dropping inconvenient, expired or failed cases.
If lawful evidence is no longer available, mark insufficient evidence; do not
archive prohibited payloads or reconstruct them from memory. Inspect before expiry
when possible, retaining only permitted judgments, identities/digests and metrics.

## Page outcome and quality GO

Every page is assessed as useful, justified empty, incorrect abstention, failed/
refused, or not assessed. A useful page has an appropriate exact sourced lead and
a coherent useful core; judge each supplementary choice too, not just the lead.
Optional sections remain empty if no offered material supports them. The bestiality
regression specifically requires recognizing an eligible supplied sense definition,
not a hard-coded output. A truly empty/unsafe packet may justify abstention, but
that is not a useful curated entry.

Quality GO requires ALL:
- Ten primary page outcomes recorded, with zero invented quotations or unauthorized
source-content writes, and zero unsupported/misattributed published explanations.
- At least 90% supported meaning across ALL proposed choices by the denominator
above; report absolute counts and uncertainty, never only assessable-subset 98%.
- Ten useful curated entries, including bestiality when suitable native definitions
are supplied. Justified empty outcomes are honest partial progress, not completion
of the owner's ten-useful-entry goal.
- Each publication has inspectable exact references, actual agent/model/persona/
instruction/attempt/date, grounded reason/basis and authority. No fake human or
consensus attribution. Failed/abstained/unchanged results remain distinguishable.

Report intermediate improvement even on NO-GO. Do not change thresholds after calls.
A changed rubric/instruction/cohort creates a new dated experiment and preserves
this version, its failures and the original 29 October report.

## Lifecycle acceptance and cost

Quality and operational verdicts are separate. The subsequent authorized multi-day
window must observe a real due assessment without a page visit, a fresh repeat
with zero inference, unchanged assessment without duplicate publication, owner hold
preventing overwrite, honest offline deferral, and expiry with permitted citation
survival. Existing news-expiry evidence may be referenced by exact IDs/dates; do not
force a fresh news choice merely to fill a demonstration. Distinguish expired
selection/cache from provider fields explicitly allowed to remain durable.

Record every call including readiness, refused/failed/repair calls, artifact/config/
profile/instruction hashes, prompt/output tokens, elapsed inference, API wait,
available memory/swap and shared daily budget. Record developer evaluation minutes
prospectively; if unavailable say unmeasured. Useful published choices per machine
minute is reported, not inferred from successful calls alone. Small ten-page results
justify another bounded experiment, not corpus-wide autonomy or model superiority.

## Coverage report and evidence handling

Use mix dd.curation.coverage with DD_NO_OBAN=1 and PHX_SERVER unset.
The report reads existing attempts and the current published receipt; it requests
neither discovery nor inference and creates no rows. Historical counts.offered is
the eligible screen pool, candidates is the bounded packet, omitted gives exclusion
counts/reasons, result gives actual selected candidate IDs. Current shown/withheld
status is a later snapshot, not historical publication-time eligibility.
Total raw API inventory and historical developer scores were not captured in every
attempt: report unknown, not zero or an invented funnel. Do not sum snapshots across
attempts or claim every available source item was considered. Reports contain only
counts, statuses, permitted local IDs/hashes and dates, never provider text/reasons.
22 changes: 19 additions & 3 deletions lib/devils_dictionary/curation/agents/persona.ex
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,23 @@ defmodule DevilsDictionary.Curation.Agents.Persona do
@selection_manifest,
~w(dossier source_refs deceased_evidence_refs template_version)
)
@sense_lead_path Path.expand(
"../../../../priv/curation/agents/personas/voltaire/v7/profile.json",
__DIR__
)
@external_resource @sense_lead_path
@sense_lead_manifest @sense_lead_path |> File.read!() |> Jason.decode!()
@sense_lead_body Map.take(
@sense_lead_manifest,
~w(dossier source_refs deceased_evidence_refs template_version)
)
@whole_bodies %{
"voltaire-whole-entry-v1" => @whole_body,
"voltaire-whole-entry-v2" => @revised_body,
"voltaire-whole-entry-v3" => @sections_body,
"voltaire-whole-entry-v4" => @relevance_body,
"voltaire-whole-entry-v5" => @selection_body
"voltaire-whole-entry-v5" => @selection_body,
"voltaire-whole-entry-v6" => @sense_lead_body
}

def manifest, do: @manifest
Expand All @@ -106,6 +117,9 @@ defmodule DevilsDictionary.Curation.Agents.Persona do
@sections_manifest["dossier"]["disclosure"] <>
"\n" <> @sections_manifest["dossier"]["selection"]

def selection_instruction("voltaire-whole-entry-v6"),
do: selection_instruction("voltaire-whole-entry-v5")

def selection_instruction("voltaire-whole-entry-v5"),
do: selection_instruction("voltaire-whole-entry-v4")

Expand Down Expand Up @@ -150,7 +164,8 @@ defmodule DevilsDictionary.Curation.Agents.Persona do
"voltaire-whole-entry-v2",
"voltaire-whole-entry-v3",
"voltaire-whole-entry-v4",
"voltaire-whole-entry-v5"
"voltaire-whole-entry-v5",
"voltaire-whole-entry-v6"
],
:persona_instruction_mismatch
) do
Expand Down Expand Up @@ -184,7 +199,8 @@ defmodule DevilsDictionary.Curation.Agents.Persona do
"voltaire-whole-entry-v2",
"voltaire-whole-entry-v3",
"voltaire-whole-entry-v4",
"voltaire-whole-entry-v5"
"voltaire-whole-entry-v5",
"voltaire-whole-entry-v6"
],
do: "single_editor_v1",
else: "proposes only"
Expand Down
Loading