Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/curation/runtime-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,22 @@ mix dd.runtime setup --model qwen3.5:9b --slug qwen3.5-9b-whole-generic-v1 \
--instruction generic-whole-entry-v1 --accept-license
```

The persona arms (#307) are v7 with only the lens changed: `hitchens-whole-entry-v1` and
`bierce-whole-entry-v1` are `Packet.grounded_with_lens/1` of each reviewed dossier's
lens (`priv/curation/agents/personas/<slug>/v1/profile.json`), compiled in, so their
policy sections are v7's byte for byte. They are comparison-only, like the generic
control, and need their own model configs:

```
mix dd.runtime setup --model qwen3.5:9b --slug qwen3.5-9b-whole-hitchens-v1 \
--instruction hitchens-whole-entry-v1 --accept-license
mix dd.runtime setup --model qwen3.5:9b --slug qwen3.5-9b-whole-bierce-v1 \
--instruction bierce-whole-entry-v1 --accept-license
mix dd.runtime compare --batch s1-2026-11-05 --pilot --dry-run \
--arms V=qwen3.5-9b-whole-voltaire-v7,G=qwen3.5-9b-whole-generic-v1,H=qwen3.5-9b-whole-hitchens-v1,B=qwen3.5-9b-whole-bierce-v1 \
--out "/Volumes/LLM Models/dictionary/run/s1-2026-11-05/batch.json"
```

D6's four policy arms, dry run first (it calls and writes nothing):

```
Expand Down
25 changes: 25 additions & 0 deletions lib/devils_dictionary/curation/runtime/packet.ex
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,29 @@ defmodule DevilsDictionary.Curation.Runtime.Packet do
# without a dossier.
@generic_whole_entry @grounded_policy

# The persona arms (#307, #305 Stage 1): v7's grounded body with a reviewed
# dossier's lens (#302), composed as `grounded_with_lens/1` composes it, so
# every policy section is v7's byte for byte. The dossiers are read by slug
# through `Agents.Persona` and compiled in; a changed lens is a new dossier
# version under a new instruction name, never these.
@personas Path.expand("../../../../priv/curation/agents/personas", __DIR__)

@persona_arms (for {instruction, slug} <- [
{"hitchens-whole-entry-v1", "hitchens"},
{"bierce-whole-entry-v1", "bierce"}
],
into: %{} do
alias DevilsDictionary.Curation.Agents.Persona
path = Persona.path(slug, 1, root: @personas)
Module.put_attribute(__MODULE__, :external_resource, path)
manifest = Persona.manifest!(slug, 1, root: @personas)

if manifest["template_version"] != instruction,
do: raise("#{slug} v1 must name #{instruction} as its template")

{instruction, @grounded_policy <> @lens_preface <> Persona.lens(manifest)}
end)

# The fixed-lead variant (#298 C9, arms L and PL): v7 with only its lead
# section changed. Where the packet names a `fixed_lead` (the applicable
# Bierce entry), the rule leads and the model chooses highlights only;
Expand Down Expand Up @@ -367,6 +390,8 @@ defmodule DevilsDictionary.Curation.Runtime.Packet do
"voltaire-whole-entry-v7" => @whole_grounded,
"voltaire-whole-entry-v7-fixed-lead-v1" => @whole_grounded_fixed_lead,
"generic-whole-entry-v1" => @generic_whole_entry,
"hitchens-whole-entry-v1" => Map.fetch!(@persona_arms, "hitchens-whole-entry-v1"),
"bierce-whole-entry-v1" => Map.fetch!(@persona_arms, "bierce-whole-entry-v1"),
"generic-editor-v1" => @generic_editor_v1,
"generic-editor-v2" => @generic_editor_v2,
"voltaire-selector-v1" =>
Expand Down
7 changes: 6 additions & 1 deletion lib/devils_dictionary/curation/whole_entry.ex
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,12 @@ defmodule DevilsDictionary.Curation.WholeEntry do
# The section output, the contract and the evidence guidance admit every
# one of them from this list: a new comparison instruction (a persona's,
# #298 C4) is pinned in `Runtime.Packet` and named here.
@comparison ["generic-whole-entry-v1", "voltaire-whole-entry-v7-fixed-lead-v1"]
@comparison [
"generic-whole-entry-v1",
"voltaire-whole-entry-v7-fixed-lead-v1",
"hitchens-whole-entry-v1",
"bierce-whole-entry-v1"
]

# The instructions whose packets name a rule-fixed lead (#298 C9).
@fixed_lead ["voltaire-whole-entry-v7-fixed-lead-v1"]
Expand Down
9 changes: 9 additions & 0 deletions priv/curation/agents/instructions/bierce-whole-entry-v1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Bierce, whole entry, version 1

A comparison-only instruction for the first persona measurement (#307, prepared 4 November 2026). No configuration can bind it and nothing it chooses is published.

- Version 7's whole-entry evidence policy and section rules, unchanged word for word: the lead decided first, positive supplied evidence for every optional choice, one evidence rule for every type, the same limits and the same 100-character reasons.
- Selection lens: the Bierce dossier's disclosure and selection rule (dossier version 1). Among supported records it favours a precise contrast, shown in the source itself, between a professed principle and the interest or conduct behind it, and an exact, economical contrast over a loud one.
- The lens breaks ties on highlights only. The Bierce-first lead rule is the lead rule's, not the lens's. It never widens eligibility, meanings or quotation rules, and never speaks for Bierce or calls anyone a hypocrite in the dictionary's voice.

Packet: the same as version 7's, up to 24 candidates and 12 meanings; 8K context, 1,024 output tokens, a 120-second deadline and a shared 30-minute daily budget.
9 changes: 9 additions & 0 deletions priv/curation/agents/instructions/hitchens-whole-entry-v1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Hitchens, whole entry, version 1

A comparison-only instruction for the first persona measurement (#307, prepared 4 November 2026). No configuration can bind it and nothing it chooses is published.

- Version 7's whole-entry evidence policy and section rules, unchanged word for word: the lead decided first, positive supplied evidence for every optional choice, one evidence rule for every type, the same limits and the same 100-character reasons.
- Selection lens: the Hitchens dossier's disclosure and selection rule (dossier version 1). Among supported records it favours one whose own source carries the evidence, such as a documented case or a dated record, and one that tests the word against a case rather than merely asserting it.
- The lens breaks ties on highlights only. It never sets the lead, never widens eligibility, meanings or quotation rules, and never speaks for Hitchens or characterises a living person.

Packet: the same as version 7's, up to 24 candidates and 12 meanings; 8K context, 1,024 output tokens, a 120-second deadline and a shared 30-minute daily budget.
15 changes: 12 additions & 3 deletions test/devils_dictionary/curation/agents/persona_dossiers_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,16 @@ defmodule DevilsDictionary.Curation.Agents.PersonaDossiersTest do
assert source["basis"] =~ "#{headword} is quoted only as a documented period prejudice"
assert Enum.any?(d["blind_spots"], &(&1 =~ headword)), label

# Neither the headword nor the passage itself, in any case.
# Neither the headword nor the passage itself, in any case, and not in
# the disclosure either: the instruction is compiled from it (#307).
lens =
Enum.map(
[d["selection"], d["positive_example"], d["negative_example"] | d["principles"]],
[
d["disclosure"],
d["selection"],
d["positive_example"],
d["negative_example"] | d["principles"]
],
&String.downcase/1
)

Expand Down Expand Up @@ -190,8 +196,11 @@ defmodule DevilsDictionary.Curation.Agents.PersonaDossiersTest do
assert String.to_integer(stated) == Enum.sum(words), label

# Locators describe paragraphs rather than quote them, as `rights` says.
# An opening single quote counts; `\u2019` and `'` stay allowed as apostrophes.
locators = source["locators"] ++ Enum.map(source["passages"], & &1["locator"])
refute Enum.any?(locators, &String.contains?(&1, ["\u201C", "\u201D", "\""])), label

refute Enum.any?(locators, &String.contains?(&1, ["\u201C", "\u201D", "\"", "\u2018"])),
label
end
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ defmodule DevilsDictionary.Curation.Runtime.ComparisonInstructionsTest do
by every runtime allowlist and by no publication path; the generic
control sees v7's guidance exactly; the fixed-lead variant differs from
v7 only in its lead section, and its contract puts the rule's lead in.
#307: the Hitchens and Bierce arms are v7 with only the lens changed.
"""
use ExUnit.Case, async: true

Expand All @@ -20,6 +21,7 @@ defmodule DevilsDictionary.Curation.Runtime.ComparisonInstructionsTest do
@v7 "voltaire-whole-entry-v7"
@generic "generic-whole-entry-v1"
@fixed "voltaire-whole-entry-v7-fixed-lead-v1"
@personas %{"hitchens" => "hitchens-whole-entry-v1", "bierce" => "bierce-whole-entry-v1"}

defp packet(instruction, extra \\ %{}) do
native =
Expand Down Expand Up @@ -77,6 +79,12 @@ defmodule DevilsDictionary.Curation.Runtime.ComparisonInstructionsTest do

assert Digest.sha256(Packet.instruction!(@fixed)) ==
"2af1ca1aee77687ef4e302253c36ded81ea64cdfcd35ab12d0447515a4754183"

assert Digest.sha256(Packet.instruction!(@personas["hitchens"])) ==
"3463bce9960a3919fbe8d9ef79aea83a06e8cfc3d15800338ee1cfdf73abe9ab"

assert Digest.sha256(Packet.instruction!(@personas["bierce"])) ==
"b02355a9a294f4f4c5deca7d07325ae705fda6dec045ab2b64e7ac92efcb57f0"
end

test "the generic control is v7's evidence policy without a dossier" do
Expand All @@ -94,6 +102,31 @@ defmodule DevilsDictionary.Curation.Runtime.ComparisonInstructionsTest do
assert Packet.grounded_with_lens(lens) == Packet.instruction!(@v7)
end

test "a persona arm is v7's policy byte for byte with its dossier's lens (#307)" do
v7 = Packet.instruction!(@v7)
v7_lens = "voltaire" |> Persona.manifest!(5) |> Persona.lens()
policy = String.replace_suffix(v7, v7_lens, "")
assert byte_size(policy) == byte_size(v7) - byte_size(v7_lens)

for {slug, instruction} <- @personas do
text = Packet.instruction!(instruction)

# The dossier on disk, read without `Persona`: disclosure, a newline
# and selection, as Voltaire's lens is compiled.
%{"template_version" => ^instruction, "dossier" => d} =
"priv/curation/agents/personas/#{slug}/v1/profile.json"
|> File.read!()
|> Jason.decode!()

lens = d["disclosure"] <> "\n" <> d["selection"]
assert lens == slug |> Persona.manifest!(1) |> Persona.lens()
assert text == policy <> lens
assert text == Packet.grounded_with_lens(lens)
assert String.starts_with?(text, Packet.grounded_policy())
refute text =~ "Voltaire"
end
end

test "the fixed-lead variant differs from v7 only by its lead clause" do
fixed = Packet.instruction!(@fixed)
assert String.replace(fixed, Packet.fixed_lead_clause(), "") == Packet.instruction!(@v7)
Expand All @@ -106,7 +139,10 @@ defmodule DevilsDictionary.Curation.Runtime.ComparisonInstructionsTest do

describe "allowlists" do
test "every runtime allowlist admits the comparison instructions" do
for instruction <- [@generic, @fixed] do
assert WholeEntry.comparison_instructions() ==
[@generic, @fixed, @personas["hitchens"], @personas["bierce"]]

for instruction <- WholeEntry.comparison_instructions() do
assert instruction in Packet.instruction_versions()
assert instruction in WholeEntry.instructions()
assert SectionOutput.enabled?(%{"instruction_version" => instruction})
Expand Down Expand Up @@ -160,6 +196,23 @@ defmodule DevilsDictionary.Curation.Runtime.ComparisonInstructionsTest do
refute Packet.prompt_sha256(packet(@generic)) == Packet.prompt_sha256(packet(@v7))
end

test "a persona arm sees exactly v7's packet, guidance and schema (#307)" do
[v7_system, v7_user] = Packet.messages(packet(@v7))

for instruction <- Map.values(@personas) do
[system, user] = Packet.messages(packet(instruction))
assert system.content == Packet.instruction!(instruction)
refute system.content == v7_system.content
assert user.content == v7_user.content
assert Contract.schema(packet(instruction)) == Contract.schema(packet(@v7))

assert Packet.evidence_fingerprint(packet(instruction)) ==
Packet.evidence_fingerprint(packet(@v7))

refute Packet.prompt_sha256(packet(instruction)) == Packet.prompt_sha256(packet(@v7))
end
end

test "v7's synthetic prompt is unchanged by the policy list" do
v7 = %{
"instruction_version" => @v7,
Expand Down
2 changes: 2 additions & 0 deletions test/devils_dictionary/curation/runtime/packet_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,11 @@ defmodule DevilsDictionary.Curation.Runtime.PacketTest do
# A pinned text never changes: an attempt's `prompt_sha256` depends on it.
test "each instruction version is held to its text" do
assert Packet.instruction_versions() == [
"bierce-whole-entry-v1",
"generic-editor-v1",
"generic-editor-v2",
"generic-whole-entry-v1",
"hitchens-whole-entry-v1",
"voltaire-selector-v1",
"voltaire-whole-entry-v1",
"voltaire-whole-entry-v2",
Expand Down