Skip to content

Curation P4: the packet carries works and examples, bounds meanings as well as candidates, and records its cut (under #254, #253) #262

Closed

Description

@holden

Dedicated implementation issue for P4 of #254, under #253 (the product rules, which govern on any conflict) and #193. Written 26 October 2026 after P1, P2 and P5 are live and P7's fixes are merged. Same shape as #255, #257 and #259. This is the last infrastructure part before P6, the first bot proposal.

1. Outcome

The frozen packet a model sees carries everything a human can choose from and nothing a human cannot: definitions, sense quotations, catalog works and accepted exemplars, under the same eligibility, with a bound on meanings as well as candidates, a deterministic cut that records what it left out, and a contract that validates every kind. A packet builds, freezes and verifies for every whole On scope among the ten pilot pages, including love (43 senses, 48 packet meanings), and the benchmark fixtures still verify. No model is called in this part; P6 calls it.

2. What you see

Nothing on the public page changes in P4. What changes is visible in two places:

2.1 An attempt's record (inference_attempts.packet_summary), the audit D9 needs

packet_version   runtime-packet-v2
instruction      <from the pinned config, never a literal>
scope            love · lexemes 9051 14794 401975 402387 1492222 · en
meanings         12 of 48 (required 3, bounded 9) · policy meanings-v2
candidates       12 of 36 · content 4, sense_quotation 3, work 3, exemplar 2
priority         Bierce LOVE/n (always in)
omitted          24 · by id with reason: over_quota 21, meaning_out_of_bound 3
hashes           packet sha256 …, candidate set sha256 …

2.2 The operator's view (mix dd.runtime packet --scope love --dry-run)

A new dry-run mode prints the table above for a slug without admitting anything, so the owner and the auditor can see exactly what an editor would be offered before P6 exists.

3. Seams

Need Existing
The packet today Runtime.Packet.build/3 gathers Bierce entries, other definitions and sense quotations; refuses {:too_many_candidates, n} above max_candidates (12, Runtime.Endpoint) and {:too_many_meanings, n} above @max_meanings 12; freeze/1, verify/1, summary/1; @version "runtime-packet-v1", @instruction_version "generic-editor-v1" as literals
What a human can choose Curation.Candidates.for_page/1 (P2): leads, highlights of the four kinds with item specs and Eligibility.check/3 results. Reuse it for candidate discovery; the packet must offer exactly what the screen offers, re-checked at freeze
The contract Runtime.Contract.schema/0 and validate/3: candidate_id, meaning_id, reason ≤ 280, up to three highlights, quote an exact substring, re-validated against Eligibility and LeadRule
Eligibility per kind Curation.Eligibility (content, sense quotation by words hash, catalog work by manifest and checksum, exemplar by accepted claim)
The pinned instruction P7's second PR pins generic-editor-v2 beside v1 and makes Packet read the instruction version from config. P4 builds on that; if it is not merged yet, read the version from config with v1 as the default and do not duplicate P7's work
Fixtures priv/curation/runtime_bench/packets/{ordinary,ambiguous,sparse,adverse}.json and plan.json; Runtime.Bench
Freshness P5's fingerprint today is the candidate-set digest candidates-v1; #259 §5 names the versioned transition to the packet digest. P4 defines packet-v2 as a second fingerprint input and P5's compare takes either, so a change of policy version re-admits once and never loops

4. Scope

  • work (page-matched catalog artworks, and P3's pinned works when they exist) and exemplar (accepted illustrates claims) as candidate kinds in Packet.build/3, discovered through Candidates.for_page/1.
  • Two bounds, both deterministic, replacing both refusals: candidates ≤ 12, meanings ≤ 12. Required meanings are kept: every chosen candidate's meaning, and the lead's. The rest are ranked by how many candidates cite them, then by sense rank on the page, then by id. A candidate whose meaning was bounded out is omitted with meaning_out_of_bound.
  • The cut: Bierce entries always in (and counted against the content quota; if Bierce entries alone exceed the candidate cap, refuse {:priority_candidates_exceed_cap, n} rather than trim, as Curation agent MVP: what an editor does, what a curated entry contains, how the page looks, how the owner and the bots share one record, and how it stays current (under #193) #253 §5a decides); quotas content 4, sense_quotation 3, work 3, exemplar 2, filled in source-priority order (Bierce, Johnson, WordNet, Wiktionary, Wikipedia; verified corpus before live; accepted claims by review date), ties by object id ascending; unused quota flows to the next kind in that order.
  • Omissions recorded on the frozen packet and the attempt summary by id and reason; never excerpt text.
  • packet_version → runtime-packet-v2; freeze/1 and verify/1 re-derive every candidate of every kind at dispatch time and refuse :packet_stale on any change.
  • Contract accepts the new kinds; a work or exemplar highlight carries no quote; a sense_quotation still needs an exact substring.
  • mix dd.runtime packet --scope <slug> --dry-run prints §2.2 and writes nothing.
  • The P5 fingerprint transition (§3 last row).
  • Benchmark fixtures: re-frozen under v2 only if they fail to verify, with the reason recorded in plan.json's notes; the plan's cases keep their meaning.

Out of scope: any model call or proposal (P6); pinning (P3); personas; changing eligibility or the lead rule; changing what the screen offers; a general poem-to-page relation (the owner's decision, #253 D-list). If a blocker needs more, write it on the PR and stop.

5. Build

Change What
lib/devils_dictionary/curation/runtime/packet.ex kinds, bounds, cut, omissions, v2, config-read instruction
lib/devils_dictionary/curation/runtime/contract.ex kinds and the quote rule
lib/devils_dictionary/curation/runtime/endpoint.ex quotas and policy versions as code defaults, configuration-overridable
lib/devils_dictionary/curation/refresh.ex accept packet-v2 as a fingerprint input beside candidates-v1
lib/mix/tasks/dd.runtime.ex the packet --dry-run subcommand
priv/curation/runtime_bench/{packets,plan.json} only if re-freezing is needed
docs/curation/runtime-stage-a.md one section: packet v2
Tests: test/devils_dictionary/curation/runtime/packet_test.exs (extend), packet_v2_scope_test.exs (new), contract_test.exs (extend) §6

6. Tests

  1. Every candidate kind the screen offers for a page appears in the packet or in the omitted list; nothing the screen does not offer appears (property over the ten pilot scopes).
  2. Bierce entries survive any cut; Bierce entries exceeding the cap refuse, never trim.
  3. Quotas, flow-down and tie-breaks are deterministic: two builds of the same scope at the same record are byte-identical.
  4. Meanings bound: love's whole On scope (five lexemes, 43 senses, 48 packet meanings) builds, freezes and verifies with 12 meanings, every chosen candidate's meaning kept; so do war (37) and hope (58).
  5. The omitted list names every dropped candidate by id with its reason; no excerpt text in the summary.
  6. verify/1 refuses :packet_stale when a work's catalog checksum changes or an exemplar's claim is withdrawn after freezing.
  7. The contract accepts a work and an exemplar highlight without a quote and rejects one with a quote; a sense_quotation without an exact substring is rejected as before.
  8. Fixtures: the four benchmark packets verify under v2, or the re-freeze is recorded and the plan's cases still match their intent.
  9. The dry-run prints §2.2 for greed (no lead, 7 quotations, 6 meanings) and for love, and writes nothing (row counts unchanged).
  10. P5's fingerprint accepts the new input: a policy version change re-admits one run and the next visit admits none.

Precommit: MIX_ENV=test MIX_TEST_PARTITION=_cp4 DD_NO_OBAN=1 mix precommit; report the count.

7. Delivery A, implementation ready for review

  1. Worktree on the external drive with the usual symlinks.
  2. Build §4–§5, tests §6, precommit green.
  3. Dry-run on the test partition for the ten pilot pages and paste the ten §2.2 tables in the PR.
  4. PR "Curation P4 (Curation P4: the packet carries works and examples, bounds meanings as well as candidates, and records its cut (under #254, #253) #262): the packet carries works and examples, bounds meanings, and records its cut". CodeRabbit and one independent review addressed. Write "PR open" on Curation MVP implementation plan: seven parts with agent-ready briefs (P1 read compositions, P2 screen, P3 pin, P4 packet, P5 freshness, P6 proposals, P7 runtime) — under #253 #254's P4 line; tick on merge. Comment here. Stop with the merge command.

8. Delivery B, after the merge

No migration, no restart (no config file change; quotas are code defaults). After git pull, one local request recompiles the published build. Then, read-only on v2: mix dd.runtime packet --scope love --dry-run, the same for war and hope, pasted here as the live evidence; and confirm P5's clock on love re-admitted exactly once for the fingerprint transition and then settled. Close this issue; tick P4 on #254.

9. Constraints

As #254's ground rules: external-drive worktrees, port 5434, private partition, never touch the 4007 server or its launch script, nothing against v2 except the read-only dry-run in Delivery B, gh pr merge is the owner's, build only this issue, report a concrete blocker instead of widening.

10. Acceptance

11. Implementation prompt

Implement #262 exactly as written. Read it in full, then #254's ground rules and P4 section, #253 §5a (the trimming policy and both bounds), §9 and D9, PR #258's Candidates module and PR #260's runtime changes, docs/curation/runtime-stage-a.md and the benchmark record, and the files in §3 and §5. Discover candidates through Curation.Candidates.for_page/1 so the packet offers exactly what the screen offers; implement both bounds and the deterministic cut with recorded omissions; move the packet to v2; read the instruction version from config; extend the contract; add the dry-run; add the P5 fingerprint transition. Call no model. Build with the tests in §6, dry-run the ten pilot pages on the test partition, run the required precommit, open the PR, address CodeRabbit and one independent review, write "PR open" on #254's P4 line, comment here, and stop with the merge command. Do not build proposals, pinning or anything in P6 or P3. Report a concrete blocker instead of broadening scope.

Activity

  1. holden commented on Oct 26, 2026

    @holden
    MemberAuthor

    Delivery A: PR #265 is open (26 Oct 2026). It is stacked on #263 and must merge after it. Once #263 lands in main, the branch is rebased onto main once.

    What #265 builds, as §4 and §5 ask:

    • runtime-packet-v2 discovers its candidates through Curation.Candidates.for_page/1, so works and accepted exemplars join definitions and quotations. Each candidate is held to the meaning the screen offers it under, and a definition is offered as the screen shows it.
    • Both bounds hold: 12 candidates and 12 meanings, with every chosen candidate's meaning kept.
    • Bierce is always in and counted against content. Too many Bierce entries refuse the packet; it is never trimmed.
    • Quotas 4/3/3/2 are filled in source-priority order, ties by object id, with unused quota flowing to the next kind.
    • What the cut leaves out is recorded on the packet and the attempt summary by id and reason, never by text.
    • verify/1 re-derives every kind against the screen, the catalog, the claim and the priority set.
    • The contract has quote_not_allowed for works and exemplars.
    • mix dd.runtime packet --scope SLUG --dry-run --instruction V prints the cut.
    • P5's fingerprint becomes packet-v2/cut-v1/meanings-v2. A candidates-v1 baseline re-admits its page once and then settles.

    Checks:

    • Tests 1–10 pass, plus the review's cases. Precommit _cp4 gives 19 doctests, 2,979 tests, 0 failures.
    • The ten pilot pages build, freeze and verify on the test partition, love, war and hope among them. The ten §2.2 tables are in the PR. Under v1, five of the ten were refused.
    • The benchmark packets were not re-frozen. All eight verify read-only against devils_dictionary_runtime_bench, and bench seed now keeps them in place.
    • One independent review was addressed (response on the PR). CodeRabbit's one finding was fixed in e31d697.

    Two questions are on the PR for the owner:

    1. Should unused quota also flow back to cut candidates? Today it flows only forward, so hope sends 8 of 14 with 4 places empty. Allowing it would be a cut-v2.
    2. Exemplars are ordered oldest acceptance first. Should that stay?

    A fact for P7's go/no-go: a v2 packet never offers a definition under a sense, so #263's ambiguous check scores a choice a P6 editor will not have.

    Delivery B (after the merge, on the owner's word): no migration and no restart.

    1. git pull, then one local request.
    2. Run the read-only dry runs for love, war and hope on v2 and paste them here.
    3. Confirm love's clock re-admitted once (fingerprint_version) and then settled.
    4. Close this issue and tick P4 on Curation MVP implementation plan: seven parts with agent-ready briefs (P1 read compositions, P2 screen, P3 pin, P4 packet, P5 freshness, P6 proposals, P7 runtime) — under #253 #254.

    Merge: gh pr merge 265 --merge, after #263 and the rebase.

  2. added a commit that references this issue on Oct 26, 2026
  3. holden commented on Oct 26, 2026

    @holden
    MemberAuthor

    Delivery B, done (26 Oct 2026).

    The three dry runs on v2

    mix dd.runtime packet --scope SLUG --dry-run --instruction generic-editor-v2. v2 holds no model config, so the instruction is named: the revised, pinned one.

    love

    packet_version   runtime-packet-v2
    instruction      generic-editor-v2
    scope            love · lexemes 9051 14794 401975 402387 1492222 · en
    meanings         12 of 48 (required 5, bounded 7) · policy meanings-v2
    candidates       10 of 32 · content 3, sense_quotation 4, work 3, exemplar 0 · policy cut-v1
    priority         bierce LOVE (n) (content-1779138, always in)
    omitted          22 · by id with reason: over_quota 22
    hashes           packet sha256 7a6061f8caa4ea5be3a248132d2f88771fb76c1510d61f2ad468502ddf75ec73, candidate set sha256 f9226fce83bbe34953bf048b876d5d3a361483485ce6cedb04576b858adce87b
    
    love: chosen and omitted, by id
    chosen
      c1  content         bierce      content-1779138               m0
      c2  content         johnson     content-1807159               m0
      c3  content         johnson     content-1807160               m1
      c4  sense_quotation wiktionary  quotation-248429-1            s33
      c5  sense_quotation wiktionary  quotation-248429-2            s33
      c6  sense_quotation wiktionary  quotation-248469-1            s30
      c7  sense_quotation wiktionary  quotation-248469-2            s30
      c8  work            wikidata    work-3734892                  s16
      c9  work            wikidata    work-3735130                  s16
      c10 work            wikidata    work-3735439                  s16
    omitted
      quotation-248482-0                sense_quotation over_quota
      quotation-248491-1                sense_quotation over_quota
      quotation-248494-0                sense_quotation over_quota
      quotation-248494-1                sense_quotation over_quota
      quotation-248526-0                sense_quotation over_quota
      quotation-248526-1                sense_quotation over_quota
      quotation-248526-2                sense_quotation over_quota
      quotation-248526-3                sense_quotation over_quota
      quotation-248575-3                sense_quotation over_quota
      quotation-248598-1                sense_quotation over_quota
      quotation-248598-2                sense_quotation over_quota
      quotation-248621-1                sense_quotation over_quota
      quotation-248676-0                sense_quotation over_quota
      quotation-248741-0                sense_quotation over_quota
      quotation-248786-1                sense_quotation over_quota
      quotation-248837-0                sense_quotation over_quota
      quotation-248953-0                sense_quotation over_quota
      quotation-249021-0                sense_quotation over_quota
      quotation-507643-0                sense_quotation over_quota
      quotation-507756-0                sense_quotation over_quota
      quotation-1734289-2               sense_quotation over_quota
      work-3735469                      work            over_quota
    
    **war**
    packet_version   runtime-packet-v2
    instruction      generic-editor-v2
    scope            war · lexemes 55026 147197 524674 876817 1036067 1037852 1201408 · en
    meanings         12 of 37 (required 4, bounded 8) · policy meanings-v2
    candidates       9 of 29 · content 3, sense_quotation 4, work 2, exemplar 0 · policy cut-v1
    priority         bierce WAR (n) (content-1779512, always in)
    omitted          20 · by id with reason: over_quota 20
    hashes           packet sha256 1fc7c48c86e30892a3ff2685a59392e02f1aea298daae8586c525988aa3fe1a5, candidate set sha256 227a3b55d926d7f4c1e5e8b2a22e3530efb807b25ba99ba8ada75ca29611b420
    
    war: chosen and omitted, by id
    chosen
      c1  content         bierce      content-1779512               m1
      c2  content         johnson     content-1828065               m0
      c3  content         johnson     content-1828066               m0
      c4  sense_quotation wiktionary  quotation-367513-0            s19
      c5  sense_quotation wiktionary  quotation-367513-2            s19
      c6  sense_quotation wiktionary  quotation-367513-3            s19
      c7  sense_quotation wiktionary  quotation-367513-4            s19
      c8  work            wikidata    work-3734261                  s6
      c9  work            wikidata    work-3734923                  s6
    omitted
      quotation-367513-5                sense_quotation over_quota
      quotation-367569-0                sense_quotation over_quota
      quotation-367737-0                sense_quotation over_quota
      quotation-367737-1                sense_quotation over_quota
      quotation-367825-1                sense_quotation over_quota
      quotation-367825-2                sense_quotation over_quota
      quotation-367825-6                sense_quotation over_quota
      quotation-367825-7                sense_quotation over_quota
      quotation-367825-8                sense_quotation over_quota
      quotation-367825-9                sense_quotation over_quota
      quotation-367825-11               sense_quotation over_quota
      quotation-367825-12               sense_quotation over_quota
      quotation-367825-13               sense_quotation over_quota
      quotation-367825-15               sense_quotation over_quota
      quotation-367840-0                sense_quotation over_quota
      quotation-368012-0                sense_quotation over_quota
      quotation-368070-0                sense_quotation over_quota
      quotation-368288-0                sense_quotation over_quota
      quotation-368295-0                sense_quotation over_quota
      quotation-1101326-0               sense_quotation over_quota
    
    **hope**
    packet_version   runtime-packet-v2
    instruction      generic-editor-v2
    scope            hope · lexemes 79836 83337 145195 402744 · en
    meanings         12 of 58 (required 4, bounded 8) · policy meanings-v2
    candidates       8 of 17 · content 4, sense_quotation 3, work 1, exemplar 0 · policy cut-v1
    priority         bierce HOPE (n) (content-1778855, always in)
    omitted          9 · by id with reason: over_quota 9
    hashes           packet sha256 f21bb3b93a17a1774c3745ee7103e1a3686dc021e2984d6394e64cf3548bbca2, candidate set sha256 1553356354e4f3b488a015250f4c24995555a4bf0b9d88e171337dd9ff288909
    
    hope: chosen and omitted, by id
    chosen
      c1  content         bierce      content-1778855               m0
      c2  content         johnson     content-1800812               m1
      c3  content         johnson     content-1800813               m1
      c4  content         johnson     content-1800814               m0
      c5  sense_quotation wiktionary  quotation-247813-6            s17
      c6  sense_quotation wiktionary  quotation-247813-7            s17
      c7  sense_quotation wiktionary  quotation-247813-8            s17
      c8  work            wikidata    work-3735474                  s11
    omitted
      content-1800815                   content         over_quota
      quotation-248027-3                sense_quotation over_quota
      quotation-248067-0                sense_quotation over_quota
      quotation-248067-1                sense_quotation over_quota
      quotation-248131-3                sense_quotation over_quota
      quotation-248152-0                sense_quotation over_quota
      quotation-248152-1                sense_quotation over_quota
      quotation-248256-0                sense_quotation over_quota
      quotation-508436-0                sense_quotation over_quota
    

    What they show:

    • The scopes match Curation P4: the packet carries works and examples, bounds meanings as well as candidates, and records its cut (under #254, #253) #262. love is five words (9051 14794 401975 402387 1492222) with 48 meanings, war has 37 meanings over seven words, and hope has 58 over four. Each builds, freezes and keeps 12 meanings.
    • Under v1 all three were refused. love and war exceed 12 meanings; hope has 17 candidates.
    • No exemplar is offered on any of the three. v2 holds no accepted illustrates claim for them yet.
    • The forward-only flow leaves places empty live. love sends 10 of 32, war 9 of 29 and hope 8 of 17, with 2, 3 and 4 places unused. This is the owner's question on Curation P4 (#262): the packet carries works and examples, bounds meanings, and records its cut #265: cut-v2 would let unused places flow back to cut quotations.
    • Ties by object id cluster quotations by sense. Within a source, quotations sort by sense id, so the first sense's quotations fill the quota. war's four quotations all come from sense 367513, hope's three from 247813, and love's four from two senses. A rule that takes one quotation per sense before a second would give the editor more meanings to choose from. That is another cut-v2 question for the owner; I did not change it.

    P5's clock on love (composition 1)

    curation_refresh_states and oban_jobs, read only:

    row 1 · composition 1 · published version 1 · configuration version 1
    state idle · generation 3 · last_outcome needs_editor · last_outcome_reason fingerprint_version
    last_completed_at 2026-10-26 13:59:58Z · refresh_after 2026-10-28 13:59:58Z · dirty_at none
    input fingerprint version packet-v2/cut-v1/meanings-v2 · superseded_on_arrival false
    
    Curation.RefreshWorker jobs
    4645  completed  generation 1  love  2026-10-26 00:03:08   (P5 Delivery B)
    4646  completed  generation 2  love  2026-10-26 00:04:21   (P5 Delivery B)
    4730  completed  generation 3  love  2026-10-26 13:59:58   (this transition)
    
    • It re-admitted exactly once. The first visit after the merge (13:59:24Z) admitted generation 3. That run found the stored candidates-v1 baseline of another version, completed as needs_editor with reason fingerprint_version, and stored the new packet-v2/cut-v1/meanings-v2 fingerprint.
    • It has settled. No job since. Asked read-only, Refresh.decision/2 answers :fresh for row 1 now, so a visit admits nothing until 28 Oct, when the window is due.
    • The published pointer did not move. Composition 1 is still at version 1.

    Delivery A and B are complete. Ticking P4 on #254 and closing this issue. The two cut-v2 questions (backward flow; one quotation per sense first) and the exemplar order stay with the owner on #265.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions