You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The context's test: a fresh agent adds a rule to the Kernel without being told how #8
Part of the epic "Epic: slice 0, foundations. The repo holds the line, its context is tested, and the platform facts are recorded".
Why. This is slice 0's own exit test, and the only measure of whether the context works. As first written it could not fail: the answer was in the repo the agent reads, and make verify passes on an untouched tree.
Done when
The task is one sentence, recorded under evals/. Its known answer, a boundary test for the rule, is held outside the repo, in the private held-out home; only its hash is recorded here.
A fresh agent given only the repo and that sentence makes a change on which the held-back test passes, make verify passes, and the agent's own new mutant is killed. Three runs out of three, with no help.
Each run records what it read, what it changed and what it cost.
A failed run changes the context, never the task, and the three runs are repeated.
Needs. The private held-out home, from the Maintainer. Run last in slice 0, after every story that changes the root AGENTS.md. Creating evals/ trips the context check on decision 0010, which is waiting on that directory for a rule of slice 3; that wait is re-pointed in this story.
Design. §9.1, "Slice 0 is done when"; §9.14, last paragraph.
This runs after #28 merges, since no ruled story changes the root AGENTS.md after that. Two things are settled for it:
evals/ is created by the new story "The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests", which starts now and re-points the one wait on evals/ in decision 0010 and the row in decision 0013. This story then adds its task sentence and hash there, and does not create the directory itself.
The private held-out home exists (2026-10-18). Its read token is the Maintainer's to mint when the eval-set story asks for it; this story's three runs read the held-back test from the same place.
Its three runs spend model capacity; run them when no builder needs the machine.
Solution Architect, delegated. On main at bc2eaed (#232). Read from the issue and its sequencing comment, the epic #3 ("Done when", "Order", "Rules for every story"), the plan review of 2026-10-19 (§2 Exit, the row on #8, the row on rows waiting on reserved directories), design §9.1 ("Slice 0 is done when"), §9.12 (the fresh reader, the second model family), §9.14 (last paragraph), §3.13 (where a held-out thing lives and why), §4.9 (the eval job's credential), docs/context-engineering.md (its last adopted row and its sources 5 and 16), decision 0013 (its last row, "not held yet"), decision 0010 (its evals row), decision 0033 (the Maintainer's own subscription, the pinned CLI), tools/check_context.py (PENDING, _is_pending, the waits on evals/), tests/support.py::pending, tests/test_evals.py as #221 left it (the two @pending tests, TheSet, LIVE), evals/AGENTS.md, tools/trial_p13_cli.sh (how a vendor's CLI is run headless at a pin with nothing of the shell reaching it, and nothing of the model's text reaching the fixture), the root AGENTS.md and kernel/AGENTS.md as they stand, and the open issues #178, #190, #226 and #237.
What this story decides. Slice 0's exit test and the only measure of the context: whether an agent given nothing but this repo and one sentence can add a rule to the Kernel the way AGENTS.md and kernel/AGENTS.md say, and pass the required check, three times out of three. The story decides what the task is and where its answer lives, how a fresh agent is given it, what the agent must leave behind, how a test in make verify judges that without a model in CI, which marks come off and which rows close, and what merges before it. It is the last story of slice 0: after it the exit is declared and the slice-1 epic opens.
1. The task is one Kernel rule the repo does not hold and no @pending test states. As first written the test could not fail, because every rule the Kernel lacks is stated by a @pending test the agent can read, and kernel/AGENTS.md tells it to take the mark off. So the task's rule is one with no test in the repo: its boundary test is the known answer, written in the private held-out home by a separate private session (as #183's cases are: no reviewer tools, no notes, nothing of it in a pull request, an issue, a session note or a shared transcript), and only its sha256 is recorded here. The rule is chosen by four criteria, which the ruling states so that the private session can choose another if the first is taken: (a) the design states it, so the agent can find the rule's words without being told how to build it; (b) it is one reason code, in one Kernel module, behind one boundary; (c) it needs no new fact and no new limit, so the task measures the context and not the schema work; (d) it is decided, so that the agent's change could stand if a story later builds it. The first choice is #237's rule, "Kernel: a merge read while an issue is held or declined is a Hold, not done (#227)": it is the Maintainer's ruling of 2026-10-20, it is one code in kernel/issue.py, it reads facts the schema already carries, and it has no test. Why: a task with a known answer, run more than once (docs/context-engineering.md, source 16); a task whose answer is in the repo measures nothing (the issue's "Why").
2. The task sentence. One sentence under evals/context/task.json, the rule in the design's words with its reason code, and nothing of how: for the first choice, "A merge of a linked pull request read while the issue is held or declined is a Hold with the reason merged_under_hold, never done." The sentence names the code because the code is the rule's name (every reason is a code, never a sentence) and the held-out test asserts it; it names no file, no test, no mutant and no document. Why: the agent is told what, as a story is, and finds how from the context; that is what is tested.
3. How a fresh agent is given it, and what it must leave. A runner, tools/context_eval.sh, modelled on tools/trial_p13_cli.sh and held to #178's list from the start: the vendor's CLI at the pin decision 0033 names, on a configuration directory of its own logged in to the Maintainer's own subscription (decision 0033), started with an empty environment but for the few names the script allows, so no key or token of the shell reaches it. The run: git archive of one commit of main into an empty directory outside every repo, so the agent has the files and no history, no remote, no notes and no other checkout; the CLI runs headless (-p) with the sentence as its whole prompt, its file and shell tools allowed inside that directory only, its network tools disallowed, and a cap on turns and on cost the fixture records. No system prompt is added, no skill is named, no hint is given: the sentence and the repo are all. The agent is expected to do what the root AGENTS.md says: read kernel/AGENTS.md, write the boundary case, change the Kernel, add the mutant, run both commands. What it leaves is the directory's diff. Why: "given only the repo and that sentence" (the issue); a fresh context that returns a short summary (§9.12); and decision 0033 for where the model credential lives.
4. How a run is judged, outside CI, with the model. After the agent stops, the runner, not the agent, in the same directory: (a) copies the held-out test in from the private home, read by the Maintainer at his own terminal (the read token is the eval job's, §4.9, and arrives with slice 1; until then the file is copied by hand), and runs it with python3 -m unittest: it must pass; (b) runs make verify on the directory: it must pass, the held-out test among the tests; (c) runs the mutation check as make verify does, and reads from its report that at least one mutant the agent added was killed, and none survived; (d) removes the held-out test before anything is recorded. A run passes when all three hold. Three runs, from the same commit and the same sentence, each a fresh directory and a fresh CLI session; the story passes at three of three. Why: the issue's three boxes; "the agent's own new mutant is killed" is the one thing that shows the agent followed the three-part rule of kernel/AGENTS.md and not only made a test pass.
5. What a run records, and how a test in CI judges it without a model. Each run leaves one record in evals/context/task.json, schema context-eval/1: the commit, the date, the CLI's version and the cap; what the agent read, as the paths of its file reads in order and the commands it ran, taken from the CLI's transcript by name and nothing else; what it changed, as git diff --stat of the directory against the export, with the names of the test, the mutant and the reason code it added; what it cost, as the CLI's own usage fields (num_turns, duration_ms, total_cost_usd, usage, modelUsage), kept by allowlist as #178 asks; and the three verdicts with the mutation report's line for the agent's mutant. No model text reaches the record: not the agent's summary, not a tool's input, not a diff. The test tests/test_evals.py::test_a_fresh_agent_given_the_repo_adds_a_rule_and_passes_the_check comes off @pending and holds the record: the file validates; the task is one sentence; its held-out test's sha256 is 64 hex; there are at least three runs on the task's hash and the newest three all pass; each run's reads name only paths that exist in the repo, each names a test, a mutant and a reason code, and each cost is a number; and no run holds a string that looks like code, a diff or prose longer than a path. tests/test_tools.py holds the assembler: a stand-in CLI transcript with model text in it yields a record with none; a run that failed any verdict is recorded as failed; a transcript the assembler cannot read records nothing. Why: CI runs no model and fetches nothing (tools/check_repo.py); what CI can hold is the fixture of the runs, as a trial's fixture is held, and that the fixture is honest about what it holds.
6. A failed run. The issue's rule: a failed run changes the context, never the task, and the three runs are repeated. The runner records the failed run too, so the record shows what was tried; the context change is a pull request of its own, with the record's failed run named in its body as the reason; then three fresh runs. The test counts the newest three on the task's hash, so an earlier failure stays in the record and does not fail the check. What "the context" means here: the root AGENTS.md, kernel/AGENTS.md, tests/AGENTS.md, docs/standards/, docs/lessons.md, the glossary, a skill. Not the design, not the Kernel. Why: the measure is of the context; a change to the task after a failure would be the POC's stub again.
7. Which marks come off, and which rows close. Off: decision 0013's last row ("not held yet"), docs/context-engineering.md's last adopted row, the two @pending marks in tests/test_evals.py that this story owns (only test_a_fresh_agent_...; test_the_review_intents_eval_set_has_its_cases is #183's second pull request and stays). Closed: the epic's box "An agent given only the repo's own context can add a rule to the Kernel and pass the required check without being told how"; design §9.1's "Slice 0 is done when" gains the date and the record's commit in the words §6's rows use; §9.14's last paragraph points at evals/context/. Not closed by this story: the other seven boxes, each its own story's.
8. Record. One new decision, numbered at enqueue (0042 as of this ruling): "The context is tested by a task run outside CI, and the runs are held as a fixture." It decides: the task is a rule the repo does not hold, with its answer held out; the runner and the judge are the Maintainer's terminal on his own subscription, never CI; a run record holds reads, changes and cost and no text of the model's; three of three is the bar; a failed run changes the context and never the task; and a change to the context after the record is re-run at the Maintainer's call, not by CI (CI cannot run a model, and cannot know which context change matters). Rows: the test above; tests/test_tools.py::test_a_context_run_record_holds_no_model_text; tests/test_evals.py::test_a_case_dropped_under_evals_in_any_file_is_caught extended to evals/context/; a row on 0033 ("the context's runs use the same login and pin").
9. Files. New: evals/context/task.json (in LIVE, schema context-eval/1), schemas/context-eval.schema.json, examples/context-eval.json with counter-examples under examples/invalid/ (a task of two sentences; a hash that is not 64 hex; a run whose reads hold a diff; a cost that is a string), tools/context_eval.sh and tools/context_eval.py (the assembler, pure: transcript and diff in, record out), docs/standards/ gains no page (the runner's runbook is a comment at the top of the script, as #178 asks of P-13's). Edited: tests/test_evals.py, tests/test_tools.py, tools/check_examples.py (LIVE), evals/AGENTS.md (rows), decision 0013, 0010 unchanged, 0033 (row), docs/context-engineering.md, the design §9.1 and §9.14, the new record. Not edited: the root AGENTS.md, kernel/AGENTS.md, kernel/ (the agents' changes are never merged; the rule is built by #237 afterwards, with its own public test).
10. What must merge before it, and what must not. Prerequisites: none of #178, #190, #226 and #237. #178 ("Trial scripts: P-13's script hardened") is not a prerequisite: the runner is a new script written to #178's list, and #178 may share code with it only after both are on main. #190 and #226 touch sim/grounding.py and tools/check_trials.py, which this story does not. #237 must not merge before the three runs are recorded, if its rule is the task: the task's rule must be absent from the Kernel at the commit the runs export. The sequence is: this story's first pull request (the schema, the runner, the assembler, the test still @pending, the task sentence with the hash) merges; the three runs are made from the commit after it; a second small pull request adds the records and takes the mark off; then #237 is built as a story. If the Maintainer would rather build #237 first, the private session takes the next rule by the four criteria of ruling 1 and the sequence is the same. Also before it: every ruled story that changes the root AGENTS.md; none does (#41 and #43 changed sim/AGENTS.md and the design, not the root).
11. Gates. Boundary tests first (the assembler's and the record's, on a synthetic transcript); export verify; a fresh Claude review and a security pass (the runner: nothing of the shell reaches the CLI, nothing of the model reaches the record, the export directory is outside every repo, the held-out test is copied in and out by the runner and never committed); Muse; AGY not needed (no kernel/ or sim/ change) unless the Maintainer wants a second family's read of the runner, in which case Flash. The pull request says "Part of #8" for the first and "Closes #8" for the second, and the second names the three runs' commit and cost.
12. Out of scope. The eval job's workflow and its read token (slice 1, eval.yml's reserved name); #183's cases and its second pull request; comparing runs with the context against runs without it (the measurement of source 5, worth a story once the runner exists); more than three runs; a rerun on every context change (the record's commit says what was tested; the Maintainer decides when to run again); the agent's changes themselves, which are discarded; any change to the task after a failure.
Needs the Maintainer? Three things, none of them a block on the first pull request: the held-out test, written in the private session from the rule of ruling 1, and its sha256 given to the builder; the three runs at his own terminal on his own subscription, when no builder needs the machine; and the choice of ruling 10's sequence for #237 (hold it until the runs, or give the task another rule).
The Maintainer's choice (2026-10-20, via the orchestration session): build and merge #237 first; #8's held-out task takes another rule, chosen by ruling 1's four criteria from the rules the repo does not hold, by the private session, and named nowhere in this repo or in any shared transcript. Ruling 10's sequence otherwise stands: the first pull request (runner, schema, sentence with the hash), then the three runs from the commit after it, then the second pull request with the records.
The Maintainer's decision (2026-11-03): hash only.evals/context/task.json holds the task sentence's sha256, never the sentence, so the held-out rule stays unnamed in this repo and the task can be rerun on a new model. The pending test and schemas/context-eval.schema.json change to match and drop the check of a backticked reason code. The runner's stale line about P-26 having no fixture is fixed in the same change.
The three runs wait on #511 (Context test: a run passes only with no outside reads or unreadable commands, and the judge runs the export's mutation runner): two gaps in the judge, both ruled real by the Factory Builder, to be fixed before any run is made.
Activity
don-petry commentedon Oct 19, 2026
Sequence (plan review, 2026-10-19)
This runs after #28 merges, since no ruled story changes the root
AGENTS.mdafter that. Two things are settled for it:evals/is created by the new story "The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests", which starts now and re-points the one wait onevals/in decision 0010 and the row in decision 0013. This story then adds its task sentence and hash there, and does not create the directory itself.Its three runs spend model capacity; run them when no builder needs the machine.
don-petry commentedon Oct 20, 2026
Working rulings (overturnable in review)
Solution Architect, delegated. On main at bc2eaed (#232). Read from the issue and its sequencing comment, the epic #3 ("Done when", "Order", "Rules for every story"), the plan review of 2026-10-19 (§2 Exit, the row on #8, the row on rows waiting on reserved directories), design §9.1 ("Slice 0 is done when"), §9.12 (the fresh reader, the second model family), §9.14 (last paragraph), §3.13 (where a held-out thing lives and why), §4.9 (the eval job's credential),
docs/context-engineering.md(its last adopted row and its sources 5 and 16), decision 0013 (its last row, "not held yet"), decision 0010 (its evals row), decision 0033 (the Maintainer's own subscription, the pinned CLI),tools/check_context.py(PENDING,_is_pending, the waits onevals/),tests/support.py::pending,tests/test_evals.pyas #221 left it (the two@pendingtests,TheSet,LIVE),evals/AGENTS.md,tools/trial_p13_cli.sh(how a vendor's CLI is run headless at a pin with nothing of the shell reaching it, and nothing of the model's text reaching the fixture), the rootAGENTS.mdandkernel/AGENTS.mdas they stand, and the open issues #178, #190, #226 and #237.What this story decides. Slice 0's exit test and the only measure of the context: whether an agent given nothing but this repo and one sentence can add a rule to the Kernel the way
AGENTS.mdandkernel/AGENTS.mdsay, and pass the required check, three times out of three. The story decides what the task is and where its answer lives, how a fresh agent is given it, what the agent must leave behind, how a test inmake verifyjudges that without a model in CI, which marks come off and which rows close, and what merges before it. It is the last story of slice 0: after it the exit is declared and the slice-1 epic opens.1. The task is one Kernel rule the repo does not hold and no
@pendingtest states. As first written the test could not fail, because every rule the Kernel lacks is stated by a@pendingtest the agent can read, andkernel/AGENTS.mdtells it to take the mark off. So the task's rule is one with no test in the repo: its boundary test is the known answer, written in the private held-out home by a separate private session (as #183's cases are: no reviewer tools, no notes, nothing of it in a pull request, an issue, a session note or a shared transcript), and only its sha256 is recorded here. The rule is chosen by four criteria, which the ruling states so that the private session can choose another if the first is taken: (a) the design states it, so the agent can find the rule's words without being told how to build it; (b) it is one reason code, in one Kernel module, behind one boundary; (c) it needs no new fact and no new limit, so the task measures the context and not the schema work; (d) it is decided, so that the agent's change could stand if a story later builds it. The first choice is #237's rule, "Kernel: a merge read while an issue is held or declined is a Hold, not done (#227)": it is the Maintainer's ruling of 2026-10-20, it is one code inkernel/issue.py, it reads facts the schema already carries, and it has no test. Why: a task with a known answer, run more than once (docs/context-engineering.md, source 16); a task whose answer is in the repo measures nothing (the issue's "Why").2. The task sentence. One sentence under
evals/context/task.json, the rule in the design's words with its reason code, and nothing of how: for the first choice, "A merge of a linked pull request read while the issue isheldordeclinedis a Hold with the reasonmerged_under_hold, neverdone." The sentence names the code because the code is the rule's name (every reason is a code, never a sentence) and the held-out test asserts it; it names no file, no test, no mutant and no document. Why: the agent is told what, as a story is, and finds how from the context; that is what is tested.3. How a fresh agent is given it, and what it must leave. A runner,
tools/context_eval.sh, modelled ontools/trial_p13_cli.shand held to #178's list from the start: the vendor's CLI at the pin decision 0033 names, on a configuration directory of its own logged in to the Maintainer's own subscription (decision 0033), started with an empty environment but for the few names the script allows, so no key or token of the shell reaches it. The run:git archiveof one commit ofmaininto an empty directory outside every repo, so the agent has the files and no history, no remote, no notes and no other checkout; the CLI runs headless (-p) with the sentence as its whole prompt, its file and shell tools allowed inside that directory only, its network tools disallowed, and a cap on turns and on cost the fixture records. No system prompt is added, no skill is named, no hint is given: the sentence and the repo are all. The agent is expected to do what the rootAGENTS.mdsays: readkernel/AGENTS.md, write the boundary case, change the Kernel, add the mutant, run both commands. What it leaves is the directory's diff. Why: "given only the repo and that sentence" (the issue); a fresh context that returns a short summary (§9.12); and decision 0033 for where the model credential lives.4. How a run is judged, outside CI, with the model. After the agent stops, the runner, not the agent, in the same directory: (a) copies the held-out test in from the private home, read by the Maintainer at his own terminal (the read token is the eval job's, §4.9, and arrives with slice 1; until then the file is copied by hand), and runs it with
python3 -m unittest: it must pass; (b) runsmake verifyon the directory: it must pass, the held-out test among the tests; (c) runs the mutation check asmake verifydoes, and reads from its report that at least one mutant the agent added was killed, and none survived; (d) removes the held-out test before anything is recorded. A run passes when all three hold. Three runs, from the same commit and the same sentence, each a fresh directory and a fresh CLI session; the story passes at three of three. Why: the issue's three boxes; "the agent's own new mutant is killed" is the one thing that shows the agent followed the three-part rule ofkernel/AGENTS.mdand not only made a test pass.5. What a run records, and how a test in CI judges it without a model. Each run leaves one record in
evals/context/task.json, schemacontext-eval/1: the commit, the date, the CLI's version and the cap; what the agent read, as the paths of its file reads in order and the commands it ran, taken from the CLI's transcript by name and nothing else; what it changed, asgit diff --statof the directory against the export, with the names of the test, the mutant and the reason code it added; what it cost, as the CLI's own usage fields (num_turns,duration_ms,total_cost_usd,usage,modelUsage), kept by allowlist as #178 asks; and the three verdicts with the mutation report's line for the agent's mutant. No model text reaches the record: not the agent's summary, not a tool's input, not a diff. The testtests/test_evals.py::test_a_fresh_agent_given_the_repo_adds_a_rule_and_passes_the_checkcomes off@pendingand holds the record: the file validates; the task is one sentence; its held-out test's sha256 is 64 hex; there are at least three runs on the task's hash and the newest three all pass; each run's reads name only paths that exist in the repo, each names a test, a mutant and a reason code, and each cost is a number; and no run holds a string that looks like code, a diff or prose longer than a path.tests/test_tools.pyholds the assembler: a stand-in CLI transcript with model text in it yields a record with none; a run that failed any verdict is recorded as failed; a transcript the assembler cannot read records nothing. Why: CI runs no model and fetches nothing (tools/check_repo.py); what CI can hold is the fixture of the runs, as a trial's fixture is held, and that the fixture is honest about what it holds.6. A failed run. The issue's rule: a failed run changes the context, never the task, and the three runs are repeated. The runner records the failed run too, so the record shows what was tried; the context change is a pull request of its own, with the record's failed run named in its body as the reason; then three fresh runs. The test counts the newest three on the task's hash, so an earlier failure stays in the record and does not fail the check. What "the context" means here: the root
AGENTS.md,kernel/AGENTS.md,tests/AGENTS.md,docs/standards/,docs/lessons.md, the glossary, a skill. Not the design, not the Kernel. Why: the measure is of the context; a change to the task after a failure would be the POC's stub again.7. Which marks come off, and which rows close. Off: decision 0013's last row ("not held yet"),
docs/context-engineering.md's last adopted row, the two@pendingmarks intests/test_evals.pythat this story owns (onlytest_a_fresh_agent_...;test_the_review_intents_eval_set_has_its_casesis #183's second pull request and stays). Closed: the epic's box "An agent given only the repo's own context can add a rule to the Kernel and pass the required check without being told how"; design §9.1's "Slice 0 is done when" gains the date and the record's commit in the words §6's rows use; §9.14's last paragraph points atevals/context/. Not closed by this story: the other seven boxes, each its own story's.8. Record. One new decision, numbered at enqueue (0042 as of this ruling): "The context is tested by a task run outside CI, and the runs are held as a fixture." It decides: the task is a rule the repo does not hold, with its answer held out; the runner and the judge are the Maintainer's terminal on his own subscription, never CI; a run record holds reads, changes and cost and no text of the model's; three of three is the bar; a failed run changes the context and never the task; and a change to the context after the record is re-run at the Maintainer's call, not by CI (CI cannot run a model, and cannot know which context change matters). Rows: the test above;
tests/test_tools.py::test_a_context_run_record_holds_no_model_text;tests/test_evals.py::test_a_case_dropped_under_evals_in_any_file_is_caughtextended toevals/context/; a row on 0033 ("the context's runs use the same login and pin").9. Files. New:
evals/context/task.json(inLIVE, schemacontext-eval/1),schemas/context-eval.schema.json,examples/context-eval.jsonwith counter-examples underexamples/invalid/(a task of two sentences; a hash that is not 64 hex; a run whose reads hold a diff; a cost that is a string),tools/context_eval.shandtools/context_eval.py(the assembler, pure: transcript and diff in, record out),docs/standards/gains no page (the runner's runbook is a comment at the top of the script, as #178 asks of P-13's). Edited:tests/test_evals.py,tests/test_tools.py,tools/check_examples.py(LIVE),evals/AGENTS.md(rows), decision 0013, 0010 unchanged, 0033 (row),docs/context-engineering.md, the design §9.1 and §9.14, the new record. Not edited: the rootAGENTS.md,kernel/AGENTS.md,kernel/(the agents' changes are never merged; the rule is built by #237 afterwards, with its own public test).10. What must merge before it, and what must not. Prerequisites: none of #178, #190, #226 and #237. #178 ("Trial scripts: P-13's script hardened") is not a prerequisite: the runner is a new script written to #178's list, and #178 may share code with it only after both are on main. #190 and #226 touch
sim/grounding.pyandtools/check_trials.py, which this story does not. #237 must not merge before the three runs are recorded, if its rule is the task: the task's rule must be absent from the Kernel at the commit the runs export. The sequence is: this story's first pull request (the schema, the runner, the assembler, the test still@pending, the task sentence with the hash) merges; the three runs are made from the commit after it; a second small pull request adds the records and takes the mark off; then #237 is built as a story. If the Maintainer would rather build #237 first, the private session takes the next rule by the four criteria of ruling 1 and the sequence is the same. Also before it: every ruled story that changes the rootAGENTS.md; none does (#41 and #43 changedsim/AGENTS.mdand the design, not the root).11. Gates. Boundary tests first (the assembler's and the record's, on a synthetic transcript); export verify; a fresh Claude review and a security pass (the runner: nothing of the shell reaches the CLI, nothing of the model reaches the record, the export directory is outside every repo, the held-out test is copied in and out by the runner and never committed); Muse; AGY not needed (no
kernel/orsim/change) unless the Maintainer wants a second family's read of the runner, in which case Flash. The pull request says "Part of #8" for the first and "Closes #8" for the second, and the second names the three runs' commit and cost.12. Out of scope. The eval job's workflow and its read token (slice 1,
eval.yml's reserved name); #183's cases and its second pull request; comparing runs with the context against runs without it (the measurement of source 5, worth a story once the runner exists); more than three runs; a rerun on every context change (the record's commit says what was tested; the Maintainer decides when to run again); the agent's changes themselves, which are discarded; any change to the task after a failure.Needs the Maintainer? Three things, none of them a block on the first pull request: the held-out test, written in the private session from the rule of ruling 1, and its sha256 given to the builder; the three runs at his own terminal on his own subscription, when no builder needs the machine; and the choice of ruling 10's sequence for #237 (hold it until the runs, or give the task another rule).
don-petry commentedon Oct 20, 2026
The Maintainer's choice (2026-10-20, via the orchestration session): build and merge #237 first; #8's held-out task takes another rule, chosen by ruling 1's four criteria from the rules the repo does not hold, by the private session, and named nowhere in this repo or in any shared transcript. Ruling 10's sequence otherwise stands: the first pull request (runner, schema, sentence with the hash), then the three runs from the commit after it, then the second pull request with the records.
don-petry commentedon Nov 3, 2026
The Maintainer's decision (2026-11-03): hash only.
evals/context/task.jsonholds the task sentence's sha256, never the sentence, so the held-out rule stays unnamed in this repo and the task can be rerun on a new model. The pending test andschemas/context-eval.schema.jsonchange to match and drop the check of a backticked reason code. The runner's stale line about P-26 having no fixture is fixed in the same change.don-petry commentedon Nov 3, 2026
The three runs wait on #511 (Context test: a run passes only with no outside reads or unreadable commands, and the judge runs the export's mutation runner): two gaps in the judge, both ruled real by the Factory Builder, to be fixed before any run is made.
The context's test: a runner, a judge, a record with no word of the m…
P-13 at 2.1.306 (pass): the CLI's pin moves from the pruned 2.1.302, …
The context's task is held by its sentence's sha256, and a run record…