Skip to content

Simulator: uncovered reviews on every seed, with the first 40 seeds still reaching every path #168

Closed

Description

@don-petry

Part of the epic "Epic: slice 0, foundations. The repo holds the line, its context is tested, and the platform facts are recorded".

Why. Found while building #158, "Kernel: spent attempts hold the item even while a refused Run is queued". Ruling 6 asked for uncovered reviews on every seed (UNCOVERED_SEEDS = 1 in sim/world.py). With #158's fix every invariant holds across all 300 seeds at that spread. But the first 40 seeds, which SIM_SEEDS=40 in a local run and tools/mutants.py (on every mutant, in CI's shards) use, no longer reach two paths that other tests ask for:

  • tests/test_simulator.py::test_invariants_hold_under_faults: no approval is stopped under a Pause (stopped holds admit only).
  • tests/test_simulator.py::test_the_seeds_serve_items_with_an_acceptance_contract_and_without: no pass reads acceptance_evidence_missing.

With a review failing half the time on every seed, fewer worlds reach an approval or a contract's evidence within 250 steps. So #158 widened the spread from one seed in six to one in three, which takes in seed 297, and not to every seed.

Done when

  • Uncovered reviews run on every seed under faults, or the spread is ruled to stay narrower, with the reason in sim/world.py's comment.
  • The first 40 seeds still reach every path the simulator's coverage tests ask for, whether by a cheaper chance per Run, by the worlds that serve those paths, or by the tests' own seeds.
  • make verify passes at SIM_SEEDS=40 and at all 300 seeds.

Activity

  1. don-petry commented on Oct 18, 2026

    @don-petry
    ContributorAuthor

    Working rulings (overturnable in review)

    Solution Architect, delegated. On main at c094a8b (#170).

    The cause. The two paths come at the end of an item's life: an approval falls due while a Pause stands, and a contract's evidence is still missing once the reviews pass. In the first 40 seeds each was reached by one seed, 13 and 7, by luck of timing. A review that fails half the time on every seed delays the end of each world, so the luck runs out. Measured on a copy of c094a8b:

    • One seed in three (today): approval stopped on seed 13 only; evidence missing on seed 7 only; across 300, on 4 and 19 seeds.
    • Every seed at 0.5: neither in the first 40.
    • Every seed at 0.3: neither. At 0.2 or 0.1: seeds 13 and 12 come back, but no seed of the first 40 holds an uncovered review at its attempts, the third path test_the_seeds_serve_a_review_that_left_a_criterion_uncovered asks for.

    So a cheaper chance trades one path for another. The worlds must serve the two paths by design.

    1. Uncovered reviews on every seed. UNCOVERED_SEEDS = 1. UNCOVERED stays 0.5. Why: #158's ruling 6. The chance per Run is what found seed 297, and a cheaper one loses the attempts path.

    2. The Pause falls due with an approval. On a seed that pauses (seed % 5 in PAUSES), a pass before the seed's minute whose Decision, asked unpaused on the facts it read, is act / issue_approval starts the Pause there: pause_from becomes that minute. The span and the reading stay as the seed gives them. Ask it in poll, before the pass's own decide. Why: I-7 matters most when an approval is due, and that is the moment the coverage test asks to see. Measured with rulings 1 to 4: seeds 10, 13, 20 and 30 in the first 40; 11 of 300.

    3. The reporter can come after the review. On a seed with a contract, while faults are on, the reporter gives no result for a criterion that lacks one on this content until every required model Lens has reviewed that content and succeeded (self.reviewed(self.runs, self.cid())). Put the test in gives_evidence, in its last branch only. An expiry, another App's result and the second result are unchanged. Mark it # GitHub: a_result_can_come_after_the_review and list it in sim/grounding.py with no trial and no fixture. Why: evidence is missing only when it is late, and with every review retried it is rarely late. A reporter's check can be queued behind a model's review on GitHub. Measured with rulings 1 to 4: seeds 7, 26 and 38 in the first 40; 24 of 300.

    4. settle confirms up to five times, not three. With ruling 3, seed 236 rests in verifying / acceptance_evidence_missing after settle. It is an agent's pull request with a criterion of kind human. Each confirmation is a writer's action, which renews the Run ceiling, and with a Lens opening threads on half its reviews the content changed after each of three. Five settle it. Why: this is the person's patience once faults stop, not a coverage test. test_work_is_never_lost_once_faults_stop still asks the same thing. If another shape of ruling 3 keeps seed 236 resting with three, keep three.

    5. The coverage tests are not touched. SEEDS stays range(1, 1 + SIM_SEEDS), and no test pins a seed. Why: a pinned seed moves with any change to a world.

    6. Boundary tests, written first, in tests/test_simulator.py, each from a hand-built world:

    • "the Pause falls due with an approval": a pause seed's world, reviewed, every check passing, pause_from ten minutes ahead. One poll: pause_from is now, the Decision is converged / paused / defer, and stopped is ["issue_approval"]. Other side: a seed with no Pause keeps pause_from None, and a pass that would admit a Run moves no Pause.
    • "the reporter can come after the review": a contract world under faults, with no review of this content. gives_evidence gives no result for a lacking criterion. After a successful review of the content, it gives one. With faults off, it gives one at once.

    No mutant: neither is a Kernel rule.

    7. Measured on a copy, with rulings 1 to 4. All 27 simulator tests pass at SIM_SEEDS=40 and at 300. Again with #169's order in brakes.py and its paused() check, only the hand-built test #169 flips fails, as #169 expects. The first 40 seeds reach: an approval stopped on 10, 13, 20 and 30; evidence missing on 7, 26 and 38; an uncovered review held at its attempts on 37. Run python3 -m tools.mutants too: it runs the simulator on the first 40, and a changed world may change which mutants it kills.

    8. The text. sim/world.py's comment on UNCOVERED says every seed and why, and drops the two paths it lost. sim/AGENTS.md says "On every seed a review's Result now and then leaves a criterion...", that a Pause on its seeds may begin as an approval falls due, and that under faults the reporter may give its result only after the review.

    9. Record. None. Decision 0029's simulator row does not name the spread, and holds as written.

    10. Out of scope. The attempts path rests on one seed of the first 40, as it does today (24 then, 37 now). File it if wanted. #169 and this story both edit sim/world.py; whichever merges second rebases and runs all 300 seeds again.

    Needs the Maintainer? None.

  2. don-petry commented on Oct 19, 2026

    @don-petry
    ContributorAuthor

    Ruling 3, replaced in the build (PR #179). The ruling had the reporter's result come after the review under faults. The review of the first build found a problem: the reporter is verify, which concludes before any model review is admitted (§4.8, I-27). So the rule contradicted the design, and it cost the simulator 7 acceptance mutants at 40 seeds.

    The PR keeps the aim, acceptance_evidence_missing reached in the first 40 seeds, through a rule GitHub already shows. On a contract seed under faults, as the approval falls due, the oldest result on that content expires with its artifact (a_result_expires_with_its_artifact). The PR body has the measurements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions