Repository navigation
The eval scratch is read part by part, the plan makes the heads folder new, and the canary's head is held apart (#626) - #632
Conversation
…r new, and the canary's head is held apart (#626) scratch refuses a scratch that is the workspace, inside it or holding it by os.path.commonpath, as heads_place does, so a heldout linked to the root is refused. The plan makes heads beside the scratch, mode 0o700, and one an earlier job left stops it scratch_unreadable. A test holds the canary's head and case apart, and #623's test says it was asked before reading the head. Record NNNN. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…s scratch itself; the canary test says why it fails A heads folder that stops the plan now leaves no scratch behind. The plan's test calls scratch directly for each refused place, since the plan's heads_place would refuse those places with the same code. The canary test asserts the case was assessed and the CLI asked in a head before reading either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n's comment says the order runs one way Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…into evals/scratch-part-by-part # Conflicts: # harness/eval/__init__.py # tools/mutants.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…into evals/scratch-part-by-part
|



Closes #626.
What changes
scratchchecks containment part by part.scratchinharness/eval/__init__.pyrefuses a scratch that is the workspace, is inside it, or holds it, usingos.path.commonpathon their real paths, asheads_placealready does. It no longer compares prefixes with a separator. Aheldoutlinked to the root was accepted before, because the root's own path ends in its separator.plancreatesheadsbeside the scratch with mode 0o700, as it creates the scratch. Aheadsfolder left by an earlier job stops the plan withscratch_unreadable, as a leftover scratch does. Hosted runners are discarded after each job, so this guards only a reused runner.test_the_canarys_head_is_apart_from_its_caseasserts that the head the CLI starts in and the case's folder share no folder below the system's temporary directory.test_each_head_is_unpacked_apart_from_the_casesasserts that the CLI was asked, in a head, before reading where it was. Two of Each case's head is unpacked outside the folder that holds the set's case files #623's mutants now fail on that assertion instead of aKeyError.Mutants
Every mutant in
tools/mutants.pystill matches its file exactly once and parses.Record NNNN, number to be given. No change to
eval.yml. No protected path is touched.Verify
Pending.
Reviews
🤖 Generated with Claude Code