Skip to content

The eval scratch is read part by part, the plan makes the heads folder new, and the canary's head is held apart (#626) - #632

Merged
don-petry merged 6 commits into
mainfrom
evals/scratch-part-by-part
Nov 8, 2026
Merged

don-petry merged 6 commits into
mainfrom
evals/scratch-part-by-part

Conversation

@don-petry

@don-petry don-petry commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Closes #626.

What changes

  • scratch checks containment part by part. scratch in harness/eval/__init__.py refuses a scratch that is the workspace, is inside it, or holds it, using os.path.commonpath on their real paths, as heads_place already does. It no longer compares prefixes with a separator. A heldout linked to the root was accepted before, because the root's own path ends in its separator.
  • The plan makes the heads folder new. plan creates heads beside the scratch with mode 0o700, as it creates the scratch. A heads folder left by an earlier job stops the plan with scratch_unreadable, as a leftover scratch does. Hosted runners are discarded after each job, so this guards only a reused runner.
  • A test holds the canary's head apart from its case. test_the_canarys_head_is_apart_from_its_case asserts that the head the CLI starts in and the case's folder share no folder below the system's temporary directory.
  • Each case's head is unpacked outside the folder that holds the set's case files #623's test says why it fails. test_each_head_is_unpacked_apart_from_the_cases asserts that the CLI was asked, in a head, before reading where it was. Two of Each case's head is unpacked outside the folder that holds the set's case files #623's mutants now fail on that assertion instead of a KeyError.

Mutants

Mutant State
the scratch is held to the workspace by its separator (new) killed
the plan makes no heads folder (new) killed
a heads folder an earlier job left is taken (new) killed
the heads folder is made open to all (new) killed
the canary unpacks its head beside its case (new) killed
the scratch may be in the workspace; a workspace inside the scratch is allowed; a workspace that is the scratch is allowed (repointed to the new line) killed
the fetch unpacks each head in the scratch; the run step looks for each head in the scratch (#623's) killed, now by an assertion

Every mutant in tools/mutants.py still matches its file exactly once and parses.

Record NNNN, number to be given. No change to eval.yml. No protected path is touched.

Verify

Pending.

Reviews

Read State
Claude + security pending
Argon pending
Muse not run (quota); owed after 2026-11-09
Astra not run (quota)
Codex automatic on push

🤖 Generated with Claude Code

View guided diff Turn on auto-fix

…r new, and the canary's head is held apart (#626)

scratch refuses a scratch that is the workspace, inside it or holding it by
os.path.commonpath, as heads_place does, so a heldout linked to the root is
refused. The plan makes heads beside the scratch, mode 0o700, and one an
earlier job left stops it scratch_unreadable. A test holds the canary's head
and case apart, and #623's test says it was asked before reading the head.
Record NNNN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner November 8, 2026 07:14
@coderabbitai

coderabbitai Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-08T08:29:04.526881Z f0e23a2 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

don-petry and others added 5 commits November 8, 2026 01:19
…s scratch itself; the canary test says why it fails

A heads folder that stops the plan now leaves no scratch behind. The plan's
test calls scratch directly for each refused place, since the plan's
heads_place would refuse those places with the same code. The canary test
asserts the case was assessed and the CLI asked in a head before reading
either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n's comment says the order runs one way

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…into evals/scratch-part-by-part

# Conflicts:
#	harness/eval/__init__.py
#	tools/mutants.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 8, 2026

Copy link
Copy Markdown

@don-petry
don-petry added this pull request to the merge queue Nov 8, 2026
Merged via the queue into main with commit a27ed8f Nov 8, 2026
25 checks passed
@don-petry
don-petry deleted the evals/scratch-part-by-part branch November 8, 2026 09:22
don-petry added a commit that referenced this pull request Nov 8, 2026
The Factory Builder gave the record 0166: #628 took 0164 and #632 took 0165. Its filename, its title and the
mention in tools/AGENTS.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The eval scratch is held part by part, the heads folder is made fresh, and the canary's head is held apart

1 participant