Skip to content

The eval scratch is held part by part, the heads folder is made fresh, and the canary's head is held apart #626

Closed

Description

@don-petry

Why

PR #624 (#623) unpacks each case's head apart from the eval scratch. Reads after its last fix round found four smaller gaps beside it:

  1. scratch() reads containment by separator. harness/eval/__init__.py's scratch still checks with startswith(... + os.sep). A heldout linked to the root would be taken as the scratch, though the root holds the workspace. heads_place was moved to os.path.commonpath for this reason. This predates Each case's head is unpacked apart from the set's cases (#623) #624.
  2. The heads folder is not made fresh. head_of makes heads with exist_ok=True, while plan makes heldout without it. A heads left by an earlier job on a reused runner would be accepted, with its own modes. GitHub's hosted runners are discarded after each job, so this cannot happen today.
  3. Nothing holds the canary's head apart from its case. harness/eval/canary.py keeps them in separate temporary directories, but no test or mutant fails if they share one. The canary's case is the public example, so the harm is small.
  4. Two of Each case's head is unpacked outside the folder that holds the set's case files #623's mutants are killed by a KeyError. In test_each_head_is_unpacked_apart_from_the_cases, "the fetch unpacks each head in the scratch" and "the run step looks for each head in the scratch" fail because the CLI is never asked, not on an assertion. An explicit assertIn("head", seen) would say why.

The change

  • scratch reads containment part by part, as heads_place does. A test links heldout to the root, and a mutant restores the separator form.
  • plan makes heads new, mode 0o700, beside heldout. A leftover one stops the step, as a leftover scratch does.
  • A test that the canary's head and case share no folder below its temporary directories, with a mutant.
  • The assertIn in Each case's head is unpacked outside the folder that holds the set's case files #623's test.

Done when

  • Each of the four, with its mutant killed where it adds one.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions