Skip to content

Hooks: a set's manifest under evals/ asks before it is edited, and a recursive copy or a move asks about what it will hold (#696) - #698

Merged
don-petry merged 3 commits into
mainfrom
hooks/eval-manifests-696
Nov 10, 2026
Merged

don-petry merged 3 commits into
mainfrom
hooks/eval-manifests-696

Conversation

@don-petry

@don-petry don-petry commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Closes #696, "Protect each eval set's manifest in the hooks, so a floor or mark edit asks first". A mistake guard: hooks are helpers (decision 0038's triage rule), and CI and review still hold the rule.

The protected edits, to config/hooks.json and tools/hooks/protected.py, are pre-approved by the Maintainer 2026-11-08, ruled by the Factory Builder. No record.

What changes

What Now
config/hooks.json evals/*/cases.json joins protected, "an eval set's manifest: its gate, bars, floors and natural marks". A session's edit to one asks first, as an edit to a rubric does. A set's other files, and a cases.json elsewhere under evals/, are not held
tools/hooks/protected.py, _cp A recursive copy (-r, -R, -a, --recursive, --archive) or a move to a path that is no directory yet reads it as a directory that may come to hold a protected path. So cp -r evals/pr-review evals/qa-lead and mv /tmp/set evals/qa-lead ask, as mkdir -p and git mv already did. A plain copy to that path still makes a file

No case file is named or read: the manifest's path alone.

Checks

  • tests/test_hooks.py::test_every_protected_path_asks_for_every_edit_tool_in_every_tree and ::test_an_edit_elsewhere_is_not_asked_about name the path, a second set's manifest, and their neighbours. The new ::test_a_copy_or_move_that_makes_a_directory_asks_about_what_it_will_hold holds the copy and the move.
  • Five new mutants. At c4c1297, all 22 mutants of config/hooks.json and of _cp are killed. make checks and check_decisions are clean, and all 217 hook tests pass. CI runs the full verify.

Reviews

Reader Findings Disposition
Argon (gemini-4-argon-high) None —
Muse (muse-wonder-full-1.0) 1, marked major, not a regression: narrowing the pattern to pr-review's set survived the tests at the first head evals/gate/cases.json is named in the test, with that narrowing as a mutant, in c4c1297
Claude review and security pass Nothing major. 1 minor: a recursive copy or a move created a new set's manifest with no ask. 1 low: the bare name cases.json now reads as text, as rubric-*.json does, so a command naming /tmp/cases.json may ask The minor is fixed in b09b0da, on the Factory Builder's word. The low is left as it is: today the only tracked cases.json files are the manifests

🤖 Generated with Claude Code

don-petry and others added 2 commits November 9, 2026 17:44
…oes (#696)

evals/*/cases.json joins config/hooks.json's protected paths: a set's gate, bars, floors and natural marks
live there, and no person need approve a change to them. A session's edit to one now asks first. Only the
manifest is held: a set's other files, and a cases.json elsewhere under evals/, are not. The hooks' tests
name the path, with a mutant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll hold (#696, after the review)

cp -r evals/pr-review evals/qa-lead, or mv /tmp/set evals/qa-lead, made a new set's manifest with no ask: _cp
read a target that is no directory yet as a file. A recursive copy (-r, -R, -a, --recursive, --archive) or a
move now reads it as a directory that may come to hold a protected path, as mkdir -p and git mv already did. A
plain copy to that path still makes a file. Tests and three mutants. Pre-approved by the Maintainer 2026-11-08,
ruled by the Factory Builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner November 9, 2026 23:54
@don-petry don-petry added the factory:hands-off A person's mark: the Factory writes nothing on this pull request (kernel/terms.py LABEL_HANDS_OFF) label Nov 9, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-09T23:54:16.909182Z
We couldn’t complete this review. Comment @codex review to try again.
b09b0da PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

… pattern to one set dies (#696, after Muse)

Muse: the protected-path test drew its example from the pattern itself and named only pr-review's manifest, so
narrowing evals/*/cases.json to evals/pr-review/cases.json survived. evals/gate/cases.json is named too, with
that narrowing as a mutant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 9, 2026

Copy link
Copy Markdown

@don-petry
don-petry added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 49923ce Nov 10, 2026
26 checks passed
@don-petry
don-petry deleted the hooks/eval-manifests-696 branch November 10, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

factory:hands-off A person's mark: the Factory writes nothing on this pull request (kernel/terms.py LABEL_HANDS_OFF)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Protect each eval set's manifest in the hooks, so a floor or mark edit asks first

1 participant