Repository navigation
Hooks: a set's manifest under evals/ asks before it is edited, and a recursive copy or a move asks about what it will hold (#696) - #698
Conversation
…oes (#696) evals/*/cases.json joins config/hooks.json's protected paths: a set's gate, bars, floors and natural marks live there, and no person need approve a change to them. A session's edit to one now asks first. Only the manifest is held: a set's other files, and a cases.json elsewhere under evals/, are not. The hooks' tests name the path, with a mutant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll hold (#696, after the review) cp -r evals/pr-review evals/qa-lead, or mv /tmp/set evals/qa-lead, made a new set's manifest with no ask: _cp read a target that is no directory yet as a file. A recursive copy (-r, -R, -a, --recursive, --archive) or a move now reads it as a directory that may come to hold a protected path, as mkdir -p and git mv already did. A plain copy to that path still makes a file. Tests and three mutants. Pre-approved by the Maintainer 2026-11-08, ruled by the Factory Builder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
… pattern to one set dies (#696, after Muse) Muse: the protected-path test drew its example from the pattern itself and named only pr-review's manifest, so narrowing evals/*/cases.json to evals/pr-review/cases.json survived. evals/gate/cases.json is named too, with that narrowing as a mutant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|



Closes #696, "Protect each eval set's manifest in the hooks, so a floor or mark edit asks first". A mistake guard: hooks are helpers (decision 0038's triage rule), and CI and review still hold the rule.
The protected edits, to
config/hooks.jsonandtools/hooks/protected.py, are pre-approved by the Maintainer 2026-11-08, ruled by the Factory Builder. No record.What changes
config/hooks.jsonevals/*/cases.jsonjoinsprotected, "an eval set's manifest: its gate, bars, floors and natural marks". A session's edit to one asks first, as an edit to a rubric does. A set's other files, and acases.jsonelsewhere underevals/, are not heldtools/hooks/protected.py,_cp-r,-R,-a,--recursive,--archive) or a move to a path that is no directory yet reads it as a directory that may come to hold a protected path. Socp -r evals/pr-review evals/qa-leadandmv /tmp/set evals/qa-leadask, asmkdir -pandgit mvalready did. A plain copy to that path still makes a fileNo case file is named or read: the manifest's path alone.
Checks
tests/test_hooks.py::test_every_protected_path_asks_for_every_edit_tool_in_every_treeand::test_an_edit_elsewhere_is_not_asked_aboutname the path, a second set's manifest, and their neighbours. The new::test_a_copy_or_move_that_makes_a_directory_asks_about_what_it_will_holdholds the copy and the move.config/hooks.jsonand of_cpare killed.make checksandcheck_decisionsare clean, and all 217 hook tests pass. CI runs the full verify.Reviews
evals/gate/cases.jsonis named in the test, with that narrowing as a mutant, in c4c1297cases.jsonnow reads as text, asrubric-*.jsondoes, so a command naming/tmp/cases.jsonmay askcases.jsonfiles are the manifests🤖 Generated with Claude Code