Skip to content

Protect each eval set's manifest in the hooks, so a floor or mark edit asks first #696

Closed

Description

@don-petry

A mistake guard, for whoever is free. The Factory Builder asked for it when ruling on the security read of #692 (a gate set holds a floor per criterion). It was already on the Maintainer's optional list.

What

Add each eval set's manifest, evals/*/cases.json, to protected in config/hooks.json, with an is such as "an eval set's manifest: its gate, bars, floors and natural marks". A session's edit to a manifest then asks first, as an edit to a rubric does.

Why

A gate set's floors (criterion_bars), bars, least of a kind and natural marks all live in its manifest under evals/, which no person need approve. A change to them between commits can lower a floor or, with the cases unchanged, publish groups whose difference is under ORIGIN_LEAST (#695: "Differences between a gate set's manifests across commits can isolate a group under ORIGIN_LEAST"). The hook only guards against a session's mistake; CI and review still hold the rule.

Notes

  • config/hooks.json is itself protected: the edit is pre-approved by the Maintainer 2026-11-08, ruled by the Factory Builder. Say so in the PR body.
  • Check that the pattern form matches what tools/hooks/protected.py reads (config/rubric-*.json is a precedent), and that the hooks' own tests cover the new path.
  • No case file is named or read; the manifest path alone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions