You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
An eval case's expected Finding has its criterion's severity and category, as a Result's must #421
Part of the epic "Epic: slice 1, first approval. A Release on next reviews a pull request to this repo, and the machine user's approval stands" (#182).
Raised by the Claude review of the pull request for "A Finding that cites a rubric criterion: is its category the criterion's, as its severity is? A ruling on §3.14" (#407).
Why. Records 0088 and 0095 make kernel/coverage.py refuse a Result whose Finding cites a criterion at another severity or category than the rubric's. An eval case's expected Finding is held to neither:
tools/eval_result.py scores a Result against the case's expected.finding and never compares that Finding's severity or category with the criterion it cites.
No check does it either. examples/eval-case.json with expected.finding.category set to delivery while citing PR-3 (functional) passes python3 -m tools.check_examples.
So a case can reward a Result that apply refuses, which breaks "what is measured is what runs" (§3.14).
What is wanted
A case whose expected Finding cites a criterion is refused unless its severity and category are the rubric's. tools/check_examples.py and the eval set's own check should both refuse it, and the eval job reads such a case as not_this_set or schema, never as a pass.
The held-out cases are checked by their owner, outside this repo, without naming any case's content here (the held-out rule).
Activity
Evals: a case's expected Finding is as its rubric gives it, read at r…