Skip to content

An eval case's expected Finding has its criterion's severity and category, as a Result's must #421

Closed

Description

@don-petry

Part of the epic "Epic: slice 1, first approval. A Release on next reviews a pull request to this repo, and the machine user's approval stands" (#182).

Raised by the Claude review of the pull request for "A Finding that cites a rubric criterion: is its category the criterion's, as its severity is? A ruling on §3.14" (#407).

Why. Records 0088 and 0095 make kernel/coverage.py refuse a Result whose Finding cites a criterion at another severity or category than the rubric's. An eval case's expected Finding is held to neither:

  • tools/eval_result.py scores a Result against the case's expected.finding and never compares that Finding's severity or category with the criterion it cites.
  • No check does it either. examples/eval-case.json with expected.finding.category set to delivery while citing PR-3 (functional) passes python3 -m tools.check_examples.

So a case can reward a Result that apply refuses, which breaks "what is measured is what runs" (§3.14).

What is wanted

  • A case whose expected Finding cites a criterion is refused unless its severity and category are the rubric's. tools/check_examples.py and the eval set's own check should both refuse it, and the eval job reads such a case as not_this_set or schema, never as a pass.
  • The held-out cases are checked by their owner, outside this repo, without naming any case's content here (the held-out rule).
  • A test and a mutant hold it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions