Skip to content

The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests #183

Closed

Description

@don-petry

Part of the epic "Epic: slice 1, first approval. A Release on next reviews a pull request to this repo, and the machine user's approval stands".

Why. A gate-deciding Intent may not serve without a held-out, scored eval set (FR-55), and the design puts the review Intent's set on slice 1's critical path (§9.1, item 5): "its cheapest source is the POC's own history: merged pull requests later reverted or fixed are known answers". No story existed. It reads another repo's history only, so it runs beside every lane, now. #8 also needs evals/, which this story creates.

Done when

  • At least 20 cases with known answers live in the private held-out home: each a facts snapshot (facts/1) and the diff of a proof-of-concept pull request that was later reverted or fixed, with the Finding a reviewer should have raised (category, severity, location, criterion of config/rubric-pr-review.json), and as many cases of a change that was right. Nothing from the home is written in this repo: only each case's hash and the count, under evals/pr-review/.
  • tools/eval_result.py scores one result/1 file against one case (raised the Finding, or covered the criterion as met), with a test that scores a right and a wrong Result; it imports nothing outside STDLIB.
  • Creating evals/ re-points the row in decision 0010 ("an Intent family is cut over only after its eval set exists") and decision 0013's row and docs/context-engineering.md's last row to this story's test or to The context's test: a fresh agent adds a rule to the Kernel without being told how #8, and tools/check_context.py passes.
  • The eval job that will read the home is named in .github/workflows/ only as a reserved name; the job itself, with the read token, is the Harness story's.

Needs. The Maintainer: a read token for the home, later, for the job; none for building the cases, which a session reads from the public proof-of-concept repo's history. Never put a case's content in a PR, an issue or a transcript (the held-out rule).

Design. PRD FR-55; design §9.1 item 5, §3.13, §4.9 credentials table; decisions 0010, 0013.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions