You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests #183
Part of the epic "Epic: slice 1, first approval. A Release on next reviews a pull request to this repo, and the machine user's approval stands".
Why. A gate-deciding Intent may not serve without a held-out, scored eval set (FR-55), and the design puts the review Intent's set on slice 1's critical path (§9.1, item 5): "its cheapest source is the POC's own history: merged pull requests later reverted or fixed are known answers". No story existed. It reads another repo's history only, so it runs beside every lane, now. #8 also needs evals/, which this story creates.
Done when
At least 20 cases with known answers live in the private held-out home: each a facts snapshot (facts/1) and the diff of a proof-of-concept pull request that was later reverted or fixed, with the Finding a reviewer should have raised (category, severity, location, criterion of config/rubric-pr-review.json), and as many cases of a change that was right. Nothing from the home is written in this repo: only each case's hash and the count, under evals/pr-review/.
tools/eval_result.py scores one result/1 file against one case (raised the Finding, or covered the criterion as met), with a test that scores a right and a wrong Result; it imports nothing outside STDLIB.
The eval job that will read the home is named in .github/workflows/ only as a reserved name; the job itself, with the read token, is the Harness story's.
Needs. The Maintainer: a read token for the home, later, for the job; none for building the cases, which a session reads from the public proof-of-concept repo's history. Never put a case's content in a PR, an issue or a transcript (the held-out rule).
Activity
Evals: the scorer reads only under the directory it runs in (#183)
Evals: the path guard in the form the analyser reads (#183)
Evals: each mutant's text is its own, a job's name is reserved too, a…
Evals: the review Intent's held-out set has its 44 cases, and the cut…
The review Intent's held-out eval set: 40 cases, by id and sha256, as…