You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A fresh held-out gate set for the Release after v0.1.0, never replayed #562
The held-out cases the Release gate scores have been run, read and tuned against while the eval was built. They now work as a development set: a score on them no longer says how the Reviewer does on a change it has not seen.
factory/v0.1.0 cuts on them, with a written caveat in the cut's notes or its record that says so.
After v0.1.0, the Release gate needs a fresh held-out set:
written apart from the Factory's sessions and kept where those sessions cannot read it, as the current set is;
scored once per Release candidate, at the gate, and never replayed to tune a prompt, a rubric, a judge or a scorer;
with decision 0120's bar carried over to it, or a new record if the bar changes.
Done when the gate for the Release after v0.1.0 reads the fresh set, and the current set is named a development set wherever the gate describes what it reads.
The fresh gate set this issue asks for is being built in the private home, apart from the Factory's sessions, as
gate-v1. Nothing about its cases will be posted here. What the Factory needs to build before it can run:
How it runs. k = 3 dispatches on one commit, pooled, every one read; a faulted dispatch closes the gate. The
public report is aggregates only; per-case outcomes go to a private ledger.
The bars. Proposed, and fixed in a decision record before the first gate run: a pooled pass rate of about 90%;
a floor on clean cases, for false positives; a floor on recall of naturally occurring defects; and a floor per
graded criterion. No bar on a single case. Together they are expected to pass a reviewer truly at 95% about 89–98%
of the time and an 85% one about 2–11%, so the gate tells about 85% from 95%, not 88% from 92%.
Its life. At most three cuts. A promotion to stable needs a pass on a set that has not failed a cut.
The preconditions are sub-issues here: 0b's surplus rule (#610), report identity and usage (#616), heads from the home
(#611), a named set and aggregate reports (#612), pooled dispatches (#613), the expected test (#614) and the tier's limits
(#615). Its access boundary (0k) is not built: by the Maintainer's choice, the set is kept apart by convention.
So "Done when" reads, for this set: the gate for the Release after v0.1.0 runs the fresh set under its own decision
record, with #610–#616 merged, rather than with decision 0120's bar carried over.
Activity
expected, and that its head holds no.git#614don-petry commentedon Nov 7, 2026
The fresh gate set this issue asks for is being built in the private home, apart from the Factory's sessions, as
gate-v1. Nothing about its cases will be posted here. What the Factory needs to build before it can run:
make verifyholds, or could and shouldhold, is not graded; that is why Only the write steps call a write: a check that keeps GitHub writes to an allowlist of callers #587–A boundary's check names a test, never a file, and a new boundary names the mutant its test kills #590 were filed.
public report is aggregates only; per-case outcomes go to a private ledger.
a floor on clean cases, for false positives; a floor on recall of naturally occurring defects; and a floor per
graded criterion. No bar on a single case. Together they are expected to pass a reviewer truly at 95% about 89–98%
of the time and an 85% one about 2–11%, so the gate tells about 85% from 95%, not 88% from 92%.
stableneeds a pass on a set that has not failed a cut.The preconditions are sub-issues here: 0b's surplus rule (#610), report identity and usage (#616), heads from the home
(#611), a named set and aggregate reports (#612), pooled dispatches (#613), the
expectedtest (#614) and the tier's limits(#615). Its access boundary (0k) is not built: by the Maintainer's choice, the set is kept apart by convention.
So "Done when" reads, for this set: the gate for the Release after v0.1.0 runs the fresh set under its own decision
record, with #610–#616 merged, rather than with decision 0120's bar carried over.