Skip to content

A fresh held-out gate set for the Release after v0.1.0, never replayed #562

Open
8 of 11 issues completed
Open

Description

@don-petry

The held-out cases the Release gate scores have been run, read and tuned against while the eval was built. They now work as a development set: a score on them no longer says how the Reviewer does on a change it has not seen.

factory/v0.1.0 cuts on them, with a written caveat in the cut's notes or its record that says so.

After v0.1.0, the Release gate needs a fresh held-out set:

  • written apart from the Factory's sessions and kept where those sessions cannot read it, as the current set is;
  • scored once per Release candidate, at the gate, and never replayed to tune a prompt, a rubric, a judge or a scorer;
  • with decision 0120's bar carried over to it, or a new record if the bar changes.

Done when the gate for the Release after v0.1.0 reads the fresh set, and the current set is named a development set wherever the gate describes what it reads.

🤖 Addressed by Claude Code

Activity

  1. don-petry commented on Nov 7, 2026

    @don-petry
    ContributorAuthor

    The fresh gate set this issue asks for is being built in the private home, apart from the Factory's sessions, as
    gate-v1. Nothing about its cases will be posted here. What the Factory needs to build before it can run:

    • What it grades. Only what a deterministic check cannot hold. A defect make verify holds, or could and should
      hold, is not graded; that is why Only the write steps call a write: a check that keeps GitHub writes to an allowlist of callers #587–A boundary's check names a test, never a file, and a new boundary names the mutant its test kills #590 were filed.
    • How it runs. k = 3 dispatches on one commit, pooled, every one read; a faulted dispatch closes the gate. The
      public report is aggregates only; per-case outcomes go to a private ledger.
    • The bars. Proposed, and fixed in a decision record before the first gate run: a pooled pass rate of about 90%;
      a floor on clean cases, for false positives; a floor on recall of naturally occurring defects; and a floor per
      graded criterion. No bar on a single case. Together they are expected to pass a reviewer truly at 95% about 89–98%
      of the time and an 85% one about 2–11%, so the gate tells about 85% from 95%, not 88% from 92%.
    • Its life. At most three cuts. A promotion to stable needs a pass on a set that has not failed a cut.

    The preconditions are sub-issues here: 0b's surplus rule (#610), report identity and usage (#616), heads from the home
    (#611), a named set and aggregate reports (#612), pooled dispatches (#613), the expected test (#614) and the tier's limits
    (#615). Its access boundary (0k) is not built: by the Maintainer's choice, the set is kept apart by convention.

    So "Done when" reads, for this set: the gate for the Release after v0.1.0 runs the fresh set under its own decision
    record, with #610–#616 merged, rather than with decision 0120's bar carried over.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions