Repository navigation
Description
Part of epic #581 (self-improving skills, Discussion #572). Raised as finding #1 of the maintainer rubber-duck review on #581 — recommended as a new story because it is one of the trust guarantees the whole "self-improving" claim rests on.
Problem
The reward-hacking guard added in #582 (CODEOWNERS over evals/) stops the proposer from editing the held-out set — but not from overfitting to it. The Phase 3 proposer (#587) reads the whole repo, so it can see every case in evals/<skill>/cases.jsonl and tune the skill markdown to pass exactly those, with no real generalization ("teaching to the test"). SkillOpt's discipline depends on a held-out set the optimizer never sees; nothing in the current plan hides or rotates the cases.
Scope
Introduce a dev/test split (or a hidden/rotating holdout) for the eval sets now rooted at evals/<skill>/ (per maintainer decision A1), so the proposer only ever sees the dev subset and the gate scores against the unseen holdout.
Acceptance criteria
- Each skill's cases are partitioned into a proposer-visible dev split and a holdout split the proposer never sees — e.g.
evals/<skill>/dev/cases.jsonlvs a separately-owned/withheldevals/<skill>/holdout/cases.jsonl, or an equivalent hidden/rotating-holdout mechanism. Document the chosen mechanism and why it is sufficient. - The scorer ([Phase 1] Build the deterministic eval scorer + offline tests (triage) #583) and the strict-improvement gate ([Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586) score against the holdout split only; any proposer prompt/context is constructed from the dev split only.
evals/validate-cases.pyvalidates both splits and enforces uniqueids across their union (a case must not appear in both).- The holdout split is CODEOWNER-gated (and, where the mechanism allows, not readable by the proposer's checkout/context) — the dev split may be proposer-visible.
evals/README.mddocuments the split, the no-overlap rule, and the de-identification requirement (per decision A3).
Dev Notes
- This is a prerequisite for the Phase 3 proposer ([Phase 3] Automate the scheduled proposer behind the safe-rollout gates #587): without it, the "improvement" the gate measures can be overfit. Land it before [Phase 3] Automate the scheduled proposer behind the safe-rollout gates #587 activates.
- Keep it data/structure/validation only — no scorer or model invocation in
evals/(consistent with [Phase 1] Define held-out eval-case format + seed prompts/triage.md case set #582).
Filed from the #581 rubber-duck review (finding 1) to make the held-out-hygiene guarantee a tracked deliverable rather than prose.
Activity
github-actions commentedon Jun 14, 2026
Thanks for filing this so carefully — capturing the held-out hygiene guarantee as a tracked story (rather than leaving it as prose in the epic) is exactly the right call, and the acceptance criteria are clear and actionable. For the implementation: the issue offers two mechanisms — a static dev/holdout directory split vs a hidden/rotating holdout — and notes they're equivalent; do you have a preference, or is there a constraint (e.g. ease of auditing, CI caching, CODEOWNERS granularity) that should drive the choice? Knowing that will help land the right structure before #587 activates.
don-petry commentedon Jun 14, 2026
Dev-Lead Implementation Plan
Issue: #691 — [Guardrail] Held-out hygiene: dev/test split so the proposer never scores against cases it can see
Scope
Establish the
evals/held-out hygiene structure: a proposer-visibledev/split and a CODEOWNER-gatedholdout/split per skill, avalidate-cases.pyvalidator enforcing no cross-splitidoverlap, andREADME.mddocumenting the mechanism, the no-overlap rule, and the de-identification requirement. Data/structure/validation only — no scorer or model invocation (consistent with #582).Chosen mechanism (AC1/AC4)
Directory split
evals/<skill>/dev/cases.jsonl(proposer-visible) vsevals/<skill>/holdout/cases.jsonl(CODEOWNER-gated). The holdout gets an explicitCODEOWNERSentry so the proposer (running asGITHUB_TOKEN) can never merge edits to it, even if broader rules loosen. The dev/holdout boundary is the structural contract the scorer (#583) and gate (#586) consume; the proposer prompt (#587) is built fromdev/only. README documents why this is sufficient and notes stronger non-readability hardening (separate access-controlled store) as future work.Implementation Checklist
evals/validate-cases.py— validate both splits per skill; valid JSONL, non-empty unique stringidper split, noidin both splits (union uniqueness)evals/example-skill/dev/cases.jsonl— de-identified example dev cases (template)evals/example-skill/holdout/cases.jsonl— de-identified example holdout cases (template)evals/README.md— document the split, no-overlap rule, de-identification requirement (decision A3), CODEOWNER gating, and the scorer/proposer contract.github/CODEOWNERS— explicitholdout/gate.github/workflows/lint.yml— register the new bats test file (sanctioned bats-list exception in AGENTS.md)Tests to Write
tests/test_validate_cases.bats:idappearing in both dev and holdout (overlap)ids within a splitidevals/tree (real example skill stays valid)Test Command
bats tests/test_validate_cases.bats(full suite: thebatsjob in.github/workflows/lint.yml); lint:bash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jun 14, 2026
Dev-Lead: Implementation Complete
Established the
evals/held-out hygiene structure (the directory did not exist yet — this story is its foundation). Data/structure/validation only; no scorer or model invocation (consistent with #582).Plan Execution
evals/validate-cases.py— validates both splits per skill: well-formed JSONL, non-empty unique stringidper split, noidshared across splits (union uniqueness), and that each skill has both a dev and holdout splitevals/example-skill/dev/cases.jsonl+holdout/cases.jsonl— de-identified template cases (disjoint ids)evals/README.md— documents the dev/holdout split, the no-overlap rule, the de-identification requirement (A3), CODEOWNER gating, and the scorer([Phase 1] Build the deterministic eval scorer + offline tests (triage) #583)/gate([Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586)/proposer([Phase 3] Automate the scheduled proposer behind the safe-rollout gates #587) contract + residual-risk/future-hardening note.github/CODEOWNERS— explicit/evals/**/holdout/gate so the proposer (running asGITHUB_TOKEN) can never merge holdout edits, even if broader rules loosen.github/workflows/lint.yml— registeredtests/test_validate_cases.batsin the bats list (the sanctioned repo-specific bats-list exception in AGENTS.md)tests/test_validate_cases.bats— 10 tests (TDD: written red first, then green)Acceptance criteria mapping
evals/<skill>/dev/cases.jsonl(proposer-visible) vsevals/<skill>/holdout/cases.jsonl(gated); mechanism + rationale inevals/README.md.validate-cases.pyvalidates both splits + union-unique ids — implemented and tested (overlap, intra-split dup, malformed JSON, missing/empty/non-string id, non-object line, missing split).Test Results
Files Changed
evals/validate-cases.py(new): the held-out hygiene validator.evals/example-skill/dev/cases.jsonl,evals/example-skill/holdout/cases.jsonl(new): de-identified template split.evals/README.md(new): mechanism, no-overlap rule, de-identification requirement, CODEOWNER gating, consumer contract.tests/test_validate_cases.bats(new): validator tests..github/CODEOWNERS: explicit holdout gate..github/workflows/lint.yml: registered the new bats test file.Notes
idis owned by the scorer story ([Phase 1] Build the deterministic eval scorer + offline tests (triage) #583); this validator is intentionally agnostic to it and governs only split structure + id discipline.holdout/files instead of using the sanctioned dev-only context builder could still see them. Directory split + CODEOWNERS removes the tampering and accidental-inclusion vectors and is sufficient for the current single-repo pipeline; true non-readability (separate access-controlled/encrypted/rotating holdout, or fetching holdout only inside the gate's isolated job) is noted as follow-up hardening on the proposer/gate stories.feat: implement issue #691 — [Guardrail] Held-out hygiene: dev/test s…
feat: implement issue #691 — Held-out hygiene: dev/holdout eval split (…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #691 — Held-out hygiene: dev/holdout eval split (…
28 remaining items
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #691 — Held-out hygiene: dev/holdout eval split (…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #691 — Held-out hygiene: dev/holdout eval split (…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #691 — Held-out hygiene: dev/holdout eval split (…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #691 — Held-out hygiene: dev/holdout eval split (…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #583 — deterministic eval scorer (triage) over #…
feat: implement issue #583 — deterministic eval scorer (triage) over #…