Skip to content

[Guardrail] Held-out hygiene: dev/test split so the proposer never scores against cases it can see #691

Closed

Description

@don-petry

Part of epic #581 (self-improving skills, Discussion #572). Raised as finding #1 of the maintainer rubber-duck review on #581 — recommended as a new story because it is one of the trust guarantees the whole "self-improving" claim rests on.

Problem

The reward-hacking guard added in #582 (CODEOWNERS over evals/) stops the proposer from editing the held-out set — but not from overfitting to it. The Phase 3 proposer (#587) reads the whole repo, so it can see every case in evals/<skill>/cases.jsonl and tune the skill markdown to pass exactly those, with no real generalization ("teaching to the test"). SkillOpt's discipline depends on a held-out set the optimizer never sees; nothing in the current plan hides or rotates the cases.

Scope

Introduce a dev/test split (or a hidden/rotating holdout) for the eval sets now rooted at evals/<skill>/ (per maintainer decision A1), so the proposer only ever sees the dev subset and the gate scores against the unseen holdout.

Acceptance criteria

  1. Each skill's cases are partitioned into a proposer-visible dev split and a holdout split the proposer never sees — e.g. evals/<skill>/dev/cases.jsonl vs a separately-owned/withheld evals/<skill>/holdout/cases.jsonl, or an equivalent hidden/rotating-holdout mechanism. Document the chosen mechanism and why it is sufficient.
  2. The scorer ([Phase 1] Build the deterministic eval scorer + offline tests (triage) #583) and the strict-improvement gate ([Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586) score against the holdout split only; any proposer prompt/context is constructed from the dev split only.
  3. evals/validate-cases.py validates both splits and enforces unique ids across their union (a case must not appear in both).
  4. The holdout split is CODEOWNER-gated (and, where the mechanism allows, not readable by the proposer's checkout/context) — the dev split may be proposer-visible.
  5. evals/README.md documents the split, the no-overlap rule, and the de-identification requirement (per decision A3).

Dev Notes

Filed from the #581 rubber-duck review (finding 1) to make the held-out-hygiene guarantee a tracked deliverable rather than prose.

Activity

  1. github-actions commented on Jun 14, 2026

    @github-actions
    Contributor

    Thanks for filing this so carefully — capturing the held-out hygiene guarantee as a tracked story (rather than leaving it as prose in the epic) is exactly the right call, and the acceptance criteria are clear and actionable. For the implementation: the issue offers two mechanisms — a static dev/holdout directory split vs a hidden/rotating holdout — and notes they're equivalent; do you have a preference, or is there a constraint (e.g. ease of auditing, CI caching, CODEOWNERS granularity) that should drive the choice? Knowing that will help land the right structure before #587 activates.

  2. don-petry commented on Jun 14, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #691 — [Guardrail] Held-out hygiene: dev/test split so the proposer never scores against cases it can see

    Scope

    Establish the evals/ held-out hygiene structure: a proposer-visible dev/ split and a CODEOWNER-gated holdout/ split per skill, a validate-cases.py validator enforcing no cross-split id overlap, and README.md documenting the mechanism, the no-overlap rule, and the de-identification requirement. Data/structure/validation only — no scorer or model invocation (consistent with #582).

    Chosen mechanism (AC1/AC4)

    Directory split evals/<skill>/dev/cases.jsonl (proposer-visible) vs evals/<skill>/holdout/cases.jsonl (CODEOWNER-gated). The holdout gets an explicit CODEOWNERS entry so the proposer (running as GITHUB_TOKEN) can never merge edits to it, even if broader rules loosen. The dev/holdout boundary is the structural contract the scorer (#583) and gate (#586) consume; the proposer prompt (#587) is built from dev/ only. README documents why this is sufficient and notes stronger non-readability hardening (separate access-controlled store) as future work.

    Implementation Checklist

    • evals/validate-cases.py — validate both splits per skill; valid JSONL, non-empty unique string id per split, no id in both splits (union uniqueness)
    • evals/example-skill/dev/cases.jsonl — de-identified example dev cases (template)
    • evals/example-skill/holdout/cases.jsonl — de-identified example holdout cases (template)
    • evals/README.md — document the split, no-overlap rule, de-identification requirement (decision A3), CODEOWNER gating, and the scorer/proposer contract
    • .github/CODEOWNERS — explicit holdout/ gate
    • .github/workflows/lint.yml — register the new bats test file (sanctioned bats-list exception in AGENTS.md)

    Tests to Write

    • tests/test_validate_cases.bats:
      • accepts a well-formed dev/holdout split
      • rejects an id appearing in both dev and holdout (overlap)
      • rejects duplicate ids within a split
      • rejects malformed JSONL / missing or empty id
      • validates the committed evals/ tree (real example skill stays valid)

    Test Command

    bats tests/test_validate_cases.bats (full suite: the bats job in .github/workflows/lint.yml); lint: bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Jun 14, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Established the evals/ held-out hygiene structure (the directory did not exist yet — this story is its foundation). Data/structure/validation only; no scorer or model invocation (consistent with #582).

    Plan Execution

    Acceptance criteria mapping

    1. dev/holdout partition — evals/<skill>/dev/cases.jsonl (proposer-visible) vs evals/<skill>/holdout/cases.jsonl (gated); mechanism + rationale in evals/README.md.
    2. scorer/gate → holdout only; proposer context → dev only — documented as the consumer contract in README (the scorer/gate/proposer components [Phase 1] Build the deterministic eval scorer + offline tests (triage) #583/[Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586/[Phase 3] Automate the scheduled proposer behind the safe-rollout gates #587 are not in this repo yet; this story lands the structural boundary they key off, per "land it before [Phase 3] Automate the scheduled proposer behind the safe-rollout gates #587 activates").
    3. validate-cases.py validates both splits + union-unique ids — implemented and tested (overlap, intra-split dup, malformed JSON, missing/empty/non-string id, non-object line, missing split).
    4. holdout CODEOWNER-gated; dev may be proposer-visible — explicit CODEOWNERS rule added.
    5. README documents split + no-overlap + de-identification — done.

    Test Results

    bats tests/test_validate_cases.bats
    1..10
    ok 1 validate-cases accepts a well-formed dev/holdout split
    ok 2 validate-cases rejects an id appearing in both dev and holdout
    ok 3 validate-cases rejects a duplicate id within a split
    ok 4 validate-cases rejects malformed JSON in a case line
    ok 5 validate-cases rejects a case missing an id
    ok 6 validate-cases rejects a case with an empty id
    ok 7 validate-cases rejects a non-object case line
    ok 8 validate-cases skips blank lines in a cases file
    ok 9 validate-cases fails when a skill has a dev split but no holdout
    ok 10 validate-cases validates the committed evals tree
    
    Full bats suite (all lint.yml files): 272 tests, 0 failures
    bash .dev-lead/scripts/dev-lead-lint.sh  → all checks passed (shellcheck 51 files, 4 agent profiles)
    npx markdownlint-cli2 evals/README.md     → 0 errors
    python3 evals/validate-cases.py           → OK: 1 skill(s), 6 case(s), no cross-split id overlap
    

    Files Changed

    • evals/validate-cases.py (new): the held-out hygiene validator.
    • evals/example-skill/dev/cases.jsonl, evals/example-skill/holdout/cases.jsonl (new): de-identified template split.
    • evals/README.md (new): mechanism, no-overlap rule, de-identification requirement, CODEOWNER gating, consumer contract.
    • tests/test_validate_cases.bats (new): validator tests.
    • .github/CODEOWNERS: explicit holdout gate.
    • .github/workflows/lint.yml: registered the new bats test file.

    Notes

    • The case payload schema beyond id is owned by the scorer story ([Phase 1] Build the deterministic eval scorer + offline tests (triage) #583); this validator is intentionally agnostic to it and governs only split structure + id discipline.
    • Residual risk (documented in README): in a single git repo the dev-only/holdout-only context isolation (AC2) is a contract the proposer/gate enforce at context-assembly time, not a hard wall — a proposer that read raw holdout/ files instead of using the sanctioned dev-only context builder could still see them. Directory split + CODEOWNERS removes the tampering and accidental-inclusion vectors and is sufficient for the current single-repo pipeline; true non-readability (separate access-controlled/encrypted/rotating holdout, or fetching holdout only inside the gate's isolated job) is noted as follow-up hardening on the proposer/gate stories.
  4. added a commit that references this issue on Jun 14, 2026
  5. added 6 commits that reference this issue on Jun 14, 2026
  6. 28 remaining items

  7. added 14 commits that reference this issue on Aug 7, 2026
  8. added a commit that references this issue on Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requestsinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions