Repository navigation
Description
Part of epic #581 (self-improving skills, Discussion #572). Raised as finding #2 of the maintainer rubber-duck review on #581.
Problem
The held-out set's protection currently rests on .github/CODEOWNERS (added in #582, now over /evals/). CODEOWNERS only requests review, and only blocks merges if branch protection's "require review from code owners" is enabled — an unstated dependency. The actual guarantee the design needs (and what #587 AC3 calls "plus a check enforces this") is a CI gate that fails any proposer-authored PR that changes a held-out path, keyed on PR author + changed paths. That belongs in its own test-covered story rather than being assumed.
Scope
Add a required CI check that hard-fails when the automated skill proposer authors a PR touching evals/, independent of whether CODEOWNERS review is enforced.
Acceptance criteria
- A CI workflow/job fails when a PR whose author is the automated proposer identity (the proposer agent / its bot account — resolve the exact author handle as part of this story) changes any path under
evals/(and, if the holdout split from the hygiene story is separately rooted, that path too). - The check is keyed on PR author + changed paths: human/CODEOWNER-authored changes to
evals/pass; proposer-authored changes to non-evals/paths (e.g.prompts/triage.md) pass; only proposer +evals/fails. batscoverage for the author/path decision logic (model ontests/test_evals_cases.batsand the repo's other workflow-logic tests), wired into thebatsjob list in.github/workflows/lint.yml.- Documented in
evals/README.mdand the propose→validate→PR runbook ([Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586) as the hard enforcement that backs the advisory CODEOWNERS rule.
Dev Notes
- Complements, does not replace, CODEOWNERS: CODEOWNERS routes review; this check is the immutability guarantee.
- Prerequisite for the Phase 3 proposer ([Phase 3] Automate the scheduled proposer behind the safe-rollout gates #587). Land it before [Phase 3] Automate the scheduled proposer behind the safe-rollout gates #587 activates so the proposer can never self-edit its own test set even if branch protection is misconfigured.
Filed from the #581 rubber-duck review (finding 2) to make the case-immutability guarantee a tracked, test-covered deliverable.
Activity
don-petry commentedon Jun 14, 2026
Dev-Lead Implementation Plan
Issue: #692 — [Guardrail] Hard CI check: fail any proposer-authored PR that touches evals/
Scope
Add a required CI gate that hard-fails any PR authored by the automated skill-proposer identity when it changes a held-out path (
evals/), keyed on PR author + changed paths, independent of CODEOWNERS.Proposer identity resolution (AC1)
evals/README.mdstates the proposer "runs asGITHUB_TOKEN", whose PR author login isgithub-actions[bot]. That is the resolved default proposer handle. Crucially it is distinct fromdonpetry-bot(the human-driven dev-lead identity that authors eval-story PRs like this one), so dev-lead/human edits toevals/keep passing. The identity list is overridable via a repo variable for when the Phase-3 proposer (#587) lands.Implementation Checklist
scripts/lib/holdout-guard.sh— pure decision logic:hg_is_proposer,hg_path_is_guarded,hg_evaluate(author + newline-delimited changed paths on stdin → pass/fail). Proposer identities and guarded prefixes read from env with safe defaults (github-actions[bot],evals/)..github/workflows/holdout-guard.yml— new repo-specific PR gate (no path filter, so it always runs as a stable required check). Resolves PR author + changed files, sources the lib, fails on proposer + guarded path.tests/test_holdout_guard.bats— bats coverage of the author/path decision matrix..github/workflows/lint.yml— add the new bats file to thebatsjob list.evals/README.md— document the hard CI enforcement that backs the advisory CODEOWNERS rule.AGENTS.md— add the new workflow as a documented repo-specific exception (mirrors token-report/pr-review-sweep exceptions).Tests to Write
evals/path → failevals/.../holdout/path → failevals/path (prompts/triage.md) → passdonpetry-bot) +evals/path → passevalsomething/does not matchevals/)Test Command
bats tests/test_holdout_guard.bats(full suite via thebatsjob list in.github/workflows/lint.yml); lint viabash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jun 14, 2026
Dev-Lead: Implementation Complete
Plan Execution
scripts/lib/holdout-guard.sh— pure author/path decision logic (hg_is_proposer,hg_path_is_guarded,hg_evaluate); proposer identities + guarded prefixes read from env with safe defaults.github/workflows/holdout-guard.yml— new repo-specific required PR gate (nopaths:filter → always runs/reports), resolves PR author + changed files, sources the lib, fails on proposer + held-out pathtests/test_holdout_guard.bats— 21 tests covering the full AC2 decision matrix.github/workflows/lint.yml— added the new bats file to thebatsjob listevals/README.md— documented the hard CI enforcement (AC4)AGENTS.md— added the new workflow as a documented repo-specific exceptionProposer identity resolution (AC1)
evals/README.mdstates the proposer "runs asGITHUB_TOKEN", whose PR author login isgithub-actions[bot]— the resolved default. It is intentionally distinct fromdonpetry-bot(the dev-lead identity that authors eval-story PRs, including this one), so human/dev-lead edits toevals/keep passing. The identity list and guarded prefixes are overridable viaHOLDOUT_PROPOSER_IDENTITIES/HOLDOUT_GUARDED_PREFIXESfor when the Phase-3 proposer (#587) lands with its own bot account.Decision matrix (AC2) — all enforced by tests
github-actions[bot])evals/...(incl.holdout/)prompts/triage.md(non-evals)donpetry-bot)evals/...Test Results
Files Changed
scripts/lib/holdout-guard.sh(new): author/path decision logic; case-insensitive identity match, prefix-boundary-safe path match, robust to changed-file streams lacking a trailing newline (a dropped final path would be a false PASS).github/workflows/holdout-guard.yml(new): the required CI gatetests/test_holdout_guard.bats(new): decision-matrix coverage.github/workflows/lint.yml: wired the new bats file into thebatsjobevals/README.md: documented the hard enforcement backing the advisory CODEOWNERS ruleAGENTS.md: documented the workflow as a repo-specific template-sync exceptionNotes
donpetry-bot) modifiesevals/README.md. Because the proposer identity isgithub-actions[bot], notdonpetry-bot, the new gate correctly passes this PR — verified by the "human/CODEOWNER changing an evals/ path" test.evals/(bothdev/andholdout/), matching "changes any path underevals/". The holdout split is rooted underevals/, so the singleevals/prefix covers it; no separately-rooted holdout path exists.docs/, so documentation landed inevals/README.md. Recommend folding the same enforcement note into the [Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586 runbook when that story lands.feat: implement issue #692 — [Guardrail] Hard CI check: fail any prop…
feat: implement issue #692 — [Guardrail] Hard CI check: fail any prop…
feat: implement issue #692 — hard CI check: fail proposer PRs touchin…