Skip to content

[Guardrail] Hard CI check: fail any proposer-authored PR that touches evals/ #692

Closed

Description

@don-petry

Part of epic #581 (self-improving skills, Discussion #572). Raised as finding #2 of the maintainer rubber-duck review on #581.

Problem

The held-out set's protection currently rests on .github/CODEOWNERS (added in #582, now over /evals/). CODEOWNERS only requests review, and only blocks merges if branch protection's "require review from code owners" is enabled — an unstated dependency. The actual guarantee the design needs (and what #587 AC3 calls "plus a check enforces this") is a CI gate that fails any proposer-authored PR that changes a held-out path, keyed on PR author + changed paths. That belongs in its own test-covered story rather than being assumed.

Scope

Add a required CI check that hard-fails when the automated skill proposer authors a PR touching evals/, independent of whether CODEOWNERS review is enforced.

Acceptance criteria

  1. A CI workflow/job fails when a PR whose author is the automated proposer identity (the proposer agent / its bot account — resolve the exact author handle as part of this story) changes any path under evals/ (and, if the holdout split from the hygiene story is separately rooted, that path too).
  2. The check is keyed on PR author + changed paths: human/CODEOWNER-authored changes to evals/ pass; proposer-authored changes to non-evals/ paths (e.g. prompts/triage.md) pass; only proposer + evals/ fails.
  3. bats coverage for the author/path decision logic (model on tests/test_evals_cases.bats and the repo's other workflow-logic tests), wired into the bats job list in .github/workflows/lint.yml.
  4. Documented in evals/README.md and the propose→validate→PR runbook ([Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586) as the hard enforcement that backs the advisory CODEOWNERS rule.

Dev Notes

Filed from the #581 rubber-duck review (finding 2) to make the case-immutability guarantee a tracked, test-covered deliverable.

Activity

  1. don-petry commented on Jun 14, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #692 — [Guardrail] Hard CI check: fail any proposer-authored PR that touches evals/

    Scope

    Add a required CI gate that hard-fails any PR authored by the automated skill-proposer identity when it changes a held-out path (evals/), keyed on PR author + changed paths, independent of CODEOWNERS.

    Proposer identity resolution (AC1)

    evals/README.md states the proposer "runs as GITHUB_TOKEN", whose PR author login is github-actions[bot]. That is the resolved default proposer handle. Crucially it is distinct from donpetry-bot (the human-driven dev-lead identity that authors eval-story PRs like this one), so dev-lead/human edits to evals/ keep passing. The identity list is overridable via a repo variable for when the Phase-3 proposer (#587) lands.

    Implementation Checklist

    • scripts/lib/holdout-guard.sh — pure decision logic: hg_is_proposer, hg_path_is_guarded, hg_evaluate (author + newline-delimited changed paths on stdin → pass/fail). Proposer identities and guarded prefixes read from env with safe defaults (github-actions[bot], evals/).
    • .github/workflows/holdout-guard.yml — new repo-specific PR gate (no path filter, so it always runs as a stable required check). Resolves PR author + changed files, sources the lib, fails on proposer + guarded path.
    • tests/test_holdout_guard.bats — bats coverage of the author/path decision matrix.
    • .github/workflows/lint.yml — add the new bats file to the bats job list.
    • evals/README.md — document the hard CI enforcement that backs the advisory CODEOWNERS rule.
    • AGENTS.md — add the new workflow as a documented repo-specific exception (mirrors token-report/pr-review-sweep exceptions).

    Tests to Write

    • proposer + evals/ path → fail
    • proposer + evals/.../holdout/ path → fail
    • proposer + non-evals/ path (prompts/triage.md) → pass
    • human/CODEOWNER (donpetry-bot) + evals/ path → pass
    • proposer + mixed evals/non-evals → fail
    • non-proposer + mixed → pass
    • case-insensitive identity match; configurable identities/prefixes via env
    • path-prefix boundary (evalsomething/ does not match evals/)

    Test Command

    bats tests/test_holdout_guard.bats (full suite via the bats job list in .github/workflows/lint.yml); lint via bash .dev-lead/scripts/dev-lead-lint.sh

  2. don-petry commented on Jun 14, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • scripts/lib/holdout-guard.sh — pure author/path decision logic (hg_is_proposer, hg_path_is_guarded, hg_evaluate); proposer identities + guarded prefixes read from env with safe defaults
    • .github/workflows/holdout-guard.yml — new repo-specific required PR gate (no paths: filter → always runs/reports), resolves PR author + changed files, sources the lib, fails on proposer + held-out path
    • tests/test_holdout_guard.bats — 21 tests covering the full AC2 decision matrix
    • .github/workflows/lint.yml — added the new bats file to the bats job list
    • evals/README.md — documented the hard CI enforcement (AC4)
    • AGENTS.md — added the new workflow as a documented repo-specific exception

    Proposer identity resolution (AC1)

    evals/README.md states the proposer "runs as GITHUB_TOKEN", whose PR author login is github-actions[bot] — the resolved default. It is intentionally distinct from donpetry-bot (the dev-lead identity that authors eval-story PRs, including this one), so human/dev-lead edits to evals/ keep passing. The identity list and guarded prefixes are overridable via HOLDOUT_PROPOSER_IDENTITIES / HOLDOUT_GUARDED_PREFIXES for when the Phase-3 proposer (#587) lands with its own bot account.

    Decision matrix (AC2) — all enforced by tests

    Author Changed paths Result
    proposer (github-actions[bot]) evals/... (incl. holdout/) FAIL
    proposer prompts/triage.md (non-evals) PASS
    proposer mixed evals + non-evals FAIL
    human / CODEOWNER (donpetry-bot) evals/... PASS
    non-proposer mixed PASS

    Test Results

    21 tests, 0 failures   (tests/test_holdout_guard.bats)
    Full bats suite (lint.yml list): 305 tests, 0 failures
    shellcheck --severity=warning -x scripts/lib/holdout-guard.sh: OK
    bash .dev-lead/scripts/dev-lead-lint.sh: all checks passed
    

    Files Changed

    • scripts/lib/holdout-guard.sh (new): author/path decision logic; case-insensitive identity match, prefix-boundary-safe path match, robust to changed-file streams lacking a trailing newline (a dropped final path would be a false PASS)
    • .github/workflows/holdout-guard.yml (new): the required CI gate
    • tests/test_holdout_guard.bats (new): decision-matrix coverage
    • .github/workflows/lint.yml: wired the new bats file into the bats job
    • evals/README.md: documented the hard enforcement backing the advisory CODEOWNERS rule
    • AGENTS.md: documented the workflow as a repo-specific template-sync exception

    Notes

  3. added 2 commits that reference this issue on Jun 14, 2026
  4. added a commit that references this issue on Jun 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions