Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .github/workflows/holdout-guard.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: Holdout Guard

# Hard CI enforcement for the held-out eval immutability guarantee (#692, epic
# #581). Fails any PR authored by the automated skill-proposer identity that
# changes a held-out path (evals/), keyed on PR author + changed paths.
#
# This COMPLEMENTS, does not replace, .github/CODEOWNERS:
# - CODEOWNERS routes review (and only blocks merges when branch protection's
# "require review from code owners" is enabled — an unstated dependency).
# - this guard is the immutability guarantee — it fails the proposer's PR
# independent of whether CODEOWNER review is enforced.
#
# No `paths:` filter on purpose: this must run (and report) on EVERY PR so it is
# a stable required status check. The author/path decision (and the early pass
# for non-proposer authors) lives in scripts/lib/holdout-guard.sh and is covered
# by tests/test_holdout_guard.bats.

on:
pull_request:
types: [opened, synchronize, reopened]

permissions:
contents: read
pull-requests: read

concurrency:
group: holdout-guard-${{ github.event.pull_request.number }}
cancel-in-progress: true

jobs:
holdout-guard:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3

- name: Enforce held-out eval immutability
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
run: |
set -euo pipefail
# shellcheck source=scripts/lib/holdout-guard.sh
source scripts/lib/holdout-guard.sh

echo "PR #${PR_NUMBER} authored by '${PR_AUTHOR}'"

changed="$(gh pr diff "$PR_NUMBER" --name-only)"
echo "Changed paths:"
printf '%s\n' "$changed" | sed 's/^/ /'

printf '%s\n' "$changed" | hg_evaluate "$PR_AUTHOR"
3 changes: 2 additions & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@ jobs:
tests/test_sweep_stuck_reviews.bats \
tests/test_validate_cases.bats \
tests/test_review_registry.bats \
tests/test_skill_evals.bats
tests/test_skill_evals.bats \
tests/test_holdout_guard.bats

validate-agent-profiles:
runs-on: ubuntu-latest
Expand Down
8 changes: 8 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,14 @@ This is the `.github-private` org infrastructure repo for `petry-projects`. It c
- **Exception:** The `bats` test list in `lint.yml` is extended with repo-specific test files (e.g.
`tests/test_push_protection.bats`) beyond the org template baseline. When adding new test files to this
repo, add them to this list. Template syncs must not reset it to the base template list.
- **Exception:** `holdout-guard.yml` (#692, epic #581) is a documented repo-specific workflow with no
corresponding org template in `standards/workflows/`. It is the hard CI enforcement behind the advisory
`.github/CODEOWNERS` rule over `evals/`: it fails any PR authored by the automated skill-proposer identity
that changes a held-out path, keyed on PR author + changed paths. It runs on every PR (no `paths:` filter)
so it stays a stable required status check. The author/path decision lives in
`scripts/lib/holdout-guard.sh` (tests: `tests/test_holdout_guard.bats`). It must not be removed by template
syncs. If the org template gains a held-out immutability equivalent, remove this exception and defer to the
template instead.
- **Exception:** `auto-rebase-retry.yml` is a documented repo-specific workflow with no corresponding org
template in `standards/workflows/`. It is a `workflow_run` handler that self-heals a *failed* Auto-rebase
run by re-running its failed jobs (bounded by `run_attempt`) — the gap left by the conflict-only sentinel
Expand Down
44 changes: 38 additions & 6 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,16 +62,48 @@ reconstructions of a behavior, not raw captured transcripts. Because the `dev/`
split is proposer-visible (and the whole repo is readable by other automation),
treat every committed case as world-readable within the org.

## The hard CI immutability gate (#692)

CODEOWNERS only **requests** review — it blocks a merge only when branch
protection's "require review from code owners" is enabled, an unstated
dependency. The actual immutability guarantee is a **CI gate** that fails any
proposer-authored PR touching a held-out path, independent of CODEOWNER review.

- **Workflow:** [`.github/workflows/holdout-guard.yml`](../.github/workflows/holdout-guard.yml)
runs on every PR (no `paths:` filter, so it is a stable required check).
- **Decision logic:** [`scripts/lib/holdout-guard.sh`](../scripts/lib/holdout-guard.sh),
keyed on **PR author + changed paths**:
- author is **not** the proposer identity → **pass** (humans/CODEOWNERS, and
the dev-lead bot authoring eval *stories*, edit `evals/` freely);
- author **is** the proposer but changes only non-`evals/` paths (e.g.
`prompts/triage.md`) → **pass**;
- author **is** the proposer **and** changes any `evals/` path → **fail**.
- **Proposer identity:** the proposer runs as `GITHUB_TOKEN`, whose PR author
login is `github-actions[bot]` (the resolved default). It is intentionally
distinct from the dev-lead bot (`donpetry-bot`), which authors the eval
stories themselves. Override the identity list (and guarded prefixes) via the
`HOLDOUT_PROPOSER_IDENTITIES` / `HOLDOUT_GUARDED_PREFIXES` env vars when the
Phase-3 proposer (#587) lands with its own bot account.
- **Coverage:** [`tests/test_holdout_guard.bats`](../tests/test_holdout_guard.bats),
wired into the `bats` job in `.github/workflows/lint.yml`.

This **complements, does not replace, CODEOWNERS**: CODEOWNERS routes review;
this gate is the immutability guarantee. Land it before the Phase-3 proposer
(#587) activates so the proposer can never self-edit its own test set even if
branch protection is misconfigured.

## Why CODEOWNER-gating the holdout is sufficient (AC1/AC4)

The held-out guarantee rests on two independent controls:

1. **Tamper protection (enforced here).** `holdout/` is CODEOWNER-gated in
[`.github/CODEOWNERS`](../.github/CODEOWNERS) with an explicit rule, on top of
the repo-wide default. The proposer runs as `GITHUB_TOKEN` and therefore
**cannot merge** any edit to a holdout file — it can neither rewrite the
holdout to match a weak skill nor move cases between splits. The reward-hacking
guard from #582 (CODEOWNERS over `evals/`) is what this rule specialises.
1. **Tamper protection (enforced here, plus a CI check enforces it).** `holdout/`
is CODEOWNER-gated in [`.github/CODEOWNERS`](../.github/CODEOWNERS) with an
explicit rule, on top of the repo-wide default, **and** the CI gate above
(#692) hard-fails any proposer-authored change to `evals/`. The proposer runs
as `GITHUB_TOKEN` and therefore **cannot merge** any edit to a holdout file —
it can neither rewrite the holdout to match a weak skill nor move cases
between splits. The reward-hacking guard from #582 (CODEOWNERS over `evals/`)
is what this rule specialises.

2. **Context isolation (enforced by the consumer).** The proposer prompt is built
from `dev/` only; the gate reads `holdout/` only. This is a contract the
Expand Down
116 changes: 116 additions & 0 deletions scripts/lib/holdout-guard.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
#!/usr/bin/env bash
# holdout-guard.sh — held-out eval immutability guard (#692, epic #581).
#
# This is the *hard* CI enforcement that backs the advisory CODEOWNERS rule over
# evals/. CODEOWNERS only routes review (and only blocks merges when branch
# protection's "require review from code owners" is enabled — an unstated
# dependency). This guard is the immutability guarantee: it fails any PR whose
# author is the automated skill-proposer identity when that PR changes a
# held-out path, keyed purely on PR author + changed paths.
#
# It complements, does not replace, CODEOWNERS:
# - CODEOWNERS routes review.
# - this guard is the immutability guarantee — independent of whether
# CODEOWNER review is enforced.
#
# Decision (hg_evaluate):
# author NOT a proposer identity -> PASS (humans/CODEOWNERS edit evals/)
# author IS a proposer + no guarded path -> PASS (e.g. prompts/triage.md)
# author IS a proposer + a guarded path -> FAIL (the held-out guarantee)
#
# Proposer identity (HOLDOUT_PROPOSER_IDENTITIES, default "github-actions[bot]"):
# The proposer runs as GITHUB_TOKEN (see evals/README.md), whose PR author
# login is github-actions[bot]. This is intentionally DISTINCT from the
# dev-lead bot (donpetry-bot), which authors the eval *stories* (like #692
# itself) and must keep passing. Override via repo variable when the Phase-3
# proposer (#587) lands with its own bot account.
#
# Guarded prefixes (HOLDOUT_GUARDED_PREFIXES, default "evals/"):
# Path prefixes (each should end with "/") that are held-out. The holdout
# split lives under evals/<skill>/holdout/, so the evals/ prefix already
# covers it; add a second prefix here if a holdout is ever separately rooted.

# hg_proposer_identities — emit the configured proposer logins, one per line.
# Accepts comma- and/or whitespace-separated values in the env var.
hg_proposer_identities() {
local raw="${HOLDOUT_PROPOSER_IDENTITIES:-github-actions[bot]}" item
# Replace commas with spaces (word splitting handles multiple whitespace)
for item in ${raw//,/ }; do
[ -n "$item" ] && printf '%s\n' "$item"
done
}

# hg_guarded_prefixes — emit the configured guarded path prefixes, one per line.
hg_guarded_prefixes() {
local raw="${HOLDOUT_GUARDED_PREFIXES:-evals/}" item
# Replace commas with spaces (word splitting handles multiple whitespace)
for item in ${raw//,/ }; do
[ -n "$item" ] && printf '%s\n' "$item"
done
}

# hg_is_proposer <login> — return 0 if the login is a configured proposer.
hg_is_proposer() {
local login="${1:-}" want identity
login="${login,,}"
[ -n "$login" ] || return 1
while IFS= read -r identity || [ -n "$identity" ]; do
[ -n "$identity" ] || continue
want="${identity,,}"
[ "$login" = "$want" ] && return 0
done < <(hg_proposer_identities)
return 1
}

# hg_path_is_guarded <path> — return 0 if the path sits under a guarded prefix.
hg_path_is_guarded() {
local path="${1:-}" prefix
[ -n "$path" ] || return 1
while IFS= read -r prefix || [ -n "$prefix" ]; do
[ -n "$prefix" ] || continue
case "$path" in
"$prefix"*) return 0 ;;
esac
done < <(hg_guarded_prefixes)
return 1
}

# hg_evaluate <author> — read newline-delimited changed paths on stdin and
# decide pass/fail. Prints a human-readable verdict. Exit 0 = pass, 1 = fail,
# 2 = usage error (missing author).
hg_evaluate() {
local author="${1:-}"
if [ -z "$author" ]; then
echo "[holdout-guard] ERROR: no PR author supplied" >&2
return 2
fi

if ! hg_is_proposer "$author"; then
echo "[holdout-guard] author '$author' is not a proposer identity — pass"
# Drain stdin so callers piping into us don't see a broken pipe.
cat >/dev/null 2>&1 || true
return 0
fi

local path offending=()
while IFS= read -r path || [ -n "$path" ]; do
[ -n "$path" ] || continue
if hg_path_is_guarded "$path"; then
offending+=("$path")
fi
done

if [ "${#offending[@]}" -gt 0 ]; then
echo "[holdout-guard] FAIL: proposer '$author' changed held-out path(s):"
local p
for p in "${offending[@]}"; do
echo " - $p"
done
echo "[holdout-guard] The proposer must never edit its own held-out eval set." >&2
echo "[holdout-guard] This is the hard guarantee behind the CODEOWNERS rule over evals/." >&2
return 1
fi

echo "[holdout-guard] proposer '$author' changed no held-out paths — pass"
return 0
}
Loading
Loading