Repository navigation
feat: implement issue #582 — [Phase 1] Define held-out eval-case format + seed prompts/triage.md case set #623
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
f9a2cab
feat: implement issue #582 — [Phase 1] Define held-out eval-case form…
donpetry-bot d2b56ba
chore: apply manual instructions [skip ci-relay]
donpetry-bot 5c45865
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry ead31e1
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry a1cb04e
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry e337611
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry 2ea5b77
refactor(evals): relocate held-out eval set to repo-root evals/ per m…
claude 3b310fd
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry 7f76e1e
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry 872b362
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry 21b8ea7
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry 9b7da2a
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry f11c907
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry a67cff3
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry d480551
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry d6d0d22
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry f342a2e
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry fe72dec
Merge branch 'main' into dev-lead/issue-582-20260612-0136
don-petry File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,7 @@ | ||
| # Default owner for all files | ||
| * @petry-projects/org-leads | ||
|
|
||
| # Held-out eval cases are the trust anchor for skill-quality measurement. | ||
| # They MUST stay separately owned: a future automated skill proposer may only | ||
| # edit the skill markdown, never this held-out set (reward-hacking guard, #582). | ||
| /evals/ @petry-projects/org-leads |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Held-out eval cases | ||
|
|
||
| This directory holds the **held-out eval-case sets** used to measure prompt-skill | ||
| quality against fixed, reward-hack-resistant expected outputs. It is the trust | ||
| anchor of the skill-quality initiative (Discussion #572): cases live here, on | ||
| their own, and are validated structurally — there is **no scorer and no model | ||
| invocation** in this directory. | ||
|
|
||
| ## Layout | ||
|
|
||
| ``` | ||
| evals/ | ||
| ├── case.schema.json # JSON-Schema (draft 2020-12) for one eval case | ||
| ├── validate-cases.py # validate a .jsonl set against the schema | ||
| └── <skill>/cases.jsonl # one case per line for that skill (e.g. triage/) | ||
| ``` | ||
|
|
||
| Each `<skill>/cases.jsonl` is [JSON Lines](https://jsonlines.org/): one JSON | ||
| object per line, each conforming to `case.schema.json`. | ||
|
|
||
| ## Case format | ||
|
|
||
| A case pairs a single triage **input** with the **expected** decision the skill | ||
| must emit. Required fields (see `case.schema.json` for the authoritative | ||
| contract — it is `additionalProperties: false`, so unknown keys are rejected): | ||
|
|
||
| | Field | Type | Notes | | ||
| | ------------------- | ------------------------- | -------------------------------------------------------------------------------------- | | ||
| | `id` | string (kebab-case) | Stable, unique identifier for the case. | | ||
| | `input` | string | The pre-fetched PR context exactly as `prompts/triage.md` consumes under `## Pre-fetched PR context`. Self-contained; triage has no tools. | | ||
| | `expected.escalate` | boolean | Whether triage must escalate (`true`) or may approve (`false`). | | ||
| | `expected.risk` | `LOW` \| `MEDIUM` \| `HIGH` | Expected risk classification. Any `HIGH` signal must pair with `escalate: true`. | | ||
| | `description` | string (optional) | What the case exercises and which `triage.md` criterion grounds the expected decision. | | ||
| | `tags` | string[] (optional) | Coverage labels, e.g. the high-risk trigger a case exercises (`auth-secrets`, `db-migration`, `security-anti-pattern`). | | ||
|
|
||
| Expected decisions are **grounded strictly in the numbered decision criteria of | ||
| `prompts/triage.md`** — not invented behavior. `triage.md` emits a deterministic | ||
| JSON object with no tools, so the expected `escalate`/`risk` can be fixed values. | ||
|
|
||
| ## Validation | ||
|
|
||
| Validation uses the same `python jsonschema` pattern that | ||
| `scripts/initiative-planner/plan.schema.json` uses: | ||
|
|
||
| ```sh | ||
| pip install 'jsonschema>=4' | ||
| python3 evals/validate-cases.py evals/triage/cases.jsonl | ||
| ``` | ||
|
|
||
| The validator checks every line against `case.schema.json` and enforces the one | ||
| cross-line invariant the set depends on: **case ids must be unique**. It runs in | ||
| CI via `tests/test_evals_cases.bats`. | ||
|
|
||
| ## Separately owned — reward-hacking guard | ||
|
|
||
| This directory is **separately owned** via `.github/CODEOWNERS`. That is a | ||
| deliberate guard: a future automated skill *proposer* may only ever edit the | ||
| skill markdown (e.g. `prompts/triage.md`), **never** this held-out set. If the | ||
| proposer could edit the cases, it could optimize the skill against a moving | ||
| target — reward hacking. Keeping the cases CODEOWNER-gated, apart from any | ||
| scorer or proposer, is what makes a measurement against them trustworthy. | ||
|
|
||
| Any change to `evals/**` therefore requires review from the designated | ||
| code owners. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| { | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "title": "Triage Eval Case", | ||
| "description": "One held-out eval case: a single triage input paired with its expected decision. Each line of evals/<skill>/cases.jsonl is one object of this shape. This is the trust anchor for skill-quality measurement and is CODEOWNER-gated separately from any scorer/proposer.", | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": ["id", "input", "expected"], | ||
| "properties": { | ||
| "id": { | ||
| "type": "string", | ||
| "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$", | ||
| "minLength": 3, | ||
| "description": "Stable, unique kebab-case identifier for the case (e.g. 'triage-auth-token')." | ||
| }, | ||
| "description": { | ||
| "type": "string", | ||
| "minLength": 5, | ||
| "description": "Human-readable note on what this case exercises and which triage.md criterion grounds the expected decision." | ||
| }, | ||
| "tags": { | ||
| "type": "array", | ||
| "items": { | ||
| "type": "string", | ||
| "minLength": 2 | ||
| }, | ||
| "uniqueItems": true, | ||
| "description": "Optional labels for coverage tracking, e.g. the high-risk trigger a case exercises ('auth-secrets', 'db-migration', 'security-anti-pattern')." | ||
| }, | ||
| "input": { | ||
| "type": "string", | ||
| "minLength": 1, | ||
| "description": "The pre-fetched PR context exactly as triage.md consumes it under '## Pre-fetched PR context'. A self-contained string; triage has no tools and reads nothing else." | ||
| }, | ||
| "expected": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": ["escalate", "risk"], | ||
| "description": "The fixed, reward-hack-resistant expected decision fields triage.md must emit.", | ||
| "properties": { | ||
| "escalate": { | ||
| "type": "boolean", | ||
| "description": "Whether triage must escalate (true) or may approve (false)." | ||
| }, | ||
| "risk": { | ||
| "type": "string", | ||
| "enum": ["LOW", "MEDIUM", "HIGH"], | ||
| "description": "Expected risk classification. Any HIGH signal must pair with escalate=true." | ||
| } | ||
| }, | ||
| "if": { | ||
| "properties": { | ||
| "risk": { "const": "HIGH" } | ||
| } | ||
| }, | ||
| "then": { | ||
| "properties": { | ||
| "escalate": { "const": true } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| {"id": "triage-docs-typo", "description": "Docs-only typo fix; touches no high-risk area (criterion 1) and has no anti-patterns (criterion 3) -> approve.", "input": "Title: Fix typo in README quick-start\n\nFiles changed (1):\n README.md (+1/-1)\n\nDiff:\n--- a/README.md\n+++ b/README.md\n@@\n-Run the the installer.\n+Run the installer.\n\nLinked issue: none. No unresolved review threads. CI: all green.", "expected": {"escalate": false, "risk": "LOW"}} | ||
| {"id": "triage-test-only", "description": "Test-only addition that addresses its linked issue; no high-risk files (criterion 1), all gates pass -> approve.", "input": "Title: Add unit tests for date formatter\n\nFiles changed (1):\n src/utils/format_date.test.ts (+42/-0)\n\nDiff adds test cases only; no production code is touched.\n\nLinked issue: #210 (add coverage for the formatter); the diff addresses it. No unresolved review threads. CI: all green.", "expected": {"escalate": false, "risk": "LOW"}} | ||
| {"id": "triage-clean-logic-fix", "description": "Non-trivial but clean logic fix; no high-risk area, no anti-pattern, linked issue addressed, all gates pass -> approve at MEDIUM.", "input": "Title: Cap retry backoff at 30s\n\nFiles changed (1):\n src/net/retry.ts (+6/-2)\n\nDiff clamps exponential backoff to a 30s ceiling. No auth, secrets, DB, or workflow files touched and no security anti-patterns. Linked issue #233 (runaway backoff) is addressed. Clear title and scope. No unresolved review threads. CI green.", "expected": {"escalate": false, "risk": "MEDIUM"}} | ||
| {"id": "triage-auth-token", "description": "Touches authentication/session tokens under src/auth/** (criterion 1, high-risk area) -> always escalate, HIGH.", "tags": ["auth-secrets"], "input": "Title: Rotate session tokens on privilege change\n\nFiles changed (2):\n src/auth/session.ts (+58/-12)\n src/auth/tokens.ts (+20/-4)\n\nDiff reworks session-token issuance and rotation in the authentication layer; both files live under src/auth/. Linked issue #240. CI green.", "expected": {"escalate": true, "risk": "HIGH"}} | ||
| {"id": "triage-db-migration", "description": "Adds a SQL migration under migrations/ altering the users schema (criterion 1, high-risk area) -> always escalate, HIGH.", "tags": ["db-migration"], "input": "Title: Add nullable phone column to users\n\nFiles changed (2):\n migrations/0007_add_user_phone.sql (+9/-0)\n src/models/user.ts (+3/-0)\n\nDiff adds a new SQL migration under migrations/ that alters the users table schema. Linked issue #251. CI green.", "expected": {"escalate": true, "risk": "HIGH"}} | ||
| {"id": "triage-sql-concat", "description": "Builds a SQL query by string concatenation of input (criterion 3, security anti-pattern) -> escalate, HIGH.", "tags": ["security-anti-pattern"], "input": "Title: Add user search endpoint\n\nFiles changed (1):\n src/api/search.py (+18/-0)\n\nDiff:\n+def search(user_id):\n+ q = 'SELECT * FROM users WHERE id = ' + user_id\n+ return db.execute(q)\n\nNo auth or migration files are touched, but the query is assembled by string concatenation of the input. Linked issue #260.", "expected": {"escalate": true, "risk": "HIGH"}} | ||
| {"id": "triage-workflow-pr-target", "description": "GitHub Actions workflow using pull_request_target with a secret and PR-head checkout (criterion 1, workflow/secrets) -> escalate, HIGH.", "tags": ["workflow-secrets"], "input": "Title: Add PR labeler workflow\n\nFiles changed (1):\n .github/workflows/labeler.yml (+30/-0)\n\nDiff adds a GitHub Actions workflow triggered on pull_request_target that checks out the PR head and uses a repository secret. Linked issue #266.", "expected": {"escalate": true, "risk": "HIGH"}} | ||
| {"id": "triage-unresolved-thread", "description": "No high-risk area or anti-pattern, but an unresolved review thread requests changes (criterion 2) -> escalate at MEDIUM.", "tags": ["unresolved-threads"], "input": "Title: Refactor pagination helper\n\nFiles changed (1):\n src/api/paginate.ts (+24/-18)\n\nNo high-risk areas are touched and there are no security anti-patterns. However an unresolved review thread requests changes to an off-by-one in the page-size clamp, not yet addressed. Linked issue #270.", "expected": {"escalate": true, "risk": "MEDIUM"}} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| #!/usr/bin/env python3 | ||
| """Validate a held-out eval-case set (JSONL) against case.schema.json. | ||
|
|
||
| Each line of the .jsonl file is one case object. Beyond per-line JSON-Schema | ||
| structural checks this enforces the one cross-line invariant the held-out set | ||
| depends on: | ||
|
|
||
| - case ids are unique (a duplicate id silently shadows a case) | ||
|
|
||
| This is intentionally a *validator only* — no scorer, no model invocation. The | ||
| held-out cases are the trust anchor of skill-quality measurement; this tool | ||
| exists so the set can be checked in CI the same way scripts/initiative-planner | ||
| checks plan.schema.json (python jsonschema, draft 2020-12). | ||
|
|
||
| Usage: validate-cases.py <cases.jsonl> [schema.json] | ||
| Exit 0 on success; non-zero with a message on failure. | ||
| """ | ||
| import json | ||
| import sys | ||
| from pathlib import Path | ||
| from typing import NoReturn | ||
|
|
||
|
|
||
| def fail(msg: str) -> NoReturn: | ||
| # stdout (not stderr) so the reason is visible in workflow logs and capturable | ||
| # by tests; the ::error:: prefix still renders as a GitHub annotation. | ||
| print(f"::error::eval cases invalid: {msg}") | ||
| sys.exit(1) | ||
|
|
||
|
|
||
| def main() -> None: | ||
| if len(sys.argv) < 2: | ||
| fail("usage: validate-cases.py <cases.jsonl> [schema.json]") | ||
| cases_path = Path(sys.argv[1]) | ||
| schema_path = Path(sys.argv[2]) if len(sys.argv) > 2 else Path(__file__).with_name("case.schema.json") | ||
|
|
||
| try: | ||
| import jsonschema | ||
| except ImportError: | ||
| fail("jsonschema not installed (pip install 'jsonschema>=4')") | ||
|
|
||
| try: | ||
| schema = json.loads(schema_path.read_text(encoding="utf-8")) | ||
| except (OSError, json.JSONDecodeError) as exc: | ||
| fail(f"could not read/parse schema {schema_path}: {exc}") | ||
|
|
||
| try: | ||
| raw = cases_path.read_text(encoding="utf-8") | ||
| except OSError as exc: | ||
| fail(f"could not read {cases_path}: {exc}") | ||
|
|
||
| validator = jsonschema.Draft202012Validator(schema) | ||
| ids: dict[str, int] = {} | ||
| count = 0 | ||
| for lineno, line in enumerate(raw.splitlines(), start=1): | ||
| if not line.strip(): | ||
| continue # tolerate blank separator lines | ||
| try: | ||
| case = json.loads(line) | ||
| except json.JSONDecodeError as exc: | ||
| fail(f"line {lineno}: not valid JSON: {exc}") | ||
| error = next(validator.iter_errors(case), None) | ||
| if error is not None: | ||
| loc = "/".join(str(p) for p in error.absolute_path) or "<root>" | ||
| fail(f"line {lineno}: schema violation at {loc}: {error.message}") | ||
| case_id = case["id"] | ||
| if case_id in ids: | ||
| fail(f"line {lineno}: duplicate case id '{case_id}' (first seen on line {ids[case_id]})") | ||
| ids[case_id] = lineno | ||
| count += 1 | ||
|
|
||
| if count == 0: | ||
| fail(f"{cases_path} contains no cases") | ||
|
|
||
| print(f"cases OK: {count} case(s), unique ids, schema-valid.") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| #!/usr/bin/env bats | ||
| # Tests for the held-out triage eval-case set (evals/). | ||
| # validate-cases.py performs JSON-Schema validation only (no scorer, no model | ||
| # invocation) so the whole suite runs offline. | ||
|
|
||
| setup() { | ||
| ROOT="$(cd "$BATS_TEST_DIRNAME/.." && pwd)" | ||
| EVALS_DIR="$ROOT/evals" | ||
| VALIDATOR="$EVALS_DIR/validate-cases.py" | ||
| SCHEMA="$EVALS_DIR/case.schema.json" | ||
| CASES="$EVALS_DIR/triage/cases.jsonl" | ||
| TMP="$(mktemp -d)" | ||
| } | ||
|
|
||
| teardown() { rm -rf "$TMP"; } | ||
|
|
||
| @test "validate-cases accepts the seed triage case set" { | ||
| run python3 "$VALIDATOR" "$CASES" | ||
| [ "$status" -eq 0 ] | ||
| [[ "$output" == *"cases OK"* ]] | ||
| } | ||
|
|
||
| @test "validate-cases rejects a case missing expected.risk" { | ||
| jq -c 'del(.expected.risk)' <(head -1 "$CASES") >"$TMP/bad.jsonl" | ||
| run python3 "$VALIDATOR" "$TMP/bad.jsonl" | ||
| [ "$status" -ne 0 ] | ||
| [[ "$output" == *"risk"* ]] | ||
| } | ||
|
|
||
| @test "validate-cases rejects an unknown top-level property" { | ||
| jq -c '.surprise = "x"' <(head -1 "$CASES") >"$TMP/bad.jsonl" | ||
| run python3 "$VALIDATOR" "$TMP/bad.jsonl" | ||
| [ "$status" -ne 0 ] | ||
| [[ "$output" == *"additional"* || "$output" == *"surprise"* ]] | ||
| } | ||
|
|
||
| @test "validate-cases rejects an invalid risk enum value" { | ||
| jq -c '.expected.risk = "CATASTROPHIC"' <(head -1 "$CASES") >"$TMP/bad.jsonl" | ||
| run python3 "$VALIDATOR" "$TMP/bad.jsonl" | ||
| [ "$status" -ne 0 ] | ||
| } | ||
|
|
||
| @test "validate-cases rejects a non-boolean escalate" { | ||
| jq -c '.expected.escalate = "yes"' <(head -1 "$CASES") >"$TMP/bad.jsonl" | ||
| run python3 "$VALIDATOR" "$TMP/bad.jsonl" | ||
| [ "$status" -ne 0 ] | ||
| } | ||
|
|
||
| @test "validate-cases rejects duplicate case ids" { | ||
| head -1 "$CASES" >"$TMP/dup.jsonl" | ||
| head -1 "$CASES" >>"$TMP/dup.jsonl" | ||
| run python3 "$VALIDATOR" "$TMP/dup.jsonl" | ||
| [ "$status" -ne 0 ] | ||
| [[ "$output" == *"unique"* || "$output" == *"duplicate"* ]] | ||
| } | ||
|
|
||
| @test "case schema validates with the same python jsonschema pattern as plan.schema.json" { | ||
| run python3 -c "import json,jsonschema,sys; jsonschema.Draft202012Validator.check_schema(json.load(open('$SCHEMA')))" | ||
| [ "$status" -eq 0 ] | ||
| } | ||
|
|
||
| @test "seed cases cover both escalate outcomes" { | ||
| has_true="$(jq -s 'any(.[]; .expected.escalate == true)' "$CASES")" | ||
| has_false="$(jq -s 'any(.[]; .expected.escalate == false)' "$CASES")" | ||
| [ "$has_true" = "true" ] | ||
| [ "$has_false" = "true" ] | ||
| } | ||
|
|
||
| @test "seed cases include a case for each named high-risk trigger" { | ||
| tags="$(jq -r 'select(.tags != null) | .tags[]' "$CASES" | sort -u)" | ||
| grep -qx 'auth-secrets' <<<"$tags" | ||
| grep -qx 'db-migration' <<<"$tags" | ||
| grep -qx 'security-anti-pattern' <<<"$tags" | ||
| } | ||
|
|
||
| @test "every high-risk (HIGH) seed case escalates" { | ||
| # A HIGH classification must never be paired with escalate=false: triage.md | ||
| # treats any HIGH signal as an always-escalate condition. | ||
| bad="$(jq -s '[.[] | select(.expected.risk == "HIGH" and .expected.escalate == false)] | length' "$CASES")" | ||
| [ "$bad" -eq 0 ] | ||
| } | ||
|
don-petry marked this conversation as resolved.
Resolved
|
||
|
|
||
| @test "validate-cases rejects a case with HIGH risk but escalate false" { | ||
| jq -c '.expected.risk = "HIGH" | .expected.escalate = false' <(head -1 "$CASES") >"$TMP/bad.jsonl" | ||
| run python3 "$VALIDATOR" "$TMP/bad.jsonl" | ||
| [ "$status" -ne 0 ] | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.