Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
# Explicit rule so the holdout stays owner-locked even if broader rules loosen;
# the dev/ split may remain proposer-visible.
/evals/**/holdout/ @petry-projects/org-leads

# LLM-judge scoring prompt(s) (#585, epic #581) — a skill-quality input on a par
# with the held-out cases: the rubric the candidate is graded against. Owner-lock
# it explicitly so the proposer can never weaken the judge it must satisfy.
/evals/judge.md @petry-projects/org-leads
Comment thread
don-petry marked this conversation as resolved.
Resolved
/evals/**/scorer.json @petry-projects/org-leads
2 changes: 2 additions & 0 deletions evals/deep-review/dev/cases.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"id":"deep-dev-migration-escalate","description":"Adds a SQL migration altering the users schema — HIGH per shared.md (DB migrations/schema). Proposer-visible dev case; never scored by the gate.","tags":["db-migration"],"input":"Title: Add nullable phone column to users\n\nFiles changed (2):\n migrations/0007_add_user_phone.sql (+9/-0)\n src/models/user.ts (+3/-0)\n\nDiff adds a new SQL migration under migrations/ that alters the users table schema. Linked issue #251. CI: green.\n\nTriage result: {\"escalate\": true, \"risk\": \"HIGH\", \"signals\": [\"db-migration\"], \"summary\": \"users schema migration\"}","expected":{"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["adds a DB migration that alters the users table schema (migrations/), a HIGH-risk area that always escalates to the security audit tier"]}}
{"id":"deep-dev-tests-only-approve","description":"Test-only addition addressing its linked issue; LOW risk, all gates pass -> approve. Proposer-visible dev case.","input":"Title: Add unit tests for date formatter\n\nFiles changed (1):\n src/utils/format_date.test.ts (+42/-0)\n\nDiff adds test cases only; no production code is touched. Linked issue #210 (add coverage for the formatter) is addressed. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": false, \"risk\": \"LOW\", \"signals\": [], \"summary\": \"tests only\"}","expected":{"decision":"approve","risk":"LOW","escalate_to_opus":false,"key_findings":["LOW-risk test-only change addressing issue #210 with all gates green; deep review approves"]}}
5 changes: 5 additions & 0 deletions evals/deep-review/holdout/cases.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{"id":"deep-auth-session-rework","description":"Reworks session-token issuance under src/auth/** — HIGH per shared.md (auth/secrets/session). deep-review.md Decision: HIGH always escalates to the security audit tier.","tags":["auth-secrets"],"input":"Title: Rotate session tokens on privilege change\n\nFiles changed (2):\n src/auth/session.ts (+58/-12)\n src/auth/tokens.ts (+20/-4)\n\nDiff reworks session-token issuance and rotation in the authentication layer; both files live under src/auth/. Linked issue #240 (rotate tokens on role change) is addressed. CI: all green. No unresolved review threads.\n\nTriage result: {\"escalate\": true, \"risk\": \"HIGH\", \"signals\": [\"auth\"], \"summary\": \"session-token rotation in auth layer\"}","expected":{"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["touches authentication/session-token handling under src/auth/**, a HIGH-risk area that must escalate to the security auditor regardless of how clean the diff looks"]}}
{"id":"deep-sql-string-concat","description":"SQL query built by string concatenation of input — security anti-pattern per shared.md HIGH taxonomy (SQL injection). Must escalate even though no auth/migration files are touched.","tags":["security-anti-pattern"],"input":"Title: Add user search endpoint\n\nFiles changed (1):\n src/api/search.py (+18/-0)\n\nDiff:\n+def search(user_id):\n+ q = 'SELECT * FROM users WHERE id = ' + user_id\n+ return db.execute(q)\n\nNo auth or migration files are touched, but the query is assembled by string concatenation of the request input. Linked issue #260. CI: green.\n\nTriage result: {\"escalate\": true, \"risk\": \"HIGH\", \"signals\": [\"sql-injection\"], \"summary\": \"query built by string concatenation\"}","expected":{"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["SQL query is built by string concatenation of untrusted input (search(user_id) -> 'SELECT ... WHERE id = ' + user_id), a SQL-injection anti-pattern","recommend a parameterized/prepared query instead of concatenation"]}}
{"id":"deep-clean-logic-approve","description":"Non-trivial but clean logic fix; MEDIUM risk, no high-risk area or anti-pattern, all decision gates pass -> approve (do not escalate). Grounds deep-review.md Decision: LOW/MEDIUM + gates pass = approve.","input":"Title: Cap retry backoff at 30s\n\nFiles changed (1):\n src/net/retry.ts (+6/-2)\n\nDiff clamps exponential backoff to a 30s ceiling. No auth, secrets, DB, or workflow files touched and no security anti-patterns. Linked issue #233 (runaway backoff) is addressed by the diff. Clear title and single coherent purpose. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": true, \"risk\": \"MEDIUM\", \"signals\": [\"non-trivial-logic\"], \"summary\": \"backoff clamp\"}","expected":{"decision":"approve","risk":"MEDIUM","escalate_to_opus":false,"key_findings":["MEDIUM-risk logic change with no high-risk area or anti-pattern; all decision gates pass (CI green, issue #233 addressed, well-structured), so deep review may approve without escalating to the security audit tier"]}}
{"id":"deep-docs-only-approve","description":"Docs-only change; LOW risk, all gates pass -> approve. Grounds the LOW branch of deep-review.md Decision.","input":"Title: Fix typo in README quick-start\n\nFiles changed (1):\n README.md (+1/-1)\n\nDiff fixes a duplicated word in the installation instructions. No code, auth, DB, or workflow files touched. Linked issue: none required for a docs typo. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": false, \"risk\": \"LOW\", \"signals\": [], \"summary\": \"docs typo\"}","expected":{"decision":"approve","risk":"LOW","escalate_to_opus":false,"key_findings":["LOW-risk docs-only change with all gates green; deep review approves"]}}
{"id":"deep-medium-gate-fail-escalate","description":"MEDIUM risk but a decision gate fails (unresolved review thread requesting changes). Per deep-review.md: LOW/MEDIUM with a failing gate -> escalate on own findings, no security-audit tier needed.","tags":["unresolved-threads"],"input":"Title: Refactor pagination helper\n\nFiles changed (1):\n src/api/paginate.ts (+24/-18)\n\nNo high-risk areas are touched and there are no security anti-patterns. However an unresolved review thread requests changes to an off-by-one in the page-size clamp that has not been addressed. Linked issue #270. CI: green.\n\nTriage result: {\"escalate\": true, \"risk\": \"MEDIUM\", \"signals\": [\"unresolved-thread\"], \"summary\": \"pagination refactor with open thread\"}","expected":{"decision":"escalate","risk":"MEDIUM","escalate_to_opus":false,"key_findings":["a decision gate fails: an unresolved review thread requests changes to an off-by-one in the page-size clamp that is not yet addressed, so deep review escalates rather than approves","risk stays MEDIUM and the security-audit tier is not required — deep review's own findings are sufficient"]}}
5 changes: 5 additions & 0 deletions evals/deep-review/scorer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"mode": "llm-judge",
"judge_prompt": "judge.md",
"pass_threshold": 0.7
}
54 changes: 54 additions & 0 deletions evals/judge.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# LLM-judge — deep-review scoring rubric

You are an automated **grader** for a PR-review skill (`prompts/deep-review.md`).
You are given an **expected reference** (what a correct review must conclude) and a
**candidate output** (what the skill actually produced for one held-out case). Your
job is to score how well the candidate matches the expected reference and emit a
single numeric score. You are NOT reviewing the PR yourself — you only grade the
candidate against the reference.

The deep-review skill emits a JSON verdict with at least `decision`
(`approve` | `escalate`), `risk` (`LOW` | `MEDIUM` | `HIGH`), and a `findings`
array. The expected reference carries the correct `decision`, `risk`, an
`escalate_to_opus` flag, and a list of `key_findings` the review must surface.

## Risk taxonomy (the same one the skill is graded against)

Use `prompts/shared.md`'s shared risk taxonomy as the ground truth:

- **HIGH** (always escalate, never approve): auth / secrets / credentials / crypto
/ tokens / session handling / `.env*`; DB migrations or schema; security
anti-patterns (SQL string concatenation, eval/exec on untrusted input,
`shell=True`, hardcoded secrets, disabled TLS verification, etc.); CI security
scanner warnings; org/standards violations; GitHub Actions security smells.
- **MEDIUM**: non-trivial logic changes, new deps, cross-module refactors.
- **LOW**: docs, comments, typos, tests-only, lockfile updates.

A correct review may **approve** only when risk is LOW or MEDIUM **and** every
decision gate passes (CI green, linked issue addressed, no unresolved threads,
well-structured). Otherwise it must **escalate**.

## How to score (0.0 – 1.0)

Weigh the candidate against the expected reference:

1. **Decision (approve vs escalate) — most important.** A candidate that gets the
approve-vs-escalate decision wrong cannot score above **0.3**, no matter how
good its prose. A correct decision is worth roughly **0.5** of the score.
2. **Risk classification.** Matching the expected `risk` (and, for HIGH,
`escalate_to_opus`/route to the security-audit tier) is worth roughly **0.2**.
3. **Key findings.** Award the remaining **~0.3** in proportion to how many of the
expected `key_findings` the candidate substantively surfaces (semantic match,
not exact wording). Missing the central finding (e.g. naming the actual
SQL-injection vector) should cost most of this band.

Do not reward extra, irrelevant, or hallucinated findings. Do not penalize a
candidate for additional *correct* detail beyond the reference.

## Output (MANDATORY)

Emit **exactly one** JSON object to stdout and nothing else. Output raw JSON — do not wrap it in prose, code fences, or any markdown formatting:

{"score": 0.0, "reason": "one sentence on what matched and what was missing"}

`score` MUST be a number in `[0, 1]`.
Loading
Loading