Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions evals/judge.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,15 @@ Use `prompts/shared.md`'s shared risk taxonomy as the ground truth:
anti-patterns (SQL string concatenation, eval/exec on untrusted input,
`shell=True`, hardcoded secrets, disabled TLS verification, etc.); CI security
scanner warnings; org/standards violations; GitHub Actions security smells.
- **MEDIUM**: non-trivial logic changes, new deps, cross-module refactors.
- **MEDIUM**: non-trivial logic changes, new deps.
- **LOW**: docs, comments, typos, tests-only, lockfile updates.

A correct review may **approve** only when risk is LOW or MEDIUM **and** every
decision gate passes (CI green, linked issue addressed, no unresolved threads,
well-structured). Otherwise it must **escalate**.
well-structured). Otherwise it must **escalate**. A **cross-module / boundary-crossing
refactor** is MEDIUM-risk but is **not** on the auto-approve path: it alters system
shape, so a correct review escalates it for human + architectural review rather than
approving.

## How to score (0.0 – 1.0)

Expand Down
2 changes: 2 additions & 0 deletions evals/pr-review/holdout/cases.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,5 @@
{"id":"triage-auth-token","description":"Touches authentication/session tokens under src/auth/** (criterion 1, high-risk area) -> always escalate, HIGH.","tags":["auth-secrets"],"input":"Title: Rotate session tokens on privilege change\n\nFiles changed (2):\n src/auth/session.ts (+58/-12)\n src/auth/tokens.ts (+20/-4)\n\nDiff reworks session-token issuance and rotation in the authentication layer; both files live under src/auth/. Linked issue #240. CI green.","expected":{"escalate":true,"risk":"HIGH"}}
{"id":"triage-db-migration","description":"Adds a SQL migration under migrations/ altering the users schema (criterion 1, high-risk area) -> always escalate, HIGH.","tags":["db-migration"],"input":"Title: Add nullable phone column to users\n\nFiles changed (2):\n migrations/0007_add_user_phone.sql (+9/-0)\n src/models/user.ts (+3/-0)\n\nDiff adds a new SQL migration under migrations/ that alters the users table schema. Linked issue #251. CI green.","expected":{"escalate":true,"risk":"HIGH"}}
{"id":"triage-sql-concat","description":"Builds a SQL query by string concatenation of input (criterion 3, security anti-pattern) -> escalate, HIGH.","tags":["security-anti-pattern"],"input":"Title: Add user search endpoint\n\nFiles changed (1):\n src/api/search.py (+18/-0)\n\nDiff:\n+def search(user_id):\n+ q = 'SELECT * FROM users WHERE id = ' + user_id\n+ return db.execute(q)\n\nNo auth or migration files are touched, but the query is assembled by string concatenation of the input. Linked issue #260.","expected":{"escalate":true,"risk":"HIGH"}}
{"id":"triage-boundary-refactor-escalate","description":"Clean structural refactor that moves code across module boundaries; no high-risk area or anti-pattern and every other gate is green, but a boundary-crossing refactor is no longer auto-approvable per shared.md — it escalates for human + architectural review. Locks the #1632 reclassification.","tags":["boundary-refactor"],"input":"Title: Move retry/backoff helpers from src/net into new src/resilience module\n\nFiles changed (4):\n src/resilience/retry.ts (+96/-0)\n src/resilience/backoff.ts (+54/-0)\n src/net/retry.ts (+0/-92)\n src/net/client.ts (+8/-8)\n\nA pure structural refactor: it relocates the retry/backoff helpers out of src/net into a new src/resilience module and repoints every importer, crossing module boundaries. No auth, secrets, DB, or workflow files are touched and there are no security anti-patterns. Linked issue #275 (carve resilience helpers into their own module) is addressed. Clear title and single coherent purpose. No unresolved review threads. CI: all green.","expected":{"escalate":true,"risk":"MEDIUM"}}
Comment thread
don-petry marked this conversation as resolved.
Resolved
{"id": "deep-boundary-refactor-escalate", "description": "Deep-tier mirror of triage-boundary-refactor-escalate. A clean, behavior-preserving refactor that crosses module boundaries with every other gate green. Per shared.md decision gate 7 the deep reviewer must escalate on structural grounds (reason code boundary-refactor), NOT approve and NOT reclassify as poorly-structured. Locks the #1632 reclassification into the deep tier, where MEDIUM-escalate nuance is adjudicated.", "tags": ["boundary-refactor"], "input": "Title: Move retry/backoff helpers from src/net into new src/resilience module\n\nFiles changed (4):\n src/resilience/retry.ts (+96/-0)\n src/resilience/backoff.ts (+54/-0)\n src/net/retry.ts (+0/-92)\n src/net/client.ts (+8/-8)\n\nA pure structural refactor: it relocates the retry/backoff helpers out of src/net into a new src/resilience module and repoints every importer, crossing module boundaries. No auth, secrets, DB, or workflow files are touched and there are no security anti-patterns. Linked issue #275 (carve resilience helpers into their own module) is addressed. Clear title and single coherent purpose. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": true, \"risk\": \"MEDIUM\", \"signals\": [\"boundary-refactor\"], \"summary\": \"behavior-preserving move of retry helpers across module boundaries\"}", "expected": {"decision": "escalate", "risk": "MEDIUM", "escalate_to_opus": false, "key_findings": ["decision gate 7 fails: the change crosses module boundaries and alters system shape, so deep review escalates for human and architectural review even though every other gate is green", "the correct reason code is boundary-refactor, not poorly-structured \u2014 the PR is well structured; it escalates on structural-change grounds, not quality grounds", "risk stays MEDIUM and the security-audit tier is not required \u2014 no high-risk area or anti-pattern is touched"]}}
14 changes: 12 additions & 2 deletions prompts/shared.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,13 @@ CI security warning) still apply and still escalate.

- Non-trivial logic changes in application code that aren't HIGH.
- New dependencies (non security-sensitive).
- Refactors crossing module boundaries.
- Test changes coupled with logic changes.

A refactor that **crosses module boundaries** is MEDIUM-risk but is **not** on the
auto-approve path: a boundary-crossing/structural refactor alters the shape of the
system, so it escalates for human and architectural review (decision gate 7 below)
rather than being auto-approved.

### LOW (auto-approve allowed if all gates pass)

- Docs only (`*.md`, `docs/**`).
Expand All @@ -121,6 +125,12 @@ You may recommend `approve` only if ALL of:
4. No unresolved review threads requesting changes.
5. No unanswered questions in human-reviewer comments.
6. PR is well-structured: clear title, description, single coherent purpose.
7. The change is not a boundary-crossing / structural refactor. A refactor that
crosses module boundaries alters the shape of the system, so it escalates for
human and architectural review rather than being auto-approved — even when
every other gate passes. Report this gate with the `boundary-refactor`
reason code, not `poorly-structured` — the change may be perfectly well
structured; it escalates because it changes system shape.

Otherwise → `escalate`.

Expand All @@ -146,7 +156,7 @@ Do not print the JSON to stdout — write it to the file. Schema:
"head_sha": "<the SHA you reviewed>",
"risk": "LOW|MEDIUM|HIGH",
"decision": "approve|escalate|skip",
"reason_codes": ["high-risk-content"|"ci-failing"|"issue-not-addressed"|"unresolved-threads"|"poorly-structured"|"no-linked-issue"|"draft"|"head-sha-changed"|"none"],
"reason_codes": ["high-risk-content"|"ci-failing"|"issue-not-addressed"|"unresolved-threads"|"poorly-structured"|"no-linked-issue"|"draft"|"head-sha-changed"|"boundary-refactor"|"none"],
"summary": "<2-3 sentence summary from your lens's perspective>",
"findings": [
{
Expand Down
Loading