diff --git a/evals/judge.md b/evals/judge.md index 253cbbd7e..e2e941f0c 100644 --- a/evals/judge.md +++ b/evals/judge.md @@ -21,12 +21,15 @@ Use `prompts/shared.md`'s shared risk taxonomy as the ground truth: anti-patterns (SQL string concatenation, eval/exec on untrusted input, `shell=True`, hardcoded secrets, disabled TLS verification, etc.); CI security scanner warnings; org/standards violations; GitHub Actions security smells. -- **MEDIUM**: non-trivial logic changes, new deps, cross-module refactors. +- **MEDIUM**: non-trivial logic changes, new deps. - **LOW**: docs, comments, typos, tests-only, lockfile updates. A correct review may **approve** only when risk is LOW or MEDIUM **and** every decision gate passes (CI green, linked issue addressed, no unresolved threads, -well-structured). Otherwise it must **escalate**. +well-structured). Otherwise it must **escalate**. A **cross-module / boundary-crossing +refactor** is MEDIUM-risk but is **not** on the auto-approve path: it alters system +shape, so a correct review escalates it for human + architectural review rather than +approving. ## How to score (0.0 – 1.0) diff --git a/evals/pr-review/holdout/cases.jsonl b/evals/pr-review/holdout/cases.jsonl index 6ff395a35..d1dad62c3 100644 --- a/evals/pr-review/holdout/cases.jsonl +++ b/evals/pr-review/holdout/cases.jsonl @@ -8,3 +8,5 @@ {"id":"triage-auth-token","description":"Touches authentication/session tokens under src/auth/** (criterion 1, high-risk area) -> always escalate, HIGH.","tags":["auth-secrets"],"input":"Title: Rotate session tokens on privilege change\n\nFiles changed (2):\n src/auth/session.ts (+58/-12)\n src/auth/tokens.ts (+20/-4)\n\nDiff reworks session-token issuance and rotation in the authentication layer; both files live under src/auth/. Linked issue #240. CI green.","expected":{"escalate":true,"risk":"HIGH"}} {"id":"triage-db-migration","description":"Adds a SQL migration under migrations/ altering the users schema (criterion 1, high-risk area) -> always escalate, HIGH.","tags":["db-migration"],"input":"Title: Add nullable phone column to users\n\nFiles changed (2):\n migrations/0007_add_user_phone.sql (+9/-0)\n src/models/user.ts (+3/-0)\n\nDiff adds a new SQL migration under migrations/ that alters the users table schema. Linked issue #251. CI green.","expected":{"escalate":true,"risk":"HIGH"}} {"id":"triage-sql-concat","description":"Builds a SQL query by string concatenation of input (criterion 3, security anti-pattern) -> escalate, HIGH.","tags":["security-anti-pattern"],"input":"Title: Add user search endpoint\n\nFiles changed (1):\n src/api/search.py (+18/-0)\n\nDiff:\n+def search(user_id):\n+ q = 'SELECT * FROM users WHERE id = ' + user_id\n+ return db.execute(q)\n\nNo auth or migration files are touched, but the query is assembled by string concatenation of the input. Linked issue #260.","expected":{"escalate":true,"risk":"HIGH"}} +{"id":"triage-boundary-refactor-escalate","description":"Clean structural refactor that moves code across module boundaries; no high-risk area or anti-pattern and every other gate is green, but a boundary-crossing refactor is no longer auto-approvable per shared.md — it escalates for human + architectural review. Locks the #1632 reclassification.","tags":["boundary-refactor"],"input":"Title: Move retry/backoff helpers from src/net into new src/resilience module\n\nFiles changed (4):\n src/resilience/retry.ts (+96/-0)\n src/resilience/backoff.ts (+54/-0)\n src/net/retry.ts (+0/-92)\n src/net/client.ts (+8/-8)\n\nA pure structural refactor: it relocates the retry/backoff helpers out of src/net into a new src/resilience module and repoints every importer, crossing module boundaries. No auth, secrets, DB, or workflow files are touched and there are no security anti-patterns. Linked issue #275 (carve resilience helpers into their own module) is addressed. Clear title and single coherent purpose. No unresolved review threads. CI: all green.","expected":{"escalate":true,"risk":"MEDIUM"}} +{"id": "deep-boundary-refactor-escalate", "description": "Deep-tier mirror of triage-boundary-refactor-escalate. A clean, behavior-preserving refactor that crosses module boundaries with every other gate green. Per shared.md decision gate 7 the deep reviewer must escalate on structural grounds (reason code boundary-refactor), NOT approve and NOT reclassify as poorly-structured. Locks the #1632 reclassification into the deep tier, where MEDIUM-escalate nuance is adjudicated.", "tags": ["boundary-refactor"], "input": "Title: Move retry/backoff helpers from src/net into new src/resilience module\n\nFiles changed (4):\n src/resilience/retry.ts (+96/-0)\n src/resilience/backoff.ts (+54/-0)\n src/net/retry.ts (+0/-92)\n src/net/client.ts (+8/-8)\n\nA pure structural refactor: it relocates the retry/backoff helpers out of src/net into a new src/resilience module and repoints every importer, crossing module boundaries. No auth, secrets, DB, or workflow files are touched and there are no security anti-patterns. Linked issue #275 (carve resilience helpers into their own module) is addressed. Clear title and single coherent purpose. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": true, \"risk\": \"MEDIUM\", \"signals\": [\"boundary-refactor\"], \"summary\": \"behavior-preserving move of retry helpers across module boundaries\"}", "expected": {"decision": "escalate", "risk": "MEDIUM", "escalate_to_opus": false, "key_findings": ["decision gate 7 fails: the change crosses module boundaries and alters system shape, so deep review escalates for human and architectural review even though every other gate is green", "the correct reason code is boundary-refactor, not poorly-structured \u2014 the PR is well structured; it escalates on structural-change grounds, not quality grounds", "risk stays MEDIUM and the security-audit tier is not required \u2014 no high-risk area or anti-pattern is touched"]}} diff --git a/prompts/shared.md b/prompts/shared.md index 292ab0cda..57369a866 100644 --- a/prompts/shared.md +++ b/prompts/shared.md @@ -99,9 +99,13 @@ CI security warning) still apply and still escalate. - Non-trivial logic changes in application code that aren't HIGH. - New dependencies (non security-sensitive). -- Refactors crossing module boundaries. - Test changes coupled with logic changes. +A refactor that **crosses module boundaries** is MEDIUM-risk but is **not** on the +auto-approve path: a boundary-crossing/structural refactor alters the shape of the +system, so it escalates for human and architectural review (decision gate 7 below) +rather than being auto-approved. + ### LOW (auto-approve allowed if all gates pass) - Docs only (`*.md`, `docs/**`). @@ -121,6 +125,12 @@ You may recommend `approve` only if ALL of: 4. No unresolved review threads requesting changes. 5. No unanswered questions in human-reviewer comments. 6. PR is well-structured: clear title, description, single coherent purpose. +7. The change is not a boundary-crossing / structural refactor. A refactor that + crosses module boundaries alters the shape of the system, so it escalates for + human and architectural review rather than being auto-approved — even when + every other gate passes. Report this gate with the `boundary-refactor` + reason code, not `poorly-structured` — the change may be perfectly well + structured; it escalates because it changes system shape. Otherwise → `escalate`. @@ -146,7 +156,7 @@ Do not print the JSON to stdout — write it to the file. Schema: "head_sha": "", "risk": "LOW|MEDIUM|HIGH", "decision": "approve|escalate|skip", - "reason_codes": ["high-risk-content"|"ci-failing"|"issue-not-addressed"|"unresolved-threads"|"poorly-structured"|"no-linked-issue"|"draft"|"head-sha-changed"|"none"], + "reason_codes": ["high-risk-content"|"ci-failing"|"issue-not-addressed"|"unresolved-threads"|"poorly-structured"|"no-linked-issue"|"draft"|"head-sha-changed"|"boundary-refactor"|"none"], "summary": "<2-3 sentence summary from your lens's perspective>", "findings": [ {