From 060669948caa26cc342a531fbf8f0e7ce33ea55e Mon Sep 17 00:00:00 2001 From: don-petry <36422719+don-petry@users.noreply.github.com> Date: Tue, 1 Sep 2026 11:09:14 +0000 Subject: [PATCH 1/2] =?UTF-8?q?feat:=20implement=20issue=20#1632=20?= =?UTF-8?q?=E2=80=94=20[Phase=203]=20Reclassify=20boundary-crossing=20refa?= =?UTF-8?q?ctors=20out=20of=20the=20auto-approvable=20MEDIUM=20tier=20in?= =?UTF-8?q?=20prompts/shared.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- evals/judge.md | 7 +++++-- evals/pr-review/holdout/cases.jsonl | 1 + prompts/shared.md | 10 +++++++++- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/evals/judge.md b/evals/judge.md index 253cbbd7e..e2e941f0c 100644 --- a/evals/judge.md +++ b/evals/judge.md @@ -21,12 +21,15 @@ Use `prompts/shared.md`'s shared risk taxonomy as the ground truth: anti-patterns (SQL string concatenation, eval/exec on untrusted input, `shell=True`, hardcoded secrets, disabled TLS verification, etc.); CI security scanner warnings; org/standards violations; GitHub Actions security smells. -- **MEDIUM**: non-trivial logic changes, new deps, cross-module refactors. +- **MEDIUM**: non-trivial logic changes, new deps. - **LOW**: docs, comments, typos, tests-only, lockfile updates. A correct review may **approve** only when risk is LOW or MEDIUM **and** every decision gate passes (CI green, linked issue addressed, no unresolved threads, -well-structured). Otherwise it must **escalate**. +well-structured). Otherwise it must **escalate**. A **cross-module / boundary-crossing +refactor** is MEDIUM-risk but is **not** on the auto-approve path: it alters system +shape, so a correct review escalates it for human + architectural review rather than +approving. ## How to score (0.0 – 1.0) diff --git a/evals/pr-review/holdout/cases.jsonl b/evals/pr-review/holdout/cases.jsonl index 6ff395a35..a28310de7 100644 --- a/evals/pr-review/holdout/cases.jsonl +++ b/evals/pr-review/holdout/cases.jsonl @@ -8,3 +8,4 @@ {"id":"triage-auth-token","description":"Touches authentication/session tokens under src/auth/** (criterion 1, high-risk area) -> always escalate, HIGH.","tags":["auth-secrets"],"input":"Title: Rotate session tokens on privilege change\n\nFiles changed (2):\n src/auth/session.ts (+58/-12)\n src/auth/tokens.ts (+20/-4)\n\nDiff reworks session-token issuance and rotation in the authentication layer; both files live under src/auth/. Linked issue #240. CI green.","expected":{"escalate":true,"risk":"HIGH"}} {"id":"triage-db-migration","description":"Adds a SQL migration under migrations/ altering the users schema (criterion 1, high-risk area) -> always escalate, HIGH.","tags":["db-migration"],"input":"Title: Add nullable phone column to users\n\nFiles changed (2):\n migrations/0007_add_user_phone.sql (+9/-0)\n src/models/user.ts (+3/-0)\n\nDiff adds a new SQL migration under migrations/ that alters the users table schema. Linked issue #251. CI green.","expected":{"escalate":true,"risk":"HIGH"}} {"id":"triage-sql-concat","description":"Builds a SQL query by string concatenation of input (criterion 3, security anti-pattern) -> escalate, HIGH.","tags":["security-anti-pattern"],"input":"Title: Add user search endpoint\n\nFiles changed (1):\n src/api/search.py (+18/-0)\n\nDiff:\n+def search(user_id):\n+ q = 'SELECT * FROM users WHERE id = ' + user_id\n+ return db.execute(q)\n\nNo auth or migration files are touched, but the query is assembled by string concatenation of the input. Linked issue #260.","expected":{"escalate":true,"risk":"HIGH"}} +{"id":"triage-boundary-refactor-escalate","description":"Clean structural refactor that moves code across module boundaries; no high-risk area or anti-pattern and every other gate is green, but a boundary-crossing refactor is no longer auto-approvable per shared.md — it escalates for human + architectural review. Locks the #1632 reclassification.","tags":["boundary-refactor"],"input":"Title: Move retry/backoff helpers from src/net into new src/resilience module\n\nFiles changed (4):\n src/resilience/retry.ts (+96/-0)\n src/resilience/backoff.ts (+54/-0)\n src/net/retry.ts (+0/-92)\n src/net/client.ts (+8/-8)\n\nA pure structural refactor: it relocates the retry/backoff helpers out of src/net into a new src/resilience module and repoints every importer, crossing module boundaries. No auth, secrets, DB, or workflow files are touched and there are no security anti-patterns. Linked issue #275 (carve resilience helpers into their own module) is addressed. Clear title and single coherent purpose. No unresolved review threads. CI: all green.","expected":{"escalate":true,"risk":"MEDIUM"}} diff --git a/prompts/shared.md b/prompts/shared.md index 292ab0cda..07975eb59 100644 --- a/prompts/shared.md +++ b/prompts/shared.md @@ -99,9 +99,13 @@ CI security warning) still apply and still escalate. - Non-trivial logic changes in application code that aren't HIGH. - New dependencies (non security-sensitive). -- Refactors crossing module boundaries. - Test changes coupled with logic changes. +A refactor that **crosses module boundaries** is MEDIUM-risk but is **not** on the +auto-approve path: a boundary-crossing/structural refactor alters the shape of the +system, so it escalates for human and architectural review (decision gate 7 below) +rather than being auto-approved. + ### LOW (auto-approve allowed if all gates pass) - Docs only (`*.md`, `docs/**`). @@ -121,6 +125,10 @@ You may recommend `approve` only if ALL of: 4. No unresolved review threads requesting changes. 5. No unanswered questions in human-reviewer comments. 6. PR is well-structured: clear title, description, single coherent purpose. +7. The change is not a boundary-crossing / structural refactor. A refactor that + crosses module boundaries alters the shape of the system, so it escalates for + human and architectural review rather than being auto-approved — even when + every other gate passes. Otherwise → `escalate`. From 3b54aee100384dc5efe6c5b557e88f382f45211a Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Wed, 2 Sep 2026 06:42:14 -0500 Subject: [PATCH 2/2] fix(shared): add boundary-refactor reason code + deep-tier holdout case MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the two review findings on this PR that the previous commit did not resolve (confirmed independently by pr-review and gemini-code-assist). 1. MAJOR — decision gate 7 escalates boundary-crossing refactors, but the reason_codes enum had no code to represent it, forcing an agent to emit an inaccurate code such as `poorly-structured` or `none`. Adds `boundary-refactor` to the enum and points gate 7 at it explicitly, with the reason the distinction matters: such a PR may be perfectly well structured — it escalates because it changes system shape, not because its quality is poor. 2. MINOR — the reclassification was locked only in the triage tier. The deep tier is where MEDIUM-escalate nuance is adjudicated, so adds `deep-boundary-refactor-escalate`, mirroring the existing triage case and the `deep-*` family's shape (decision/risk/escalate_to_opus/key_findings). Verified: evals/validate-cases.py green (13 skills, 108 cases, no cross-split id overlap); markdownlint clean. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015YTfTumtrmd5jdFX9W1XvT --- evals/pr-review/holdout/cases.jsonl | 1 + prompts/shared.md | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/evals/pr-review/holdout/cases.jsonl b/evals/pr-review/holdout/cases.jsonl index a28310de7..d1dad62c3 100644 --- a/evals/pr-review/holdout/cases.jsonl +++ b/evals/pr-review/holdout/cases.jsonl @@ -9,3 +9,4 @@ {"id":"triage-db-migration","description":"Adds a SQL migration under migrations/ altering the users schema (criterion 1, high-risk area) -> always escalate, HIGH.","tags":["db-migration"],"input":"Title: Add nullable phone column to users\n\nFiles changed (2):\n migrations/0007_add_user_phone.sql (+9/-0)\n src/models/user.ts (+3/-0)\n\nDiff adds a new SQL migration under migrations/ that alters the users table schema. Linked issue #251. CI green.","expected":{"escalate":true,"risk":"HIGH"}} {"id":"triage-sql-concat","description":"Builds a SQL query by string concatenation of input (criterion 3, security anti-pattern) -> escalate, HIGH.","tags":["security-anti-pattern"],"input":"Title: Add user search endpoint\n\nFiles changed (1):\n src/api/search.py (+18/-0)\n\nDiff:\n+def search(user_id):\n+ q = 'SELECT * FROM users WHERE id = ' + user_id\n+ return db.execute(q)\n\nNo auth or migration files are touched, but the query is assembled by string concatenation of the input. Linked issue #260.","expected":{"escalate":true,"risk":"HIGH"}} {"id":"triage-boundary-refactor-escalate","description":"Clean structural refactor that moves code across module boundaries; no high-risk area or anti-pattern and every other gate is green, but a boundary-crossing refactor is no longer auto-approvable per shared.md — it escalates for human + architectural review. Locks the #1632 reclassification.","tags":["boundary-refactor"],"input":"Title: Move retry/backoff helpers from src/net into new src/resilience module\n\nFiles changed (4):\n src/resilience/retry.ts (+96/-0)\n src/resilience/backoff.ts (+54/-0)\n src/net/retry.ts (+0/-92)\n src/net/client.ts (+8/-8)\n\nA pure structural refactor: it relocates the retry/backoff helpers out of src/net into a new src/resilience module and repoints every importer, crossing module boundaries. No auth, secrets, DB, or workflow files are touched and there are no security anti-patterns. Linked issue #275 (carve resilience helpers into their own module) is addressed. Clear title and single coherent purpose. No unresolved review threads. CI: all green.","expected":{"escalate":true,"risk":"MEDIUM"}} +{"id": "deep-boundary-refactor-escalate", "description": "Deep-tier mirror of triage-boundary-refactor-escalate. A clean, behavior-preserving refactor that crosses module boundaries with every other gate green. Per shared.md decision gate 7 the deep reviewer must escalate on structural grounds (reason code boundary-refactor), NOT approve and NOT reclassify as poorly-structured. Locks the #1632 reclassification into the deep tier, where MEDIUM-escalate nuance is adjudicated.", "tags": ["boundary-refactor"], "input": "Title: Move retry/backoff helpers from src/net into new src/resilience module\n\nFiles changed (4):\n src/resilience/retry.ts (+96/-0)\n src/resilience/backoff.ts (+54/-0)\n src/net/retry.ts (+0/-92)\n src/net/client.ts (+8/-8)\n\nA pure structural refactor: it relocates the retry/backoff helpers out of src/net into a new src/resilience module and repoints every importer, crossing module boundaries. No auth, secrets, DB, or workflow files are touched and there are no security anti-patterns. Linked issue #275 (carve resilience helpers into their own module) is addressed. Clear title and single coherent purpose. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": true, \"risk\": \"MEDIUM\", \"signals\": [\"boundary-refactor\"], \"summary\": \"behavior-preserving move of retry helpers across module boundaries\"}", "expected": {"decision": "escalate", "risk": "MEDIUM", "escalate_to_opus": false, "key_findings": ["decision gate 7 fails: the change crosses module boundaries and alters system shape, so deep review escalates for human and architectural review even though every other gate is green", "the correct reason code is boundary-refactor, not poorly-structured \u2014 the PR is well structured; it escalates on structural-change grounds, not quality grounds", "risk stays MEDIUM and the security-audit tier is not required \u2014 no high-risk area or anti-pattern is touched"]}} diff --git a/prompts/shared.md b/prompts/shared.md index 07975eb59..57369a866 100644 --- a/prompts/shared.md +++ b/prompts/shared.md @@ -128,7 +128,9 @@ You may recommend `approve` only if ALL of: 7. The change is not a boundary-crossing / structural refactor. A refactor that crosses module boundaries alters the shape of the system, so it escalates for human and architectural review rather than being auto-approved — even when - every other gate passes. + every other gate passes. Report this gate with the `boundary-refactor` + reason code, not `poorly-structured` — the change may be perfectly well + structured; it escalates because it changes system shape. Otherwise → `escalate`. @@ -154,7 +156,7 @@ Do not print the JSON to stdout — write it to the file. Schema: "head_sha": "", "risk": "LOW|MEDIUM|HIGH", "decision": "approve|escalate|skip", - "reason_codes": ["high-risk-content"|"ci-failing"|"issue-not-addressed"|"unresolved-threads"|"poorly-structured"|"no-linked-issue"|"draft"|"head-sha-changed"|"none"], + "reason_codes": ["high-risk-content"|"ci-failing"|"issue-not-addressed"|"unresolved-threads"|"poorly-structured"|"no-linked-issue"|"draft"|"head-sha-changed"|"boundary-refactor"|"none"], "summary": "<2-3 sentence summary from your lens's perspective>", "findings": [ {