Repository navigation
feat: implement issue #585 — [Phase 2] Extend the harness to prompts/deep-review.md via an LLM-judge scorer #745
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
f846114
feat: implement issue #585 — [Phase 2] Extend the harness to prompts/…
donpetry-bot 8fc1e1a
fix(reviews): address review comments [skip ci-relay]
donpetry-bot 4833817
chore: apply manual instructions [skip ci-relay]
donpetry-bot 8af004a
Merge branch 'main' into dev-lead/issue-585-20260614-2325
don-petry dad627c
Merge branch 'main' into dev-lead/issue-585-20260614-2325
don-petry 41e7c78
Merge branch 'main' into dev-lead/issue-585-20260614-2325
don-petry File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| {"id":"deep-dev-migration-escalate","description":"Adds a SQL migration altering the users schema — HIGH per shared.md (DB migrations/schema). Proposer-visible dev case; never scored by the gate.","tags":["db-migration"],"input":"Title: Add nullable phone column to users\n\nFiles changed (2):\n migrations/0007_add_user_phone.sql (+9/-0)\n src/models/user.ts (+3/-0)\n\nDiff adds a new SQL migration under migrations/ that alters the users table schema. Linked issue #251. CI: green.\n\nTriage result: {\"escalate\": true, \"risk\": \"HIGH\", \"signals\": [\"db-migration\"], \"summary\": \"users schema migration\"}","expected":{"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["adds a DB migration that alters the users table schema (migrations/), a HIGH-risk area that always escalates to the security audit tier"]}} | ||
| {"id":"deep-dev-tests-only-approve","description":"Test-only addition addressing its linked issue; LOW risk, all gates pass -> approve. Proposer-visible dev case.","input":"Title: Add unit tests for date formatter\n\nFiles changed (1):\n src/utils/format_date.test.ts (+42/-0)\n\nDiff adds test cases only; no production code is touched. Linked issue #210 (add coverage for the formatter) is addressed. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": false, \"risk\": \"LOW\", \"signals\": [], \"summary\": \"tests only\"}","expected":{"decision":"approve","risk":"LOW","escalate_to_opus":false,"key_findings":["LOW-risk test-only change addressing issue #210 with all gates green; deep review approves"]}} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| {"id":"deep-auth-session-rework","description":"Reworks session-token issuance under src/auth/** — HIGH per shared.md (auth/secrets/session). deep-review.md Decision: HIGH always escalates to the security audit tier.","tags":["auth-secrets"],"input":"Title: Rotate session tokens on privilege change\n\nFiles changed (2):\n src/auth/session.ts (+58/-12)\n src/auth/tokens.ts (+20/-4)\n\nDiff reworks session-token issuance and rotation in the authentication layer; both files live under src/auth/. Linked issue #240 (rotate tokens on role change) is addressed. CI: all green. No unresolved review threads.\n\nTriage result: {\"escalate\": true, \"risk\": \"HIGH\", \"signals\": [\"auth\"], \"summary\": \"session-token rotation in auth layer\"}","expected":{"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["touches authentication/session-token handling under src/auth/**, a HIGH-risk area that must escalate to the security auditor regardless of how clean the diff looks"]}} | ||
| {"id":"deep-sql-string-concat","description":"SQL query built by string concatenation of input — security anti-pattern per shared.md HIGH taxonomy (SQL injection). Must escalate even though no auth/migration files are touched.","tags":["security-anti-pattern"],"input":"Title: Add user search endpoint\n\nFiles changed (1):\n src/api/search.py (+18/-0)\n\nDiff:\n+def search(user_id):\n+ q = 'SELECT * FROM users WHERE id = ' + user_id\n+ return db.execute(q)\n\nNo auth or migration files are touched, but the query is assembled by string concatenation of the request input. Linked issue #260. CI: green.\n\nTriage result: {\"escalate\": true, \"risk\": \"HIGH\", \"signals\": [\"sql-injection\"], \"summary\": \"query built by string concatenation\"}","expected":{"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["SQL query is built by string concatenation of untrusted input (search(user_id) -> 'SELECT ... WHERE id = ' + user_id), a SQL-injection anti-pattern","recommend a parameterized/prepared query instead of concatenation"]}} | ||
| {"id":"deep-clean-logic-approve","description":"Non-trivial but clean logic fix; MEDIUM risk, no high-risk area or anti-pattern, all decision gates pass -> approve (do not escalate). Grounds deep-review.md Decision: LOW/MEDIUM + gates pass = approve.","input":"Title: Cap retry backoff at 30s\n\nFiles changed (1):\n src/net/retry.ts (+6/-2)\n\nDiff clamps exponential backoff to a 30s ceiling. No auth, secrets, DB, or workflow files touched and no security anti-patterns. Linked issue #233 (runaway backoff) is addressed by the diff. Clear title and single coherent purpose. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": true, \"risk\": \"MEDIUM\", \"signals\": [\"non-trivial-logic\"], \"summary\": \"backoff clamp\"}","expected":{"decision":"approve","risk":"MEDIUM","escalate_to_opus":false,"key_findings":["MEDIUM-risk logic change with no high-risk area or anti-pattern; all decision gates pass (CI green, issue #233 addressed, well-structured), so deep review may approve without escalating to the security audit tier"]}} | ||
| {"id":"deep-docs-only-approve","description":"Docs-only change; LOW risk, all gates pass -> approve. Grounds the LOW branch of deep-review.md Decision.","input":"Title: Fix typo in README quick-start\n\nFiles changed (1):\n README.md (+1/-1)\n\nDiff fixes a duplicated word in the installation instructions. No code, auth, DB, or workflow files touched. Linked issue: none required for a docs typo. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": false, \"risk\": \"LOW\", \"signals\": [], \"summary\": \"docs typo\"}","expected":{"decision":"approve","risk":"LOW","escalate_to_opus":false,"key_findings":["LOW-risk docs-only change with all gates green; deep review approves"]}} | ||
| {"id":"deep-medium-gate-fail-escalate","description":"MEDIUM risk but a decision gate fails (unresolved review thread requesting changes). Per deep-review.md: LOW/MEDIUM with a failing gate -> escalate on own findings, no security-audit tier needed.","tags":["unresolved-threads"],"input":"Title: Refactor pagination helper\n\nFiles changed (1):\n src/api/paginate.ts (+24/-18)\n\nNo high-risk areas are touched and there are no security anti-patterns. However an unresolved review thread requests changes to an off-by-one in the page-size clamp that has not been addressed. Linked issue #270. CI: green.\n\nTriage result: {\"escalate\": true, \"risk\": \"MEDIUM\", \"signals\": [\"unresolved-thread\"], \"summary\": \"pagination refactor with open thread\"}","expected":{"decision":"escalate","risk":"MEDIUM","escalate_to_opus":false,"key_findings":["a decision gate fails: an unresolved review thread requests changes to an off-by-one in the page-size clamp that is not yet addressed, so deep review escalates rather than approves","risk stays MEDIUM and the security-audit tier is not required — deep review's own findings are sufficient"]}} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| { | ||
| "mode": "llm-judge", | ||
| "judge_prompt": "judge.md", | ||
| "pass_threshold": 0.7 | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # LLM-judge — deep-review scoring rubric | ||
|
|
||
| You are an automated **grader** for a PR-review skill (`prompts/deep-review.md`). | ||
| You are given an **expected reference** (what a correct review must conclude) and a | ||
| **candidate output** (what the skill actually produced for one held-out case). Your | ||
| job is to score how well the candidate matches the expected reference and emit a | ||
| single numeric score. You are NOT reviewing the PR yourself — you only grade the | ||
| candidate against the reference. | ||
|
|
||
| The deep-review skill emits a JSON verdict with at least `decision` | ||
| (`approve` | `escalate`), `risk` (`LOW` | `MEDIUM` | `HIGH`), and a `findings` | ||
| array. The expected reference carries the correct `decision`, `risk`, an | ||
| `escalate_to_opus` flag, and a list of `key_findings` the review must surface. | ||
|
|
||
| ## Risk taxonomy (the same one the skill is graded against) | ||
|
|
||
| Use `prompts/shared.md`'s shared risk taxonomy as the ground truth: | ||
|
|
||
| - **HIGH** (always escalate, never approve): auth / secrets / credentials / crypto | ||
| / tokens / session handling / `.env*`; DB migrations or schema; security | ||
| anti-patterns (SQL string concatenation, eval/exec on untrusted input, | ||
| `shell=True`, hardcoded secrets, disabled TLS verification, etc.); CI security | ||
| scanner warnings; org/standards violations; GitHub Actions security smells. | ||
| - **MEDIUM**: non-trivial logic changes, new deps, cross-module refactors. | ||
| - **LOW**: docs, comments, typos, tests-only, lockfile updates. | ||
|
|
||
| A correct review may **approve** only when risk is LOW or MEDIUM **and** every | ||
| decision gate passes (CI green, linked issue addressed, no unresolved threads, | ||
| well-structured). Otherwise it must **escalate**. | ||
|
|
||
| ## How to score (0.0 – 1.0) | ||
|
|
||
| Weigh the candidate against the expected reference: | ||
|
|
||
| 1. **Decision (approve vs escalate) — most important.** A candidate that gets the | ||
| approve-vs-escalate decision wrong cannot score above **0.3**, no matter how | ||
| good its prose. A correct decision is worth roughly **0.5** of the score. | ||
| 2. **Risk classification.** Matching the expected `risk` (and, for HIGH, | ||
| `escalate_to_opus`/route to the security-audit tier) is worth roughly **0.2**. | ||
| 3. **Key findings.** Award the remaining **~0.3** in proportion to how many of the | ||
| expected `key_findings` the candidate substantively surfaces (semantic match, | ||
| not exact wording). Missing the central finding (e.g. naming the actual | ||
| SQL-injection vector) should cost most of this band. | ||
|
|
||
| Do not reward extra, irrelevant, or hallucinated findings. Do not penalize a | ||
| candidate for additional *correct* detail beyond the reference. | ||
|
|
||
| ## Output (MANDATORY) | ||
|
|
||
| Emit **exactly one** JSON object to stdout and nothing else. Output raw JSON — do not wrap it in prose, code fences, or any markdown formatting: | ||
|
|
||
| {"score": 0.0, "reason": "one sentence on what matched and what was missing"} | ||
|
|
||
| `score` MUST be a number in `[0, 1]`. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.