Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/pr-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,17 @@ jobs:
- name: Checkout agent repo
uses: actions/checkout@v5

- name: Install review engine CLI
- name: Install review engine CLIs
run: |
# Primary engine must install successfully; duck engine is best-effort.
case "$REVIEW_ENGINE" in
claude)
npm install -g @anthropic-ai/claude-code
gh extension install github/gh-copilot || true
;;
copilot)
gh extension install github/gh-copilot
npm install -g @anthropic-ai/claude-code || true
;;
*)
echo "::error::Unknown REVIEW_ENGINE=$REVIEW_ENGINE"
Expand Down
30 changes: 26 additions & 4 deletions AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,26 +21,48 @@ and **Copilot**.
Tier 1: Triage (~15s, no tools, pre-fetched context)
└─ clean? → single confirmation → approve + auto-merge
└─ concerns? ↓
Tier 2: Deep review (~2 min, full agentic)
Tier 2: Deep review + Rubber duck (~2 min, parallel, cross-engine)
└─ synthesize verdicts from both engines
└─ clean? → approve + auto-merge
└─ HIGH risk? ↓
Tier 3: Security audit (~3 min, full agentic)
└─ final decision → approve or escalate
```

### Cross-engine adversarial review (Rubber Duck)

At tier 2, two reviewers analyze the PR **in parallel** using different
model families. The primary engine runs the deep review; the **opposite**
engine runs an adversarial "rubber duck" review. A synthesis step merges
both verdicts before deciding whether to approve or escalate.

This approach is inspired by [GitHub Copilot CLI's rubber duck feature](https://github.blog/ai-and-ml/github-copilot/github-copilot-cli-combines-model-families-for-a-second-opinion/),
which shows cross-model review closes ~75% of the performance gap between
model tiers. Different model families have different blind spots — running
both catches issues that either alone would miss.

The rubber duck is **always the opposite engine**: if `REVIEW_ENGINE=claude`,
the duck is Copilot (GPT-5.4), and vice versa. No extra configuration needed.

**Graceful degradation:** if the rubber duck fails (missing credentials,
CLI not installed, timeout), the cascade continues with the primary deep
review only. The duck is additive, never blocking.

### Engine model mapping

| Tier | Claude engine | Copilot engine |
| Tier | Claude primary | Copilot primary |
|---|---|---|
| Triage | Haiku 4.5 | GPT-5-mini |
| Deep review | Sonnet 4.6 | GPT-5.2 |
| Rubber duck | GPT-5.4 (cross) | Sonnet 4.6 (cross) |
| Synthesis | Sonnet 4.6 | GPT-5.2 |
| Security audit | Opus 4.6 | GPT-5.4 |
| Action / single review | Sonnet 4.6 / Opus 4.6 | GPT-5.2 / GPT-5.4 |

**Cost profile:**
- ~80% of PRs: triage + single confirm (2 calls, ~30s)
- ~15% of PRs: + deep review (3 calls, ~2.5 min)
- ~5% of PRs: + security audit (4 calls, ~5.5 min)
- ~15% of PRs: + deep + duck + synthesis (5 calls, ~3 min)
- ~5% of PRs: + security audit (6 calls, ~6 min)

4. **Post-review actions** — after the review is posted, the action tier takes
additional actions depending on the decision:
Expand Down
12 changes: 10 additions & 2 deletions prompts/cascade-action.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ read that verdict and post the review to GitHub.
- `$REVIEW_CYCLE` — integer.
- `$MAX_REVIEW_CYCLES` — integer.
- `$FINAL_RESULT` — path to the JSON verdict from the resolving tier.
- `$FINAL_TIER` — `deep` or `audit` — which tier made the final call.
- `$FINAL_TIER` — `deep+duck`, `deep`, or `audit` — which tier made the final call.
- `$ENGINE_LABEL` — human-readable label for the cascade models (for footer).
- `$DUCK_ENGINE` — which engine ran the rubber duck (`claude` or `copilot`).
- `$DUCK_MODEL` — which model ran the rubber duck.
- `$TRIAGE_RESULT` — JSON from the triage tier (for context).

## Steps
Expand All @@ -40,8 +42,14 @@ read that verdict and post the review to GitHub.
### Summary
<from the verdict's summary>

### Cross-engine agreement
<If $FINAL_TIER is "deep+duck" and the verdict JSON has an "agreement" field,
report the agreement level and highlight findings where both engines converged.
If $FINAL_TIER is "deep" (no duck), omit this section.>

### Findings
<from the verdict's findings, grouped by severity>
<from the verdict's findings, grouped by severity. If findings have a "sources"
array, note which engine(s) flagged each finding.>

### CI status
<from the verdict or from PR metadata>
Expand Down
110 changes: 110 additions & 0 deletions prompts/rubber-duck.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Cross-engine adversarial review (Rubber Duck)

You are a cross-engine adversarial reviewer — the "rubber duck." You run on a
**different model family** from the primary deep reviewer, providing an
independent second opinion. Your job is to catch issues the primary reviewer's
model family is likely to miss: different blind spots, different strengths.

Default to skepticism. Assume the diff has gaps until the evidence says
otherwise. Prioritize concrete findings over vague concerns.

## Inputs (environment variables)

- `$PR_URL` — the PR to review.
- `$PR_HEAD_SHA` — the head commit SHA.
- `$DRY_RUN` — `true` or `false`.
- `$REVIEW_CYCLE` — integer.
- `$MAX_REVIEW_CYCLES` — integer.
- `$TRIAGE_RESULT` — JSON output from the triage tier, including its
`signals` array explaining why it escalated.
- `$PRIOR_REVIEW_BODY` — prior review body if this is a re-review (empty if first).
- `$PRIOR_REVIEW_SHA` — prior SHA if re-review.
- `$OUTPUT_FILE` — absolute path where you MUST write your JSON verdict.

## Scope

You review **exactly one PR**: `$PR_URL`. You have `gh` CLI available.

**FORBIDDEN**: `gh search prs`, `gh pr list`, `gh pr status`, or any
enumeration. No actions on other PRs.

## Steps

1. Read `$TRIAGE_RESULT` to understand why triage escalated — but don't limit
yourself to those signals. Your value comes from finding what others miss.
2. `gh pr view "$PR_URL" --json number,title,body,author,isDraft,baseRefName,headRefName,headRefOid,url,repository,labels,reviewDecision,mergeable,mergeStateStatus,statusCheckRollup,reviewRequests,reviews,comments,commits,closingIssuesReferences,additions,deletions,changedFiles,files`
3. `gh pr diff "$PR_URL"` — read the diff thoroughly.
4. Fetch linked issues if any.
5. Check `statusCheckRollup` for CI status.

## Adversarial focus areas

Go beyond the standard review checklist. Specifically look for:

- **Subtle logic errors** — off-by-one, race conditions, null/undefined paths
that only manifest under specific conditions.
- **Implicit assumptions** — does the code assume ordering, uniqueness, or
availability that isn't guaranteed?
- **Missing error paths** — what happens when the network is down, the file
doesn't exist, the API returns unexpected shapes?
- **Security blind spots** — TOCTOU, path traversal, prototype pollution,
deserialization attacks, timing side channels.
- **Integration risks** — does this change interact safely with the rest of the
system? Are there downstream consumers that will break?
- **Testing gaps** — are the tests actually testing the right thing? Do they
cover the failure modes?

## Risk classification

Same taxonomy as the primary reviewer:

### HIGH → always escalate
- Auth/secrets/credentials/crypto/tokens/`.env*`
- DB migrations/schema changes
- Security anti-patterns (injection, eval, shell=True, hardcoded secrets, etc.)
- CI security scanner warnings
- Org/project standards violations
- GitHub Actions security smells

### MEDIUM → approve if all gates pass
- Non-trivial logic changes, new deps, cross-module refactors

### LOW → approve if all gates pass
- Docs, comments, typos, tests-only, lockfile updates

## Decision gates

Approve only if ALL:
1. Risk is LOW or MEDIUM (never HIGH)
2. All required CI checks are green
3. Linked issue substantively addressed
4. No unresolved review threads
5. Well-structured PR

Otherwise → escalate.

## Output

Write a JSON object to `$OUTPUT_FILE`:

```json
{
"tier": "rubber-duck",
"model": "<the model you are running as>",
"risk": "LOW|MEDIUM|HIGH",
"decision": "approve|escalate",
"reason_codes": ["..."],
"summary": "2-3 sentences — focus on what you found that the primary might miss",
"findings": [
{
"severity": "info|minor|major|critical",
"category": "...",
"message": "...",
"file": "path or null",
"line": "number or null"
}
]
}
```

Write with `cat > "$OUTPUT_FILE" <<'JSON' ... JSON`. Ensure it parses with `jq`.
98 changes: 98 additions & 0 deletions prompts/synthesize-duck.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Synthesize deep review + rubber duck verdicts

Two reviewers analyzed the same PR in parallel using **different model
families** — one as the primary deep reviewer, the other as the adversarial
"rubber duck." Your job: read both verdicts, reconcile them, and produce
a single combined verdict.

## Inputs (environment variables)

- `$PR_URL` — the PR under review.
- `$PR_HEAD_SHA` — the head commit SHA.
- `$DRY_RUN` — `true` or `false`.
- `$REVIEW_CYCLE` — integer.
- `$MAX_REVIEW_CYCLES` — integer.
- `$TRIAGE_RESULT` — JSON from the triage tier.
- `$DEEP_RESULT` — path to the primary deep review JSON.
- `$DUCK_RESULT` — path to the rubber duck review JSON.
- `$OUTPUT_FILE` — path where you MUST write your combined verdict.
- `$DUCK_ENGINE` — which engine ran the rubber duck (`claude` or `copilot`).
- `$DUCK_MODEL` — which model ran the rubber duck.

## Steps

1. Read both JSON files:
```
jq . "$DEEP_RESULT"
jq . "$DUCK_RESULT"
```
2. Compare the two verdicts across all dimensions.
3. Synthesize a combined verdict following the rules below.
4. Write the combined JSON to `$OUTPUT_FILE`.

## Synthesis rules

### Risk
- `risk` = the **higher** of the two. If either says HIGH, the combined
risk is HIGH. HIGH > MEDIUM > LOW.

### Decision
- If **either** reviewer says `escalate` → combined decision is `escalate`.
- Both must say `approve` for the combined decision to be `approve`.

### Escalation
- `escalate_to_opus` = `true` if combined risk is HIGH, OR if either reviewer
flagged a finding with severity `critical`.

### Findings
- **Union** all findings from both reviewers.
- **Deduplicate**: if both flagged the same file+line+category, keep one entry
and add a `"sources": ["deep", "rubber-duck"]` field noting both agreed.
Cross-engine agreement on a finding is a strong signal — apply this exact
severity mapping to the deduplicated finding: `info` → `minor`, `minor` →
`major`, and leave `major` / `critical` unchanged.
- For findings from only one reviewer, add `"sources": ["deep"]` or
`"sources": ["rubber-duck"]`.
- Otherwise preserve the original severity.

### Summary
Write a 2-4 sentence combined summary that:
- Notes the overall risk assessment and whether the reviewers agreed or disagreed.
- Highlights any findings where both engines converged (strongest signal).
- Calls out findings unique to the rubber duck (cross-model diversity value).

## Output

Write a JSON object to `$OUTPUT_FILE`:

```json
{
"tier": "combined",
"primary_engine": "<REVIEW_ENGINE>",
"duck_engine": "<DUCK_ENGINE>",
"risk": "LOW|MEDIUM|HIGH",
"decision": "approve|escalate",
"escalate_to_opus": true|false,
"reason_codes": ["..."],
"agreement": "full|partial|divergent",
"summary": "2-4 sentences",
"findings": [
{
"severity": "info|minor|major|critical",
"category": "...",
"message": "...",
"file": "path or null",
"line": "number or null",
"sources": ["deep", "rubber-duck"]
}
]
}
```

`agreement` values:
- `full` — same risk AND same decision
- `partial` — same decision but different risk, OR same risk but different decision
- `divergent` — different risk AND different decision

Write with `cat > "$OUTPUT_FILE" <<'JSON' ... JSON`. Ensure it parses with `jq`.
After writing, exit. Do not do anything else.
44 changes: 42 additions & 2 deletions scripts/engine.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@
# Sourced by review-one-pr.sh — provides:
# run_triage <prompt_file> — no-tool tier (stdout capture)
# run_agentic <prompt_file> <model> — full-tool tier (stdout)
# run_duck <prompt_file> <model> — cross-engine adversarial (stdout)
# ENGINE_* env vars for model names and labels
# DUCK_ENGINE / DUCK_MODEL for rubber-duck cross-engine review

REVIEW_ENGINE="${REVIEW_ENGINE:-claude}"
export REVIEW_ENGINE
Expand All @@ -17,17 +19,23 @@ case "$REVIEW_ENGINE" in
ENGINE_AUDIT_MODEL="claude-opus-4-6"
ENGINE_ACTION_MODEL="claude-sonnet-4-6"
ENGINE_SINGLE_MODEL="claude-opus-4-6"
ENGINE_LABEL="triage: haiku 4.5 → deep: sonnet 4.6 → audit: opus 4.6"
ENGINE_LABEL="triage: haiku 4.5 → deep: sonnet 4.6 + duck: gpt-5.4 → audit: opus 4.6"
ENGINE_SINGLE_LABEL="single-reviewer mode: opus 4.6"
# Cross-engine rubber duck: always the opposite engine
DUCK_ENGINE="copilot"
DUCK_MODEL="gpt-5.4"
;;
copilot)
ENGINE_TRIAGE_MODEL="gpt-5-mini"
ENGINE_DEEP_MODEL="gpt-5.2"
ENGINE_AUDIT_MODEL="gpt-5.4"
ENGINE_ACTION_MODEL="gpt-5.2"
ENGINE_SINGLE_MODEL="gpt-5.4"
ENGINE_LABEL="triage: gpt-5-mini → deep: gpt-5.2 → audit: gpt-5.4"
ENGINE_LABEL="triage: gpt-5-mini → deep: gpt-5.2 + duck: sonnet 4.6 → audit: gpt-5.4"
ENGINE_SINGLE_LABEL="single-reviewer mode: gpt-5.4"
# Cross-engine rubber duck: always the opposite engine
DUCK_ENGINE="claude"
DUCK_MODEL="claude-sonnet-4-6"
;;
*)
echo "::error::Unknown REVIEW_ENGINE='$REVIEW_ENGINE' (expected: claude or copilot)"
Expand All @@ -38,6 +46,7 @@ esac
export ENGINE_TRIAGE_MODEL ENGINE_DEEP_MODEL ENGINE_AUDIT_MODEL
export ENGINE_ACTION_MODEL ENGINE_SINGLE_MODEL
export ENGINE_LABEL ENGINE_SINGLE_LABEL
export DUCK_ENGINE DUCK_MODEL

echo " engine: $REVIEW_ENGINE ($ENGINE_LABEL)"

Expand Down Expand Up @@ -83,3 +92,34 @@ run_agentic() {
;;
esac
}

# run_duck <prompt_file> <model>
# Cross-engine adversarial "rubber duck" review.
# Always uses the OPPOSITE engine from REVIEW_ENGINE. Output to stdout.
# Strips the opposing engine's credentials to prevent cross-engine leakage.
run_duck() {
local prompt_file="$1"
local model="$2"
case "$DUCK_ENGINE" in
claude)
unset COPILOT_GITHUB_TOKEN 2>/dev/null || true
timeout 300 claude --print \
--model "$model" \
--permission-mode acceptEdits \
--allowed-tools "Bash,Read,Grep,Glob" \
--max-turns 25 \
< "$prompt_file"
;;
copilot)
unset CLAUDE_CODE_OAUTH_TOKEN 2>/dev/null || true
timeout 300 copilot \
-p "$(cat "$prompt_file")" \
--model "$model" \
-s --allow-all --no-ask-user
;;
*)
echo "::error::Unknown DUCK_ENGINE='$DUCK_ENGINE'" >&2
return 1
;;
esac
}
Loading
Loading