Repository navigation
Description
Problem
The pr-review cascade hands a PR to a human (needs-human-review + CODEOWNERS request) whenever cascade-action returns decision=escalate (scripts/post-pr-review.sh, the DECISION = escalate branch) or the review-cycle cap is reached (scripts/review-one-pr.sh, "human attention needed"). Both are model-judgment outcomes, and they reach a human with no second opinion from a stronger model. Over the last 30 days that was roughly 1–2 escalations a day across the org.
Claude Fable 5.2 is the strongest available model. It tied Opus 5.5 on the routine held-out sets (model-ab run 42866988770: triage 0.8 vs 0.8, deep-review 0.8 vs 0.8) at roughly 2–3x the cost. So it is wrong as a per-PR default, but a good fit for the few PRs that would otherwise escalate.
Proposal: tier 4, "adjudicate"
Add a step that runs only when the cascade's verdict would escalate to a human:
- Fable gets the PR diff plus every prior tier's output (deep, rubber-duck, security audit when it ran, and the
cascade-actionverdict) through a newprompts/adjudicate.md. It returns structured JSON (decision: approve | request_changes | escalate,risk,summary,findings), the same contract shape as the other tiers. Its output is never parsed from discussion text. - Only a confirmed
escalatefrom Fable proceeds to the existing human escalation. Anapproveorrequest_changesreplaces the verdict that gets posted. - Resolve the model with
ai_model_for_family fable(added by "enable Claude Fable 5.2 as the preferred Fable", feat(models): enable Claude Fable 5.2 as the preferred Fable #2494). Never hard-pin an id.
Guardrails:
- Fable may not approve a PR whose security audit reported HIGH risk; that still escalates.
- One adjudication per head SHA, and tier 4 never re-runs itself.
- A run that rate-limits, errors, or returns invalid JSON falls back to today's human escalation. It never approves by default.
- The posted review records
FINAL_TIER=adjudicate.
Out of scope: mechanical and safety escalations go straight to a human, as today: single-review failure after retries, label-write failure, and the dev-lead guards (no-op, test tamper or regression, rebase exhaustion, channel pin, sync stub).
Acceptance criteria
- AC-1: Tier 4 in the cascade. Implement it in
scripts/review-one-pr.sh/scripts/post-pr-review.shfor both thedecision=escalatepath and the max-cycles path, behind an Actions variablePR_REVIEW_ADJUDICATEthat defaults to off. Every guardrail above has a bats test, including fallback to human escalation on error, rate limit or invalid JSON. - AC-2: Observability. The run log and the escalation or review comment state that tier 4 ran, its model (the resolved id), its decision, and whether it overrode the escalation. Token records go to
TOKEN_LOG_FILEunder tieradjudicate. - AC-3: Docs. Add tier 4 to
docs/engine-configuration.mdand to the pr-review cascade docs, including how to turn it on and off.
Notes
- Volume and cost: about 34 judgment escalations and about 48 max-cycle escalations in 30 days (counted by comment marker, overlapping). Fable 5.2 lists at $8/$40 per MTok, a small amount at this volume. CI draws on the shared subscription quota, so watch quota, not dollars.
- Depends on feat(models): enable Claude Fable 5.2 as the preferred Fable #2494 for the
fablefamily. Start after it merges. - Related: "Model rollout policy file and ADR defining 'default for a family + tier' and the automated promotion and pause rules" ([Phase A] Model rollout policy file and ADR defining 'default for a family + tier' and the automated promotion and pause rules #2116).
🤖 Generated with Claude Code
Activity
github-actions commentedon Nov 10, 2026
Thanks for the detailed proposal and for including the cost and volume data, which makes the trade-offs easy to follow. The guardrails and the replay-eval gate are well thought out. One question to help us scope this: when Fable's adjudication disagrees with what the human reviewer would have done, such as a false approval, what outcome matters most to you? For example, is a zero-false-approval bar a hard requirement for any rollout, or could a small, reviewed tolerance be acceptable if agreement is high?
don-petry commentedon Nov 12, 2026
Dev-Lead Implementation Plan
Issue: #2495 — pr-review: Fable adjudication tier before escalating to a human
Scope
Add an opt-in tier 4 (
adjudicate, behind the Actions variablePR_REVIEW_ADJUDICATE, off by default). It runs Fable (ai_model_for_family fable) only when the cascade verdict would escalate to a human, or when the review-cycle cap is reached. Its approve or request_changes replaces the posted verdict. A confirmed escalate, or any failure, falls through to today's human escalation.Implementation Checklist
scripts/lib/adjudicate.sh(new): enable flag; anescalates_to_humanpredicate shared with post-pr-review; one-per-SHA and since-reset dedup; strict JSON validation; HIGH-audit approve guard;run_adjudication, which returns ok, rate-limited, error, invalid-json or engine-unsupported, reads only$OUTPUT_FILEand never parses stdout; note and marker builder (<!-- pr-review-agent adjudicate v1 sha=… -->, FINAL_TIER=adjudicate)prompts/adjudicate.md(new): the contract{decision: approve|request_changes|escalate, risk, summary, findings}, written to$OUTPUT_FILEscripts/review-one-pr.sh: run tier 4 before posting a cascade verdict that would escalate to a human (cascade-action, both tier-2 and audit paths, plus single-review). Also run it at the max-cycles cap. It does not run under DRY_RUN or diagnose at the cap, and the cap path skips it when tier 4 already ran since the last reset, so it never re-runs itself.scripts/post-pr-review.sh: use the shared predicate. An.adjudication.decision=request_changesposts a fix-request. Every approve, fix-request or escalation body appends the tier-4 note (model, decision, overrode, outcome).scripts/lib/model-sandbox.sh: keep a sanitized.adjudicationobject through verdict validationscripts/engine.sh: pin--effort highfor theadjudicatetier. Token records already go toTOKEN_LOG_FILEunder the tier nameadjudicate..github/workflows/pr-review.yml:PR_REVIEW_ADJUDICATE: ${{ vars.PR_REVIEW_ADJUDICATE || 'false' }}; add the new bats file tolint.ymldocs/engine-configuration.md,docs/pr-review-agent/pr-review-agent.mdTests to Write
tests/test_adjudicate.bats: validation (good, bad decision, non-JSON, missing fields); HIGH-audit guard; predicate; per-SHA dedup that ignores forged non-bot markers; since-reset; run_adjudication ok, rate-limit, error, invalid JSON and non-claude fallbacks; resolves the fable family; token record under tieradjudicateTest Command
bats tests/test_adjudicate.bats tests/test_post_pr_review_escalation.bats tests/test_review_one_pr_escalation_exit.bats(full:bash scripts/run-bats.sh), plusbash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Nov 12, 2026
Dev-Lead: Progress (in progress — not yet durable)
Plan Execution (intended)
PR_REVIEW_ADJUDICATE(default off) on both the cascade-escalate path and the max-cycles path; Fable is resolved withai_model_for_family fableFiles touched (this workspace, not yet pushed)
scripts/lib/adjudicate.sh(new): predicate shared with post-pr-review, strict JSON validation, HIGH-audit guard, per-SHA and since-reset dedup,run_adjudication(ok, rate-limited, error, invalid-json or engine-unsupported; reads only$OUTPUT_FILE), note/marker/log builders, verdict building, strip of model-written recordsprompts/adjudicate.md(new): the contract{decision, risk, summary, findings}scripts/review-one-pr.sh: tier 4 before the three cascadepost_verdictcalls and at the cycle cap (never under DRY_RUN/diagnose)scripts/post-pr-review.sh: an adjudicatedrequest_changesbecomes a fix-request; the tier-4 note goes on every approval, fix-request and escalationscripts/lib/model-sandbox.sh: keeps a sanitized.adjudicationrecord.github/workflows/pr-review.yml(PR_REVIEW_ADJUDICATEenv),lint.yml(new bats files)docs/engine-configuration.md,docs/pr-review-agent/pr-review-agent.mdtests/test_adjudicate.bats,tests/test_review_one_pr_adjudicate.bats, plus additions totest_post_pr_review_escalation.batsandtest_model_sandbox.batsNotes
--effort highfor theadjudicatetier inengine.sh. If Fable 5.2 rejects that flag, every adjudication would error and fall back. That is worth a follow-up once it is verified live.escalatethat would go to a human, since it reaches the samedecision=escalatebranch.risk=HIGHblocks an approve instead..adjudicationfield is stripped before tier 4 decides, so a prompt-injected model cannot self-claim an adjudication.mainin this workspace (test_engine_chain_config,test_model_sandboxsetup tests, and fivetest_review_one_pr_*suites that stop at the maintainer-comment gate). They are not caused by this change; CI is authoritative.don-petry commentedon Nov 12, 2026
Dev-Lead: Implementation Complete — PR #2517
The implementation for issue #2495 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
c71a524ae73ccda2ee1a2b8c395c1b8d093c9b80main: 13 file(s), +1590/-8 linesReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
fix(reviews): address tier-4 adjudicate review findings
test(reviews): the HIGH-risk delegation contract checks the shared ru…
feat: implement issue #2495 — pr-review: Fable adjudication tier befo…