Skip to content

pr-review: Fable adjudication tier before escalating to a human #2495

Closed

Description

@don-petry

Problem

The pr-review cascade hands a PR to a human (needs-human-review + CODEOWNERS request) whenever cascade-action returns decision=escalate (scripts/post-pr-review.sh, the DECISION = escalate branch) or the review-cycle cap is reached (scripts/review-one-pr.sh, "human attention needed"). Both are model-judgment outcomes, and they reach a human with no second opinion from a stronger model. Over the last 30 days that was roughly 1–2 escalations a day across the org.

Claude Fable 5.2 is the strongest available model. It tied Opus 5.5 on the routine held-out sets (model-ab run 42866988770: triage 0.8 vs 0.8, deep-review 0.8 vs 0.8) at roughly 2–3x the cost. So it is wrong as a per-PR default, but a good fit for the few PRs that would otherwise escalate.

Proposal: tier 4, "adjudicate"

Add a step that runs only when the cascade's verdict would escalate to a human:

  1. Fable gets the PR diff plus every prior tier's output (deep, rubber-duck, security audit when it ran, and the cascade-action verdict) through a new prompts/adjudicate.md. It returns structured JSON (decision: approve | request_changes | escalate, risk, summary, findings), the same contract shape as the other tiers. Its output is never parsed from discussion text.
  2. Only a confirmed escalate from Fable proceeds to the existing human escalation. An approve or request_changes replaces the verdict that gets posted.
  3. Resolve the model with ai_model_for_family fable (added by "enable Claude Fable 5.2 as the preferred Fable", feat(models): enable Claude Fable 5.2 as the preferred Fable #2494). Never hard-pin an id.

Guardrails:

  • Fable may not approve a PR whose security audit reported HIGH risk; that still escalates.
  • One adjudication per head SHA, and tier 4 never re-runs itself.
  • A run that rate-limits, errors, or returns invalid JSON falls back to today's human escalation. It never approves by default.
  • The posted review records FINAL_TIER=adjudicate.

Out of scope: mechanical and safety escalations go straight to a human, as today: single-review failure after retries, label-write failure, and the dev-lead guards (no-op, test tamper or regression, rebase exhaustion, channel pin, sync stub).

Acceptance criteria

  • AC-1: Tier 4 in the cascade. Implement it in scripts/review-one-pr.sh / scripts/post-pr-review.sh for both the decision=escalate path and the max-cycles path, behind an Actions variable PR_REVIEW_ADJUDICATE that defaults to off. Every guardrail above has a bats test, including fallback to human escalation on error, rate limit or invalid JSON.
  • AC-2: Observability. The run log and the escalation or review comment state that tier 4 ran, its model (the resolved id), its decision, and whether it overrode the escalation. Token records go to TOKEN_LOG_FILE under tier adjudicate.
  • AC-3: Docs. Add tier 4 to docs/engine-configuration.md and to the pr-review cascade docs, including how to turn it on and off.

Notes

🤖 Generated with Claude Code

Activity

  1. github-actions commented on Nov 10, 2026

    @github-actions
    Contributor

    Thanks for the detailed proposal and for including the cost and volume data, which makes the trade-offs easy to follow. The guardrails and the replay-eval gate are well thought out. One question to help us scope this: when Fable's adjudication disagrees with what the human reviewer would have done, such as a false approval, what outcome matters most to you? For example, is a zero-false-approval bar a hard requirement for any rollout, or could a small, reviewed tolerance be acceptable if agreement is high?

  2. don-petry commented on Nov 12, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2495 — pr-review: Fable adjudication tier before escalating to a human

    Scope

    Add an opt-in tier 4 (adjudicate, behind the Actions variable PR_REVIEW_ADJUDICATE, off by default). It runs Fable (ai_model_for_family fable) only when the cascade verdict would escalate to a human, or when the review-cycle cap is reached. Its approve or request_changes replaces the posted verdict. A confirmed escalate, or any failure, falls through to today's human escalation.

    Implementation Checklist

    • scripts/lib/adjudicate.sh (new): enable flag; an escalates_to_human predicate shared with post-pr-review; one-per-SHA and since-reset dedup; strict JSON validation; HIGH-audit approve guard; run_adjudication, which returns ok, rate-limited, error, invalid-json or engine-unsupported, reads only $OUTPUT_FILE and never parses stdout; note and marker builder (<!-- pr-review-agent adjudicate v1 sha=… -->, FINAL_TIER=adjudicate)
    • prompts/adjudicate.md (new): the contract {decision: approve|request_changes|escalate, risk, summary, findings}, written to $OUTPUT_FILE
    • scripts/review-one-pr.sh: run tier 4 before posting a cascade verdict that would escalate to a human (cascade-action, both tier-2 and audit paths, plus single-review). Also run it at the max-cycles cap. It does not run under DRY_RUN or diagnose at the cap, and the cap path skips it when tier 4 already ran since the last reset, so it never re-runs itself.
    • scripts/post-pr-review.sh: use the shared predicate. An .adjudication.decision=request_changes posts a fix-request. Every approve, fix-request or escalation body appends the tier-4 note (model, decision, overrode, outcome).
    • scripts/lib/model-sandbox.sh: keep a sanitized .adjudication object through verdict validation
    • scripts/engine.sh: pin --effort high for the adjudicate tier. Token records already go to TOKEN_LOG_FILE under the tier name adjudicate.
    • .github/workflows/pr-review.yml: PR_REVIEW_ADJUDICATE: ${{ vars.PR_REVIEW_ADJUDICATE || 'false' }}; add the new bats file to lint.yml
    • Docs: docs/engine-configuration.md, docs/pr-review-agent/pr-review-agent.md

    Tests to Write

    • tests/test_adjudicate.bats: validation (good, bad decision, non-JSON, missing fields); HIGH-audit guard; predicate; per-SHA dedup that ignores forged non-bot markers; since-reset; run_adjudication ok, rate-limit, error, invalid JSON and non-claude fallbacks; resolves the fable family; token record under tier adjudicate
    • post-pr-review: request_changes posts a fix-request with delegation off; an escalate or fallback note lands in the human-escalation comment; approve body carries FINAL_TIER=adjudicate
    • review-one-pr cap path: flag off is unchanged; approve overrides (exit 0, no label); escalate and invalid JSON fall back to human (exit 101); already adjudicated since reset goes straight to human

    Test Command

    bats tests/test_adjudicate.bats tests/test_post_pr_review_escalation.bats tests/test_review_one_pr_escalation_exit.bats (full: bash scripts/run-bats.sh), plus bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Nov 12, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet. The automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    • Tier 4 "adjudicate" added behind PR_REVIEW_ADJUDICATE (default off) on both the cascade-escalate path and the max-cycles path; Fable is resolved with ai_model_for_family fable
    • Tests were written first and confirmed red, then made green locally

    Files touched (this workspace, not yet pushed)

    • scripts/lib/adjudicate.sh (new): predicate shared with post-pr-review, strict JSON validation, HIGH-audit guard, per-SHA and since-reset dedup, run_adjudication (ok, rate-limited, error, invalid-json or engine-unsupported; reads only $OUTPUT_FILE), note/marker/log builders, verdict building, strip of model-written records
    • prompts/adjudicate.md (new): the contract {decision, risk, summary, findings}
    • scripts/review-one-pr.sh: tier 4 before the three cascade post_verdict calls and at the cycle cap (never under DRY_RUN/diagnose)
    • scripts/post-pr-review.sh: an adjudicated request_changes becomes a fix-request; the tier-4 note goes on every approval, fix-request and escalation
    • scripts/lib/model-sandbox.sh: keeps a sanitized .adjudication record
    • .github/workflows/pr-review.yml (PR_REVIEW_ADJUDICATE env), lint.yml (new bats files)
    • docs/engine-configuration.md, docs/pr-review-agent/pr-review-agent.md
    • Tests: tests/test_adjudicate.bats, tests/test_review_one_pr_adjudicate.bats, plus additions to test_post_pr_review_escalation.bats and test_model_sandbox.bats

    Notes

    • Deviations from the plan:
      • I did not pin --effort high for the adjudicate tier in engine.sh. If Fable 5.2 rejects that flag, every adjudication would error and fall back. That is worth a follow-up once it is verified live.
      • Tier 4 also covers a single-review escalate that would go to a human, since it reaches the same decision=escalate branch.
    • Interpretation of "never re-runs itself":
      • Cascade path: one adjudication per head SHA.
      • Cap path: additionally, no second adjudication once tier 4 has run since the last reset (a human approval or escalation).
      • At the cap there is no security audit, so a prior verdict carrying risk=HIGH blocks an approve instead.
    • Hardening: a cascade verdict's own .adjudication field is stripped before tier 4 decides, so a prompt-injected model cannot self-claim an adjudication.
    • Local environment: several existing suites fail identically on main in this workspace (test_engine_chain_config, test_model_sandbox setup tests, and five test_review_one_pr_* suites that stop at the maintainer-comment gate). They are not caused by this change; CI is authoritative.
  4. don-petry commented on Nov 12, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2517

    The implementation for issue #2495 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions