Repository navigation
Description
Story
As a review-engine maintainer,
I want the audit and single-reviewer tiers to run on claude-opus-5-5, and Fable 5 removed from every default model chain,
so that the engine stops depending on Fable, which Opus 5.5 outperforms at lower cost, and reviews use one current model family end to end.
Decision (supersedes the conditional "Option C" evaluation)
The maintainer has decided to deprecate Fable 5. This story is no longer "swap only if the A/B passes". The swap is the change. The audit A/B (below) is still run and recorded as rollout evidence before stable promotion, but it does not gate this PR.
Current state on main (after #1972)
Model defaults live in scripts/lib/engine-models.sh (_ai_models_default), not engine.sh:
| Key | Default chain today | After this story |
|---|---|---|
audit |
claude-fable-5,claude-opus-4-8,claude-opus-4-7 |
claude-opus-5-5,claude-opus-4-8,claude-opus-4-7 |
single |
claude-fable-5,claude-opus-4-8,claude-opus-4-7 |
claude-opus-5-5,claude-opus-4-8,claude-opus-4-7 |
No other default chain names Fable.
Acceptance Criteria
-
Defaults. In
scripts/lib/engine-models.sh, change theclaude:auditandclaude:singledefaults toclaude-opus-5-5,claude-opus-4-8,claude-opus-4-7. Theopus-4-8hop is the known-good fallback, as in [Phase 2] Swap the deep tier to claude-opus-5-5 in engine.sh (model, chain, label, effort) #1898.ENGINE_AUDIT_MODEL,ENGINE_SINGLE_MODELand theengine:log label follow automatically from the chains, so confirm the log line readsaudit: opus 5.5 [opus 4.8, opus 4.7]. -
Reasoning effort.
claude-opus-5-5defaults tomediumreasoning effort underclaude --print. That is whyrun_agenticinscripts/engine.shpins--effort high, but today it does so only for the deep tier ([ "$tier" = "deep" ]). Extend the pin to theaudittier and to the single-reviewer path, wherever that path invokes Claude, so that moving off Fable does not silently lower their reasoning effort. Add bats cases asserting that the audit and single invocations pass--effort high, and that triage and action still do not. -
Remove Fable-specific text. Rewrite the "Fable 5" model notes in
engine-models.sh(the adaptive-thinking and cache-prefix rules) and theengine.shcomment nearset_engine_configthat cites them, so they describe the current defaults. Keep any note that still applies to Opus 5.5. -
Deprecation warning. When a configured chain names a
claude-fable-*model (throughAI_MODELS_CLAUDEorCLAUDE_<TIER>_MODEL_CHAIN), log one::warning::per run saying Fable is deprecated and naming the Opus 5.5 default. Still honour the configured model: this is a warning, not a rejection, so an operator can use it as a stop-gap. Add a bats case for it. -
Pricing stays. Keep the
claude-fable-5rows inscripts/lib/model-pricing.tsv, because historical token records still need them. Don't add or change any Fable price. -
Tests. Update everything that pins Fable as a default:
tests/dev-lead/unit/test_engine_new_models.bats:ENGINE_AUDIT_MODEL/ENGINE_SINGLE_MODEL, and which model the audit/single chains start with;tests/dev-lead/unit/test_engine_chain.bats: the audit rate-limit hop tests;tests/dev-lead/unit/test_engine_models.bats: the pinned chains.
Rewrite each audit fallback test to start from
claude-opus-5-5while keeping what it checks, such as the rate-limit hop toopus-4-8and all three hops exhausting. Leave the historical fixtures intests/fixtures/et-baseline/and the initiative-planner fixture as they are, because they are recorded data. -
Docs. Update the defaults table in
docs/engine-configuration.mdand any Fable mention inAGENTS.mdordocs/pr-review-agent/setup.md. Afterwards,git grep -in fable -- scripts .github prompts personas docsshould find only the pricing rows, the deprecation warning and a changelog-style note. -
Validation.
shellcheck -S warningand actionlint are clean, and everytest_engine*.batsandtest_model_dispatchpass. The PR states that it is inert until autocut movespr-review/v1-next/dev-lead/v139-next.
Rollout evidence (maintainer-gated, not part of the PR)
- Once feat: implement issue #1950 — [Phase 2] Add a workflow_dispatch runner for model-ab.sh plus a liveness/effort probe, so the Opus 5.5 non-regression A/B actually executes #1952 (the
model-ab.ymlrunner) lands, the maintainer dispatchesmodel-ab.sh claude-opus-5-5 claude-fable-5 auditand records the per-set scores and exit code here. It follows the epic cost cap of at most 3 runs per arm, on the frozen audit held-out set, with the judge held fixed (AB_JUDGE_MODEL) and no prompt or corpus edits. - The change rides
next→ rings under the existing health gate. feat: implement issue #1951 — [Phase 2] Canary go/no-go report: compare deep-tier cost, cache-read cost and latency by model from token-usage artifacts #1953's canary report covers audit-tier cost and latency. If the A/B shows a regression, stop promotion and report it on this issue. Don't roll back automatically.
Dev Notes
- Don't dispatch
model-ab.yml, Canary Rollout or any release step from the PR. - [Phase 3] Sonnet 5.5 rollout: price claude-sonnet-5-5 and make it the default Sonnet in every Claude chain #1978 (Sonnet 5.5) edits the same defaults function and test file. Whichever PR lands second rebases onto the first. [Phase 3] Sonnet 5.5 rollout: price claude-sonnet-5-5 and make it the default Sonnet in every Claude chain #1978 leaves
auditandsingleto this story. - Keep Bash 3.2 compatibility (no
${,,}, guard empty arrays).
Likely target surface
scripts/lib/engine-models.sh,scripts/engine.sh(effort pin and comment)tests/dev-lead/unit/test_engine_new_models.bats,test_engine_chain.bats,test_engine_models.batsdocs/engine-configuration.md
Originally prepared by the BMAD Scrum Master for epic #1895 as a conditional evaluation. Re-scoped by the maintainer on 2026-09-29 to deprecate Fable 5 outright.
Activity
[-][Phase 3] Evaluate Option C: audit/single tier Fable 5 → Opus 5.5, metric-gated[/-][+][Phase 3] Deprecate Fable 5: move the audit and single tiers to Opus 5.5 and remove Fable from every default chain[/+]don-petry commentedon Sep 29, 2026
Dev-Lead: implementation deferred — automation PR queue is full
No pull request was opened for issue #1901. The org-wide
open-automation-PR queue is currently at its signed-off cap (see
standards/pr-limits.jsoninpetry-projects/.github).This issue stays labeled
dev-leadand will be re-attempted automatically oncethe queue drains — no action is required.
don-petry commentedon Sep 30, 2026
Dev-Lead Implementation Plan
Issue: #1901 — [Phase 3] Deprecate Fable 5: move the audit and single tiers to Opus 5.5 and remove Fable from every default chain
Scope
Swap the
claude:auditandclaude:singledefault chains from Fable 5 toclaude-opus-5-5,claude-opus-4-8,claude-opus-4-7, extend the--effort highpin to the audit and single tiers, add a Fable-deprecation::warning::, and update the notes/docs — leaving pricing rows and recorded fixtures/audit data untouched.Implementation Checklist
scripts/lib/engine-models.sh:claude:audit+claude:singledefaults →claude-opus-5-5,claude-opus-4-8,claude-opus-4-7scripts/lib/engine-models.sh: rewrite the "Fable 5" adaptive-thinking/cache-prefix notes to describe the current opus-5-5 defaults + a Fable-deprecation (changelog-style) notescripts/lib/engine-models.sh: addai_models_fable_deprecation(pure detector for a configuredclaude-fable-*viaAI_MODELS_CLAUDE/CLAUDE_<TIER>_MODEL_CHAIN)scripts/engine.sh: extend the--effort highpin inrun_agenticfrom deep-only todeep|audit|single; update its commentscripts/engine.sh: update theset_engine_configcomment citing the Fable notes; wire the one-warning-per-run Fable deprecation::warning::scripts/lib/model-pricing.tsv: unchanged (keep theclaude-fable-5*row)docs/engine-configuration.md: update theaudit/singlerows in the defaults tableTests to Write / Update
tests/dev-lead/unit/test_engine_new_models.bats:ENGINE_AUDIT_MODEL/ENGINE_SINGLE_MODEL= opus-5-5; audit/single chain starts + fallback hops rewritten from opus-5-5tests/dev-lead/unit/test_engine_chain.bats: audit rate-limit hop from opus-5-5; effort tests — audit + single now pass--effort high, triage/action still do nottests/dev-lead/unit/test_engine_models.bats: pinned default chains for audit/single; new Fable-deprecation warning case (warns once, still honours the configured model)Test Command
bash scripts/run-bats.sh tests/dev-lead/unit/test_engine_new_models.bats tests/dev-lead/unit/test_engine_chain.bats tests/dev-lead/unit/test_engine_models.bats tests/dev-lead/unit/test_model_dispatch.bats(andbash .dev-lead/scripts/dev-lead-lint.sh)Note
The two historical audit docs
docs/initiatives/in-loop-fetch-audit.mdanddocs/initiatives/pr-review-bughunter-audit.mdstill name Fable as recorded/frozen data (the first is CODEOWNERS-locked and pinned by a regression test); per the AC #6 principle of preserving recorded data, they are left unchanged.don-petry commentedon Sep 30, 2026
Dev-Lead: Implementation Complete — PR #1980
The implementation for issue #1901 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
ae0bcb73b172a8825f4ebd70640114a5ff2b39d1main: 6 file(s), +169/-53 linesReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
feat: implement issue #1901 — [Phase 3] Deprecate Fable 5: move the a…
feat(models): enable Claude Fable 5.2 as the preferred Fable (#2494)