Skip to content

[Phase 3] Deprecate Fable 5: move the audit and single tiers to Opus 5.5 and remove Fable from every default chain #1901

Closed

Description

@github-actions

Story

As a review-engine maintainer,
I want the audit and single-reviewer tiers to run on claude-opus-5-5, and Fable 5 removed from every default model chain,
so that the engine stops depending on Fable, which Opus 5.5 outperforms at lower cost, and reviews use one current model family end to end.

Decision (supersedes the conditional "Option C" evaluation)

The maintainer has decided to deprecate Fable 5. This story is no longer "swap only if the A/B passes". The swap is the change. The audit A/B (below) is still run and recorded as rollout evidence before stable promotion, but it does not gate this PR.

Current state on main (after #1972)

Model defaults live in scripts/lib/engine-models.sh (_ai_models_default), not engine.sh:

Key Default chain today After this story
audit claude-fable-5,claude-opus-4-8,claude-opus-4-7 claude-opus-5-5,claude-opus-4-8,claude-opus-4-7
single claude-fable-5,claude-opus-4-8,claude-opus-4-7 claude-opus-5-5,claude-opus-4-8,claude-opus-4-7

No other default chain names Fable.

Acceptance Criteria

  1. Defaults. In scripts/lib/engine-models.sh, change the claude:audit and claude:single defaults to claude-opus-5-5,claude-opus-4-8,claude-opus-4-7. The opus-4-8 hop is the known-good fallback, as in [Phase 2] Swap the deep tier to claude-opus-5-5 in engine.sh (model, chain, label, effort) #1898. ENGINE_AUDIT_MODEL, ENGINE_SINGLE_MODEL and the engine: log label follow automatically from the chains, so confirm the log line reads audit: opus 5.5 [opus 4.8, opus 4.7].

  2. Reasoning effort. claude-opus-5-5 defaults to medium reasoning effort under claude --print. That is why run_agentic in scripts/engine.sh pins --effort high, but today it does so only for the deep tier ([ "$tier" = "deep" ]). Extend the pin to the audit tier and to the single-reviewer path, wherever that path invokes Claude, so that moving off Fable does not silently lower their reasoning effort. Add bats cases asserting that the audit and single invocations pass --effort high, and that triage and action still do not.

  3. Remove Fable-specific text. Rewrite the "Fable 5" model notes in engine-models.sh (the adaptive-thinking and cache-prefix rules) and the engine.sh comment near set_engine_config that cites them, so they describe the current defaults. Keep any note that still applies to Opus 5.5.

  4. Deprecation warning. When a configured chain names a claude-fable-* model (through AI_MODELS_CLAUDE or CLAUDE_<TIER>_MODEL_CHAIN), log one ::warning:: per run saying Fable is deprecated and naming the Opus 5.5 default. Still honour the configured model: this is a warning, not a rejection, so an operator can use it as a stop-gap. Add a bats case for it.

  5. Pricing stays. Keep the claude-fable-5 rows in scripts/lib/model-pricing.tsv, because historical token records still need them. Don't add or change any Fable price.

  6. Tests. Update everything that pins Fable as a default:

    • tests/dev-lead/unit/test_engine_new_models.bats: ENGINE_AUDIT_MODEL/ENGINE_SINGLE_MODEL, and which model the audit/single chains start with;
    • tests/dev-lead/unit/test_engine_chain.bats: the audit rate-limit hop tests;
    • tests/dev-lead/unit/test_engine_models.bats: the pinned chains.

    Rewrite each audit fallback test to start from claude-opus-5-5 while keeping what it checks, such as the rate-limit hop to opus-4-8 and all three hops exhausting. Leave the historical fixtures in tests/fixtures/et-baseline/ and the initiative-planner fixture as they are, because they are recorded data.

  7. Docs. Update the defaults table in docs/engine-configuration.md and any Fable mention in AGENTS.md or docs/pr-review-agent/setup.md. Afterwards, git grep -in fable -- scripts .github prompts personas docs should find only the pricing rows, the deprecation warning and a changelog-style note.

  8. Validation. shellcheck -S warning and actionlint are clean, and every test_engine*.bats and test_model_dispatch pass. The PR states that it is inert until autocut moves pr-review/v1-next / dev-lead/v139-next.

Rollout evidence (maintainer-gated, not part of the PR)

Dev Notes

Likely target surface

  • scripts/lib/engine-models.sh, scripts/engine.sh (effort pin and comment)
  • tests/dev-lead/unit/test_engine_new_models.bats, test_engine_chain.bats, test_engine_models.bats
  • docs/engine-configuration.md

Originally prepared by the BMAD Scrum Master for epic #1895 as a conditional evaluation. Re-scoped by the maintainer on 2026-09-29 to deprecate Fable 5 outright.

Activity

  1. changed the title [-][Phase 3] Evaluate Option C: audit/single tier Fable 5 → Opus 5.5, metric-gated[/-] [+][Phase 3] Deprecate Fable 5: move the audit and single tiers to Opus 5.5 and remove Fable from every default chain[/+] on Sep 29, 2026
  2. don-petry commented on Sep 29, 2026

    @don-petry
    Collaborator

    Dev-Lead: implementation deferred — automation PR queue is full

    No pull request was opened for issue #1901. The org-wide
    open-automation-PR queue is currently at its signed-off cap (see
    standards/pr-limits.json in petry-projects/.github).

    This issue stays labeled dev-lead and will be re-attempted automatically once
    the queue drains — no action is required.

  3. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Sep 30, 2026
  4. don-petry commented on Sep 30, 2026

    @don-petry
    Collaborator

    Dev-Lead Implementation Plan

    Issue: #1901 — [Phase 3] Deprecate Fable 5: move the audit and single tiers to Opus 5.5 and remove Fable from every default chain

    Scope

    Swap the claude:audit and claude:single default chains from Fable 5 to claude-opus-5-5,claude-opus-4-8,claude-opus-4-7, extend the --effort high pin to the audit and single tiers, add a Fable-deprecation ::warning::, and update the notes/docs — leaving pricing rows and recorded fixtures/audit data untouched.

    Implementation Checklist

    • scripts/lib/engine-models.sh: claude:audit + claude:single defaults → claude-opus-5-5,claude-opus-4-8,claude-opus-4-7
    • scripts/lib/engine-models.sh: rewrite the "Fable 5" adaptive-thinking/cache-prefix notes to describe the current opus-5-5 defaults + a Fable-deprecation (changelog-style) note
    • scripts/lib/engine-models.sh: add ai_models_fable_deprecation (pure detector for a configured claude-fable-* via AI_MODELS_CLAUDE / CLAUDE_<TIER>_MODEL_CHAIN)
    • scripts/engine.sh: extend the --effort high pin in run_agentic from deep-only to deep|audit|single; update its comment
    • scripts/engine.sh: update the set_engine_config comment citing the Fable notes; wire the one-warning-per-run Fable deprecation ::warning::
    • scripts/lib/model-pricing.tsv: unchanged (keep the claude-fable-5* row)
    • docs/engine-configuration.md: update the audit/single rows in the defaults table

    Tests to Write / Update

    • tests/dev-lead/unit/test_engine_new_models.bats: ENGINE_AUDIT_MODEL/ENGINE_SINGLE_MODEL = opus-5-5; audit/single chain starts + fallback hops rewritten from opus-5-5
    • tests/dev-lead/unit/test_engine_chain.bats: audit rate-limit hop from opus-5-5; effort tests — audit + single now pass --effort high, triage/action still do not
    • tests/dev-lead/unit/test_engine_models.bats: pinned default chains for audit/single; new Fable-deprecation warning case (warns once, still honours the configured model)

    Test Command

    bash scripts/run-bats.sh tests/dev-lead/unit/test_engine_new_models.bats tests/dev-lead/unit/test_engine_chain.bats tests/dev-lead/unit/test_engine_models.bats tests/dev-lead/unit/test_model_dispatch.bats (and bash .dev-lead/scripts/dev-lead-lint.sh)

    Note

    The two historical audit docs docs/initiatives/in-loop-fetch-audit.md and docs/initiatives/pr-review-bughunter-audit.md still name Fable as recorded/frozen data (the first is CODEOWNERS-locked and pinned by a regression test); per the AC #6 principle of preserving recorded data, they are left unchanged.

  5. don-petry commented on Sep 30, 2026

    @don-petry
    Collaborator

    Dev-Lead: Implementation Complete — PR #1980

    The implementation for issue #1901 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  6. added a commit that references this issue on Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requestsinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions