Repository navigation
Open
Labels
initiativeEpic / initiative tracking issueEpic / initiative tracking issue
Description
Story
As a maintainer of the review cascade,
I want claude-haiku-5-5 as the default triage model with Haiku 4.5 as its first fallback, while the llm-judge stays on Haiku 4.5,
so that triage gets the cheaper model once it is proven non-regressive, and the swap changes the generator only, so skill-eval score trends do not all shift on swap day.
Acceptance Criteria
ai_models_chain claude triageinscripts/lib/engine-models.shreturnsclaude-haiku-5-5,claude-haiku-4-5-20251001,claude-sonnet-5-5,claude-sonnet-5, and thehaikufamily resolves toclaude-haiku-5-5.- When
EVAL_JUDGE_CMDis not overridden,scripts/evals/run-eval.shinvokes the judge with the triage chain pinned toclaude-haiku-4-5-20251001, independent ofCLAUDE_TRIAGE_MODEL_CHAINand theengine-models.shdefault. The pin carries a# model-pin-ok:justification soscripts/check-model-pins.shpasses. An explicitEVAL_JUDGE_CMDoverride is still honoured, andmodel-ab.sh'sAB_JUDGE_MODELshim is unchanged. - If the evidence story chose a non-default triage effort, the engine's claude triage effort is pinned to that level. Otherwise no effort flag is added.
docs/engine-configuration.mdand the comment block inengine-models.shdescribe the new chain and the judge pin.- Chain tests (
tests/dev-lead/unit/test_engine_chain.bats,test_engine_chain_config.bats,test_engine_models.bats,test_engine_new_models.bats) are updated and pass. A bats test shows the judge stays on Haiku 4.5 with the new chain head.scripts/check-model-pins.shpasses.ET_BASELINE_MODELand the last-resort fallback pin inscripts/engine.share unchanged. - The PR description links the A/B evidence, the shadow report numbers against the epic thresholds, and the merged refusal-escalation PR.
Tasks / Subtasks
- Update the triage chain default and the family mapping (AC: test issue from agent #1)
- Pin the default judge command to Haiku 4.5 (AC: Go-live improvements for PR review agent #2)
- Follow the shim pattern
model-ab.shuses to re-pin the chain for the judge
- Follow the shim pattern
- Apply the effort decision (AC: Add @claude delegation, auto-merge, and rebase handling #3)
- Update docs and tests (AC: Optimize review: small-PR and incremental fast paths #4, feat: add Copilot engine support via REVIEW_ENGINE toggle #5)
- Assemble evidence links (AC: feat: feature ideation framework — adversarial proposer/challenger/synthesizer #6)
Dev Notes
- The judge pin is in this PR because the research pass recommended it ('in the same PR as the swap'). Today
run-eval.sh:107setsEVAL_JUDGE_CMD="${EVAL_JUDGE_CMD:-run_triage}", and 10 of 13evals/*/scorer.jsonusellm-judge, so a bare chain swap would also swap the judge. - Fixed-anchor guardrail (discussion Enable Haiku 5.5 #2095): the judge moves only through its own story (14 days stable plus >= 95% calibration agreement), never as a side effect.
- The chain head is set in
scripts/lib/engine-models.sh(theclaude:triage)case) and the family map (haiku → triage). Only the chain head changes (no-version-pins rule). - Refusal escalation must already be on
main. - A merged engine change is inert until the channel tag moves (Script-only changes never reach the fleet: autocut watches the reusable blob, but agents execute scripts/ pinned at agent_ref #1592). The canary story owns the cut.
Project Structure Notes
Same shape as the #1978 Sonnet 5.5 default-swap PR, plus a small change in scripts/evals/run-eval.sh.
References
- scripts/lib/engine-models.sh
- scripts/engine.sh
- scripts/evals/run-eval.sh
- scripts/evals/model-ab.sh
- docs/engine-configuration.md
- tests/dev-lead/unit/test_engine_chain.bats
- tests/dev-lead/unit/test_engine_chain_config.bats
- tests/dev-lead/unit/test_engine_models.bats
- tests/dev-lead/unit/test_engine_new_models.bats
- scripts/check-model-pins.sh
- .github/workflows/skill-eval-report.yml
- issue [Phase 3] Sonnet 5.5 rollout: price claude-sonnet-5-5 and make it the default Sonnet in every Claude chain #1978
Likely target surface
scripts/lib/engine-models.shscripts/evals/run-eval.shdocs/engine-configuration.md
Story prepared by the BMAD Scrum Master (Bob) for epic #2113. Status: ready-for-dev.
Metadata
Metadata
Assignees
Labels
initiativeEpic / initiative tracking issueEpic / initiative tracking issue
Activity