Skip to content

[Phase 2] Make claude-haiku-5-5 the head of the claude:triage chain and pin the eval judge to Haiku 4.5 in the same PR #2123

Open

Description

@github-actions

Story

As a maintainer of the review cascade,
I want claude-haiku-5-5 as the default triage model with Haiku 4.5 as its first fallback, while the llm-judge stays on Haiku 4.5,
so that triage gets the cheaper model once it is proven non-regressive, and the swap changes the generator only, so skill-eval score trends do not all shift on swap day.

Acceptance Criteria

  1. ai_models_chain claude triage in scripts/lib/engine-models.sh returns claude-haiku-5-5,claude-haiku-4-5-20251001,claude-sonnet-5-5,claude-sonnet-5, and the haiku family resolves to claude-haiku-5-5.
  2. When EVAL_JUDGE_CMD is not overridden, scripts/evals/run-eval.sh invokes the judge with the triage chain pinned to claude-haiku-4-5-20251001, independent of CLAUDE_TRIAGE_MODEL_CHAIN and the engine-models.sh default. The pin carries a # model-pin-ok: justification so scripts/check-model-pins.sh passes. An explicit EVAL_JUDGE_CMD override is still honoured, and model-ab.sh's AB_JUDGE_MODEL shim is unchanged.
  3. If the evidence story chose a non-default triage effort, the engine's claude triage effort is pinned to that level. Otherwise no effort flag is added.
  4. docs/engine-configuration.md and the comment block in engine-models.sh describe the new chain and the judge pin.
  5. Chain tests (tests/dev-lead/unit/test_engine_chain.bats, test_engine_chain_config.bats, test_engine_models.bats, test_engine_new_models.bats) are updated and pass. A bats test shows the judge stays on Haiku 4.5 with the new chain head. scripts/check-model-pins.sh passes. ET_BASELINE_MODEL and the last-resort fallback pin in scripts/engine.sh are unchanged.
  6. The PR description links the A/B evidence, the shadow report numbers against the epic thresholds, and the merged refusal-escalation PR.

Tasks / Subtasks

Dev Notes

  • The judge pin is in this PR because the research pass recommended it ('in the same PR as the swap'). Today run-eval.sh:107 sets EVAL_JUDGE_CMD="${EVAL_JUDGE_CMD:-run_triage}", and 10 of 13 evals/*/scorer.json use llm-judge, so a bare chain swap would also swap the judge.
  • Fixed-anchor guardrail (discussion Enable Haiku 5.5 #2095): the judge moves only through its own story (14 days stable plus >= 95% calibration agreement), never as a side effect.
  • The chain head is set in scripts/lib/engine-models.sh (the claude:triage) case) and the family map (haiku → triage). Only the chain head changes (no-version-pins rule).
  • Refusal escalation must already be on main.
  • A merged engine change is inert until the channel tag moves (Script-only changes never reach the fleet: autocut watches the reusable blob, but agents execute scripts/ pinned at agent_ref #1592). The canary story owns the cut.

Project Structure Notes

Same shape as the #1978 Sonnet 5.5 default-swap PR, plus a small change in scripts/evals/run-eval.sh.

References

Likely target surface

  • scripts/lib/engine-models.sh
  • scripts/evals/run-eval.sh
  • docs/engine-configuration.md

Story prepared by the BMAD Scrum Master (Bob) for epic #2113. Status: ready-for-dev.

Metadata

Metadata

Assignees

No one assigned

    Labels

    initiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions