Skip to content

[Acceptance] Re-run Haiku 5.5 through the finished pipeline and prove a known-bad candidate pauses it (negative control) #2132

Open

Description

@github-actions

Story

As a maintainer accepting the automated pipeline,
I want Haiku 5.5 re-run end to end through Phases A–D, plus a known-bad candidate that must pause the pipeline,
so that we have evidence that the pipeline promotes a good model with zero human actions and stops a bad one before it reaches any ring.

Acceptance Criteria

  1. The pipeline is force-onboarded (detection workflow_dispatch with model_id=claude-haiku-5-5, incumbent claude-haiku-4-5-20251001) and runs every stage. Its decision and stage results on the onboarding issue match the hand-run evidence (A/B exit 0, shadow bars met, cost and latency within tolerance), and the number of human actions on the happy path is recorded as 0.
  2. Because Haiku 5.5 is already the chain head after the hand rollout, the promotion step is recorded as a no-op (no PR opened, no channel moved). Any other outcome is reported as a defect.
  3. Negative control: a known-bad candidate (a configured stub or deliberately incompatible model id that fails preflight or the quality gate) is force-onboarded. The pipeline pauses with the expected reason, opens no chain-head PR and moves no channel.
  4. A summary comment on the epic records both runs against the pipeline success metrics (0 human actions; 0 wrongly promoted).

Tasks / Subtasks

Dev Notes

  • Maintainer decision (discussion Enable Haiku 5.5 #2095): Haiku 5.5 is done by hand, then re-run through the pipeline as its acceptance test, together with a negative control.
  • The offline negative-control test lives in the auto-promotion story; this is the live run.
  • Spend counts toward the $25 triage cap per model.
  • Hands-off: it spends tokens and is the maintainer's acceptance decision.

Project Structure Notes

Operational. No code changes.

References

  • discussion Enable Haiku 5.5 #2095 (Scope: negative control; Haiku 5.5 acceptance)
  • .github/workflows/model-ab.yml
  • scripts/lib/engine-models.sh

Story prepared by the BMAD Scrum Master (Bob) for epic #2113. Status: ready-for-dev.

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-lead:hands-offExclude from dev-lead agent automationinitiativeEpic / initiative tracking issueinitiative:holdInitiative driver must not release this issue (human hold)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions