Skip to content

[Phase 1] Add a triage effort knob to the engine and a false-approve / false-escalate breakdown to model-ab.sh #2106

Closed

Description

@github-actions

Story

As a maintainer judging a triage-model swap,
I want to A/B the candidate at a chosen reasoning effort and see false-approves apart from false-escalates,
so that the effort setting is picked from data rather than inherited from the CLI default, and a missed escalation, which costs more than an extra one, cannot hide inside an aggregate score.

Acceptance Criteria

  1. run_triage in scripts/engine.sh passes --effort <level> on the claude branch only when an opt-in env var (for example CLAUDE_TRIAGE_EFFORT) is set to a valid level. When it is unset the CLI invocation is byte-for-byte unchanged (bats asserts both cases).
  2. scripts/evals/model-ab.sh accepts an optional candidate-arm effort, applied only to the candidate arm, and records it in the JSON evidence object.
  3. For the triage set, the evidence JSON reports per arm the counts of false-approves (expected escalate, predicted approve) and false-escalates (expected approve, predicted escalate).
  4. The accept/regression rule stays >= per set, with exit codes 0 / 1 / 2 unchanged (the Stage 5 gate stays as it is, per discussion Enable Haiku 5.5 #2095). The confusion counts are reported only and are judged against the epic's false-approve metric in the evidence story.
  5. The held-out checksum guard still covers cases.jsonl, scorer.json and evals/judge.md. Tests in tests/test_model_ab.bats cover the new evidence fields offline (no network) and show the exit code is unaffected by them.

Tasks / Subtasks

Dev Notes

  • Today run_agentic pins --effort high only for deep/audit/single. Triage keeps the CLI default, which on Haiku 5.5 is medium with adaptive thinking (research pass, gap 4.1).
  • Eval safeguard (overfitting): the held-out set is 5 triage cases, so do NOT choose effort by best score. The decision rule recorded in the evidence story is: of the effort arms that pass non-regression, pick the cheapest. Shadow traffic then confirms the choice. The held-out cases, scorer and judge stay immutable: there is no tuning input, and the checksum guard is unchanged.
  • Keep model-ab.yml's rule of no prompt or corpus override input. Effort is a generator setting, not a corpus or judge knob.
  • Testing: bats with stubbed EVAL_ENGINE_CMD / EVAL_JUDGE_CMD, following the existing offline tests.

Project Structure Notes

The engine change is opt-in and inert by default, so it is safe to merge before the swap.

References

  • scripts/engine.sh
  • scripts/evals/model-ab.sh
  • scripts/evals/run-eval.sh
  • tests/test_model_ab.bats
  • evals/triage/holdout/cases.jsonl
  • docs/evals/model-ab-regression.md

Likely target surface

  • scripts/engine.sh
  • scripts/evals/model-ab.sh
  • tests/test_model_ab.bats

Story prepared by the BMAD Scrum Master (Bob) for epic #2101. Status: ready-for-dev.

Metadata

Metadata

Assignees

No one assigned

    Labels

    initiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions