Repository navigation
Closed
Labels
initiativeEpic / initiative tracking issueEpic / initiative tracking issue
Description
Story
As a maintainer judging a triage-model swap,
I want to A/B the candidate at a chosen reasoning effort and see false-approves apart from false-escalates,
so that the effort setting is picked from data rather than inherited from the CLI default, and a missed escalation, which costs more than an extra one, cannot hide inside an aggregate score.
Acceptance Criteria
run_triageinscripts/engine.shpasses--effort <level>on the claude branch only when an opt-in env var (for exampleCLAUDE_TRIAGE_EFFORT) is set to a valid level. When it is unset the CLI invocation is byte-for-byte unchanged (bats asserts both cases).scripts/evals/model-ab.shaccepts an optional candidate-arm effort, applied only to the candidate arm, and records it in the JSON evidence object.- For the
triageset, the evidence JSON reports per arm the counts of false-approves (expected escalate, predicted approve) and false-escalates (expected approve, predicted escalate). - The accept/regression rule stays
>=per set, with exit codes 0 / 1 / 2 unchanged (the Stage 5 gate stays as it is, per discussion Enable Haiku 5.5 #2095). The confusion counts are reported only and are judged against the epic's false-approve metric in the evidence story. - The held-out checksum guard still covers
cases.jsonl,scorer.jsonandevals/judge.md. Tests intests/test_model_ab.batscover the new evidence fields offline (no network) and show the exit code is unaffected by them.
Tasks / Subtasks
- Add the opt-in triage effort env to run_triage (AC: test issue from agent #1)
- Plumb candidate-arm effort through model-ab.sh (AC: Go-live improvements for PR review agent #2)
- Compute and report the confusion counts for triage without changing the exit semantics (AC: Add @claude delegation, auto-merge, and rebase handling #3, Optimize review: small-PR and incremental fast paths #4, feat: add Copilot engine support via REVIEW_ENGINE toggle #5)
Dev Notes
- Today
run_agenticpins--effort highonly for deep/audit/single. Triage keeps the CLI default, which on Haiku 5.5 ismediumwith adaptive thinking (research pass, gap 4.1). - Eval safeguard (overfitting): the held-out set is 5 triage cases, so do NOT choose effort by best score. The decision rule recorded in the evidence story is: of the effort arms that pass non-regression, pick the cheapest. Shadow traffic then confirms the choice. The held-out cases, scorer and judge stay immutable: there is no tuning input, and the checksum guard is unchanged.
- Keep
model-ab.yml's rule of no prompt or corpus override input. Effort is a generator setting, not a corpus or judge knob. - Testing: bats with stubbed
EVAL_ENGINE_CMD/EVAL_JUDGE_CMD, following the existing offline tests.
Project Structure Notes
The engine change is opt-in and inert by default, so it is safe to merge before the swap.
References
- scripts/engine.sh
- scripts/evals/model-ab.sh
- scripts/evals/run-eval.sh
- tests/test_model_ab.bats
- evals/triage/holdout/cases.jsonl
- docs/evals/model-ab-regression.md
Likely target surface
scripts/engine.shscripts/evals/model-ab.shtests/test_model_ab.bats
Story prepared by the BMAD Scrum Master (Bob) for epic #2101. Status: ready-for-dev.
Metadata
Metadata
Assignees
Labels
initiativeEpic / initiative tracking issueEpic / initiative tracking issue
Activity
github-actions commentedon Oct 9, 2026
Superseded by the re-planned initiative epic #2113 (re-planned from idea discussion #2095).