Repository navigation
Closed
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Description
Story
As a release operator,
I want a strict-improvement comparator plus a documented manual propose->validate->PR procedure, with a test proving a worse candidate is rejected,
so that completes the idea's Deliverable 2 — proving the gate rejects regressions before any automation is built.
Acceptance Criteria
- A comparator (skills/evals/gate.sh, or a run-eval mode) scores an incumbent skill file and a candidate skill file against the same held-out set and exits non-zero unless the candidate STRICTLY beats the incumbent — ties and regressions are rejected.
- An offline test feeds a deliberately-worse candidate and asserts rejection, and a strictly-better candidate and asserts acceptance.
- A runbook under docs/ documents the manual flow: edit one skill file, run the gate locally, then open a PR that goes through the existing pr-review + CODEOWNER gates — reusing the idea->initiative pipeline unchanged.
- The runbook states the reward-hacking invariant explicitly: the proposer may edit only the skill markdown, never the cases.jsonl or the judge prompt.
Tasks / Subtasks
- Implement the strict-beat comparator over incumbent vs candidate scores (AC: test issue from agent #1)
- Add offline accept/reject tests proving ties and regressions are rejected (AC: Go-live improvements for PR review agent #2)
- Write docs/initiatives/skill-self-improvement.md runbook cross-linked to the human gates (AC: Add @claude delegation, auto-merge, and rebase handling #3, Optimize review: small-PR and incremental fast paths #4)
Dev Notes
- Strict improvement (no ties promoted) is the core gate from the idea and SkillOpt's held-out-validation discipline — encode 'candidate_score > incumbent_score', not '>='.
- Reuse the existing human-gated PR pipeline documented in docs/initiatives/idea-to-initiative-pipeline.md (two opt-in label gates, pr-review, CODEOWNER). This adds NO release-tag changes and is strictly more conservative than today's un-gated hand edits.
- The comparator is deterministic plumbing around run-eval.sh; the human dry-run is operator work, but the gate logic and the regression-rejection proof are fully automatable and testable here (so this story is not hands_off).
- Follow AGENTS.md bash + shellcheck standards; register any new bats in lint.yml.
Project Structure Notes
Adds gate logic beside run-eval.sh and a runbook under docs/initiatives/ alongside idea-to-initiative-pipeline.md and agentic-release-strategy.md.
References
- docs/initiatives/idea-to-initiative-pipeline.md
- skills/evals/run-eval.sh
- tests/test_initiative_planner.bats
- AGENTS.md
Likely target surface
skills/evals/gate.shdocs/initiatives/skill-self-improvement.mdtests/test_skill_evals.bats
Story prepared by the BMAD Scrum Master (Bob) for epic #581. Status: ready-for-dev.
Metadata
Metadata
Assignees
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Activity
don-petry commentedon Jun 18, 2026
Dev-Lead Implementation Plan
Issue: #586 — [Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof
Scope
Add a deterministic strict-improvement comparator that scores an incumbent vs candidate skill file against the same held-out set and exits non-zero unless the candidate STRICTLY beats the incumbent (ties + regressions rejected), plus an offline accept/reject proof and an operator runbook.
Implementation Checklist
scripts/evals/run-eval.sh— add aSKILL_PROMPT_FILEenv override so the scorer can score an arbitrary skill file against a skill's held-out cases (back-compat default unchanged)scripts/evals/gate.sh— new strict-beat comparator: runs run-eval.sh twice (incumbent, candidate) over the same holdout, emits a JSON verdict, exits 0 only whencandidate_score > incumbent_scoredocs/initiatives/skill-self-improvement.md— operator runbook: edit one skill markdown -> run gate locally -> open PR through existing pr-review + CODEOWNER gates; states the reward-hacking invariant (proposer edits only the skill markdown, never cases.jsonl / judge.md)Tests to Write (offline, stub engine, no network)
>, not>=)Test Command
bash scripts/run-bats.sh tests/test_skill_evals.bats(full suite via.github/workflows/lint.ymlbats list); lint viabash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jun 18, 2026
Dev-Lead: Implementation Complete
Plan Execution
SKILL_PROMPT_FILEoverride toscripts/evals/run-eval.sh(back-compat defaultprompts/<skill>.md) so an arbitrary skill file can be scored against the same held-out setscripts/evals/gate.sh— strict-improvement comparator: scores incumbent vs candidate over the SAMEholdout/cases.jsonl, accepts (exit 0) only whencandidate_score > incumbent_score; ties and regressions reject (exit 1); hard errors exit 2 (AC test issue from agent #1)tests/test_skill_evals.bats: strictly-better → accept, regression → reject, tie → reject (proves strict>not>=), score-0 candidate → reject (AC Go-live improvements for PR review agent #2)docs/initiatives/skill-self-improvement.mdrunbook: edit one skill markdown → run gate locally → PR through existing pr-review + CODEOWNER gates, reusing the idea→initiative pipeline unchanged (AC Add @claude delegation, auto-merge, and rebase handling #3), and stating the reward-hacking invariant explicitly — proposer edits only the skill markdown, nevercases.jsonlorjudge.md(AC Optimize review: small-PR and incremental fast paths #4)Test Results
Files Changed
scripts/evals/gate.sh(new): strict-beat comparator, deterministic plumbing around run-eval.sh; emits a JSON verdict{skill, incumbent_score, candidate_score, accepted, verdict}scripts/evals/run-eval.sh: addedSKILL_PROMPT_FILEenv override (default unchanged); held-out cases stay read-onlytests/test_skill_evals.bats: +1 override test, +8 gate tests (accept/reject/tie/score-0/verdict-shape/hard-errors/holdout-immutability)docs/initiatives/skill-self-improvement.md(new): runbook + reward-hacking invariant, cross-linked to idea-to-initiative-pipeline.md, evals/README.md, holdout-guard.ymlNotes
scripts/evals/gate.sh(besiderun-eval.sh) rather than the issue's hintedskills/evals/path, since the repo's eval tooling lives underscripts/evals/and there is noskills/tree.>and is directly covered by a dedicated test; ties and regressions are both rejected.::error::annotations are emitted at top level (not inside command substitutions) so they surface on stdout/CI; verified manually against an empty-cases skill (exit 2).feat: implement issue #586 — [Phase 2] Strict-improvement gate + manu…