Skip to content

[Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586

Closed

Description

@github-actions

Story

As a release operator,
I want a strict-improvement comparator plus a documented manual propose->validate->PR procedure, with a test proving a worse candidate is rejected,
so that completes the idea's Deliverable 2 — proving the gate rejects regressions before any automation is built.

Acceptance Criteria

  1. A comparator (skills/evals/gate.sh, or a run-eval mode) scores an incumbent skill file and a candidate skill file against the same held-out set and exits non-zero unless the candidate STRICTLY beats the incumbent — ties and regressions are rejected.
  2. An offline test feeds a deliberately-worse candidate and asserts rejection, and a strictly-better candidate and asserts acceptance.
  3. A runbook under docs/ documents the manual flow: edit one skill file, run the gate locally, then open a PR that goes through the existing pr-review + CODEOWNER gates — reusing the idea->initiative pipeline unchanged.
  4. The runbook states the reward-hacking invariant explicitly: the proposer may edit only the skill markdown, never the cases.jsonl or the judge prompt.

Tasks / Subtasks

Dev Notes

  • Strict improvement (no ties promoted) is the core gate from the idea and SkillOpt's held-out-validation discipline — encode 'candidate_score > incumbent_score', not '>='.
  • Reuse the existing human-gated PR pipeline documented in docs/initiatives/idea-to-initiative-pipeline.md (two opt-in label gates, pr-review, CODEOWNER). This adds NO release-tag changes and is strictly more conservative than today's un-gated hand edits.
  • The comparator is deterministic plumbing around run-eval.sh; the human dry-run is operator work, but the gate logic and the regression-rejection proof are fully automatable and testable here (so this story is not hands_off).
  • Follow AGENTS.md bash + shellcheck standards; register any new bats in lint.yml.

Project Structure Notes

Adds gate logic beside run-eval.sh and a runbook under docs/initiatives/ alongside idea-to-initiative-pipeline.md and agentic-release-strategy.md.

References

  • docs/initiatives/idea-to-initiative-pipeline.md
  • skills/evals/run-eval.sh
  • tests/test_initiative_planner.bats
  • AGENTS.md

Likely target surface

  • skills/evals/gate.sh
  • docs/initiatives/skill-self-improvement.md
  • tests/test_skill_evals.bats

Story prepared by the BMAD Scrum Master (Bob) for epic #581. Status: ready-for-dev.

Activity

  1. don-petry commented on Jun 18, 2026

    @don-petry
    Collaborator

    Dev-Lead Implementation Plan

    Issue: #586 — [Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof

    Scope

    Add a deterministic strict-improvement comparator that scores an incumbent vs candidate skill file against the same held-out set and exits non-zero unless the candidate STRICTLY beats the incumbent (ties + regressions rejected), plus an offline accept/reject proof and an operator runbook.

    Implementation Checklist

    • scripts/evals/run-eval.sh — add a SKILL_PROMPT_FILE env override so the scorer can score an arbitrary skill file against a skill's held-out cases (back-compat default unchanged)
    • scripts/evals/gate.sh — new strict-beat comparator: runs run-eval.sh twice (incumbent, candidate) over the same holdout, emits a JSON verdict, exits 0 only when candidate_score > incumbent_score
    • docs/initiatives/skill-self-improvement.md — operator runbook: edit one skill markdown -> run gate locally -> open PR through existing pr-review + CODEOWNER gates; states the reward-hacking invariant (proposer edits only the skill markdown, never cases.jsonl / judge.md)

    Tests to Write (offline, stub engine, no network)

    • strictly-better candidate is ACCEPTED (exit 0)
    • regressed candidate is REJECTED (exit non-zero)
    • a TIE is REJECTED (strict >, not >=)
    • verdict JSON carries incumbent_score, candidate_score, accepted flag
    • missing args / missing candidate file is a hard error
    • gate never writes the held-out cases file

    Test Command

    bash scripts/run-bats.sh tests/test_skill_evals.bats (full suite via .github/workflows/lint.yml bats list); lint via bash .dev-lead/scripts/dev-lead-lint.sh

  2. don-petry commented on Jun 18, 2026

    @don-petry
    Collaborator

    Dev-Lead: Implementation Complete

    Plan Execution

    • Added SKILL_PROMPT_FILE override to scripts/evals/run-eval.sh (back-compat default prompts/<skill>.md) so an arbitrary skill file can be scored against the same held-out set
    • Implemented scripts/evals/gate.sh — strict-improvement comparator: scores incumbent vs candidate over the SAME holdout/cases.jsonl, accepts (exit 0) only when candidate_score > incumbent_score; ties and regressions reject (exit 1); hard errors exit 2 (AC test issue from agent #1)
    • Added offline accept/reject proofs to tests/test_skill_evals.bats: strictly-better → accept, regression → reject, tie → reject (proves strict > not >=), score-0 candidate → reject (AC Go-live improvements for PR review agent #2)
    • Wrote docs/initiatives/skill-self-improvement.md runbook: edit one skill markdown → run gate locally → PR through existing pr-review + CODEOWNER gates, reusing the idea→initiative pipeline unchanged (AC Add @claude delegation, auto-merge, and rebase handling #3), and stating the reward-hacking invariant explicitly — proposer edits only the skill markdown, never cases.jsonl or judge.md (AC Optimize review: small-PR and incremental fast paths #4)

    Test Results

    Full bats suite (lint.yml list, 28 files): 1..516  — all pass, exit 0
    tests/test_skill_evals.bats: 25/25 ok (9 new gate/override tests)
    shellcheck scripts/evals/gate.sh scripts/evals/run-eval.sh: clean
    .dev-lead/scripts/dev-lead-lint.sh: all checks passed
    

    Files Changed

    • scripts/evals/gate.sh (new): strict-beat comparator, deterministic plumbing around run-eval.sh; emits a JSON verdict {skill, incumbent_score, candidate_score, accepted, verdict}
    • scripts/evals/run-eval.sh: added SKILL_PROMPT_FILE env override (default unchanged); held-out cases stay read-only
    • tests/test_skill_evals.bats: +1 override test, +8 gate tests (accept/reject/tie/score-0/verdict-shape/hard-errors/holdout-immutability)
    • docs/initiatives/skill-self-improvement.md (new): runbook + reward-hacking invariant, cross-linked to idea-to-initiative-pipeline.md, evals/README.md, holdout-guard.yml

    Notes

    • The gate is placed at scripts/evals/gate.sh (beside run-eval.sh) rather than the issue's hinted skills/evals/ path, since the repo's eval tooling lives under scripts/evals/ and there is no skills/ tree.
    • Tie-rejection is enforced via jq strict > and is directly covered by a dedicated test; ties and regressions are both rejected.
    • Hard-error ::error:: annotations are emitted at top level (not inside command substitutions) so they surface on stdout/CI; verified manually against an empty-cases skill (exit 2).
    • No release-tag changes; the local gate is an advisory pre-flight — merge authority stays with pr-review + CODEOWNER, strictly more conservative than today's un-gated hand edits.
  3. added a commit that references this issue on Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions