Repository navigation
Closed
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Description
Story
As a Epic #581's strict-improvement skill gate,
I want a skill_candidate artifact type registered with the skill eval rubric and a pass/score output channel,
so that a candidate skill edit is reviewed by the same review brain and gated on a strict improvement.
Acceptance Criteria
- A skill_candidate registry entry maps to a skill rubric (Epic Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581's eval criteria) and a pass/score output channel.
- The content_ref adapter accepts a candidate skill edit (diff or file) and presents it to the rubric via scripts/engine.sh.
- The output channel emits a pass/fail + score the gate can consume (not a GitHub PR review).
- pr_diff and plan_json behavior are untouched; the registration is purely additive.
- bats coverage exercises skill_candidate resolution and a sample producing a score; the new test is registered in the lint.yml bats list.
Tasks / Subtasks
- Author the skill rubric under prompts/ from Epic Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581's eval criteria (strict-improvement gate), aligned with that initiative's eval-case format. (AC: test issue from agent #1)
- Add the skill_candidate registry entry {rubric, pass/score channel} and a content_ref adapter for a candidate skill edit. (AC: test issue from agent #1, Go-live improvements for PR review agent #2)
- Implement the pass/score output channel (gate signal, not a GitHub review). (AC: Add @claude delegation, auto-merge, and rebase handling #3)
- Add bats coverage, register it in lint.yml, and confirm the pr_diff and plan_json guards stay green. (AC: Optimize review: small-PR and incremental fast paths #4, feat: add Copilot engine support via REVIEW_ENGINE toggle #5)
Dev Notes
- Epic Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581 ('Eval-gated, human-reviewed self-improving skills') defines the eval criteria; its phase issues — [Phase 1] Define held-out eval-case format + seed prompts/triage.md case set #582 (held-out eval-case format), [Phase 1] Build the deterministic eval scorer + offline tests (triage) #583 (deterministic scorer), [Phase 2] Extend the harness to prompts/deep-review.md via an LLM-judge scorer #585 (LLM-judge for prompts/deep-review.md), [Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586 (strict-improvement gate) — are where rubric/scorer semantics live. The skill rubric here must align with that eval format, not invent a parallel one.
- Output is a gate signal (pass/score) consumed by Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581's propose->validate->PR runbook — do not post a GitHub review and do not call post-pr-review.sh.
- Additive only: registering skill_candidate must not touch the pr_diff or plan_json registry entries.
- Reuse scripts/engine.sh routing (run_agentic) for invocation.
Project Structure Notes
New rubric prompt under prompts/ (e.g. prompts/skill-review.md); registry entry + adapter + pass/score channel in scripts/lib/. Sequenced after plan_json so the registry grows one additive type at a time.
References
- prompts/deep-review.md
- scripts/engine.sh#run_agentic
- scripts/lib/review-registry.sh
- tests/
Likely target surface
prompts/skill-review.mdscripts/lib/review-registry.shtests/
Story prepared by the BMAD Scrum Master (Bob) for epic #610. Status: ready-for-dev.
Metadata
Metadata
Assignees
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Activity
don-petry commentedon Jun 17, 2026
Dev-Lead Implementation Plan
Issue: #615 — [Phase 2] Register skill_candidate artifact type (skill rubric + pass/score channel)
Scope
Additively register a third review artifact_type,
skill_candidate, in the rubric registry — binding a new skill-eval rubric (aligned to Epic #581's strict-improvement eval criteria) to a pass/score output channel the gate consumes — mirroring theplan_json(#614) pattern, withpr_diffandplan_jsonleft untouched.Implementation Checklist
prompts/skill-review.md— new skill rubric: the review brain scores a candidate skill edit (diff or file) against Epic Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581's eval criteria ([Phase 1] Define held-out eval-case format + seed prompts/triage.md case set #582 case format, [Phase 2] Extend the harness to prompts/deep-review.md via an LLM-judge scorer #585 judge scoring), emittingverdict(pass/fail) + numericscore[0,1] + findings. Reuses theevals/judge.mdscoring vocabulary; not a parallel format.scripts/lib/review-registry.tsv— add one data rowskill_candidate<TAB>prompts/skill-review.md<TAB>scripts/post-skill-score.sh; bumpschema_version2 → 3. pr_diff/plan_json rows unchanged.scripts/post-skill-score.sh— pass/score output channel (gate signal, not a GitHub review). Validates verdict ∈ {pass,fail} + numeric score ∈ [0,1], writes the score JSON toSKILL_SCORE_OUTPUTfor the strict-improvement gate ([Phase 2] Strict-improvement gate + manual propose->validate->PR runbook with regression-rejection proof #586) to consume. Never calls post-pr-review.sh / gh pr review.scripts/evals/review-skill.sh— content_ref adapter: accepts a candidate skill edit (diff or file), resolves {rubric, output_channel} from the registry, presents it to the rubric viaengine.sh run_agentic(deep tier, no new model selector), delivers via the resolved channel.scripts/lib/README.md— document theskill_candidaterow in the artifact-contract table.Tests to Write
tests/test_skill_review.bats— registry resolution (skill_candidate → rubric + channel), additive guards (pr_diff AND plan_json still resolve), driver wiring (sources registry + engine.sh, calls run_agentic deep tier, never post-pr-review.sh), output-channel scoring over a sample producing a pass/fail + score, and rejection of malformed/out-of-range scores.tests/test_skill_review.batsin thelint.ymlbats list (AC feat: add Copilot engine support via REVIEW_ENGINE toggle #5).Test Command
bats tests/test_skill_review.bats(full suite via thelint.ymlbats list); lint viabash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jun 17, 2026
Dev-Lead: Implementation Complete
Registered a third review artifact_type,
skill_candidate, additively besidepr_diffandplan_json— mirroring theplan_json(#614) pattern.Plan Execution
prompts/skill-review.md— the review brain scores a candidate skill edit (diff or file) against Epic Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581's strict-improvement eval criteria, aligned toevals/judge.md(0–1 score) andprompts/shared.md(risk taxonomy); not a parallel format.skill_candidateregistry row{rubric: prompts/skill-review.md, output_channel: scripts/post-skill-score.sh}and bumpedschema_version2 → 3.scripts/post-skill-score.sh— emits a machine-readable pass/fail + numeric score the strict-improvement gate consumes viaSKILL_SCORE_OUTPUT. Never a GitHub review.scripts/evals/review-skill.sh— presents the candidate edit to the rubric viaengine.sh run_agentic(deep tier, no new model selector), delivers via the registry-resolved channel.tests/test_skill_review.bats(30 tests) and registered it in thelint.ymlbats list.scripts/lib/README.md.Acceptance Criteria
skill_candidate→ skill rubric + pass/score channel. ✓scripts/engine.sh. ✓pr_diffandplan_jsonrows byte-for-byte unchanged; purely additive (guarded by tests). ✓skill_candidateresolution and a sample producing a score; registered inlint.yml. ✓Test Results
Files Changed
prompts/skill-review.md(new): strict-improvement skill rubric.scripts/post-skill-score.sh(new): pass/score output channel; validates verdict ∈ {pass,fail} and score ∈ [0,1].scripts/evals/review-skill.sh(new): content_ref adapter reusing engine.sh deep-tier routing.tests/test_skill_review.bats(new): 30 tests — registry resolution, additive guards for pr_diff/plan_json, driver wiring, output-channel scoring + rejection of malformed/out-of-range scores.scripts/lib/review-registry.tsv: +1 data row, schema_version 2 → 3.scripts/lib/README.md: documented theskill_candidaterow and helper examples..github/workflows/lint.yml: registered the new bats file.Notes
post-plan-findings.shdefers verdict semantics. No evals/ paths are touched, so the holdout guard is not engaged.feat: implement issue #615 — [Phase 2] Register skill_candidate artif…
feat: implement issue #615 — [Phase 2] Register skill_candidate artif…