Skip to content

[Phase 2] Register skill_candidate artifact type (skill rubric + pass/score channel) #615

Closed

Description

@github-actions

Story

As a Epic #581's strict-improvement skill gate,
I want a skill_candidate artifact type registered with the skill eval rubric and a pass/score output channel,
so that a candidate skill edit is reviewed by the same review brain and gated on a strict improvement.

Acceptance Criteria

  1. A skill_candidate registry entry maps to a skill rubric (Epic Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581's eval criteria) and a pass/score output channel.
  2. The content_ref adapter accepts a candidate skill edit (diff or file) and presents it to the rubric via scripts/engine.sh.
  3. The output channel emits a pass/fail + score the gate can consume (not a GitHub PR review).
  4. pr_diff and plan_json behavior are untouched; the registration is purely additive.
  5. bats coverage exercises skill_candidate resolution and a sample producing a score; the new test is registered in the lint.yml bats list.

Tasks / Subtasks

Dev Notes

Project Structure Notes

New rubric prompt under prompts/ (e.g. prompts/skill-review.md); registry entry + adapter + pass/score channel in scripts/lib/. Sequenced after plan_json so the registry grows one additive type at a time.

References

  • prompts/deep-review.md
  • scripts/engine.sh#run_agentic
  • scripts/lib/review-registry.sh
  • tests/

Likely target surface

  • prompts/skill-review.md
  • scripts/lib/review-registry.sh
  • tests/

Story prepared by the BMAD Scrum Master (Bob) for epic #610. Status: ready-for-dev.

Activity

  1. don-petry commented on Jun 17, 2026

    @don-petry
    Collaborator

    Dev-Lead Implementation Plan

    Issue: #615 — [Phase 2] Register skill_candidate artifact type (skill rubric + pass/score channel)

    Scope

    Additively register a third review artifact_type, skill_candidate, in the rubric registry — binding a new skill-eval rubric (aligned to Epic #581's strict-improvement eval criteria) to a pass/score output channel the gate consumes — mirroring the plan_json (#614) pattern, with pr_diff and plan_json left untouched.

    Implementation Checklist

    Tests to Write

    • tests/test_skill_review.bats — registry resolution (skill_candidate → rubric + channel), additive guards (pr_diff AND plan_json still resolve), driver wiring (sources registry + engine.sh, calls run_agentic deep tier, never post-pr-review.sh), output-channel scoring over a sample producing a pass/fail + score, and rejection of malformed/out-of-range scores.
    • Register tests/test_skill_review.bats in the lint.yml bats list (AC feat: add Copilot engine support via REVIEW_ENGINE toggle #5).

    Test Command

    bats tests/test_skill_review.bats (full suite via the lint.yml bats list); lint via bash .dev-lead/scripts/dev-lead-lint.sh

  2. don-petry commented on Jun 17, 2026

    @don-petry
    Collaborator

    Dev-Lead: Implementation Complete

    Registered a third review artifact_type, skill_candidate, additively beside pr_diff and plan_json — mirroring the plan_json (#614) pattern.

    Plan Execution

    • Authored the skill rubric prompts/skill-review.md — the review brain scores a candidate skill edit (diff or file) against Epic Initiative: Eval-gated, human-reviewed self-improving skills (SkillOpt-style) #581's strict-improvement eval criteria, aligned to evals/judge.md (0–1 score) and prompts/shared.md (risk taxonomy); not a parallel format.
    • Added the skill_candidate registry row {rubric: prompts/skill-review.md, output_channel: scripts/post-skill-score.sh} and bumped schema_version 2 → 3.
    • Implemented the pass/score output channel scripts/post-skill-score.sh — emits a machine-readable pass/fail + numeric score the strict-improvement gate consumes via SKILL_SCORE_OUTPUT. Never a GitHub review.
    • Implemented the content_ref adapter scripts/evals/review-skill.sh — presents the candidate edit to the rubric via engine.sh run_agentic (deep tier, no new model selector), delivers via the registry-resolved channel.
    • Added tests/test_skill_review.bats (30 tests) and registered it in the lint.yml bats list.
    • Documented the row in scripts/lib/README.md.

    Acceptance Criteria

    • AC1 — registry maps skill_candidate → skill rubric + pass/score channel. ✓
    • AC2 — adapter accepts a candidate skill edit (diff or file), presents it to the rubric via scripts/engine.sh. ✓
    • AC3 — channel emits pass/fail + score (gate signal, not a GitHub PR review). ✓
    • AC4 — pr_diff and plan_json rows byte-for-byte unchanged; purely additive (guarded by tests). ✓
    • AC5 — bats exercises skill_candidate resolution and a sample producing a score; registered in lint.yml. ✓

    Test Results

    tests/test_skill_review.bats: 1..30  — all ok (30/30)
    
    Full lint.yml bats list (27 files): 477/477 ok
    dev-lead-lint.sh: shellcheck (60 files) + agent profiles — all checks passed
    lint.yml: valid YAML
    

    Files Changed

    • prompts/skill-review.md (new): strict-improvement skill rubric.
    • scripts/post-skill-score.sh (new): pass/score output channel; validates verdict ∈ {pass,fail} and score ∈ [0,1].
    • scripts/evals/review-skill.sh (new): content_ref adapter reusing engine.sh deep-tier routing.
    • tests/test_skill_review.bats (new): 30 tests — registry resolution, additive guards for pr_diff/plan_json, driver wiring, output-channel scoring + rejection of malformed/out-of-range scores.
    • scripts/lib/review-registry.tsv: +1 data row, schema_version 2 → 3.
    • scripts/lib/README.md: documented the skill_candidate row and helper examples.
    • .github/workflows/lint.yml: registered the new bats file.

    Notes

  3. added 2 commits that reference this issue on Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions