Skip to content

[Phase 1] Regression guard locking pr_diff behavior through the contract #613

Closed

Description

@github-actions

Story

As a reviewer of changes to the review agent,
I want an automated regression guard that pins pr_diff's routing and output through the contract,
so that registering new artifact types can never silently regress production PR review.

Acceptance Criteria

  1. A new bats test asserts the registry resolves pr_diff to the prompts cascade + the post-pr-review output channel.
  2. The test asserts the pr_diff verdict/marker contract (decision approve|escalate, risk LOW|MEDIUM|HIGH, idempotency marker format) is unchanged.
  3. The new test file is added to the lint.yml repo-specific bats list so CI runs it.
  4. shellcheck and the full bats suite pass.

Tasks / Subtasks

Dev Notes

  • Model the new test after tests/test_review_cycle.bats, tests/test_ci_status.bats, and tests/test_initiative_planner.bats; reuse fixtures under tests/fixtures/.
  • AGENTS.md notes the bats list in lint.yml is intentionally extended with repo-specific files (e.g. tests/test_push_protection.bats) beyond the org template, and template syncs must not reset it — so add the new file to that list.
  • This guard is the explicit precondition from discussion 💡 Generalize pr-review into a Context-Adaptive "Review" Agent (PRs, Plans, Skills) #607 for registering plan_json/skill_candidate: those stories are blocked until this passes.

Project Structure Notes

New test lives in tests/ following the existing .bats convention; the only other touch is the lint.yml bats list.

References

  • tests/test_review_cycle.bats
  • tests/test_initiative_planner.bats
  • tests/fixtures
  • .github/workflows/lint.yml
  • AGENTS.md#workflow-files

Likely target surface

  • tests/test_review_registry.bats
  • .github/workflows/lint.yml

Story prepared by the BMAD Scrum Master (Bob) for epic #610. Status: ready-for-dev.

Activity

  1. don-petry commented on Jun 15, 2026

    @don-petry
    Collaborator

    Dev-Lead Implementation Plan

    Issue: #613 — [Phase 1] Regression guard locking pr_diff behavior through the contract

    Scope

    Add a regression guard that pins pr_diff's verdict/marker contract (decision approve|escalate, risk LOW|MEDIUM|HIGH, idempotency-marker format) so registering new artifact types can never silently regress production PR review.

    Current state (important)

    Implementation Checklist

    • Extend tests/test_review_registry.bats with a verdict/marker contract section that drives scripts/post-pr-review.sh:
      • approve and escalate are accepted decisions (DRY_RUN echoes decision + risk)
      • any other decision is rejected non-zero; skip is a no-op (exit 100)
      • risk values LOW|MEDIUM|HIGH are surfaced unchanged
      • the idempotency marker format <!-- pr-review-agent v1 sha=<HEX> --> (+ detection regex / superseded sentinel) is unchanged
    • Confirm tests/test_review_registry.bats remains in the lint.yml bats list (AC Add @claude delegation, auto-merge, and rebase handling #3 — already present).

    Tests to Write

    • tests/test_review_registry.bats: verdict-contract tests (accept approve/escalate, reject unknown decision, skip no-op, risk surfaced)
    • tests/test_review_registry.bats: marker-format guard (canonical marker + detection regex + superseded sentinel)

    Test Command

    bats tests/test_review_registry.bats (and the full bats suite from lint.yml) + shellcheck --severity=warning -x + bash .dev-lead/scripts/dev-lead-lint.sh

    Note

    This is an additive regression guard (test-only, no production code change), so the Phase-2 red→green TDD cycle is noted as the documented exception — the guard must pass against today's correct behavior and fail only if the contract regresses.

  2. don-petry commented on Jun 15, 2026

    @don-petry
    Collaborator

    Dev-Lead: Implementation Complete

    Plan Execution

    What the guard pins (AC #2)

    Drives scripts/post-pr-review.sh (the resolved pr_diff output_channel) to lock its contract so registering a new artifact_type can never silently regress production PR review:

    • Decision vocabulary is exactly approve | escalate; an unknown decision is rejected non-zero; skip is an explicit no-op (exit 100).
    • Risk LOW|MEDIUM|HIGH round-trips unchanged, and HIGH is treated specially (bypasses AI delegation → human escalation).
    • Idempotency marker format is unchanged: canonical <!-- pr-review-agent v1 sha=$PR_HEAD_SHA -->, detection regex pr-review-agent v1 sha=[a-f0-9]+, and the <!-- pr-review-agent superseded --> sentinel.

    Behavioral cases run hermetically: approve/escalate use DRY_RUN=true (exits before any gh/network call); skip / unknown-decision use DRY_RUN=false because DRY_RUN=true short-circuits before decision validation. Verified the marker guard fails on a v1→v2 mutation.

    Test Results

    $ bats <full lint.yml bats list>
    1..398
    ... (0 failures)
    
    $ find scripts -name "*.sh" -exec shellcheck --severity=warning -x {} +
    SHELLCHECK OK
    
    $ bash .dev-lead/scripts/dev-lead-lint.sh
      [lint] shellcheck on 55 file(s)...
      [lint] validating 4 agent profile(s)...
      [lint] all checks passed
    
    $ bats tests/test_review_registry.bats
    1..22
    ok 14 verdict contract: decision=approve is accepted and surfaced (DRY_RUN)
    ok 15 verdict contract: decision=escalate is accepted and surfaced (DRY_RUN)
    ok 16 verdict contract: an unknown decision is rejected non-zero
    ok 17 verdict contract: decision=skip is an explicit no-op (exit 100), not a verdict
    ok 18 verdict contract: risk LOW|MEDIUM|HIGH are surfaced unchanged (DRY_RUN)
    ok 19 risk contract: HIGH risk bypasses AI delegation (forces human escalation)
    ok 20 marker contract: post-pr-review.sh emits the canonical v1 sha marker
    ok 21 marker contract: prior-agent items are detected by the v1 sha=[a-f0-9]+ regex
    ok 22 marker contract: superseded items carry the superseded sentinel
    

    Files Changed

    • tests/test_review_registry.bats: +88 lines — write_verdict helper + 9 verdict/marker contract tests (decision vocabulary, risk vocabulary, idempotency marker format).

    Notes

  3. added a commit that references this issue on Jun 15, 2026
  4. added 2 commits that reference this issue on Jul 9, 2026
  5. added 3 commits that reference this issue on Aug 2, 2026
  6. 1 remaining item

  7. added 14 commits that reference this issue on Aug 7, 2026
  8. added a commit that references this issue on Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions