Skip to content

[Phase 4] Promotion mechanism: informational to blocking after two clean audit cycles #647

Closed

Description

@don-petry

Story

As a org standards maintainer,
I want track structural-check false positives across audit cycles and provide a documented, human-gated promotion to a blocking check,
so that the check only becomes enforcing once proven low-false-positive, addressing the adversarial false-positive risk.

Acceptance Criteria

  1. The audit summary issue records, per cycle, the count of structural findings and any confirmed false positives, so two consecutive clean cycles are observable from the issue history.
  2. docs/initiatives/agents-md-validation.md documents the promotion gate: informational becomes blocking only after 2 consecutive audit cycles with zero confirmed false positives AND explicit maintainer sign-off (never automatic).
  3. The mechanism to flip to blocking is a discrete, reviewable change (switch the linter to failing mode / add the required status check) that is gated behind the documented criteria; this story delivers the mechanism and documentation, not the promotion decision.
  4. The story does not itself enable blocking; the actual flip remains a human decision recorded against the documented criteria.
  5. Per-cycle structural finding counts and each false-positive determination are recorded durably and tamper-evidently — an append-only, committed cycle log (e.g. under docs/initiatives/) rather than only an editable summary issue — and every false-positive confirmation names the maintainer who made it, so the 'two consecutive clean cycles' claim is independently auditable and its baseline record is immutable.

Tasks / Subtasks

Dev Notes

  • The promotion decision is human-gated by design — the discussion mandates promoting to blocking only after 2 audit cycles of zero false positives; encode the criteria and the toggle, but never auto-enable.
  • Reuse Story 3's informational summary section as the per-cycle record; add explicit finding/false-positive counts so 'two clean cycles' is auditable from issue history.
  • The failing-mode flag added to the linter in Story 2 is the toggle point; flipping the local CI check (Story 4) to a required status check is the branch-protection side of promotion — document both, enable neither in this story.
  • This mirrors the repo's convention that activation/enforcement is a deliberate human decision, consistent with the initiative-planner's inert-until-human-activates pattern.
  • Guard the promotion gate against reward-hacking: the 'zero false positives over 2 cycles' inputs must not live only in an editable issue. Persist per-cycle counts and false-positive calls to an append-only committed log and attribute each false-positive confirmation to a named maintainer, so a clean-cycle claim cannot be fabricated by silently editing or suppressing findings.

Project Structure Notes

Doc edits to docs/initiatives/agents-md-validation.md and summary-output edits in scripts/aw-standards-sync.sh; the toggle stays disabled. No branch-protection change is applied here.

References

  • docs/initiatives/idea-to-initiative-pipeline.md
  • scripts/aw-standards-sync.sh
  • .github/workflows/lint.yml

Likely target surface

  • docs/initiatives/agents-md-validation.md (new/edit)
  • scripts/aw-standards-sync.sh

Story prepared by the BMAD Scrum Master (Bob) for epic #642. Status: ready-for-dev.

Activity

  1. don-petry commented on Sep 7, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #647 — [Phase 4] Promotion mechanism: informational to blocking after two clean audit cycles

    Scope

    Deliver the human-gated informational → blocking promotion mechanism and its documentation for the AGENTS.md structural check — per-cycle finding counts surfaced in the audit summary, an append-only committed cycle log attributing every false-positive determination to a named maintainer, and a discrete (disabled) blocking toggle gated behind the documented 2-clean-cycle + maintainer-sign-off criteria. The story enables nothing.

    Implementation Checklist

    Tests to Write

    • tests/agents_md_cycle_log.bats (new) — clean streak detected; a dirty cycle breaks the streak; missing maintainer fails validate; eligibility never signals auto-promotion and always states sign-off is still required
    • test/scripts/compliance-audit/agents-md-structural-lint.bats — assert the summary now emits the per-cycle structural finding count and links the cycle log

    Test Command

    bats test/scripts/compliance-audit/ tests/agents_md_cycle_log.bats tests/test_agents_md_rules_config.bats tests/agents_md_lint.bats (plus bash .dev-lead/scripts/dev-lead-lint.sh)

  2. don-petry commented on Sep 7, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    Test Results

    ok 27 a confirmed false positive in the latest cycle breaks the clean streak
    ok 28 eligibility always states sign-off is required and never auto-promotes (AC #4)
    ok 29 the committed cycle log exists and is well-formed
    ok 30 the committed cycle log is not promotion-eligible on merge (ships inert)
    ok 31 rule-set file exists
    ok 32 rule set parses as valid JSON
    ok 33 _schema_version is a positive integer
    ok 34 an inline _note documents the file is inert (no linter consumes it yet)
    ok 35 all required top-level keys exist
    ok 36 scope is 'structural' (AC #3)
    ok 37 out_of_scope is a non-empty array explicitly excluding semantic/content checks (AC #3)
    ok 38 spec_snapshot records a source and a date (AC #1 task 2)
    ok 39 rules is a non-empty array
    ok 40 every rule has a unique id, a description and an element
    ok 41 every rule is tagged exactly 'required' or 'recommended' (AC #2)
    ok 42 at least one required and at least one recommended rule exist (AC #2)
    ok 43 the required rules exist and are 'required'
    ok 44 contested section-presence rules stay 'recommended', never 'required' (Dev Notes)
    ok 45 promotion gate encodes two clean cycles and maintainer sign-off (AC #1 task 4)
    ok 46 promotion default_severity starts informational (non-blocking) (epic scope)
    

    Full run: 271 bats pass, 0 failures across test/scripts/compliance-audit/, tests/test_agents_md_rules_config.bats, tests/agents_md_lint.bats, tests/agents_md_cycle_log.bats. bash .dev-lead/scripts/dev-lead-lint.sh passes (shellcheck clean); yamllint clean on the workflow.

    Files Changed

    • docs/initiatives/agents-md-validation.md: promotion gate fully documented (§4.1–4.3); status → Phase 4; references updated
    • docs/initiatives/agents-md-validation-cycle-log.md (new): append-only committed cycle log, inert on merge
    • scripts/agents-md-cycle-log.sh (new): pure reader/validator for the cycle log; never promotes
    • scripts/compliance-audit.sh: per-cycle structural finding count + cycle-log pointer in the audit summary
    • scripts/lib/agents-md-rules.json: promotion.cycle_log + disabled promotion.toggle
    • test/scripts/compliance-audit/agents-md-structural-lint.bats: +5 tests (per-cycle count, cycle-log link, counter helper)
    • tests/agents_md_cycle_log.bats (new): 12 contract tests for validate/eligibility
    • .github/workflows/agents-md-rules-tests.yml: CI wiring for the new artifacts

    Notes

    • This story arms nothing. promotion.default_severity stays informational, promotion.toggle.enabled is false, and no branch-protection change is applied. The committed cycle log records zero cycles, so agents-md-cycle-log.sh eligibility … 2 reports clean-cycles-met=false. The flip remains a deliberate, human-gated decision recorded against the documented criteria.
    • The audit cannot itself confirm false positives (a human review), so it records the machine finding count; the maintainer's confirmed-false-positive determination is logged durably — with attribution — in the append-only cycle log, closing the reward-hacking hole an editable summary issue would leave open.
    • Reference-script path note: the issue's likely-surface listed scripts/aw-standards-sync.sh, but Phase 3 wired the structural linter into scripts/compliance-audit.sh (the live audit), so the per-cycle summary change landed there, consistent with the existing Phase 3 integration.
  3. added a commit that references this issue on Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions