Skip to content

compliance-audit ignores the derived <id>:hands-off label family — missing persona opt-out labels are never reported or remediated #1139

Closed

Description

@don-petry

Story

As the org,
I want the compliance audit to check and remediate the derived <id>:hands-off label family, not just the seven fixed labels,
so that a repo missing a persona opt-out label is reported and fixed automatically instead of being discovered by hand.

Filed from combsmith/.github-private#1644 (epic combsmith/.github-private#1643). No new standard is needed — standards/github-settings.md §"Labels — Standard Set" already mandates both the fixed set and the derived family. What is missing is the check and the fix.

The gap (verified 2026-09-15)

check_labels() in scripts/compliance-audit.sh iterates REQUIRED_LABEL_SPECS — seven hardcoded entries (security, dependencies, scorecard, bug, enhancement, documentation, in-progress). The derived family is never consulted. The only opt-out label the audit knows is the DEV_LEAD_HANDS_OFF_LABEL special case, which exists so a finding issue can carry it — not as a compliance check.

scripts/compliance-remediate.sh keeps a second, independent hardcoded copy of the same seven labels (LABEL_COLORS / LABEL_DESCS), so even once the audit reports a missing persona label, the remediator cannot create it.

Consequence, from the labels API on 2026-09-15 — nine personas exist, so nine <id>:hands-off labels are expected per repo:

Repo <id>:hands-off labels
.github, .github-private, bmad-bgreat-suite, TalkTerm 9
markets, ContentTwin, repo-template, broodly, incubator 1 (dev-lead:hands-off only)

The weekly audit (compliance-audit-and-improvement.yml) ran green on 2026-09-11 with five repos in that state and filed no finding, because no check covers the family.

Why the labels are absent differs per repo, and is mostly tracked elsewhere:

  • ContentTwin, repo-template, broodly, incubator have no apply-repo-settings caller stub yet → #1045 Step D: add apply-repo-settings.yml to DEPLOYABLE_WORKFLOWS (Step C done — v1-stable exists) #1128.
  • markets' workflow is disabled_manually and its last run (2026-08-31) failed for a missing GH_TOKEN_ADMIN. Being disabled looks like a deliberate human choice — report it, do not re-enable it.
  • TalkTerm self-healed on 2026-09-14 (run 34844937654) once the check-suites 403 of .github-private#1209 was fixed. Before that it failed weekly, having aborted mid-run.

That last case is the reason for AC #5: the applier should not lose label provisioning because an unrelated settings step failed first.

Acceptance Criteria

  1. check_labels() checks the derived <id>:hands-off family in addition to the fixed set, deriving it from the persona manifests exactly as apply-repo-settings.sh (persona_opt_out_label_configs()) does. Adding a persona must require no edit to any label list — the constraint from Provision <id>:hands-off labels so persona opt-out actually works #756.
  2. One source of truth. The audit and the remediator must not hold two hardcoded label tables. Extract the fixed set and the derivation into a shared helper both source.
  3. Fail closed. If the manifests cannot be read, emit an error or finding. Never treat an unreadable manifest list as "no persona labels required" — mirror the guard apply-repo-settings.sh already has for a failed derivation.
  4. Remediation creates the missing labels through the existing idempotent path. The read-only default stands (compliance-audit.sh closes live findings: fail-open closure, umbrella title mismatch, no execution guard #1036 AC Add workflow, environment, and orchestration guidance #4): no mutation without --apply, and --dry-run reports without mutating.
  5. A failing settings step must not skip label provisioning. Check the step ordering and failure handling in apply-repo-settings.sh: if a hard failure (e.g. the check-suites PATCH) can abort the run before apply_labels(), decouple them so labels are still applied and the run still reports the failure. If ordering already guarantees this, say so in the PR with the evidence and treat the AC as met.
  6. Before/after evidence in the PR description: the <id>:hands-off count per repo from the labels API, for all nine repos.
  7. shellcheck --severity=warning -x clean. Tests in the existing bats suite cover: a repo missing one persona label produces a finding; unreadable manifests fail closed; a compliant repo produces none.

Out of scope

Budget guard

  1. Shared derivation helper + the audit check + tests (AC Addressing PR comments #1–Add coding standards & principles section to AGENTS.md #3, docs: document branch protection rules, rulesets, and auto-merge policy #7). This is the commit that matters.
  2. Wire the remediator to the shared list (AC Add workflow, environment, and orchestration guidance #4).
  3. The applier decoupling (AC Add stacked PR strategy and Epic-level workflow guidance #5) and the evidence table (AC feat: add Structured Logging and CQRS standards #6).

If you run low on budget, stop after any step with a green PR and file the rest as a follow-up. Do not attempt all three and time out.

References

Activity

  1. don-petry commented on Sep 16, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #1139 — compliance-audit ignores the derived <id>:hands-off label family — missing persona opt-out labels are never reported or remediated

    Scope

    Extract the fixed label set + the persona opt-out derivation into one shared helper that the audit, remediator, and applier all source; teach check_labels() to check the derived <id>:hands-off family (fail-closed on unreadable manifests); wire the remediator to the shared list; verify the applier already decouples label provisioning from a failing settings step.

    Implementation Checklist

    • scripts/lib/labels.sh — new shared source of truth: STANDARD_LABEL_SPECS (fixed 7) + persona_opt_out_label_configs() (moved from applier) + std_label_spec() lookup helper + persona manifest config (PERSONA_MANIFEST_REPO/REF, PERSONA_OPT_OUT_COLOR)
    • scripts/apply-repo-settings.sh — source the lib; drop the now-shared copies (persona_opt_out_label_configs, manifest vars, color) and build apply_labels' fixed set from STANDARD_LABEL_SPECS; keep its existing cache wrapper untouched
    • scripts/compliance-audit.sh — source the lib; replace REQUIRED_LABEL_SPECS; rewrite check_labels() to check the fixed set + the derived persona family, failing closed with an error finding when derivation fails
    • scripts/compliance-remediate.sh — source the lib; drop LABEL_COLORS/LABEL_DESCS; look up color/desc via std_label_spec() (fixed set + derived family) so a missing persona label can be created
    • AC Add stacked PR strategy and Epic-level workflow guidance #5 — audit apply-repo-settings.sh step ordering; confirm apply_labels cannot be aborted by an earlier failing step (already guaranteed by apply_repo's step || FAILED_STEPS+=(...) — apply-repo-settings.sh aborts before the ruleset self-heal — 30/30 CI runs failed, pr-quality convergence has never executed #1038); document evidence in PR
    • AC feat: add Structured Logging and CQRS standards #6 — before/after <id>:hands-off count per repo (all nine) in the PR description

    Tests to Write

    • test/scripts/compliance-audit/labels-hands-off-family.bats — a repo missing one persona label produces a missing-label-<id>:hands-off finding; unreadable manifests fail closed with an error finding; a compliant repo (all fixed + derived present) produces none
    • test/scripts/compliance-remediate/labels-derived.bats — remediator resolves color/desc for a derived <id>:hands-off label from the shared list; dry-run mutates nothing
    • Existing test/scripts/apply-repo-settings/apply-repo-settings.bats must stay green after the lib extraction

    Test Command

    bats --print-output-on-failure test/scripts/compliance-audit/ test/scripts/apply-repo-settings/ test/scripts/compliance-remediate/ + shellcheck --severity=warning -x scripts/*.sh scripts/lib/*.sh

  2. don-petry commented on Sep 16, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    • Shared source of truth — new scripts/lib/labels.sh holds the fixed set (STANDARD_LABEL_SPECS), the derived-family derivation (persona_opt_out_label_configs, moved verbatim from the applier), and a std_label_spec() lookup. The audit, remediator, and applier all source it; the three separate hardcoded label tables are gone (AC#2).
    • Audit — check_labels() now checks the fixed set and the derived <id>:hands-off family, derived from the persona manifests exactly as apply-repo-settings.sh does (AC#1). It fails closed: an unreadable/garbled manifest listing emits a persona-opt-out-derivation-failed error finding rather than reading as "no persona labels required" (AC#3).
    • Remediator — remediate_label() resolves colour+description via std_label_spec() (fixed set + derived family), so a missing persona label can now be created through the existing idempotent --force path; read-only/--dry-run default preserved (AC#4).
    • Applier decoupling (AC#5) — no code change needed. apply_repo() already runs apply_labels second (before apply_check_suite_prefs, last) and guards every step with || FAILED_STEPS+=(...), so a failing settings/check-suites step cannot abort label provisioning (the apply-repo-settings.sh aborts before the ruleset self-heal — 30/30 CI runs failed, pr-quality convergence has never executed #1038 fix). Existing apply-repo-settings.bats tests already lock this in ("apply_repo runs every step even when an EARLY step fails"). Evidence to go in the PR body.

    AC#6 — before evidence (<id>:hands-off count per repo, labels API, 2026-09-16)

    Expected family size = 9 (nine personas: business-analyst, dev-lead, devops-lead, pr-review, qa-lead, scrum-master, security-lead, solution-architect, sre-lead).

    Repo before Repo before
    .github 9 markets 1
    .github-private 9 ContentTwin 1
    bmad-bgreat-suite 9 repo-template 1
    TalkTerm 9 broodly 1
    incubator 1

    Matches the issue's 2026-09-15 table. After this change the audit reports 8 missing-label-<id>:hands-off findings on each of the five 1-label repos (family now checked); the remediator/applier can create them. "After" counts follow once the applier/remediator run on the fleet (out of scope for this script-only PR; markets stays disabled per the issue — reported, not re-enabled).

    Local check output (provisional — not the pushed run)

    shellcheck --severity=warning -x scripts/{compliance-audit,compliance-remediate,apply-repo-settings}.sh scripts/lib/labels.sh  → clean
    bats test/scripts/compliance-audit/         → 230 ok / 0 fail
    bats test/scripts/apply-repo-settings/      →  60 ok / 0 fail
    bats test/scripts/compliance-remediate/     →  26 ok / 0 fail
    .dev-lead/scripts/dev-lead-lint.sh          → all checks passed
    

    Files touched (this workspace, not yet pushed)

    • scripts/lib/labels.sh (new): shared label source of truth + std_label_spec().
    • scripts/compliance-audit.sh: source the lib; drop REQUIRED_LABEL_SPECS; check_labels() checks the derived family + fails closed.
    • scripts/compliance-remediate.sh: source the lib; drop LABEL_COLORS/LABEL_DESCS; remediate_label() uses std_label_spec().
    • scripts/apply-repo-settings.sh: source the lib; drop the now-shared derivation + persona vars; build apply_labels fixed set from STANDARD_LABEL_SPECS (its cache untouched).
    • test/scripts/compliance-audit/labels-hands-off-family.bats (new): missing persona label → finding; unreadable manifests → fail closed; compliant repo → none.
    • test/scripts/compliance-remediate/labels-derived.bats (new): remediator creates a derived label; std_label_spec lookup; fail-closed derivation.

    Notes

    • One deliberate strengthening: the shared derivation also fails closed when the listing parses to zero personas (not just on an API/jq error), since an org always has personas — a garbled empty body must not silently yield an empty family. Pinned by a new test.
    • Budget: all three budget-guard steps completed within budget; no follow-up needed.
  3. don-petry commented on Sep 16, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete — PR #1140

    The implementation for issue #1139 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  4. added a commit that references this issue on Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions