Skip to content

Compliance: ruleset-drift-code-quality-required_status_checks #772

Closed

Description

@don-petry

Compliance Finding

Category: rulesets
Severity: error
Check: ruleset-drift-code-quality-required_status_checks

Detail

Ruleset code-quality parameter required_status_checks has drifted from the codified standard: expected CodeQL, SonarCloud, agent-shield / AgentShield, dependency-audit / Detect ecosystems, actual missing: dependency-audit / Detect ecosystems. The codified JSON (standards/rulesets/code-quality.json) is the source of truth (#575/#580); run scripts/apply-rulesets.sh --repo petry-projects/.github to converge.

Standard Reference

standards/github-settings.md#code-quality--required-checks-ruleset-all-repositories

Remediation

Please review the linked standard and bring this repository into compliance.

See the full standards documentation for implementation guidance.


This issue was automatically created by the weekly compliance audit.

Activity

  1. don-petry commented on Jul 17, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #772 — Compliance: ruleset-drift-code-quality-required_status_checks

    Scope

    Fix the root cause of the code-quality ruleset drift: this repo's .github/workflows/dependency-audit.yml is still the old inline workflow (job detect → check context Detect ecosystems), whereas every other self-hosted reusable in .github (agent-shield, auto-rebase, dependabot-*) is a thin caller stub. Migrating it to the caller-stub convention makes it produce dependency-audit / Detect ecosystems, matching the codified standards/rulesets/code-quality.json.

    Root-cause analysis

    • Codified source of truth standards/rulesets/code-quality.json requires dependency-audit / Detect ecosystems (already correct — no change needed).
    • Live code-quality ruleset on the repo requires Detect ecosystems (the old inline name).
    • The inline dependency-audit.yml runs detect (displayName "Detect ecosystems") directly → GitHub check context Detect ecosystems.
    • The reusable-caller pattern (a job named dependency-audit calling dependency-audit-reusable.yml, whose detect job is "Detect ecosystems") composes to dependency-audit / Detect ecosystems — exactly the codified name. agent-shield.yml already does this (agent-shield / AgentShield) and shows no drift.

    Implementation Checklist

    • Add drift-guard test to test/scripts/apply-rulesets/apply-rulesets.bats asserting .github/workflows/dependency-audit.yml is a caller stub producing dependency-audit / Detect ecosystems
    • Convert .github/workflows/dependency-audit.yml to a thin caller stub (job dependency-audit → ./.github/workflows/dependency-audit-reusable.yml local ref), matching the agent-shield/auto-rebase self-host convention

    Tests to Write

    • dependency-audit.yml defines a job dependency-audit that uses: the local reusable (red against current inline file)
    • dependency-audit.yml has no inline detect job (i.e., no longer publishes the bare Detect ecosystems context)

    Test Command

    npx bats test/scripts/apply-rulesets/apply-rulesets.bats

    Out of scope (operational, admin-token)

    Converging the live ruleset (scripts/apply-rulesets.sh --repo petry-projects/.github) is a manual admin-token step per standards/ruleset-remediation-runbook.md and is not something the dev-lead PR flow performs. After this change merges and the workflow publishes dependency-audit / Detect ecosystems, that step can run without bricking PRs.

  2. don-petry commented on Jul 17, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete

    Root cause

    The code-quality ruleset drift was caused by this repo's own .github/workflows/dependency-audit.yml still being the pre-centralization inline workflow: it ran a top-level detect job (displayName "Detect ecosystems") directly, so GitHub published the check context as the bare Detect ecosystems. Every other self-hosted reusable in .github (agent-shield.yml, auto-rebase.yml, dependabot-*.yml) is already a thin caller stub, so their check names compose correctly (e.g. agent-shield / AgentShield) and show no drift. Migrating dependency-audit.yml to the same caller-stub shape makes it publish dependency-audit / Detect ecosystems — exactly the context codified in standards/rulesets/code-quality.json.

    Plan Execution

    • Added drift-guard tests asserting dependency-audit.yml is a caller stub producing dependency-audit / Detect ecosystems (red first)
    • Converted .github/workflows/dependency-audit.yml to a thin caller stub (job dependency-audit → ./.github/workflows/dependency-audit-reusable.yml local ref), matching the agent-shield.yml self-host convention
    • Full bats suite (49 files) + required dev-lead lint pass

    Test Results

    1..20
    ok 1 code-quality.json: requires exactly the 4 codified contexts
    ok 2 code-quality.json: does NOT require 'Dev-Lead Agent' (the retired divergence)
    ok 3 pr-quality.json + code-quality.json carry both bypass actors (OrgAdmin + Integration)
    ok 4 pr-quality.json: omits 'automatic_copilot_code_review_enabled' (Free-plan API rejects it)
    ok 5 release-channel-tags.json: tag ruleset, generic refs/tags/** include, update+deletion only
    ok 6 release-channel-tags.json: carries OrgAdmin + both Integration apps (automation 3167543 + release-manager 4193127)
    ok 7 dependency-audit.yml: is a caller stub with job 'dependency-audit' using the reusable
    ok 8 dependency-audit.yml: carries no inline 'detect' job (would publish bare 'Detect ecosystems')
    ok 9 apply --repo: creates rulesets when absent (POST per file)
    ok 10 apply --repo: updates when present (PUT by id)
    ok 11 apply: --dry-run makes no write calls
    ok 12 apply: a bare <repo-name> resolves to $ORG/<name>
    ok 13 apply: a name filter applies only that ruleset
    ok 14 apply: default set is the fleet allowlist only — release-channel-tags is NOT swept in
    ok 15 apply: release-channel-tags applies only when named explicitly
    ok 16 apply: unknown ruleset name errors
    ok 17 apply --all: iterates every non-archived org repo
    ok 18 apply: --all and --repo are mutually exclusive
    ok 19 apply: a bare token after --all is a ruleset-name filter (unknown → errors)
    ok 20 ruleset_id_by_name: resolves id from the list
    

    Full sweep: all 49 bats suites green (incl. ruleset-contents-drift.bats 10/10, the suite for the audit function that raised this finding). .dev-lead/scripts/dev-lead-lint.sh → all checks passed.

    Files Changed

    • .github/workflows/dependency-audit.yml: inline detect/audit workflow → thin caller stub delegating to the org reusable; publishes dependency-audit / Detect ecosystems. No with:/secrets: needed (the reusable declares no inputs and only contents: read); detection/audit behavior is unchanged (the reusable is the org source of truth for that logic).
    • test/scripts/apply-rulesets/apply-rulesets.bats: added two drift-guard tests tying the codified required context to the workflow that must produce it.

    Notes

    • Operational follow-up (out of dev-lead scope): the live ruleset on petry-projects/.github still lists the old Detect ecosystems context. Converging it is a manual admin-token step per standards/ruleset-remediation-runbook.md (scripts/apply-rulesets.sh --repo petry-projects/.github); compliance-remediate.sh intentionally skips the rulesets category. Now that the workflow publishes dependency-audit / Detect ecosystems, that step converges the ruleset without bricking PRs.
    • Merge ordering: because the live ruleset currently requires the bare Detect ecosystems, this PR should be merged with admin bypass (already a bypass actor on .github), then the apply-rulesets step run, so future PRs gate on the codified name.
  3. don-petry commented on Jul 24, 2026

    @don-petry
    ContributorAuthor

    Weekly Compliance Audit (2026-07-24)

    This finding is still open.

    Detail: Ruleset code-quality parameter required_status_checks has drifted from the codified standard: expected CodeQL, SonarCloud, agent-shield / AgentShield, dependency-audit / Detect ecosystems, actual missing: dependency-audit / Detect ecosystems. The codified JSON (standards/rulesets/code-quality.json) is the source of truth (#575/#580); run scripts/apply-rulesets.sh --repo petry-projects/.github to converge.

    Standard: standards/github-settings.md#code-quality--required-checks-ruleset-all-repositories

  4. don-petry commented on Jul 31, 2026

    @don-petry
    ContributorAuthor

    Weekly Compliance Audit (2026-07-31)

    This finding is still open.

    Detail: Ruleset code-quality parameter required_status_checks has drifted from the codified standard: expected CodeQL, SonarCloud, agent-shield / AgentShield, dependency-audit / Detect ecosystems, actual missing: dependency-audit / Detect ecosystems. The codified JSON (standards/rulesets/code-quality.json) is the source of truth (#575/#580); run scripts/apply-rulesets.sh --repo petry-projects/.github to converge.

    Standard: standards/github-settings.md#code-quality--required-checks-ruleset-all-repositories

  5. don-petry commented on Aug 2, 2026

    @don-petry
    ContributorAuthor

    Operational follow-up complete

    Ran the manual admin-token convergence step called out as out-of-scope in the dev-lead plan:

    GH_TOKEN=<admin> ./scripts/apply-rulesets.sh --repo petry-projects/.github
    

    Output:

    [apply-rulesets] repo=petry-projects/.github dir=.../standards/rulesets dry_run=false
      update ruleset 'code-quality' (id 14872169) on petry-projects/.github
      update ruleset 'pr-quality' (id 14872168) on petry-projects/.github
    [apply-rulesets] done (2 ruleset(s))
    

    Verified live code-quality ruleset now requires exactly the codified set:

    SonarCloud
    CodeQL
    agent-shield / AgentShield
    dependency-audit / Detect ecosystems
    

    The dependency-audit / Detect ecosystems context is no longer missing. Closing — next weekly compliance audit should come back clean.

  6. added 3 commits that reference this issue on Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    compliance-auditAutomated compliance audit findingdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions