Skip to content

[Phase 2] Build the standalone AGENTS.md structural linter #644

Closed

Description

@don-petry

Story

As a compliance-audit maintainer,
I want implement a deterministic shell linter that validates a single AGENTS.md file against the Phase 1 rule set,
so that any repo's AGENTS.md can be checked for structural drift with zero LLM cost and reproducible output.

Acceptance Criteria

  1. A new shell script (e.g. scripts/agents-md-lint.sh) accepts an AGENTS.md path and validates required section presence, valid Markdown heading hierarchy (exactly one H1, no skipped heading levels), and cross-reference integrity (relative links and referenced files such as CLAUDE.md / SKILL.md resolve).
  2. All checks are driven by the Phase 1 rule-set data file, not hardcoded, so rule changes need no code edits.
  3. The linter emits deterministic, severity-tagged, machine-readable findings and defaults to informational mode (exit 0 / never fails the caller); a mode flag can switch it to failing (exit nonzero) for later phases.
  4. Pure classification/parse helpers are sourceable and covered by bats tests with fixtures — a structurally-valid AGENTS.md plus one fixture per failure mode (missing required section, skipped heading level, dangling cross-reference); tests use no network.
  5. shellcheck --severity=warning -x passes for the new script, and the new bats test file is registered in the lint.yml bats job via scripts/run-bats.sh.

Tasks / Subtasks

Dev Notes

  • Follow the repo's established pattern of sourceable pure classification helpers exercised by pure bats tests with synthetic fixtures — see tests/template_stub_drift.bats (setup sources the script; asserts over inputs; no network).
  • The lint.yml shellcheck job runs find scripts -name '*.sh' -exec shellcheck --severity=warning -x {} +, so the new script must be clean under that.
  • The bats job invokes scripts/run-bats.sh with an explicit file list; add the new test file there — run-bats.sh tolerates a not-yet-present file with a ::warning:: so the list can be added in the same PR.
  • Keep the linter self-contained and file-path-in / findings-out: it must NOT enumerate repos or hit the network (that is the audit integration's job in Phase 3). This keeps it unit-testable and reusable by both the audit and the local CI self-check.
  • Cross-reference integrity should resolve relative links/paths against the AGENTS.md's own directory; treat http(s) links and pure '#anchor' references as out of scope for existence checks (mirrors how the repo's own validators skip URLs/anchors).

Project Structure Notes

New scripts/agents-md-lint.sh alongside other scripts/*.sh; new tests/agents_md_lint.bats and tests/fixtures/agents-md/ alongside existing fixtures (tests/fixtures/). Registered in .github/workflows/lint.yml.

References

  • tests/template_stub_drift.bats
  • scripts/run-bats.sh
  • .github/workflows/lint.yml
  • scripts/initiative-planner/validate-plan.py

Likely target surface

  • scripts/agents-md-lint.sh (new)
  • tests/agents_md_lint.bats (new)
  • .github/workflows/lint.yml

Story prepared by the BMAD Scrum Master (Bob) for epic #642. Status: ready-for-dev.

Activity

  1. don-petry commented on Sep 1, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #644 — [Phase 2] Build the standalone AGENTS.md structural linter

    Scope

    Implement a deterministic, self-contained shell linter (scripts/agents-md-lint.sh) that validates a single AGENTS.md file against the Phase 1 rule set (scripts/lib/agents-md-rules.json), with sourceable pure helpers, machine-readable severity-tagged findings, and an informational (default, exit 0) vs. failing (exit nonzero) mode flag.

    Repo-reality notes

    The issue's Dev Notes reference tests/template_stub_drift.bats, scripts/run-bats.sh, and .github/workflows/lint.yml, which do not exist in this repo. The faithful equivalents here are:

    • shellcheck --severity=warning -x is already enforced on scripts/**/*.sh by the ci.yml ShellCheck job (auto-covers the new script).
    • Bats gating is done per-feature via a dedicated *-tests.yml workflow (e.g. agents-md-rules-tests.yml); the new bats file is registered by adding a sibling agents-md-lint-tests.yml.

    Implementation Checklist

    • scripts/agents-md-lint.sh — sourceable pure helpers (heading extraction, single-H1, heading-hierarchy, fenced-closure, link classification, slugify, cross-reference resolution, section-presence, canonical detection) + guarded main that reads the rule-set JSON and dispatches checks by rule id/element/level
    • Informational/failing mode flag + deterministic, severity-tagged, machine-readable findings output
    • tests/fixtures/agents-md/ — valid/ (+ a resolvable sibling file) and one fixture per failure mode: missing-section/, skipped-heading/, dangling-xref/
    • .github/workflows/agents-md-lint-tests.yml — installs bats+jq and runs the new bats file (registers it in CI)
    • .markdownlint-cli2.yaml — ignore tests/fixtures/** so intentionally-malformed fixtures don't fail the Lint job

    Tests to Write

    • tests/agents_md_lint.bats — sources the script (guarded main) and exercises each pure helper over the fixtures (no network), plus main in informational mode (exit 0) and failing mode (exit nonzero on a required-rule failure, exit 0 on valid)

    Test Command

    bats tests/agents_md_lint.bats (plus shellcheck --severity=warning -x scripts/agents-md-lint.sh and bash .dev-lead/scripts/dev-lead-lint.sh)

  2. don-petry commented on Sep 1, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: issue #644 implementation failed — will retry

    The engine failed (engine-error — e.g. a timeout or transient engine error). This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

    Last 40 lines of engine output (redacted)
        [copilot] calling gh copilot (model=openai/o4-mini, timeout=2100s, flags=--yolo)
    Error: Classic Personal Access Tokens (ghp_) are not supported by Copilot.
    
    The GH_TOKEN environment variable contains a classic PAT.
    Please use a Fine-Grained Personal Access Token or another authentication method.
    
    To fix this, you can:
      • Replace the token in GH_TOKEN with a fine-grained PAT
      • Unset GH_TOKEN and run 'gh auth login' to authenticate
      • Unset GH_TOKEN and start 'copilot', then use the '/login' command
    
    
  3. don-petry commented on Sep 1, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #644 — [Phase 2] Build the standalone AGENTS.md structural linter

    Scope

    Build a deterministic, zero-LLM-cost shell linter (scripts/agents-md-lint.sh) that validates a single AGENTS.md against the Phase 1 rule set (scripts/lib/agents-md-rules.json) — driven by that data file, with sourceable pure helpers, machine-readable severity-tagged findings, and an informational (exit 0) / failing (exit nonzero) mode flag.

    Repo-convention note

    The issue references lint.yml + scripts/run-bats.sh; those are org-template names. This repo actually runs shellcheck over scripts/**/*.sh in ci.yml's ShellCheck job (auto-covers the new script) and gates bats via per-feature *-tests.yml workflows. AGENTS.md structural validation already has .github/workflows/agents-md-rules-tests.yml, so the new bats file will be registered there (its natural, cohesive home) rather than inventing lint.yml.

    Implementation Checklist

    • scripts/agents-md-lint.sh (new) — sourceable pure helpers (slugify, heading extraction ignoring code fences, single-H1, heading-hierarchy, cross-reference integrity, fenced-code closure, section-presence, org-import, rule-level lookup) + BASH_SOURCE-guarded main
    • Rule-set-driven driver: iterate rules from agents-md-rules.json, dispatch by rule id, tag each finding with the rule's level (severity) and honor applies_to
    • --mode informational (default, always exit 0) / --mode failing (exit nonzero only on required-level findings); machine-readable TSV findings
    • tests/fixtures/agents-md/ — valid file + one fixture per failure mode (missing required section, skipped heading level, dangling cross-reference)
    • tests/agents_md_lint.bats (new) — source the helpers, assert over fixtures, no network
    • .github/workflows/agents-md-rules-tests.yml — register the new bats file + add trigger paths for the script/test/fixtures

    Tests to Write

    • valid fixture → no required-level findings; exit 0 in both modes
    • missing-section fixture → recommended-level finding (reported, never fails)
    • skipped-heading fixture → required finding; failing mode exits nonzero
    • dangling-cross-reference fixture → required finding
    • pure-helper unit tests (slugify, heading extraction skips fences, single-H1 count) + rule-level lookup from the data file

    Test Command

    bats tests/agents_md_lint.bats (full: bats tests/), plus bash .dev-lead/scripts/dev-lead-lint.sh (shellcheck)

  4. don-petry commented on Sep 1, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete

    Phase 2 standalone AGENTS.md structural linter is implemented, rule-set-driven, bats-tested, and CI-registered.

    Plan Execution

    Deviation from the issue's file names (documented in the plan)

    The issue references lint.yml + scripts/run-bats.sh; those are org-template names and do not exist in this repo's CI. This repo runs shellcheck over scripts/**/*.sh in ci.yml's ShellCheck job (auto-covers the new script) and gates bats via per-feature *-tests.yml workflows. I therefore registered tests/agents_md_lint.bats in the existing .github/workflows/agents-md-rules-tests.yml — the cohesive home for AGENTS.md structural validation — and added trigger paths for the new script, test, and fixtures. Also excluded the deliberately-malformed fixtures from **/*.md markdownlint via .markdownlint-cli2.yaml ignores so the CI Lint job stays green.

    Test Results

    $ bats tests/agents_md_lint.bats
    1..24
    ok 1 amdl_slugify lowercases, hyphenates spaces, strips punctuation
    ...
    ok 20 missing required section is a recommended finding — reported but never fails
    ok 21 skipped heading level is a required finding that fails --mode failing
    ok 22 dangling cross-reference is a required finding that fails --mode failing
    ok 23 default mode is informational (exit 0 even with required findings)
    ok 24 a missing target file is reported as an error (exit 2)
    
    $ bats tests/          # full suite
    602 passing, 0 failing
    
    $ shellcheck --severity=warning -x scripts/agents-md-lint.sh   # clean
    $ bash .dev-lead/scripts/dev-lead-lint.sh                      # all checks passed
    

    Sample deterministic output (--mode informational, exit 0):

    $ scripts/agents-md-lint.sh tests/fixtures/agents-md/dangling-xref.md
    required	cross-reference-integrity	4	relative reference "./does-not-exist.md" does not resolve to an existing path
    recommended	security-section-present	-	no section heading matching /security|secret/ is present
    recommended	testing-section-present	-	no section heading matching /test|tdd/ is present
    

    Files Changed

    • scripts/agents-md-lint.sh (new): the standalone, zero-token structural linter — sourceable pure helpers + rule-set-driven CLI.
    • tests/agents_md_lint.bats (new): 24 pure/CLI tests over synthetic fixtures, no network.
    • tests/fixtures/agents-md/{valid,missing-section,skipped-heading,dangling-xref,CLAUDE}.md (new): one valid file + one per failure mode + an xref target.
    • .github/workflows/agents-md-rules-tests.yml: run the new bats file and add trigger paths for the script/test/fixtures.
    • .markdownlint-cli2.yaml: ignore tests/fixtures/agents-md/** (fixtures are intentionally malformed).

    Notes

    • In scope by design: file-path-in / findings-out. No repo enumeration, no network — that is Phase 3's audit integration. Relative links resolve against the AGENTS.md's own directory; http(s)/scheme URLs and out-of-file #anchor targets are skipped for existence checks.
    • applies_to is honored: org-repo-import-consistency (downstream) is skipped under the default --scope canonical and applied with --scope downstream.
    • An unknown rule id in the data file surfaces a loud "rule-set/linter drift" finding rather than being silently dropped (fail-loud).
    • The linter stays informational everywhere; the informational→blocking promotion is Phase 4, and wiring into Standards Sync + a canonical-AGENTS.md CI self-check is Phase 3.
  5. added a commit that references this issue on Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions