Skip to content

Org standard: automated spam / low-effort outside-PR handling (spam-pr-guard) #1200

Open
0 of 7 issues completed

Description

@don-petry

Why

Public org repos accept PRs from anyone. Today spam and low-effort outside PRs sit open until a maintainer notices them (the stale-manager window is ~37 days), attract third-party bot noise (Qodo, CodeRabbit, cubic), and consume maintainer attention. The motivating incident is petry-projects/.github-private#1976 (2026-09-29): a name-as-title, unfilled-template, typo-introducing PR from a brand-new FIRST_TIME_CONTRIBUTOR. Nothing in our automation acted on it, and the existing description-quality check was fooled — a naive keyword grep over the raw body matches the template's own headings and HTML comments, so an empty template scored 0/5 missing sections.

What — a fleet-vendored capability on the org-standard shape

spam-pr-guard ships as a reusable workflow hosted in petry-projects/.github and consumed by every org repo via a thin caller stub that pins the per-agent canary channel. This matches how pr-auto-review, persona-mention, agent-shield, add-to-project, and the other first-party capabilities already ship — see standards/ci-standards.md § Reusable workflow versioning and standards/canary-rings.json.

Layered defense (layer scope unchanged; delivery shape realigned)

  • Layer 0 (native, no code): GitHub-native controls — require workflow approval for outside contributors; per-repo PR caps with a bypass list. Phase 0.
  • Layer 1 (spam-pr-guard): a Class-1, pull_request_target, metadata-only reusable workflow that runs a deterministic (no-LLM) scorer and maps a PR to low / medium (needs-info label + comment) / high (spam label + close). Shipped dry_run: true first; dry_run is a stub input, defaulting ON so consumers can flip it per-repo as the canary advances. Phases 1–3.
  • Layer 2: fix the description-quality check so an unfilled template is no longer read as complete. Phase 1 — delivered as scripts/lib/safety-checks.sh::sc_description_missing, a sourceable helper the scorer calls.
  • Layer 3: a short stale window for unanswered needs-info outside PRs, plus weekly reporting to tune thresholds. Phase 6.

Delivery shape (vendored from .github, consumed by stub, released on canary tags)

Artifact Path in petry-projects/.github Role
Reusable workflow .github/workflows/spam-pr-guard-reusable.yml on: workflow_call: entry point. inputs.dry_run (bool, default true), inputs.agent_ref (string, default spam-pr-guard/v1-stable). permissions: {} top-level; job-level grants only what the layer actually writes. Checks out petry-projects/.github@${{ inputs.agent_ref }} to source scripts/lib/safety-checks.sh + scripts/spam-pr-score.sh at the pinned channel. Pattern: .github/workflows/add-to-project-reusable.yml:62-99.
Scorer scripts/spam-pr-score.sh Deterministic signal → score mapper. Phase 2.
Safety-checks lib scripts/lib/safety-checks.sh sc_description_missing + future sc_* signals. Phase 1 (PR #1218 in flight).
Caller stub (template) standards/workflows/spam-pr-guard.yml Thin stub emitted by scripts/seed-repo-template.sh into each consumer. Pins uses: petry-projects/.github/.github/workflows/spam-pr-guard-reusable.yml@spam-pr-guard/v1-stable # NOSONAR(githubactions:S7637) first-party channel ref (the NOSONAR marker is mandatory; seed-repo-template.sh:60-67 fails closed without it). Passes with: { dry_run: true, agent_ref: spam-pr-guard/v1-stable }. Shape mirrors standards/workflows/pr-auto-review.yml:1-81.
Paired tests workflow .github/workflows/spam-pr-guard-tests.yml Path-filtered bats + shellcheck; triggers on changes to the reusable, scripts, tests, fixtures, and the stub. Model: .github/workflows/pr-auto-review-tests.yml.
User-facing standard standards/spam-pr-guard.md (dedicated file; or new section in standards/ci-standards.md if scope stays small) Signal table, action mapping, config knobs, security invariant, false-positive appeal path. Ships automatically on the next scripts/cut-standards-release.sh cut vX.Y.Z → standards/v1-stable. Phase 4.
Interaction contract interaction-contracts/spam-pr-guard.yml + row in docs/agentic-interaction-model.md §4 Class-1, metadata-only invariant, budget: none. Phase 3.
Registry entry standards/canary-rings.json .agents["spam-pr-guard"] host: petry-projects/.github, reusable: .github/workflows/spam-pr-guard-reusable.yml, run_workflow: <caller-stub display name>, standard #548 gate, rings next → ring0 → ring1 → stable. Phase 3.
Deploy wiring scripts/deploy-standard-workflows.sh DEPLOYABLE_WORKFLOWS=(...) (lines 116-129) + SKIP_SELF_MANAGED entries for .github/.github-private where appropriate Live propagation of the stub to existing repos. Phase 3.
Ring-pins wiring scripts/lib/ring-pins.sh RING_REUSABLES=(...) (lines 32-50) Alphabetical insertion; bats asserts equality to `.agents

Canary rings (fleet default; dogfood at the source)

Per the registry pattern (standards/canary-rings.json lines 20–55 for dev-lead, 949–1017 for pr-auto-review):

  • next → petry-projects/.github-private (dogfood at the source — pr-auto-review, persona-mention, agent-shield, etc. all route next through .github-private).
  • ring0 → petry-projects/.github (this repo).
  • ring1 → petry-projects/TalkTerm, petry-projects/bmad-bgreat-suite.
  • stable → * (every other consumer).
  • require_confirmation: true on ring1→stable so the human go/no-go to enable label/close ships through the fleet rather than through a code flip (Phase 5).

Promotion is automatic through the registry's graduated dwell/sample gate (scripts/canary-rollout.sh); canary-rollout.sh autocut cuts spam-pr-guard/vX.Y.Z from the host-main-HEAD blob + agent_ref_paths (default [scripts/, prompts/, personas/]) and moves the next channel.

Success metrics (initiative-level, measurable)

  • Median time-to-close for spam outside PRs: from ~37 days (stale-manager) to < 24h.
  • False-positive rate (legitimate PRs auto-closed): < 2% of auto-closes.
  • Reopened-after-auto-close rate: tracked, target < 5%.
  • Zero false positives on legitimate first-time PRs during the DRY_RUN soak on next + ring0, validated against >= 10 real outside-PR events.
  • Third-party bot noise on spam PRs driven to near zero (closed before bots react).

Cost cap / budget bound

The spam-pr-guard reusable is deterministic and adds zero incremental LLM spend — it invokes no engine (contrast scripts/stale-manager.sh, which calls Claude). It is event-driven only (pull_request_target, no schedule, no polling) and runs at most once per PR open/edit/reopen event; staged actions are idempotent (one marker-keyed comment per PR). Fork PRs never reach agents (no secrets), so there is no agent-churn budget to bound; the workflow declares budget: none in its interaction contract with that justification.

Safety

The scorer is metadata-only with no PR-code execution (validated by actions/checkout v7 defaults that block fork checkout in pull_request_target), fails closed (an unreadable signal escalates to a human, never scores "not spam"), and never self-triggers (§7 rule 1). Automation never blocks users; the live-mode flip is encoded in the stub's with.dry_run input, overridable per-ring, and require_confirmation: true holds the ring1→stable promotion for a human go/no-go.

Untracked prerequisites

  • Repo-admin action (GitHub UI/API settings, not code): on ring-0 .github-private enable 'Require approval for workflows from all outside contributors', and configure a per-repo PR cap for users without write access with a bypass list including dependabot[bot] and org-leads (where GitHub supports it). Covered operationally by Phase 0.
  • Close the motivating spam PR petry-projects/.github-private#1976 by hand as part of Phase 0 (cross-repo; cannot be wired as a blocked_by edge).

Phase index (realigned)


Planned from idea discussion #1198 by the BMAD Scrum Master initiative-planner, realigned to the fleet vendored-reusable + caller-stub + canary-rings shape. Inert until a maintainer adds initiative:auto.

Activity

  1. added
    initiative:autoArms an initiative epic for the initiative-driver auto-release
    on Oct 1, 2026
  2. don-petry commented on Oct 5, 2026

    @don-petry
    ContributorAuthor

    Initiative driver — stale in-flight slot(s) detected

    The following open sub-issue(s) still carry the dev-lead label but have
    had no dev-lead activity for ≥ 72h, so they are occupying an in-flight
    slot without progressing: #1202.

    No labels were changed. A human should decide whether to split-and-close, retry,
    or park each one so the slot is freed for the rest of the epic.

  3. added a commit that references this issue on Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    initiativeinitiative:autoArms an initiative epic for the initiative-driver auto-release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions