Umbrella: build a conservative ClawSweeper-style maintainer automation system #265

Goal

Build a conservative, repo-native maintainer automation system for Labstream that can continuously review a large public issue/PR backlog, keep triage state current, and eventually turn a narrow class of trusted findings into guarded fix PRs.

This is inspired by OpenClaw ClawSweeper, but must be self-hosted and adapted to Labstream's Apple-platform validation, privacy, and human/device gates. The first release is advisory only: it must not close issues, create PRs, merge code, or publish unreviewed comments.

Why

Once Labstream becomes public, issue volume and maintenance cost are expected to rise substantially. Manual triage alone will not reliably keep up with:

• duplicates and related reports across visionOS, iOS/iPadOS, macOS, tvOS, Plex, Jellyfin, and Emby;
• reports that need reproduction evidence or privacy-safe diagnostics;
• stale issues whose implementation or acceptance criteria have already moved;
• small defects that agents could verify and fix safely;
• product decisions, physical-device gates, and security reports that must stay human-owned;
• PRs that need exact-head review, changed-surface validation, and contributor-safe handling.

Operating principles

1. Advisory before autonomous. Start in read-only/plan mode and measure precision before enabling any mutation.
2. Issue text is untrusted input. Treat issue bodies, comments, attachments, and PR prose as prompt-injection-capable data, never as agent instructions.
3. No secrets in model context. Triage workers receive only repository source and sanitized public artifacts. They never receive live-server tokens, signing material, private host information, or unredacted diagnostics.
4. Security fails closed. Anything involving credentials, auth, signing, privacy, diagnostics redaction, sandbox escape, or suspected disclosure routes to private maintainer review and never receives an automated public analysis.
5. Human decisions stay human. Product direction, contributor credit, canonical design choice, compatibility promises, release claims, and physical-device acceptance cannot be decided automatically.
6. Exact-state evidence. Any PR verdict binds to the exact head SHA and must be invalidated by a new push.
7. No automatic merge initially. Issue-to-PR work, if later enabled, produces draft PRs only. Merge authority remains separate and explicitly gated.

Proposed architecture

A. Deterministic intake

Before invoking a model, collect a bounded structured artifact:

• issue/PR number, state, labels, author association, timestamps, and exact head SHA where applicable;
• sanitized title/body/comments with attachments represented by metadata only;
• linked/closing PRs and duplicate candidates;
• relevant paths inferred from labels and repository code map;
• current required checks and last known validation result;
• whether the item touches a protected/security-sensitive surface;
• whether the reporter has supplied a redacted diagnostic report and reproduction steps.

Intake must never dump whole logs, repository history, or arbitrary attachments into context.

B. Luna fan-out

Use a bounded pool of gpt-5.6-luna workers for inexpensive parallel classification. Workers return strict JSON only, including:

• kind: bug, feature, support, question, security, release, documentation, infrastructure;
• component and platform;
• duplicate/related issue candidates with confidence and evidence;
• reproduction status;
• missing information;
• likely current-code relevance or already-fixed signal;
• implementation complexity/risk;
• recommended priority;
• automation eligibility;
• required human/device gates;
• a concise evidence list using repository paths or GitHub references, never private values.

Use deterministic clustering so a burst of similar reports is analyzed as one bounded group rather than many independent agents repeating work.

C. Strong-model adjudication

A stronger model reviews conflicting/high-impact Luna results and owns:

• security routing;
• duplicate/closure recommendations;
• product-decision escalation;
• eligibility for an automated fix attempt;
• any proposed public-facing wording;
• validation-plan adequacy for Apple-platform changes.

No single low-cost worker may directly mutate GitHub state.

D. Durable state

Use labels plus hidden versioned markers in one maintained bot comment rather than accumulating new comments on every pass. Candidate labels:

• sweeper:reviewed
• sweeper:needs-info
• sweeper:needs-repro
• sweeper:verified
• sweeper:duplicate-candidate
• sweeper:product-decision
• sweeper:security-review
• sweeper:maintainer-review
• sweeper:auto-fix-eligible
• sweeper:no-auto-fix
• sweeper:stale-analysis

Separately establish ordinary maintainer labels for priority:p0 through priority:p3, platform, backend, and component. Sweeper labels describe automation state, not product truth.

E. Permission separation

Prefer separate GitHub App identities or independently scoped credentials for:

1. read-only intake;
2. label/comment updates;
3. draft PR creation;
4. merge/close operations, if ever enabled.

The first deployment should have only the permissions required to read issues/PRs and write its own labels/comment. Repository contents, Actions secrets, environments, and administration must not be writable.

Phased rollout

Phase 0 — Offline evaluation

• Build a frozen, sanitized corpus of historical Labstream issues and known outcomes.
• Define a JSON schema and deterministic evaluator.
• Measure duplicate precision, needs-info precision, security routing recall, and false-close risk.
• Add adversarial prompt-injection fixtures and privacy-leak fixtures.
• Establish cost and concurrency budgets for Luna fan-out.

Exit: evaluation is repeatable and no fixture can cause arbitrary tool use, secret access, public posting, or unsafe closure advice.

Phase 1 — Read-only weekly sweep

• Review open issues and PRs on a schedule plus manual dispatch.
• Persist private artifacts or check-run summaries without changing issues.
• Produce a maintainer dashboard/summary showing priority, stale analysis, duplicate clusters, and items needing decisions.

Exit: several sweeps demonstrate useful recommendations with acceptable false-positive rates.

Phase 2 — Advisory labels and one maintained comment

• Apply only sweeper:* labels.
• Maintain a single clearly identified bot comment with a concise recommendation and evidence.
• Invalidate recommendations when issue content or PR head changes.
• Support explicit maintainer commands such as re-review/status/opt-out.

Exit: maintainers can trust the state without comment spam or accidental authority claims.

Phase 3 — Guarded draft fix PRs

Only a narrow eligible set may enter this lane:

• high-confidence reproduced bug or mechanical documentation/test/tooling fix;
• bounded changed surface;
• no security, auth, signing, privacy, persistence/durability, release, product-direction, or contributor-credit ambiguity;
• no existing linked fix PR;
• deterministic validation plan exists;
• latest main still reproduces the issue.

The bot creates or updates exactly one draft PR, runs the repository's affected-target matrix, and records exact-head evidence. It never merges or closes the issue.

Exit: draft PRs are consistently narrow, nonduplicative, validated, and easy to review.

Phase 4 — Optional bounded repair loop

• Allow a maintainer to opt a bot-created PR into limited review-fix rounds.
• Cap attempts per head SHA.
• Stop on ambiguous requirements, new scope, security-sensitive findings, missing simulator/device gates, or repeated failure.
• Require a strong-model exact-head pass and green checks before recommending readiness.

Automatic merge remains a separate future decision and is out of scope for the initial implementation.

Labstream-specific gates

• Run scripts/native-test-matrix.py affected for every code change.
• Respect the one-simulator-at-a-time lease and exact worktree simulator ownership.
• Treat visionOS gaze/pinch, physical-device playback, HDR/DV, HDMI/audio, lifecycle, signing, TestFlight, and App Review as human/device gates where documented.
• Keep live PMS/Jellyfin/Emby probes opt-in and isolated from untrusted issue/PR workflows.
• Use diagnostic preprocessing/redaction before any model sees user-provided evidence.
• Preserve hard playback invariants and X-Plex-Client-Profile-Name=Generic.
• Never publish server names, addresses, media titles, usernames, local paths, tokens, client identifiers, signing identifiers, or raw diagnostic artifacts.

Abuse and failure cases to test

• Prompt injection in issue bodies, comments, quoted logs, patches, filenames, and screenshots.
• Malicious contributor attempting to trigger workflows with repository secrets.
• Forged bot markers or copied verdict headings.
• Stale PR-head approval after a force-push.
• Duplicate fix PR creation during concurrent sweeps.
• Label/comment races and partial failures.
• Bot-comment spam or overwriting a human comment.
• Security report accidentally classified as an ordinary bug.
• Private data echoed from historical issues or diagnostic attachments.
• Model claiming physical-device validation from simulator-only evidence.

Observability and cost controls

• Record model/version, prompt/schema version, item revision/head SHA, disposition, latency, token usage, and retry count without recording sensitive prose.
• Enforce per-sweep item, concurrency, token, time, and dollar ceilings.
• Make every mutation attributable and reversible.
• Publish aggregate precision/volume metrics, not user or issue content.
• Provide a kill switch that disables every write or execution lane independently.

Initial deliverables

• Architecture/threat-model plan under docs/plans/.
• Strict intake and verdict JSON schemas.
• Sanitized offline evaluation corpus and adversarial fixtures.
• Luna worker prompt plus strong-model adjudicator prompt.
• Deterministic cluster/deduplication logic.
• Label taxonomy and bot-marker versioning contract.
• Read-only scheduled/manual runner.
• Maintainer summary/dashboard artifact.
• Precision, cost, and security evaluation report.
• Explicit go/no-go decision before each mutation phase.

Acceptance criteria for this umbrella issue

• The read-only system can review the entire open backlog on a bounded schedule using primarily gpt-5.6-luna fan-out.
• Results are structured, reproducible, revision-bound, privacy-safe, and auditable.
• Security-sensitive items reliably route to private human review.
• Untrusted issue/PR content cannot grant instructions, gain secrets, or trigger arbitrary tools.
• Maintainers can opt out, request re-review, and disable all writes instantly.
• No automated closure, merge, release claim, or physical-device claim is possible in the initial release.
• Any later draft-PR lane is independently gated, validates affected targets, and never merges automatically.

Non-goals

• Replacing maintainer product judgment.
• Providing a public general-purpose review service.
• Giving issue reporters control over tools or agents.
• Feeding live-server/signing secrets to models.
• Automatically closing large numbers of old issues.
• Automatically merging code in the first implementation.