Skip to content

[qa-lead S3] Serve the pull_request surface so qa-lead runs without being summoned #1646

Closed

Description

@don-petry

Story

As a maintainer whose test-risk blind spots only get caught when someone remembers to summon the QA Lead,
I want qa-lead to advise automatically on PRs with real test surface,
so that a persona with a proven hit rate stops depending on human recall.

Do not start this story until #1644 and #1645 have merged. They are hard gates, for the reasons in Dev Notes.

Why

Measured 2026-07-18 → 2026-09-01 in .github-private: 2 agent executions, ever. Yet the single non-soak advisory reshaped 4 of the 9 stories under epic #1402 (#1403 and #1406 carry "Enhancements from QA Lead review" sections; #1410 and #1411 exist because of it). Quality is not the problem. Reachability is: mention is the only surface served, so the persona runs only when a human types a handle.

personas/qa-lead/persona.yml already declares the surface:

- surface: pull_request
  events: [opened, ready_for_review, synchronize]
  enabled: true
  mode: advisory
  notes: "Risk assessment + test-coverage review comments on PRs with test surface."

Nothing dispatches it. personas/qa-lead/interaction.yml (from #1404/#1418) records this honestly: "personas/qa-lead/persona.yml declares broader advisory surfaces … that the router does not yet wire — the contract records what is actually deployed, not the aspiration."

Acceptance Criteria

  1. A Class 1 event-driven path (per docs/agentic-interaction-model.md §2) runs qa-lead's advisory on pull requests in .github-private — no cron, no timer_role. The classification table in that document gains a row for it, in the machine-checkable shape [Phase 3] Add scripts/validate-interaction-model.sh + bats, wired into lint.yml as a PR-triggered check #1406's validator parses.
  2. It reuses persona-runner-reusable.yml — the one runner for every persona (persona-standards §4.1). Do not add a qa-lead-specific runner; a workflow per persona is precisely the drift the manifest exists to prevent.
  3. It is narrowly gated. It must not comment on every PR. At minimum it requires real test surface (changed paths under tests/, or a source change with no corresponding test change) and skips trivial/verbatim stub-sync PRs. The heuristic is a pure, sourced helper with a bats suite covering: a test-touching PR (fires), a docs-only PR (does not), a verbatim stub-sync PR (does not), and a source-change-without-tests PR (fires). State the heuristic in the PR and justify it.
  4. The opt-out works. qa-lead:hands-off on a PR suppresses the advisory, verified by a test. This is why [qa-lead S1] Provision <id>:hands-off labels across the fleet — the opt-out is inert where the persona actually runs #1644 gates this story.
  5. One advisory per PR, not one per push. Re-running on synchronize must not stack comments. Honor the idempotency_key and concurrency_lane already declared in personas/qa-lead/interaction.yml; prefer updating or skipping over posting again.
  6. It respects the automation budget and human gates. pr_has_escalation_label in scripts/lib/pr-automation-budget.sh is the canonical check — reuse it, do not re-derive. needs-human-review, dev-lead:needs-human, and the budget-exhaustion marker all suppress.
  7. The recursion guard holds. The advisory carries <!-- persona:qa-lead --> as its first body line, enforced by the workflow, not the prompt — the read/write split in persona-runner-reusable.yml is load-bearing (feat: implement issue #846 — [Phase 2] Cache the LSP index and enforce the 30s cold-start SLA with auto-skip #860: prompt-only marker enforcement produced 1,481 acks in 4.5h). The new path must not weaken it.
  8. personas/qa-lead/interaction.yml is updated to record pull_request as deployed, so the contract keeps matching reality.
  9. Noise is measured, not assumed. After a defined soak window, report how many advisories fired, how many were no-action, and the share of PRs that drew one — using the no-action-comment classifier [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411 built. If the no-action share is high, the heuristic in AC Add @claude delegation, auto-merge, and rebase handling #3 is wrong and should be tightened before this goes any further than .github-private.

Tasks / Subtasks

Dev Notes

Project Structure Notes

New pure helper under scripts/lib/ + bats under tests/ (registered in lint.yml); a new Class 1 caller under .github/workflows/; edits to docs/agentic-interaction-model.md, personas/qa-lead/interaction.yml, personas/qa-lead/README.md. No new cron.

References


⚠️ Update — 2026-09-01: this story contradicts a stated cost decision. Do not dispatch without a ruling.

Epic #1627 (Onboard solution-architect…, planned from Discussion #1360 and scoped by the maintainer refresh comment of 2026-08-31) states, under Cost cap (this cut — §9):

Mention-routed only: a run happens when — and only when — a human types the handle. No cron, no per-PR fan-out, no standing token cost. Explicitly do NOT wire pull_request: synchronize in this cut.

This story proposes exactly the opposite for qa-lead: wiring pull_request: [opened, ready_for_review, synchronize].

The tension is narrower than it first looks, and worth stating precisely:

So the question is not whether the cost posture is right — it is whether a persona with demonstrated per-invocation value earns an exception to it, and at what trigger width. Options, cheapest first:

  1. Hold at mention-only. Consistent with Onboard solution-architect as an addressable advisory persona (repo-local first cut) #1627; qa-lead stays pull-only and its throughput problem is unsolved. Closes this story.
  2. Narrow the trigger hard — pull_request: [opened, ready_for_review] only, dropping synchronize (the specific event Onboard solution-architect as an addressable advisory persona (repo-local first cut) #1627 names), gated on the test-surface heuristic in AC Add @claude delegation, auto-merge, and rebase handling #3. One advisory per PR at most, no per-push cost. This honours the letter and most of the spirit of the cap.
  3. As written — including synchronize, with the AC feat: add Copilot engine support via REVIEW_ENGINE toggle #5 idempotency guard preventing comment stacking.

Option 2 is the recommendation: it removes the standing per-push cost the cap targets while still ending the "nobody remembers to summon it" failure mode. Whichever is chosen, record it here and reconcile it against #1627 §9 so the fleet has one cost posture, not two.


✅ RULING — 2026-09-07: per-PR trigger APPROVED. This story is unblocked.

The maintainer has approved the per-PR trigger for qa-lead, overriding the mention-only cost
posture in epic #1627 §9 for this persona. Implement the story.

Trigger scope — build exactly this

on:
  pull_request:
    types: [opened, ready_for_review]

synchronize is deliberately NOT wired. "Per PR" means one advisory per pull request, not one
per push. This is the narrow reading of the approval and the cheaper one:

The original AC #1's event list is superseded by the two events above. AC #5 (idempotency) still
applies
and is now cheap to satisfy: with no synchronize, at most one advisory per PR is the
natural state — but still guard it, because ready_for_review can fire after opened on the same PR
(open as draft, then mark ready). That exact sequence is the required test case.

Why this persona earns the exception — record it in the PR

Epic #1627's cap governs an unmeasured persona whose own counter-metric watches for advisories
asserting doctrine without evidence. qa-lead is the opposite case, and the PR description should say
so plainly so the two cost postures are reconcilable rather than contradictory:

Do NOT cite the eval score as evidence either way

qa-lead's held-out score is 0.333 (2/6) as of #1656. That number must not be used to justify or
question this rollout: 2 of its 6 cases never produced an advisory at all (a context-plumbing
artifact), and the 2 genuine misses were measured on the Haiku tier while production runs Opus.
Engine parity is being fixed under #1686. Treat the gate as not-yet-meaningful, and do not gate
this story on it.

The AC #3 heuristic is now the most important part of this story

With the trigger approved, the heuristic is the only thing standing between "qa-lead becomes useful"
and "qa-lead becomes noise." Hold to the original AC #3 — a pure, sourced helper with bats covering a
test-touching PR (fires), a docs-only PR (does not), a verbatim stub-sync PR (does not), and a
source-change-without-tests PR (fires) — and state the heuristic plainly in the PR description.

Bear in mind the held-out set contains an explicit negative control
(qa-lead-hold-well-tested-refactor) whose correct behaviour is to say nothing is needed. A qa-lead
that comments on every PR would be a regression even if each comment is individually correct.

Scope reminders

Activity

  1. don-petry commented on Sep 7, 2026

    @don-petry
    CollaboratorAuthor

    Per PR triggger is approved by human

  2. don-petry commented on Sep 7, 2026

    @don-petry
    CollaboratorAuthor

    [dev-lead] Plan — [qa-lead S3] serve the pull_request surface

    Honoring the 2026-09-07 ruling: build EXACTLY on: pull_request: [opened, ready_for_review] — synchronize is deliberately not wired (one advisory as a PR enters review, not one per push). Not citing the eval score. The test-surface heuristic is the load-bearing part — the line between "qa-lead becomes useful" and "qa-lead becomes noise."

    Approach (TDD, red→green)

    1. Pure, sourced test-surface heuristic (scripts/lib/qa-lead-test-surface.sh) + bats (tests/test_qa_lead_test_surface.bats) covering the four AC Add @claude delegation, auto-merge, and rebase handling #3 cases: test-touching PR fires, docs-only skips, verbatim stub-sync skips, source-without-tests fires. (AC Add @claude delegation, auto-merge, and rebase handling #3)
    2. Pure gate (scripts/qa-lead-advisory-gate.sh) composing the heuristic with the suppressors, unit-tested (tests/test_qa_lead_advisory_gate.bats): opt-out qa-lead:hands-off (AC Optimize review: small-PR and incremental fast paths #4), one-advisory-per-PR marker scan (AC feat: add Copilot engine support via REVIEW_ENGINE toggle #5), and human/budget gates reusing the canonical pr_has_escalation_label (AC feat: feature ideation framework — adversarial proposer/challenger/synthesizer #6).
    3. Class 1 caller (.github/workflows/qa-lead-pr-advisory.yml): a gate job → an advise job that reuses the one persona-runner-reusable.yml (no qa-lead-specific runner) (AC Go-live improvements for PR review agent #2). The runner's read/write split mechanically stamps <!-- persona:qa-lead --> — recursion guard is the workflow, not the prompt (AC Workflow should skip no-op PRs in MAX_PRS count #7). Caller concurrency lane serializes opened+ready_for_review.
    4. Model bookkeeping: §4 Class 1 row in docs/agentic-interaction-model.md (AC test issue from agent #1); personas/qa-lead/interaction.yml records pull_request as deployed (AC Add Claude Code workflow #8); README documents the served surface, two-week soak, and noise metrics (AC feat: inline Claude Code workflow to access repo secrets directly #9).

    Verification

    bats (26 new tests green), validate-interaction-model.sh, the Python interaction-contract validator, validate-personas.py, repo-wide shellcheck --severity=warning -x, and dev-lead-lint.sh — all green locally. CI owns git.

  3. don-petry commented on Sep 7, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #1697

    The implementation for issue #1646 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  4. added 3 commits that reference this issue on Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requestsinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions