Repository navigation
Description
Story
As a maintainer whose test-risk blind spots only get caught when someone remembers to summon the QA Lead,
I want qa-lead to advise automatically on PRs with real test surface,
so that a persona with a proven hit rate stops depending on human recall.
Do not start this story until #1644 and #1645 have merged. They are hard gates, for the reasons in Dev Notes.
Why
Measured 2026-07-18 → 2026-09-01 in .github-private: 2 agent executions, ever. Yet the single non-soak advisory reshaped 4 of the 9 stories under epic #1402 (#1403 and #1406 carry "Enhancements from QA Lead review" sections; #1410 and #1411 exist because of it). Quality is not the problem. Reachability is: mention is the only surface served, so the persona runs only when a human types a handle.
personas/qa-lead/persona.yml already declares the surface:
- surface: pull_request
events: [opened, ready_for_review, synchronize]
enabled: true
mode: advisory
notes: "Risk assessment + test-coverage review comments on PRs with test surface."Nothing dispatches it. personas/qa-lead/interaction.yml (from #1404/#1418) records this honestly: "personas/qa-lead/persona.yml declares broader advisory surfaces … that the router does not yet wire — the contract records what is actually deployed, not the aspiration."
Acceptance Criteria
- A Class 1 event-driven path (per
docs/agentic-interaction-model.md§2) runs qa-lead's advisory on pull requests in.github-private— no cron, notimer_role. The classification table in that document gains a row for it, in the machine-checkable shape [Phase 3] Add scripts/validate-interaction-model.sh + bats, wired into lint.yml as a PR-triggered check #1406's validator parses. - It reuses
persona-runner-reusable.yml— the one runner for every persona (persona-standards §4.1). Do not add a qa-lead-specific runner; a workflow per persona is precisely the drift the manifest exists to prevent. - It is narrowly gated. It must not comment on every PR. At minimum it requires real test surface (changed paths under
tests/, or a source change with no corresponding test change) and skips trivial/verbatim stub-sync PRs. The heuristic is a pure, sourced helper with a bats suite covering: a test-touching PR (fires), a docs-only PR (does not), a verbatim stub-sync PR (does not), and a source-change-without-tests PR (fires). State the heuristic in the PR and justify it. - The opt-out works.
qa-lead:hands-offon a PR suppresses the advisory, verified by a test. This is why [qa-lead S1] Provision <id>:hands-off labels across the fleet — the opt-out is inert where the persona actually runs #1644 gates this story. - One advisory per PR, not one per push. Re-running on
synchronizemust not stack comments. Honor theidempotency_keyandconcurrency_lanealready declared inpersonas/qa-lead/interaction.yml; prefer updating or skipping over posting again. - It respects the automation budget and human gates.
pr_has_escalation_labelinscripts/lib/pr-automation-budget.shis the canonical check — reuse it, do not re-derive.needs-human-review,dev-lead:needs-human, and the budget-exhaustion marker all suppress. - The recursion guard holds. The advisory carries
<!-- persona:qa-lead -->as its first body line, enforced by the workflow, not the prompt — the read/write split inpersona-runner-reusable.ymlis load-bearing (feat: implement issue #846 — [Phase 2] Cache the LSP index and enforce the 30s cold-start SLA with auto-skip #860: prompt-only marker enforcement produced 1,481 acks in 4.5h). The new path must not weaken it. personas/qa-lead/interaction.ymlis updated to recordpull_requestas deployed, so the contract keeps matching reality.- Noise is measured, not assumed. After a defined soak window, report how many advisories fired, how many were no-action, and the share of PRs that drew one — using the no-action-comment classifier [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411 built. If the no-action share is high, the heuristic in AC Add @claude delegation, auto-merge, and rebase handling #3 is wrong and should be tightened before this goes any further than
.github-private.
Tasks / Subtasks
- Write the test-surface heuristic as a pure sourced helper + bats suite, registered in
lint.yml's bats list. (AC: Add @claude delegation, auto-merge, and rebase handling #3) - Add the Class 1 caller that invokes
persona-runner-reusable.ymlonpull_request, gated on the heuristic, the opt-out label, and the budget/human-gate markers. (AC: test issue from agent #1, Go-live improvements for PR review agent #2, Optimize review: small-PR and incremental fast paths #4, feat: feature ideation framework — adversarial proposer/challenger/synthesizer #6) - Implement per-PR idempotency using the declared key/lane. (AC: feat: add Copilot engine support via REVIEW_ENGINE toggle #5)
- Add the classification row to
docs/agentic-interaction-model.mdand confirmscripts/validate-interaction-model.shpasses. (AC: test issue from agent #1) - Update
personas/qa-lead/interaction.yml. (AC: Add Claude Code workflow #8) - Update
personas/qa-lead/README.mdto describe the now-served surface. (AC: Add Claude Code workflow #8) - Define the soak window and report the noise metrics at its end. (AC: feat: inline Claude Code workflow to access repo secrets directly #9)
Dev Notes
- Scope this to
.github-privateonly. This repo is thenextring — the smallest blast radius. Generalizing to the fleet is a separate change inpetry-projects/.githuband is gated on the ring soak ([promote persona-mention 1/3] Soak through the rings (next → ring0 → ring1 → stable) .github#783), which has not moved since 2026-07-19. - Why S1 and S2 gate this. Turning on an automatic trigger before the opt-out label exists means there is no way to tell the persona to stop — and the router's opt-out check silently reads a missing label as "not opted out." Turning it on before a quality baseline exists means the "before" number is contaminated, exactly the contamination [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411 was promoted to a gated story to prevent. Both gates come straight from this repo's own precedent; do not route around them.
- The failure mode to design against is noise, not silence. [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411 measured ~12% no-action agent comments across the fleet and is building the classifier to track it. A qa-lead that comments on every PR would be a regression even if every comment is individually correct — the manifest's own summary says "calculating risk vs value" and the held-out set contains an explicit negative control (
qa-lead-hold-well-tested-refactor) whose expected behavior is to say nothing is needed. mode: advisorymeans advisory. No writes, no labels, no blocking. The agent runs withgithub.token(which cannot post) and the workflow posts with the PAT — keep that split intact.- Note the identity caveat: until [qa-lead S7] Stop posting persona advisories under a human maintainer's PAT #1650 lands, these advisories post as
don-petry. Broadening the trigger multiplies human-attributed bot comments in the audit trail. Land [qa-lead S7] Stop posting persona advisories under a human maintainer's PAT #1650 first if it is ready; it is not a hard gate, but say in the PR which order actually happened.
Project Structure Notes
New pure helper under scripts/lib/ + bats under tests/ (registered in lint.yml); a new Class 1 caller under .github/workflows/; edits to docs/agentic-interaction-model.md, personas/qa-lead/interaction.yml, personas/qa-lead/README.md. No new cron.
References
docs/agentic-interaction-model.md§2 (Class 1), §3 (parse contract), §7 (feat: implement issue #846 — [Phase 2] Cache the LSP index and enforce the 30s cold-start SLA with auto-skip #860 rules).github/workflows/persona-runner-reusable.yml— the shared runner and the read/write splitpersonas/qa-lead/interaction.yml,personas/qa-lead/persona.ymlscripts/lib/pr-automation-budget.sh—pr_has_escalation_labelscripts/lib/pr-runaway-detect.sh— the detect-only posture to mirrordocs/postmortems/2026-06-pr-860-runaway.md- [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411 (no-action-comment classifier), [Phase 3] Add scripts/validate-interaction-model.sh + bats, wired into lint.yml as a PR-triggered check #1406 (interaction-model validator)
- petry-projects/.github
standards/persona-standards.md§4.1
⚠️ Update — 2026-09-01: this story contradicts a stated cost decision. Do not dispatch without a ruling.
Epic #1627 (Onboard solution-architect…, planned from Discussion #1360 and scoped by the maintainer refresh comment of 2026-08-31) states, under Cost cap (this cut — §9):
Mention-routed only: a run happens when — and only when — a human types the handle. No cron, no per-PR fan-out, no standing token cost. Explicitly do NOT wire
pull_request: synchronizein this cut.
This story proposes exactly the opposite for qa-lead: wiring pull_request: [opened, ready_for_review, synchronize].
The tension is narrower than it first looks, and worth stating precisely:
- That decision is scoped to "this cut" and to
solution-architect— a persona with no measured track record, whose epic's own counter-metric watches for advisories asserting doctrine without evidence. Holding it to mention-only is prudent. qa-leadis the opposite case: measured. Two agent executions in six weeks, and the single non-soak advisory reshaped 4 of the 9 stories under epic Formalize the agentic PR-interaction model (event-first, timers-as-backstop) — clear, repeatable, enforceable #1402 ([Phase 1] Write docs/agentic-interaction-model.md — the event-first standard, three trigger classes, the GITHUB_TOKEN bridge rule, and the timer contract #1403 and [Phase 3] Add scripts/validate-interaction-model.sh + bats, wired into lint.yml as a PR-triggered check #1406 carry "Enhancements from QA Lead review" sections; [Phase 4] Add a stall canary + rollback trigger for the Class-2 timer changes, before narrowing them #1410 and [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411 exist because of it). The constraint on it is reachability, not quality.- Discussion Architectural integrity in the agentic delivery model — what enforces our standards, and where does the Solution Architect fit? #1360 §5 names the same problem directly: "
qa-leadhas sat atdraftwithout a reusable; a persona that only exists as a manifest changes nothing."
So the question is not whether the cost posture is right — it is whether a persona with demonstrated per-invocation value earns an exception to it, and at what trigger width. Options, cheapest first:
- Hold at mention-only. Consistent with Onboard
solution-architectas an addressable advisory persona (repo-local first cut) #1627;qa-leadstays pull-only and its throughput problem is unsolved. Closes this story. - Narrow the trigger hard —
pull_request: [opened, ready_for_review]only, droppingsynchronize(the specific event Onboardsolution-architectas an addressable advisory persona (repo-local first cut) #1627 names), gated on the test-surface heuristic in AC Add @claude delegation, auto-merge, and rebase handling #3. One advisory per PR at most, no per-push cost. This honours the letter and most of the spirit of the cap. - As written — including
synchronize, with the AC feat: add Copilot engine support via REVIEW_ENGINE toggle #5 idempotency guard preventing comment stacking.
Option 2 is the recommendation: it removes the standing per-push cost the cap targets while still ending the "nobody remembers to summon it" failure mode. Whichever is chosen, record it here and reconcile it against #1627 §9 so the fleet has one cost posture, not two.
✅ RULING — 2026-09-07: per-PR trigger APPROVED. This story is unblocked.
The maintainer has approved the per-PR trigger for qa-lead, overriding the mention-only cost
posture in epic #1627 §9 for this persona. Implement the story.
Trigger scope — build exactly this
on:
pull_request:
types: [opened, ready_for_review]synchronize is deliberately NOT wired. "Per PR" means one advisory per pull request, not one
per push. This is the narrow reading of the approval and the cheaper one:
- It is the single event epic Onboard
solution-architectas an addressable advisory persona (repo-local first cut) #1627 §9 names explicitly ("Explicitly do NOT wire
pull_request: synchronize"), so this stays as close to that cap as an approved per-PR trigger can. - A test-risk advisory assesses the change as a whole; re-issuing it on every push is repetition, not
new signal — and repetition is what the noise metric in [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411 exists to catch. - Adding
synchronizelater is a one-line change if re-assessment per push proves genuinely wanted.
The original AC #1's event list is superseded by the two events above. AC #5 (idempotency) still
applies and is now cheap to satisfy: with no synchronize, at most one advisory per PR is the
natural state — but still guard it, because ready_for_review can fire after opened on the same PR
(open as draft, then mark ready). That exact sequence is the required test case.
Why this persona earns the exception — record it in the PR
Epic #1627's cap governs an unmeasured persona whose own counter-metric watches for advisories
asserting doctrine without evidence. qa-lead is the opposite case, and the PR description should say
so plainly so the two cost postures are reconcilable rather than contradictory:
- Its one production-tier advisory reshaped 4 of the 9 stories under epic Formalize the agentic PR-interaction model (event-first, timers-as-backstop) — clear, repeatable, enforceable #1402 — [Phase 1] Write docs/agentic-interaction-model.md — the event-first standard, three trigger classes, the GITHUB_TOKEN bridge rule, and the timer contract #1403 and [Phase 3] Add scripts/validate-interaction-model.sh + bats, wired into lint.yml as a PR-triggered check #1406
carry "Enhancements from QA Lead review" sections; [Phase 4] Add a stall canary + rollback trigger for the Class-2 timer changes, before narrowing them #1410 and [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411 exist because of it, with [Phase 4] Add a stall canary + rollback trigger for the Class-2 timer changes, before narrowing them #1410's
AC Add @claude delegation, auto-merge, and rebase handling #3 quoting it as the acceptance bar. - Its measured failure mode is reachability, not quality: 2 agent executions in six weeks, because
mentionis the only surface served.
Do NOT cite the eval score as evidence either way
qa-lead's held-out score is 0.333 (2/6) as of #1656. That number must not be used to justify or
question this rollout: 2 of its 6 cases never produced an advisory at all (a context-plumbing
artifact), and the 2 genuine misses were measured on the Haiku tier while production runs Opus.
Engine parity is being fixed under #1686. Treat the gate as not-yet-meaningful, and do not gate
this story on it.
The AC #3 heuristic is now the most important part of this story
With the trigger approved, the heuristic is the only thing standing between "qa-lead becomes useful"
and "qa-lead becomes noise." Hold to the original AC #3 — a pure, sourced helper with bats covering a
test-touching PR (fires), a docs-only PR (does not), a verbatim stub-sync PR (does not), and a
source-change-without-tests PR (fires) — and state the heuristic plainly in the PR description.
Bear in mind the held-out set contains an explicit negative control
(qa-lead-hold-well-tested-refactor) whose correct behaviour is to say nothing is needed. A qa-lead
that comments on every PR would be a regression even if each comment is individually correct.
Scope reminders
- Reuse
persona-runner-reusable.yml— one runner for every persona (persona-standards §4.1). Do
not add a qa-lead-specific runner. .github-privateonly. Fleet rollout is gated on the ring soak ([promote persona-mention 1/3] Soak through the rings (next → ring0 → ring1 → stable) .github#783).- AC feat: inline Claude Code workflow to access repo secrets directly #9's soak report stands: after the soak window, report advisories fired, no-action share, and
share of PRs drawing one, using [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411's classifier. If the no-action share is high, the heuristic is
wrong and gets tightened before this goes anywhere else.
Activity
don-petry commentedon Sep 7, 2026
Per PR triggger is approved by human
don-petry commentedon Sep 7, 2026
[dev-lead] Plan — [qa-lead S3] serve the pull_request surface
Honoring the 2026-09-07 ruling: build EXACTLY
on: pull_request: [opened, ready_for_review]—synchronizeis deliberately not wired (one advisory as a PR enters review, not one per push). Not citing the eval score. The test-surface heuristic is the load-bearing part — the line between "qa-lead becomes useful" and "qa-lead becomes noise."Approach (TDD, red→green)
scripts/lib/qa-lead-test-surface.sh) + bats (tests/test_qa_lead_test_surface.bats) covering the four AC Add @claude delegation, auto-merge, and rebase handling #3 cases: test-touching PR fires, docs-only skips, verbatim stub-sync skips, source-without-tests fires. (AC Add @claude delegation, auto-merge, and rebase handling #3)scripts/qa-lead-advisory-gate.sh) composing the heuristic with the suppressors, unit-tested (tests/test_qa_lead_advisory_gate.bats): opt-outqa-lead:hands-off(AC Optimize review: small-PR and incremental fast paths #4), one-advisory-per-PR marker scan (AC feat: add Copilot engine support via REVIEW_ENGINE toggle #5), and human/budget gates reusing the canonicalpr_has_escalation_label(AC feat: feature ideation framework — adversarial proposer/challenger/synthesizer #6)..github/workflows/qa-lead-pr-advisory.yml): agatejob → anadvisejob that reuses the onepersona-runner-reusable.yml(no qa-lead-specific runner) (AC Go-live improvements for PR review agent #2). The runner's read/write split mechanically stamps<!-- persona:qa-lead -->— recursion guard is the workflow, not the prompt (AC Workflow should skip no-op PRs in MAX_PRS count #7). Caller concurrency lane serializesopened+ready_for_review.docs/agentic-interaction-model.md(AC test issue from agent #1);personas/qa-lead/interaction.ymlrecordspull_requestas deployed (AC Add Claude Code workflow #8); README documents the served surface, two-week soak, and noise metrics (AC feat: inline Claude Code workflow to access repo secrets directly #9).Verification
bats(26 new tests green),validate-interaction-model.sh, the Python interaction-contract validator,validate-personas.py, repo-wideshellcheck --severity=warning -x, anddev-lead-lint.sh— all green locally. CI owns git.don-petry commentedon Sep 7, 2026
Dev-Lead: Implementation Complete — PR #1697
The implementation for issue #1646 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
2f6908f7ad61407b086eacfbad03a223b7122cdfReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
feat: implement issue #1646 — [qa-lead S3] Serve the pull_request sur…
feat: implement issue #1646 — [qa-lead S3] Serve the pull_request sur…
feat: implement issue #1646 — [qa-lead S3] Serve the pull_request sur…