From 14e100e1421b5945c988f519616b789077a2d184 Mon Sep 17 00:00:00 2001 From: brady gaster Date: Fri, 21 Aug 2026 13:13:12 -0700 Subject: [PATCH] docs(e2e): amend E4 for #1811/#1812, formalize E1 Rule D gate, add folder README E4: add Condition 0 dual-role justification (#1811), extend Phase 2 to include `plan activate`, add Phase 3e roster-provenance gate for #1812, add contamination-table growth note, strip citations to nonexistent `squad-e2e-runbook.md`. E1: promote Rule D from `NOT VERIFIED (future work)` to a formal gate on E1 with a required-fields NOT VERIFIED escape hatch for detection-hang cases. README: explain vestigial E-series numbering (E2 folded into E1, E3 rehearsal became E4). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7532d43f-3da4-4087-9d0b-5a3c38bbf9e9 --- .squad/e2e/E1-merge-continuation-relay.md | 61 ++++- .squad/e2e/E4-agent-binding-verification.md | 267 +++++++++++++++----- .squad/e2e/README.md | 40 +++ 3 files changed, 307 insertions(+), 61 deletions(-) create mode 100644 .squad/e2e/README.md diff --git a/.squad/e2e/E1-merge-continuation-relay.md b/.squad/e2e/E1-merge-continuation-relay.md index 9a5ef84d0..4b95f9612 100644 --- a/.squad/e2e/E1-merge-continuation-relay.md +++ b/.squad/e2e/E1-merge-continuation-relay.md @@ -11,7 +11,7 @@ The merge-continuation relay — which had **never once succeeded** before tonight — worked end to end on 2026-08-21. -**E1 verdict: PASS** — all tested items passed. One item (epic closure / Rule D) was **not verified** due to an infrastructure hang, not a Squad defect. +**E1 verdict: PASS (with one gate NOT VERIFIED)** — 7 of 8 tested items passed. The eighth — **Rule D / epic closure** — was **NOT VERIFIED** due to a `detection` infrastructure hang, not a Squad defect. Rule D is now a **formal gate on this procedure** (see *Rule D — the gate and its escape hatch* below), and any re-run must score it explicitly rather than treating an unobserved run as green. --- @@ -88,4 +88,61 @@ Operational guidance: if the `detection` job has not completed within ~10 minute Epic closure was not re-tested after the fix. The session ended while the `detection` job hung. This is **not a Squad regression** — the infra hang prevented observation, not a logic failure. -This item remains open for a future E2 experiment. +Rule D is now a formal gate on E1 (see next section). It is **not** getting its own scenario number — an earlier draft of the E-series referred to a "future E2 experiment," but that scenario was never authored. Rule D shares E1's trigger (the merge-continuation relay closing the last leaf), so verifying it separately would either duplicate E1 or fabricate the state — neither is worth a distinct procedure. See `.squad/e2e/README.md` for the numbering history. + +--- + +## Rule D — the gate and its escape hatch + +**Added post-execution, 2026-08-21.** Any re-run of E1 MUST score Rule D explicitly rather than defaulting to green on absence of contrary evidence. + +### The gate + +**Trigger.** All leaf tasks under an epic have been merged (their child issues CLOSED) and the merge-continuation relay has been observed to dispatch (i.e. E1's other 7 gates have passed on the final leaf). + +**Rule D verdict — three possible outcomes:** + +- ✅ **PASS** — the epic issue is CLOSED within a stated observation window (default: 10 minutes after the last leaf's merge-continuation relay run reaches `success`). The closure is attributed to Squad (either commented by the coordinator or effected via a Squad-authored PR/dispatch), not a manual close. +- ❌ **FAIL** — all leaves are CLOSED, the observation window has expired, the epic is still OPEN, and no `Squad —` workflow run in that window shows a closure attempt. This is a real Rule D defect: closure logic exists but did not fire, or fired against the wrong target. +- ⚠️ **NOT VERIFIED** — the observation was blocked by an infrastructure hang (`detection` on `Install ripgrep` is the known offender). Record the observation-blocking job's URL and step and the exact minute the run was cancelled. A NOT VERIFIED verdict is *not* a green: it is a documented absence of signal, and it does not close #1758.2 or any successor. + +### The escape hatch (required text on any NOT VERIFIED result) + +```yaml +rule_d: + verdict: NOT VERIFIED + reason: "detection job hung >N min on step ''; observation window expired before closure could be witnessed" + blocked_by_job_url: "https://github.com///actions/runs//job/" + blocked_by_step: "" + cancelled_at_utc: "" + observation_window_min: 10 + window_started_utc: "" # when the relay run reached success + window_ended_utc: "" # cancelled_at_utc for hang; +window_min otherwise +``` + +**Do not omit any of these fields on a NOT VERIFIED.** An escape hatch that accepts blank fields becomes a way to score anything as NOT VERIFIED. If any of the required text is missing, treat the result as **FAIL** (blocked-observation is a claim; unsubstantiated blocked-observation is silence). + +### PASS observation (the mandatory extractions) + +```powershell +$FIXTURE = "bradygaster/aspiregregator-squad-e2e" +$EPIC = + +# The relay run's success timestamp — the start of the observation window. +gh run list --repo $FIXTURE --workflow "Squad" --limit 20 ` + --json databaseId,name,conclusion,updatedAt ` + --jq '.[] | select(.conclusion=="success" and (.name|test("implement|relay"; "i"))) | + [(.databaseId|tostring),.name,.updatedAt] | join(" | ")' 2>$null + +# Epic state at the end of the observation window. +gh issue view $EPIC --repo $FIXTURE --json state,closedAt,timelineItems ` + --jq '{state,closedAt,closer:(.timelineItems[]|select(.__typename=="ClosedEvent")|.actor.login)}' 2>$null +``` + +**PASS attribution rule.** `closer` must be a Squad-owned actor (`app/github-actions` posting a Squad workflow closure, or the coordinator's Squad-authored PR merging). A closure attributed to a human operator during the observation window is **not** a Rule D PASS — it is a Rule D FAIL that was manually papered over, and E1's whole point is that closure was supposed to be autonomous. + +### FAIL observation + +Record the full `gh run list --workflow "Squad"` output for the observation window along with the epic's `state` and `openIssues` count. A FAIL says the closure logic did not fire — that is the finding. + +--- diff --git a/.squad/e2e/E4-agent-binding-verification.md b/.squad/e2e/E4-agent-binding-verification.md index 999100a47..5cd7477a3 100644 --- a/.squad/e2e/E4-agent-binding-verification.md +++ b/.squad/e2e/E4-agent-binding-verification.md @@ -1,8 +1,21 @@ -# E4 — Agent-Binding Verification (#1784) +# E4 — Agent-Binding Verification (#1784, #1812) > **Author:** Sims (E2E Test Engineer) > **Written:** 2026-08-21, from the E3 long-path rehearsal -> **Status:** ✅ **EXECUTED 2026-08-21 — result recorded below.** +> **Status:** ✅ **EXECUTED 2026-08-21 (n=1) — result recorded below. Amended 2026-08-21 (post-run) for #1811 and #1812; re-run required for n=2.** + +## Amendment log + +| Date | Change | Reason | +|---|---|---| +| 2026-08-21 (initial) | Executed, PASS at n=1, two qualifications recorded | see result below | +| 2026-08-21 (post-run) | Phase 2 extended by one command (`plan activate`); new Phase 3e roster-provenance gate; contamination-table growth note; Condition 0 dual-role justification; `squad-e2e-runbook.md` citations replaced with self-contained text; scope-limitation section updated | #1811, #1812 — the n=1 PASS was structurally silent on the `plan activate` roster-read defect, and Condition 0's timing guarantee was undocumented | + +> ⚠️ **The n=1 PASS below scored the procedure that stopped at `plan implementation`.** The +> amended procedure (Phase 2 now includes `plan activate`; Phase 3 now includes a +> roster-provenance gate) has **not yet been executed**. Any claim about #1812 requires a fresh +> live run against the amended procedure — do not cite the n=1 record as evidence of the +> provenance gate's behaviour. --- @@ -18,6 +31,20 @@ Run against fixture `bradygaster/aspiregregator-squad-e2e`, seed issue **#22**, | 2 | No `squad:lead`/`devrel`/`reviewer` in timeline `labeled` events | ✅ 15 events, all `squad` | | 3 | Activation summary reports missing-label prerequisite gap | ✅ reported verbatim | +> 🛑 **Condition 0 does three jobs, not one — evaluate it first (#1811).** It is a **liveness +> floor**, a **seed-exclusion census**, and — the reason it must come first — the +> **premature-read guard**. The #1784 measurement is a `safe_outputs` effect: **job 5 of 6** +> in the gh-aw pipeline (`pre_activation`, `activation`, `agent`, `detection`, `safe_outputs`, +> `conclusion`). `agent ✅` lands *before* the label state is written. Reading at that moment +> fails **in both directions**: a label census returns a **false RED** (the three pre-existing +> `squad:lead/devrel/reviewer` labels look like a fresh reproduction), and a timeline `labeled` +> query returns a **vacuous GREEN** (empty event list). Switching instruments does not help — +> the hazard is temporal, not instrumental. Condition 0 catches both because a premature read +> yields zero Squad-authored issues → INCONCLUSIVE → stop before 1–3 are scored. **Wait for +> all six jobs, including `conclusion`; `agent ✅` is not sufficient.** Do not refactor Condition +> 0 into "just a liveness floor" and move it later — it looks harmless because the guard is +> silent when it works. + Same-fixture control: **E3 produced `lead, lead, devrel` on this exact column**; E4 produced `McManus, Keaton, Fenster, Hockney…`. One prompt change (#1789) between them. @@ -180,26 +207,34 @@ appeared. The root cause reproducing itself precisely. --- -## ⚠️ Scope limitation — what E4 does NOT cover +## ⚠️ Scope — what E4 covers, and what it does NOT + +**As amended, E4 verifies two things:** -**E4 verifies the task-level `Agent` binding only.** +1. **Task-level `Agent` binding at `plan implementation`** (#1784). Historical scope, unchanged. +2. **Roster provenance at `plan activate`** (#1812). Added post-#1811. The gate compares the + activation summary's `"Roster set read from …"` line against the fixture's actual + `.squad/team.md → ## Members → Name` column, as a set. -Epic **#17** received `squad:lead`, but the `plan program` artifact (run `32433011339`) contains -**no owner or agent assignment whatsoever** — verified by direct read. The epic's owner is -therefore minted at **`plan activate`**, which is **downstream of where E4 stops**. +**E4 does NOT cover:** -⇒ **A green E4 does not mean "no owner label leaks anywhere."** It means *"the `Agent` column at -`plan implementation` is clean."* Epic-level owner minting is a **separate, unverified surface** -requiring its own check on a full-path run. +- **`plan validate` / `plan accept …` / `implement` / merge / post-merge relay.** Those add + ~27 min and are on E1's beat (the merge-continuation relay path), plus follow-on paths that + are **not** unattended-safe (see the E4 Follow-on section for #1787 sibling-epic refill). +- **Rule D — epic closure after last leaf merges.** That is a formal gate on E1. +- **Every label surface downstream of `plan activate`.** E4 stops immediately after `plan + activate` runs. -E4's scope remains correct and sufficient for its purpose — `devrel` appears **in the Agent -column**, so the dispositive signal is present where E4 looks. But do not overstate the result -when reporting it. +**Why the boundary is drawn at `plan activate`.** The two defects E4 is engineered to catch — +role-token leakage in the `Agent` column (#1784) and roster-provenance falsehood in the +activation summary (#1812) — both surface no later than the `plan activate` job's own +comment. Continuing past it does not improve E4's signal for either defect and does bring in +the autonomy blockers documented in the Follow-on section. -**The leak originates in the `Agent` column at `plan implementation`** (`:913`), and propagates -into minted labels at `plan activate` (`:730`). That's why this procedure stops at -`plan implementation` — it catches the defect at its source rather than at its symptom, which -is what makes a ~27 min run sufficient instead of a ~54 min one. +**Historical note (n=1 record only).** The original procedure stopped at `plan implementation` +and produced a PASS at n=1. That record is preserved because it is real evidence for its own +scope. It is **not** evidence for the amended procedure — the roster-provenance gate has +never been executed as a formal gate. Re-run required. --- @@ -285,15 +320,19 @@ Role-appropriateness requires a **manual read of each task's text against its bi ## Budget -**~27 minutes**, derived from E3 rather than estimated: E3's +**~31 minutes**, adapted from E3's measurement: E3's `research → triage → plan program → plan implementation` sub-window ran 00:18:10 → 00:44:54 = -**26.7 min** (runs `32432129404` → `32433493989`), plus fixture refresh. +**26.7 min** (runs `32432129404` → `32433493989`), plus fixture refresh. **Add ~4 min for the +`plan activate` step** appended in this amendment; E4 (n=1) measured `plan activate` at +`~2m43s` (run `32471509974`, 2026-08-21 10:11:53Z → 10:14:36Z). Round up to ~4 min to cover +between-step gh-aw pickup latency. -Add ~12 min if you enter out of order — see *Required entry sequence* in the runbook. +Add ~12 min if you enter out of order — see *Ordering — mandatory, not stylistic* in Phase 2. -**E4 is unattended-safe.** It stays entirely on the planning path, so it creates no worker PR -and therefore hits neither the `action_required` approval gate nor the `detection` hang. -E3 needed **zero** human interventions across 8 runs. +**E4 is unattended-safe.** All five commands are on the planning track, so E4 creates no +worker PR and therefore hits neither the `action_required` approval gate nor the `detection` +hang. E3 needed **zero** human interventions across 8 runs. The `plan activate` step mints +labels on the fixture; that is expected fixture state change, not an intervention. --- @@ -330,9 +369,11 @@ So the compiled lock **and** the committed source markdown are separately stale. the workflow came from. It does **not** mean the fixture is current. **Never** treat it as evidence of freshness. -**④ Do not trust `gh aw update`.** It has a 3-day cooldown that silently skips the source pull, -then compiles happily and exits 0 (runbook gotcha #1 — an instance of the silent-success -pattern, gotcha #10). Fetch via `gh api`, then `gh aw compile`. +**④ Do not trust `gh aw update`.** It has a 3-day cooldown that **silently skips** the source +pull, then compiles happily and exits 0. This is an instance of the silent-success pattern that +runs through this document: **a tool that returns success without doing the requested work is +worse than one that fails loudly**, because the operator has no signal to investigate. Fetch +via `gh api`, then `gh aw compile`. ### 0b. Establish the staleness baseline — compare, never hardcode @@ -467,23 +508,30 @@ If no: what seed shape would have been needed — ## Phase 2 — Run the short path -**Run exactly these four commands, in this order, and stop.** +**Run exactly these five commands, in this order, and stop.** ``` /squad research /squad triage /squad plan program -/squad plan implementation ← the leak site; STOP HERE +/squad plan implementation ← the #1784 leak site +/squad plan activate ← the #1812 provenance site; STOP HERE ``` -> **Do not continue to `plan validate` / `accept` / `activate`.** Those add ~27 min and -> verify nothing additional for #1784 — the `Agent` column is already emitted at -> `plan implementation`. +> **Do not continue to `plan validate` / `plan accept …` / `implement`.** Those add ~27 min and +> verify nothing additional for E4 — the `Agent` column is emitted at `plan implementation` +> and the roster-provenance line is emitted at `plan activate`. **The ordering is mandatory, not stylistic.** The long path enforces preconditions: `plan program` on a fresh issue halts and tells you to run `triage` first; `triage` halts and demands -`research` first. **Those halts are correct behaviour, not bugs** — but they cost ~12 min if -you trip them. +`research` first; `plan activate` refuses to run without `plan accept` in later revisions, but +at the time of writing runs directly after `plan implementation`. **Those halts are correct +behaviour, not bugs** — but they cost ~12 min if you trip them. + +> ⚠️ **After each command, wait for all six pipeline jobs to complete** (`pre_activation`, +> `activation`, `agent`, `detection`, `safe_outputs`, `conclusion`). **`agent ✅` is not +> sufficient** — see the Condition 0 justification at the top of this document. A run where +> `conclusion` has not yet completed is a **premature-read hazard** on the very next command. Capture the run ID after each command: @@ -519,7 +567,7 @@ Write-Host "`n=== Agent column values ===" Read the `Agent` column by eye against the accepted-values list above. Do not automate the judgement — a regex that "finds no forbidden tokens" in a plan that failed to render is a -silent-success trap (gotcha #10). **Confirm the plan actually contains a task table first.** +silent-success trap. **Confirm the plan actually contains a task table first.** > 🛑 **Ignore Squad's own "Validation Pre-check" table.** In E3 it printed > `Agent assignments valid (cast Names) | ✅ (lead, lead, devrel)` — asserting three role @@ -559,11 +607,12 @@ Every `squad:{x}` on a **newly created** issue must have `{x}` = a lowercased ro | `squad:devrel` | E1 + E3 | #8, #20 | | `squad:reviewer` | **E1 only** | #9 | -> Note: at `plan implementation` the plan is only *proposed* — labels are minted later, at -> `plan activate`. So E4 may legitimately create **no new labelled issues at all**. That is -> expected, and it means the label check is **weak corroboration only**. -> **The `Agent` column is authoritative for E4.** If labels look clean but the `Agent` column -> is dirty, that is a **FAIL** — the leak simply hasn't propagated yet. +> ⚠️ **This table grows.** Every E4 run mints labels at `plan activate` (that step is now part +> of E4 as of the 2026-08-21 amendment). **After running E4, append the newly-minted labels to +> this table before the next run** — otherwise the second run's contamination filter is stale +> and either passes labels it should catch or catches labels it should ignore. If unsure, +> re-derive the table from the fixture: `gh label list --repo $FIXTURE` gives repo-global +> presence; scope by issue via the same `E4_START`-cutoff query used above. ### 3c. Roster cross-check @@ -583,17 +632,102 @@ argument no longer holds** — re-derive the criterion before judging the run. gh aw logs squad --repo $FIXTURE --artifacts all --count 1 --output $EVIDENCE ``` +### 3e. Roster-provenance gate at `plan activate` (#1812) + +**This is the gate added in the 2026-08-21 amendment. It has never been executed as a formal +gate.** The n=1 PASS above scored the pre-amendment procedure that stopped at +`plan implementation`; that procedure was structurally silent on #1812. + +**The claim under test.** The `plan activate` summary posts a line of the shape: + +> *"Roster set read from `.squad/team.md` (`## Members` → `Name` column): ``"* + +**What #1812 measured (2026-08-21, run `32471509974`, seed #22):** the summary reported +`{lead, reviewer, devrel, security, docs}` while the fixture's actual `Name` column is +`{Keaton, McManus, Fenster, Hockney, Kint}`. Five-for-five mismatch, and the minted labels +were a strict subset of the *false* set — i.e. the code acted on the falsely-reported roster. +This is not a mere reporting bug; the false provenance is the tell for a hardcoded lookup. + +**The gate is two-part. Both parts must pass.** + +- **(A) Provenance line exists.** The activation summary must contain the line above. + **Absence is a FAIL** — a summary that no longer makes the claim is a different bug we still + want to catch (silent-success class: an activate that stopped reporting where it read from is + indistinguishable from an activate that stopped reading anything). +- **(B) Reported set equals fixture set.** Normalize both sides: trim whitespace, lowercase, + drop backticks and commas, treat as an unordered set. The reported set must equal the set + extracted from the fixture's `.squad/team.md → ## Members → Name` column. + +```powershell +# Grab the plan activate summary comment on the seed issue. +gh issue view $ISSUE --repo $FIXTURE --json comments ` + --jq '.comments[-1].body' 2>$null | + Set-Content (Join-Path $EVIDENCE "plan-activate-comment.md") + +$summary = [IO.File]::ReadAllText((Join-Path $EVIDENCE "plan-activate-comment.md")) + +# (A) presence — the exact phrasing may vary; anchor on the two invariants. +$claim = [regex]::Match($summary, '(?im)Roster set read from.*?team\.md.*?Name.*?:\s*(?.+?)\r?\n') +if (-not $claim.Success) { + Write-Host "GATE 3e (A): FAIL — no 'Roster set read from …' line in activation summary" +} else { + Write-Host "GATE 3e (A): PASS — provenance line present" + + # (B) set-equality. + $b = gh api "repos/$FIXTURE/contents/.squad/team.md" --jq '.content' 2>$null + $roster = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($b)) + + # Extract the ## Members → Name column. Trim to that section only. + $members = [regex]::Match($roster, '(?ms)^##\s*Members\s*\r?\n(?.*?)(?=^##|\z)').Groups['body'].Value + # Take the first non-header cell of each row; the Name column is convention position 1. + $realNames = [regex]::Matches($members, '(?m)^\|\s*([^|`\s][^|]*?)\s*\|') | + ForEach-Object { $_.Groups[1].Value.Trim('` ').ToLower() } | + Where-Object { $_ -notin @('name','---',':---',':---:','---:') } + + $reportedRaw = $claim.Groups['set'].Value + $reportedNames = ($reportedRaw -split '[,`]') | + ForEach-Object { $_.Trim('` ,').Trim().ToLower() } | + Where-Object { $_.Length -gt 0 } + + $real = [System.Collections.Generic.HashSet[string]]::new([string[]]$realNames) + $reported = [System.Collections.Generic.HashSet[string]]::new([string[]]$reportedNames) + $eq = $real.SetEquals($reported) + Write-Host "GATE 3e (B): $(if ($eq) {'PASS'} else {'FAIL'}) — reported=[$($reportedNames -join ',')] real=[$($realNames -join ',')]" +} +``` + +**Verdict interpretation.** + +- **Both (A) and (B) PASS** → the `plan activate` roster read is honest. #1812 is (this run, + n=1) not reproduced. +- **(A) PASS, (B) FAIL** → **the #1812 defect is confirmed live**, and the false-provenance + claim is the strongest evidence: the activate step *cited* the fixture's `team.md` and + produced a set that isn't in it. Do not accept an argument that the mismatch is cosmetic — + cite this section back. +- **(A) FAIL** → separate silent-success bug filed against Procedures. Do not proceed to score + this run for #1812 either way — you have no readable signal. + +**Read this by eye too.** As with the `Agent` column, the automated set comparison is a +double-check, not the judgement. A regex that "finds no mismatch" against an empty comment +scores green. **Confirm the activate comment actually rendered a summary first.** + +> ⚠️ **This gate assumes the activation summary's phrasing is stable.** The regex anchors on +> `"Roster set read from"` and `"team.md"` and `"Name"`. If Procedures changes the phrasing, +> update the anchor **before** re-running; a silently non-matching anchor scores green. + --- ## Evidence to capture | Artifact | Why | |---|---| -| Run IDs + conclusions for all 4 commands | Reproducibility; timing against the ~27 min budget | -| `plan-implementation-comment.md` | **Primary evidence** — the `Agent` column verbatim | +| Run IDs + conclusions for all **five** commands | Reproducibility; timing against the ~31 min budget | +| `plan-implementation-comment.md` | **Primary #1784 evidence** — the `Agent` column verbatim | +| `plan-activate-comment.md` | **Primary #1812 evidence** — the roster-provenance line verbatim | | Full `squad:*` label list | Corroborating evidence | | Phase 0b comparison output (all four `MATCH`) | Proves the fixture was actually refreshed | | Roster `DevRel`/`devrel` check output | Keeps the dispositive argument falsifiable | +| Gate 3e set-comparison output (reported vs real) | **Primary #1812 evidence** — falsifiable in text | | `safeoutputs.jsonl` | Raw agent output | | Epic-shape record from Phase 1 | Feeds the follow-on; **not** part of the verdict | @@ -601,20 +735,34 @@ gh aw logs squad --repo $FIXTURE --artifacts all --count 1 --output $EVIDENCE ```yaml scenario: E4 -purpose: "#1784 agent-binding verification" -fix_pr: -fixture_refreshed: true # all four surfaces MATCH in Phase 0b -runs: [ research, triage, plan_program, plan_implementation ] +purpose: "#1784 agent-binding verification + #1812 activate roster-provenance verification" +fix_prs: # PR numbers for the #1784 and #1812 fixes (may be same or distinct) + "#1784": "" + "#1812": "" +fixture_refreshed: true # all four surfaces MATCH in Phase 0b +runs: [ research, triage, plan_program, plan_implementation, plan_activate ] run_ids: [] + +# #1784 gate — pre-existing (Conditions 1–3 in n=1 record) agent_column_values: [] forbidden_tokens_found: [] -squad_devrel_present: # dispositive -squad_reviewer_present: # dispositive -squad_lead_present: # ambiguous — does not alone determine FAIL -new_issues_only: true # labels scoped to issues created by THIS run -verdict: PASS|PARTIAL|FAIL -epics_produced: # observational only -scope_note: "task-level Agent binding only; epic-level owner minting not covered" +squad_devrel_present: # dispositive +squad_reviewer_present: # dispositive +squad_lead_present: # ambiguous — does not alone determine FAIL +new_issues_only: true # labels scoped to issues created by THIS run +verdict_1784: PASS|PARTIAL|FAIL + +# #1812 gate — Phase 3e (new in the 2026-08-21 amendment) +gate_3e_a_provenance_line_present: +gate_3e_b_roster_set_equal: +gate_3e_reported_set: [] # verbatim as reported by activation summary +gate_3e_real_set: [] # verbatim from fixture .squad/team.md ## Members Name column +verdict_1812: PASS|FAIL|INCONCLUSIVE # INCONCLUSIVE if 3e(A) FAIL + +# Combined +verdict: PASS|PARTIAL|FAIL # PARTIAL if one gate PASS, the other FAIL +epics_produced: # observational only +scope_note: "task-level Agent binding + plan-activate roster-provenance only; validate/accept/implement/Rule D not covered" notes: "" ``` @@ -623,8 +771,8 @@ notes: "" # Follow-on (SEPARATE) — #1779 / PR #1787 sibling-epic refill > ⚠️ **This is NOT part of E4.** It has its own criterion, its own path, and its own risk -> profile. **E4's pass/fail is purely #1784.** Do not let this dilute it, and do not report a -> combined verdict. +> profile. **E4's pass/fail is #1784 + #1812 combined (verdict schema in "Verdict record").** +> Do not let this dilute it, and do not report a combined verdict. If Phase 1 produced ≥2 sibling epics, the resulting tree is *also* the shape needed to exercise #1779 — a genuine efficiency win, since no fixture has had it yet. Reuse it **only after** E4 @@ -638,9 +786,8 @@ has returned its own verdict. unstarted, no further dispatch. Sketch: merge a leaf under Epic A, then confirm the worker surfaces Epic B's work. -Cross-reference the `#1779` decision rule and the `#1772`-vs-`#1779` discriminator in the -runbook — #1779 produces tasks that were **never dispatched**; #1772 produces dispatches that -were **probe-only**. They can coexist; check both. +The `#1779`-vs-`#1772` discriminator: #1779 produces tasks that were **never dispatched**; +#1772 produces dispatches that were **probe-only**. They can coexist; check both. ### 🛑 This follow-on is NOT unattended-safe @@ -668,8 +815,10 @@ unattended-safe; this does not. | Ref | | |---|---| -| **#1784** | Owner/Agent binding fix (#1759) ineffective live — the defect under test | +| **#1784** | Owner/Agent binding fix (#1759) ineffective live — the original defect (CLOSED, but see #1812) | +| **#1812** | `plan activate` reads a hardcoded roster, not `.squad/team.md` — re-diagnoses #1784 (OPEN) | +| **#1811** | Condition 0 dual-role documentation — closed in-doc via the Condition 0 note above (OPEN) | | **#1759** | The original binding fix — structurally correct, failed live | | **#1779** | Sibling-epic refill boundary — follow-on only | | **PR #1787** | Refill dispatch slots from the root, not the parent epic | -| `squad-e2e-runbook.md` | Long-path budget, entry sequence, unattended-path table, gotchas #1–#10 | +| **PR #1789** | Prompt-side agent-binding fix (validate + implementation-plan planner) | diff --git a/.squad/e2e/README.md b/.squad/e2e/README.md new file mode 100644 index 000000000..ac2876fd0 --- /dev/null +++ b/.squad/e2e/README.md @@ -0,0 +1,40 @@ +# `.squad/e2e/` — End-to-End Test Procedures + +**Owner:** Sims (E2E Test Engineer) +**Purpose:** Written, committed, re-runnable procedures for verifying the gh-aw / Squad integration against **existing** repositories. Fixture-side and new-repo/activation procedures are Booster's beat, not here. + +## Files in this folder + +| File | Kind | Status | +|---|---|---| +| `E1-merge-continuation-relay.md` | E-scenario procedure + evidence | ✅ Executed 2026-08-21 — 7 of 8 gates PASS, Rule D formalized post-run | +| `E4-agent-binding-verification.md` | E-scenario procedure + evidence | ✅ Executed 2026-08-21 — n=1 PASS; amended post-run for #1811 and #1812 (Phase 2 extended, Phase 3e added); re-run required for n=2 | +| `windows-test-baseline.md` | **Not an E-scenario** — Windows human-interpretation runbook | Reference document; interprets `npm test` output, does not run against gh-aw | +| `README.md` (this file) | Folder guide | — | + +## Where E2 and E3 are — and why they aren't here + +The E-series numbering is **chronological in the operator's head, not a coverage plan**. That produced a folder that reads as though E2 and E3 were promised and never delivered. They were not promised. What actually happened: + +- **E2 (Rule D / epic closure).** E1's evidence table left Rule D as `NOT VERIFIED` because the `detection` job hung on `Install ripgrep` before closure could be observed. A note in E1 called this *"a future E2 experiment"*, which some readers reasonably interpreted as a scenario slot. In practice, Rule D shares E1's trigger — the same merge-continuation relay closing the same last leaf — so a separate E2 would either duplicate E1's setup or fabricate the closure state. **E2 was folded back into E1 as a formal gate (see the `Rule D — the gate and its escape hatch` section of `E1-merge-continuation-relay.md`).** There is no E2 procedure and there will not be one unless Rule D develops a trigger of its own. + +- **E3 (long-path rehearsal).** E3 was a live rehearsal of the full 8-run `research → triage → plan program → plan implementation → plan validate → plan accept scope → plan accept implementation → plan activate` walk against the `aspiregregator-squad-e2e` fixture on 2026-08-21. It surfaced #1784 (the token-leak defect that became E4). **The rehearsal's evidence never landed in this folder** — it lived on the operator's disk and became the input for E4 rather than a committed procedure of its own. E3 references you see in `E4-agent-binding-verification.md` (e.g., *"E3 produced `lead, lead, devrel`"*) are citations to that rehearsal record, not to a file that exists here. + +- **E5 and beyond.** No procedures are currently under authorship. The `#1787` sibling-epic refill scenario is documented as a `Follow-on` inside `E4-agent-binding-verification.md` rather than as a standalone `E5-…` file, because it is not unattended-safe and cannot share E4's execution budget. If it graduates to a first-class E-scenario in the future, it should get its own numbered file at that time. + +## Rules of thumb for future E-scenarios + +- **A procedure without a committed file has no procedure.** Rehearsal notes on someone's disk are not evidence. If it matters enough to run, it matters enough to commit. +- **Number by chronological authorship, not by pretending the numbering is a plan.** Gaps in the number sequence are honest information, not a claim of missing work — this file exists so future readers stop asking "where's E2?" +- **The status column in this file is the source of truth for what has been executed and when.** Do not let a scenario document drift from its own `Status:` header. +- **`windows-test-baseline.md` is intentionally here** despite not being an E-scenario. It sits with the E-scenarios because operators running E-scenarios need it. Its own header calls out that it is a human-interpretation runbook, not a procedure — do not let its file location seed the "so where are E2/E3/E5+?" question. + +## Fixture + +All E-scenarios currently target `bradygaster/aspiregregator-squad-e2e`. Fixture-refresh procedure is embedded in E4's Phase 0. Do not point an E-scenario at a different fixture without first documenting why the existing fixture cannot express the scenario — a bent fixture proves less than an honest gap. + +## Related + +- `.squad/decisions.md` — team decisions, including the fixture-fate decision (2026-08-20) +- `.squad/agents/sims/charter.md` — E2E ownership boundary (I don't own unit tests; that is FIDO's beat) +- Live issues that shape what runs next: #1784 (CLOSED), #1811, #1812, #1817