[Agents Extension] Add test scenarios using the cli-interactive-tester tool

.github/skills/agent-scenario-tests/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: agent-scenario-tests
+license: MIT
+metadata:
+  version: "1.0"
+  # Bump major on breaking prompt/trigger changes; bump minor on new references or mapping rules.
+description: >-
+  **WORKFLOW SKILL** — Runs the azure.ai.agents extension's cli-interactive-tester
+  scenarios locally as a PR regression check. Resolves the current branch's PR,
+  maps changed files to impacted scenario tags, drives the matching scenarios
+  through the cli-interactive-tester MCP server, and posts a results comment on the PR.
+
+  INVOKES: git CLI, gh CLI, cli-interactive-tester MCP tools (list_scenarios,
+  load_scenario, run_pre_hooks, start_session, send_action, finish_session,
+  run_post_hooks), ask_user.
+
+  USE FOR: run agent scenarios, scenario regression check, cli-interactive-tester,
+  test agents extension PR, run impacted scenarios, check agents extension for regressions,
+  agent scenario tests, validate azure.ai.agents change.
+
+  DO NOT USE FOR: azd core preflight (use azd-preflight), changelog (use changelog-generation),
+  creating PRs (use pull-request), authoring brand-new scenarios from scratch without a code
+  change, running scenarios for any extension other than azure.ai.agents.
+---
+
+# agent-scenario-tests
+
+Runs the `azure.ai.agents` extension's interactive CLI scenarios as a **local**
+PR regression gate and reports the results back on the pull request.
+
+## Overview
+
+The `azure.ai.agents` extension ships goal-based scenarios for the
+[cli-interactive-tester](https://github.com/coreai-microsoft/cli-interactive-tester)
+MCP server under `cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/`.
+These scenarios are **never run in CI** — they need the tester MCP server, a populated
+`profile.local.yaml`, manual `az`/`gh` login, and (for Tier 2) real Azure resources.
+
+This skill is the **opt-in, run-locally** flow a PR submitter uses to check their change
+for regressions. It:
+
+1. Resolves the current branch's PR link.
+2. Maps the PR's changed files to the impacted scenario **tag set** (`cmd:*` / `tier:*`).
+3. Enumerates and drives only the impacted scenarios through the tester.
+4. Posts a per-scenario results comment back on the PR.
+
+It is cost- and side-effect-aware: Tier 0 is free/offline, Tier 1 needs Azure auth but
+provisions nothing, and **Tier 2 incurs Azure cost and is only run after explicit user
+confirmation**.
+
+> This skill drives scenarios **deliberately, with user consent**. That is different from
+> the extension's `AGENTS.md` rule that coding agents must not invoke scenarios on their
+> own during ordinary work — here the user has explicitly asked for a scenario run.
+
+{{ references/prerequisites.md }}
+
+{{ references/workflow.md }}
+
+{{ references/impact-mapping.md }}
+
+{{ references/running-scenarios.md }}
+
+{{ references/reporting.md }}
+
+## Exit Criteria
+
+- The current branch's PR was resolved (or the user supplied one / chose to skip the comment).
+- The impacted scenario set was derived from the PR diff and **confirmed by the user**
+  (including an explicit cost acknowledgement before any Tier 2 run).
+- Every selected scenario was driven to completion with a recorded PASS/FAIL, duration, and
+  any findings, and a `FINAL-REPORT.md` was written under `.reports/<run-timestamp>/`.
+- A results comment was posted on the PR (unless the user opted out), and any Tier 2 run was
+  followed by `2Z-teardown-down` so no Azure resources are left running.

.github/skills/agent-scenario-tests/references/impact-mapping.md

@@ -0,0 +1,76 @@
+<!-- cspell:ignore nextstep exterrors -->
+# Impact mapping — changed files → scenario tags
+
+Goal: from a PR's changed-file list, derive the **smallest** scenario tag set that still
+covers the change, plus the tier ceiling for cost gating.
+
+All paths below are relative to `cli/azd/extensions/azure.ai.agents/`.
+
+## 1. Command source → `cmd:*` tag
+
+Files under `internal/cmd/` map to the command they implement:
+
+| Changed file (glob) | Tag(s) | Notes |
+| --- | --- | --- |
+| `internal/cmd/init*.go` | `cmd:init` | Includes `init_from_code*`, `init_from_templates*`, `init_models`, `init_locations`, `init_validate`, `init_copy`, `init_foundry_resources_helpers`. |
+| `internal/cmd/show.go` | `cmd:show` | |
+| `internal/cmd/invoke*.go` | `cmd:invoke` | `invoke.go`, `invoke_raw.go`. |
+| `internal/cmd/run.go` | `cmd:run` | |
+| `internal/cmd/session.go` | `cmd:sessions` | |
+| `internal/cmd/files.go` | `cmd:files` | |
+| `internal/cmd/monitor*.go` | `cmd:monitor` | `monitor.go`, `monitor_format.go`. |
+| `internal/cmd/update.go` | `cmd:endpoint` | `update.go` defines `endpoint update`. |
+| `internal/cmd/doctor*.go` | `cmd:doctor` | `doctor.go`, `doctor_format.go`. |
+| `internal/cmd/eval*.go` | `cmd:eval` | `eval.go`, `eval_init.go`, `eval_run.go`, `eval_list.go`, `eval_show.go`, etc. Tier 2 (needs a deployed agent + Foundry endpoint). |
+| `internal/cmd/optimize*.go` | `cmd:optimize` | `optimize.go`, `optimize_apply.go`, `optimize_status.go`, etc. Tier 2 (submits a cloud optimization job). |
+| `internal/cmd/sample*.go` | `cmd:sample` | `sample.go`, `sample_list.go`. |
+| `internal/cmd/code*.go` | `cmd:code` | `code.go` (code download). |
+| `internal/cmd/delete*.go` | `cmd:delete` | `delete.go` (agent deletion). |
+| `internal/cmd/version.go` | `cmd:version` | |
+| `internal/cmd/root.go` | `cmd:help` + broad | Touches the whole command tree — treat as broad (see §3). |
+| `internal/cmd/listen.go` | — | gRPC host entrypoint; not scenario-testable. |
+
+## 2. Changed command with NO scenario coverage (gaps)
+
+These commands have **no** scenario in the suite yet. If the PR touches them, you cannot
+run a regression check — **report the gap** and recommend the author add a scenario
+(per the extension `AGENTS.md`), rather than silently passing:
+
+| Changed file (glob) | Uncovered command |
+| --- | --- |
+| `internal/cmd/mcp.go` | `mcp start` (hidden/preview) |
+
+## 3. Shared / cross-cutting code → broaden
+
+Changes outside a single command file affect many flows. When the diff touches any of
+these, broaden the impacted set (and ask the user how wide to go):
+
+| Changed file (glob) | Broaden to |
+| --- | --- |
+| `internal/cmd/helpers.go`, `internal/cmd/agent_context.go`, `internal/cmd/agent_endpoint.go`, `internal/cmd/*_context.go` | All `cmd:*` for commands that resolve project/agent context — at minimum `cmd:init`, `cmd:invoke`, `cmd:show`, `cmd:doctor`. |
+| `internal/cmd/root.go`, `internal/cmd/banner.go`, `internal/cmd/nextstep_output.go` | Run a Tier 0 smoke set (`tier:0`) across all commands. |
+| `internal/pkg/**`, `internal/project/**`, `internal/exterrors/**` | Map by what the package feeds: parsers/manifests → `cmd:init`; deployment/project target → `cmd:provision` + `cmd:deploy` (Tier 2). When unclear, propose a Tier 0/1 sweep and ask before any Tier 2. |
+| `go.mod` / `go.sum` / dependency bumps | Tier 0 smoke + ask whether a fuller sweep is warranted. |
+| files **outside** `cli/azd/extensions/azure.ai.agents/` (e.g. `cli/azd/` core) | This skill is scoped to the agents extension; note that core changes may need core azd testing instead, and proceed only with the agents-relevant subset. |
+
+## 4. Tier ceiling (cost gate)
+
+From the impacted `cmd:*` set, decide the **highest tier to offer**:
+
+- Default to **Tier 0 + Tier 1** for any change to a covered command (free + auth-only).
+- Offer **Tier 2** only when the change can plausibly affect cloud behavior — i.e. it
+  touches `cmd:invoke`, `cmd:sessions`, `cmd:files`, `cmd:monitor`, `cmd:endpoint`,
+  `cmd:show`, `cmd:run`, `cmd:eval`, `cmd:optimize`, `cmd:doctor` provisioned paths,
+  deployment/project code, or the provision/deploy flow. `cmd:eval` and `cmd:optimize`
+  are Tier 2-only (no offline happy path beyond their negative-path validation
+  scenarios). Tier 2 always requires the explicit cost confirmation in
+  `workflow.md` Step 4.
+- A pure Tier 0 change (e.g. `version.go`, help text, `sample list` formatting) should run
+  Tier 0 only.
+
+## 5. Translate tags → run list
+
+Combine the derived `cmd:*` tags with the chosen tier tags and call
+`list_scenarios(tags=[...])`. Example: an `invoke.go` change approved for Tier 2 →
+`list_scenarios(tags=["cmd:invoke"])`, then keep the Tier 0/1 results plus the Tier 2
+`22-*` scenarios, prefixed by `20-setup` and suffixed by `2Z-teardown`.

.github/skills/agent-scenario-tests/references/prerequisites.md

@@ -0,0 +1,56 @@
+# Prerequisites
+
+Verify these before doing anything else. If a hard prerequisite is missing, stop and tell
+the user exactly what to fix — do **not** try to work around it.
+
+### Repo location
+
+1. Locate the scenarios directory:
+   `cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/`.
+   Resolve its absolute path; everything else is relative to it.
+2. Note the **WSL path** of that directory for MCP tool arguments. On Windows hosts the
+   tester runs inside WSL, so a Windows path like
+   `C:\Repos\azure-dev\...\scenarios\00-version.yaml` must be passed as
+   `/mnt/c/Repos/azure-dev/.../scenarios/00-version.yaml`. On macOS/Linux use the native
+   absolute path. See `running-scenarios.md` § Path style.
+
+### Tooling
+
+| Requirement | Check | If missing |
+| --- | --- | --- |
+| `git` + `gh` CLIs | `gh auth status` | Ask the user to run `gh auth login`. |
+| cli-interactive-tester MCP server | The `list_scenarios` / `start_session` MCP tools are available to you | Stop. Tell the user to register the cli-interactive-tester MCP server (see its README) and re-run. |
+| `profile.local.yaml` | File exists in the scenarios dir | Stop. Tell the user to `cp profile.local.yaml.example profile.local.yaml` and set `prefix` + `subscription`. |
+| Native Linux `azd` in WSL (Windows only) | `azd version` inside the tester returns a dev build, not a Windows `.exe` interop version | The skill automatically runs `setup-wsl.sh` (Step 1b) to rebuild from source. If you need to run it manually: `bash setup-wsl.sh` from the scenarios directory inside WSL. Symlinking to `azd.exe` does not work (causes git safe.directory, TTY, and file-locking errors). |
+
+### Auth (tier-dependent — only enforce for tiers actually selected)
+
+- **Tier 0** needs no auth.
+- **Tier 1 / Tier 2** read from / write to Azure. A human must `az login` inside WSL
+  **before** the run (the agent cannot complete the browser sign-in). If the selected set
+  includes Tier 1/2, remind the user to `az login` first.
+- **Manifest scenarios** (`10-init-from-manifest-url`, `10-init-flags-agent-name-model`)
+  download from GitHub and can fall back to the `gh` CLI; they need `gh auth login` inside
+  WSL. Their `pre` hook fails fast if it isn't set up.
+
+### Profiles
+
+The scenarios reference `{prefix}`, `{subscription}`, `{region}`, `{model}`, `{tenant}`
+(optional) and `{shared_agent_name}` via placeholders. You must:
+
+1. Read both `profile.yaml` (checked-in defaults) and `profile.local.yaml` (developer
+   overrides) and **merge them, local overriding shared**.
+2. Derive `shared_agent_name = "{prefix}-{shared_agent_suffix}-{ts}"` where `{ts}` is
+   a compact timestamp of the form `MMDDHHmm` (e.g. `07141038`). This isolates
+   concurrent runs so two agents on the same machine don't collide on Azure resource
+   names or working directories.
+3. Derive `fixtures_dir` = the tester-side absolute path of the `fixtures/` subdirectory
+   inside the scenarios directory. On Windows (where the tester runs inside WSL) this is
+   the WSL-translated path (e.g. `/mnt/c/Repos/azure-dev/.../fixtures`); on native
+   Linux/macOS it is the regular absolute path. Apply the same path-style logic used for
+   scenario paths (see `running-scenarios.md` § Path style).
+   Scenario pre-hooks use `{fixtures_dir}` to locate test fixture files.
+4. Pass the merged map (including `shared_agent_name` and `fixtures_dir`) as `session_vars`
+   on **every** `load_scenario`, `run_pre_hooks`, `start_session`, and `run_post_hooks`
+   call. Omitting it leaves placeholders unresolved and the run executes against literal
+   `{prefix}` strings.

.github/skills/agent-scenario-tests/references/reporting.md

@@ -0,0 +1,51 @@
+# Reporting
+
+Two outputs: a local `FINAL-REPORT.md` artifact and a PR comment.
+
+## FINAL-REPORT.md
+
+Write to `<scenarios-dir>/.reports/<run-timestamp>/FINAL-REPORT.md` (the `.reports/` tree is
+git-ignored). Include:
+
+- Run header: timestamp, PR number/URL, branch, base ref, the derived tag set, and the
+  tiers actually run.
+- A per-tier table of scenarios with columns: `Scenario | Tier | Result | Duration | Findings`.
+- A short "Coverage gaps" section listing any changed command(s) with no scenario (from
+  `impact-mapping.md` §2), so the author knows to add one.
+- Links to the per-scenario `tester-reports/<run_name>/` folders for screenshots/HTML.
+
+## PR comment
+
+Post with `gh pr comment <number> --body-file <path>` (use a temp file to preserve
+formatting). Keep it scannable — full detail lives in the artifact. Suggested shape:
+
+```markdown
+## 🧪 Agent scenario regression check
+
+**Branch:** `<headRef>` → `<baseRef>` · **Run:** `<run-timestamp>`
+**Impacted tags:** `cmd:invoke`, `cmd:sessions` · **Tiers run:** 0, 1, 2
+
+| Scenario | Tier | Result | Duration |
+| --- | --- | --- | --- |
+| 00-version | 0 | ✅ PASS | 4s |
+| 22-invoke-remote | 2 | ✅ PASS | 1m 12s |
+| 22-invoke-new-session | 2 | ❌ FAIL | 1m 40s |
+
+**Findings**
+- `22-invoke-new-session`: `--new-conversation` still recalled the prior name — memory
+  was not reset. (screenshot: …)
+
+**Coverage gaps:** this PR also touches `mcp.go`, which has no scenario — consider adding one.
+
+<sub>Run locally via the `agent-scenario-tests` skill. Not run in CI.</sub>
+```
+
+Rules:
+
+- Use ✅ PASS / ❌ FAIL (and ⚠️ for a scenario that completed but raised a non-fatal finding).
+- **Never** soften a real regression to make the table green. A scenario that failed because
+  of the PR's change is a FAIL — report it and recommend fixing the code, not the scenario.
+- If the user opted out of posting (or there's no PR), write only the artifact and print the
+  summary to the user instead.
+- Mention any Tier 2 teardown status explicitly (e.g. "`2Z-teardown-down` ran — no resources
+  left provisioned") so the reader knows nothing is still costing money.

.github/skills/agent-scenario-tests/references/running-scenarios.md

@@ -0,0 +1,89 @@
+<!-- cspell:ignore defaultyourvalue -->
+# Running scenarios through the tester
+
+This mirrors the scenarios `README.md` ("Driving conventions" and "Orchestrating a fleet
+run"). Follow it so the run actually *tests* the CLI instead of papering over its bugs.
+
+## Path style (Windows → WSL)
+
+On Windows the tester drives CLIs through tmux **inside WSL**, and it resolves every
+path-shaped MCP argument on the WSL side. Pass POSIX paths:
+
+| Orchestrator OS | Pass to MCP tools |
+| --- | --- |
+| Windows | `/mnt/c/Repos/azure-dev/.../scenarios/00-version.yaml` |
+| macOS / Linux | native absolute path |
+
+This applies to `path:` on `load_scenario` / `run_pre_hooks` / `run_post_hooks` and to
+`scenario_path:` on `start_session`. If `load_scenario` returns `Scenario file not found`,
+the path style is almost certainly the cause — translate `C:\…` → `/mnt/c/…` and retry once
+before fanning out.
+
+## Per-scenario loop
+
+For each selected scenario:
+
+1. `load_scenario(path=<wsl path>, session_vars=<merged profile>)` — also tells you whether
+   the scenario declares `pre`/`post` hooks.
+2. If it has `pre` hooks: `run_pre_hooks(path=…, session_vars=…)`. Hooks run host-side,
+   sequentially, fail-fast (unless `continue_on_error: true`).
+3. `start_session(scenario_path=…, session_vars=…, run_name=<scenario-stem>, output_dir=<wsl .reports path>)`.
+   - `run_name` = the YAML filename without `.yaml` (e.g. `00-version`, `21-show-json`).
+   - For scenarios that start two sessions (`27-run-local-and-invoke-local`), suffix the
+     `run_name` with a role tag (`…-run`, `…-invoke`).
+   - `output_dir` = WSL path of `<scenarios-dir>/.reports/<run-timestamp>/tester-reports`.
+     Reuse the **same** `<run-timestamp>` across every scenario in the run.
+4. Drive the session's `goals:` with `send_action` / `select_by_text` / screenshots, then
+   `finish_session`.
+5. If it has `post` hooks: `run_post_hooks(path=…, session_vars=…)`.
+
+## Driving conventions (fail-loud)
+
+- **The scenario goals are the contract.** A scenario PASSES only when the product's actual
+  behavior matches what the goals describe. If the goals say "expect error X" and the
+  product prints a different error (even a reasonable one), that is a FAIL. If the goals
+  reference a flag or subcommand that no longer exists, that is a FAIL. The driving agent's
+  job is to **verify** goals were met, not to **rationalize** why they weren't. Do not mark
+  a scenario as PASSED with an "observation" when the goals were not achieved — observations
+  are for incidental notes on scenarios that genuinely passed all their goals.
+- **Don't verify/retry after a `select`.** Reading back the echo and "correcting" a pick
+  hides the very picker defect the test exists to catch. Send the action and let downstream
+  prompts surface any failure.
+- **Treat a select miss as a hard failure.** `select_by_text` is fail-loud
+  (`ERROR during 'select': …`). Report a finding and stop that scenario — do **not** retry
+  with a different `choice_text`/`choice_index`.
+- **Never retry a failed scenario.** If a scenario fails (command errors, unexpected output,
+  non-zero exit), report the finding and move on. Do **not** re-run the scenario hoping for
+  a different result — unless the scenario's `goals:` explicitly instruct a retry. Retrying
+  masks flaky behavior and makes the test suite unreliable as a regression signal.
+- **Never adapt around broken goals.** If the goals instruct you to run a command or flag
+  that does not exist, or expect output that does not appear, fail the scenario. Do not
+  substitute an alternative command, skip the broken step, or invent a workaround. The
+  scenario must be updated by a human — the driving agent must not silently patch over it.
+- **Prefer `choice_text` over `choice_index`** (indices shift between releases).
+- **Clear a pre-filled text field before typing** (e.g. the agent-name prompt); otherwise
+  your value *appends* to the default (`defaultyourvalue`).
+- **Pause before the first cloud-creating action.** The Step 4 cost confirmation covers
+  this; never enter a Tier 2 provision flow without it.
+
+## Parallelism & ordering
+
+- **Tier 0 / Tier 1** (`parallel-safe`): fan out in small waves (4–6 at a time), one
+  sub-agent per scenario, each with a distinct descriptive `session_id` **postfixed with a
+  timestamp** (e.g. `fleet-10-init-from-code-1752434100`). The timestamp (Unix epoch
+  seconds) prevents collisions when multiple agent sessions drive the tester concurrently.
+  No `instance_id` is needed — each scenario's `cwd` already isolates itself (defaults to
+  the `-main` suffix).
+- **Same scenario N times** in parallel: pass `instance_id="1"`, `"2"`, … See the README's
+  parallel-readiness section for which scenarios support it.
+- **Tier 2** (`serial-only`): never parallelize. Run `20-setup-deploy-shared-agent` first,
+  then `21-…2D-` serially (they share one deployed agent and mutate shared session/file/
+  endpoint state), then `2Z-teardown-down` last.
+- **Validate the recipe with one scenario before fanning out** — confirm `load_scenario` →
+  `start_session` → one `send_action` round-trips for a single Tier 0 scenario first.
+
+## Capture per scenario
+
+Record, for the report: the scenario stem, tier, PASS/FAIL, wall-clock **duration**
+(`start_session` → `finish_session` incl. hooks, formatted `Hh Mm Ss`), and any
+`report_finding` text (confusing UX, errors, doc mismatches).