[Agents Extension] Add test scenarios using the cli-interactive-tester tool by trangevi · Pull Request #8524 · Azure/azure-dev

Travis Angevine (trangevi) · 2026-06-02T18:16:58Z

No description provided.

Signed-off-by: trangevi <trangevi@microsoft.com>

…ll add back later Signed-off-by: trangevi <trangevi@microsoft.com>

Signed-off-by: trangevi <trangevi@microsoft.com>

Copilot

Pull request overview

Adds a comprehensive, goal-based suite of manual interactive test scenarios for the azure.ai.agents azd extension, designed to be driven via the cli-interactive-tester MCP server. This codifies repeatable end-to-end command flows (from offline help/version checks through Tier 2 cloud provision/deploy/invoke) along with a profile/override mechanism and supporting fixtures.

Changes:

Introduces a tiered scenario catalog (00-, 10-, 2x-) with tagging conventions for selective runs and fleet orchestration.
Adds shared profile defaults (profile.yaml), a local override template (profile.local.yaml.example), and gitignore rules for local profiles and run artifacts.
Adds a minimal “from-code” Python fixture used by scaffold-only init scenarios, and documents scenario usage in both the scenarios README and the extension AGENTS.md.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
cli/azd/extensions/azure.ai.agents/AGENTS.md	Documents the existence/intent of the manual cli-interactive-tester scenario suite and how contributors should use it.
cli/azd/extensions/azure.ai.agents/cspell.yaml	Adds a new word to prevent false-positive spellcheck failures from scenario docs.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/.gitignore	Ignores local profiles and tester output artifacts.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/README.md	Provides full orchestration guidance (tiers, tags, WSL path rules, auth prerequisites, hooks, and fleet mode).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/profile.yaml	Defines repo-shared default profile values (region/model/shared suffix).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/profile.local.yaml.example	Provides a template for per-user/per-CI identifying values (prefix/subscription/optional tenant).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/fixtures/from-code/app.py	Minimal Python source fixture for “init from existing code” scenarios.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/fixtures/from-code/requirements.txt	Minimal requirements file to ensure Python project detection during init-from-code flows.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-version.yaml	Tier 0 scenario for `azd ai agent version`.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-help-root.yaml	Tier 0 scenario validating root help output/command discovery.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-sample-list-text.yaml	Tier 0 scenario for `sample list` text rendering.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-sample-list-json-filters.yaml	Tier 0 scenario for `sample list` JSON output and filtering flags.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-doctor-empty-dir.yaml	Tier 0 scenario for `doctor` behavior in an empty directory.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-doctor-local-only.yaml	Tier 0 scenario for `doctor --local-only`.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-init-validate-mutually-exclusive.yaml	Tier 0 negative-path scenario validating init argument conflicts.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-init-validate-no-prompt-missing.yaml	Tier 0 negative-path scenario validating `--no-prompt` missing inputs behavior.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-init-picker-navigation.yaml	Tier 0 scenario focusing on init picker UX (filtering, navigation, abort behavior).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-template-python.yaml	Tier 1 scenario scaffolding from a Python template (auth required; stops before provision).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-template-dotnet.yaml	Tier 1 scenario scaffolding from a .NET template (auth required; stops before provision).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-from-manifest-url.yaml	Tier 1 scenario scaffolding from a GitHub manifest URL (auth + gh auth prerequisite).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-from-code.yaml	Tier 1 scenario for “use code in current directory” flow using seeded fixture.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-flags-agent-name-model.yaml	Tier 1 scenario validating `--agent-name`/`--model` overrides when initializing from a manifest URL.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-deploy-mode-code.yaml	Tier 1 scenario validating interactive code-deploy mode prompts (entry point/runtime).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/20-setup-deploy-shared-agent.yaml	Tier 2 setup scenario that provisions and deploys a shared agent used by subsequent Tier 2 scenarios.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/21-show.yaml	Tier 2 scenario validating `show` table output.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/21-show-json.yaml	Tier 2 scenario validating `show --output json`.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/22-invoke-remote.yaml	Tier 2 scenario validating remote `invoke`.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/22-invoke-new-session.yaml	Tier 2 scenario validating session vs conversation memory semantics for invoke.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/22-invoke-input-file.yaml	Tier 2 scenario validating `invoke -f` request-body-from-file behavior.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/23-sessions-lifecycle.yaml	Tier 2 scenario validating the sessions lifecycle command group.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/24-files-lifecycle.yaml	Tier 2 scenario validating the files lifecycle command group.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/25-monitor-console.yaml	Tier 2 scenario validating monitor console logs.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/25-monitor-system.yaml	Tier 2 scenario validating monitor system/container events.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/26-endpoint-update.yaml	Tier 2 scenario validating `endpoint update` behavior (patching without new version).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/27-run-local-and-invoke-local.yaml	Tier 2 scenario validating `run` + `invoke --local` with allocated ports and two sessions.
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/2A-doctor-provisioned-all-pass.yaml	Tier 2 scenario validating doctor against a provisioned project (with a known-acceptable warning).
cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/2Z-teardown-down.yaml	Tier 2 teardown scenario to destroy resources and clean the shared working directory.

Signed-off-by: trangevi <trangevi@microsoft.com>

github-actions · 2026-06-05T21:51:37Z

📋 Prioritization Note

Thanks for the contribution! The linked issue isn't in the current milestone yet.
Thank you for logging this issue; our team is reviewing it. If you need urgent prioritization, tag Rick Winter (@RickWinter) and Kristen Womack (@kristenwomack) to let us know.

Adds a workflow skill under .github/skills/agent-scenario-tests/ that resolves the current branch's PR, maps changed files to impacted cli-interactive-tester scenario tags, drives the matching scenarios through the tester MCP server, and posts a results comment on the PR. Cost-aware: Tier 2 runs only after explicit user confirmation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Use subshell cd into module directory before go build (fixes 'outside main module' error when script is run from the scenarios directory) - Use full 40-char commit hash (git rev-parse HEAD) instead of --short, matching the required version format '<semver> (commit <full-hash>)' Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Jon Gallant (jongio)

One note on the new setup-wsl.sh build script below.

Jon Gallant (jongio) · 2026-07-13T20:02:39Z

+VERSION="0.0.0-dev.0"
+LDFLAGS="-X 'github.com/azure/azure-dev/cli/azd/internal.Version=${VERSION} (commit ${COMMIT})'"
+
+GOOS=linux GOARCH=amd64 go build \


Verified this step fails as written. go build "$AZD_DIR" runs with the scenarios directory as cwd, which is inside the azureaiagent module, but $AZD_DIR points at the separate github.com/azure/azure-dev/cli/azd module. go build resolves the path argument against the main module and errors with directory ... outside main module or its selected dependencies, so the azd core build never runs. The extension's own build.sh (same directory) avoids this by cd'ing into the module first and building .. Do the same here, for example go build -C "$AZD_DIR" -o /usr/local/bin/azd .. Separately, writing to /usr/local/bin needs root on the default WSL user, so this likely needs sudo or a user-writable install location.

# Conflicts: # cli/azd/extensions/azure.ai.agents/cspell.yaml

Copilot

Pull request overview

Copilot reviewed 59 out of 59 changed files in this pull request and generated 29 comments.

@@ -0,0 +1,12 @@
+# Tier 0 (offline) — verify `azd ai agent code download --help` prints usage.
+name: "code-download-help"
+command: "azd ai agent code download --help"


@@ -0,0 +1,12 @@
+# Tier 0 (offline) — verify `azd ai agent delete --help` prints usage.
+name: "delete-help"
+command: "azd ai agent delete --help"


@@ -0,0 +1,12 @@
+# Tier 0 (offline) — verify `azd ai agent endpoint show --help` prints usage.
+name: "endpoint-show-help"
+command: "azd ai agent endpoint show --help"


+#
+# Precondition: 20-setup-deploy-shared-agent.yaml has been run successfully.
+name: "endpoint-show"
+command: "azd ai agent endpoint show"


+#
+# Precondition: 20-setup-deploy-shared-agent.yaml has been run successfully.
+name: "code-download-not-code-based"
+command: "azd ai agent code download --dest ~/working/azd-agents-shared/code-download-test"


+
+| Requirement | Check | If missing |
+| --- | --- | --- |
+| `git` + `gh` CLIs | `gh auth status` | Ask the user to run `gh auth login`. |


+- **Same scenario N times** in parallel: pass `instance_id="1"`, `"2"`, … See the README's
+  parallel-readiness section for which scenarios support it.


+- **Record a time-to-complete per scenario.** Capture wall-clock duration for
+  every scenario (from `start_session` to `finish_session`, including pre/post
+  hooks) and include it as a `Duration` column in the per-scenario tables of
+  `FINAL-REPORT.md`. Use `Hh Mm Ss` formatting (e.g. `3m 21s`, `1h 04m 12s`).


+Record, for the report: the scenario stem, tier, PASS/FAIL, wall-clock **duration**
+(`start_session` → `finish_session` incl. hooks, formatted `Hh Mm Ss`), and any
+`report_finding` text (confusing UX, errors, doc mismatches).


Jon Gallant (jongio)

One item still open on setup-wsl.sh: step 1 builds azd core straight into /usr/local/bin/azd. On a default (non-root) WSL user that directory is root-owned, so go build -o /usr/local/bin/azd fails with permission denied, and set -euo pipefail aborts the script before the extension build. The documented flow (bash setup-wsl.sh as the normal WSL user, no sudo) won't complete as written. Install to a user-writable location like ~/.local/bin/azd (and note the PATH requirement), or sudo install the built binary.

Copilot

Pull request overview

Copilot reviewed 59 out of 59 changed files in this pull request and generated 6 comments.

+# --- Step 1: Build azd core ---
+echo "▸ Building azd core (linux/amd64)..."
+
+COMMIT=$(cd "$REPO_ROOT" && git rev-parse HEAD 2>/dev/null || echo "0000000000000000000000000000000000000000")
+VERSION="0.0.0-dev.0"
+LDFLAGS="-X 'github.com/azure/azure-dev/cli/azd/internal.Version=${VERSION} (commit ${COMMIT})'"
+
+(cd "$AZD_DIR" && GOOS=linux GOARCH=amd64 go build \
+    -ldflags="$LDFLAGS" \
+    -o /usr/local/bin/azd \
+    .)


+# Copy extension.yaml (azd needs it to discover the extension)
+cp "$EXTENSION_DIR/extension.yaml" "$EXTENSION_INSTALL_DIR/extension.yaml"


+| Namespace | Values | Meaning |
+|---|---|---|
+| `tier:N` | `tier:0`, `tier:1`, `tier:2` | The tier the scenario belongs to (same axis as the directory's three sections above). Use this to express cost / auth profile in one tag. |
+| `cmd:*` | `cmd:init`, `cmd:show`, `cmd:invoke`, `cmd:sessions`, `cmd:files`, `cmd:monitor`, `cmd:endpoint`, `cmd:run`, `cmd:doctor`, `cmd:eval`, `cmd:optimize`, `cmd:sample`, `cmd:down`, `cmd:provision`, `cmd:deploy`, `cmd:version`, `cmd:help` | The top-level `azd ai agent` (or `azd`) command(s) the scenario exercises. Multi-command scenarios (e.g. `27-run-local-and-invoke-local` runs both `run` and `invoke --local`; `20-setup` runs `init` + `provision` + `deploy`) carry multiple `cmd:*` tags. |


+| `internal/cmd/init*.go` | `cmd:init` | Includes `init_from_code*`, `init_from_templates*`, `init_models`, `init_locations`, `init_validate`, `init_copy`, `init_foundry_resources_helpers`. |
+| `internal/cmd/show.go` | `cmd:show` | |
+| `internal/cmd/invoke*.go` | `cmd:invoke` | `invoke.go`, `invoke_raw.go`. |
+| `internal/cmd/run.go` | `cmd:run` | |
+| `internal/cmd/session.go` | `cmd:sessions` | |
+| `internal/cmd/files.go` | `cmd:files` | |
+| `internal/cmd/monitor*.go` | `cmd:monitor` | `monitor.go`, `monitor_format.go`. |
+| `internal/cmd/update.go` | `cmd:endpoint` | `update.go` defines `endpoint update`. |
+| `internal/cmd/doctor*.go` | `cmd:doctor` | `doctor.go`, `doctor_format.go`. |


Jon Gallant (jongio)

One open item remains on setup-wsl.sh. Step 1 still builds azd core straight into /usr/local/bin/azd:

go build -ldflags="$LDFLAGS" -o /usr/local/bin/azd .

On a default non-root WSL user, /usr/local/bin is root-owned, so this fails with permission denied. With set -euo pipefail the script aborts there, before the extension build, so the documented bash setup-wsl.sh flow won't complete as written. Install to a user-writable path like ~/.local/bin/azd (and note the PATH requirement), or gate the copy behind sudo.

The rest of the scenario files are additive and self-contained. The other nits floating around (the gpt-4.1-mini vs gpt-5.4-mini model-name mismatch between the docs and profile.yaml, and cmd:code / cmd:delete missing from the tag taxonomy and impact mapping) are worth a quick sweep but aren't blocking.

The skill workflow now includes Step 1b which executes setup-wsl.sh via the tester at the start of every run. This guarantees WSL always has the latest local code compiled as native Linux binaries, eliminating stale binary issues when code changes between runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The scenario goals are the contract — a scenario PASSES only when the product's actual behavior matches what the goals describe. The driving agent must verify goals were met, not rationalize why they weren't. Added 'never adapt around broken goals' rule: if goals reference a command/flag that doesn't exist or expect output that doesn't appear, fail the scenario instead of working around it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The shared_agent_name derivation now includes a compact timestamp (MMDDHHmm) so concurrent runs on the same machine get unique Azure resource names and working directories, preventing deployment conflicts. Format: {prefix}-{shared_agent_suffix}-{ts} Example: alice-basic-responses-07141038 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Picker navigation: update goals to test wrapping behavior (intentional) instead of expecting bounded clamping - fixtures_dir: replace AZD_AGENTS_FIXTURES env var with {fixtures_dir} session_var auto-derived by the orchestrator (4 scenarios updated) - Container deploy: clarify that startup command prompt is expected; only entry-point/runtime prompts are code-deploy-only - Make fixtures_dir derivation platform-neutral (not WSL-specific) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- eval init -> eval generate, --max-iterations -> --max-candidates (stale flags) - Remove agent.yaml references; agent definition now inline in azure.yaml - Rename 10-init-from-manifest-url -> 10-init-from-azure-yaml-url with correct URL - Update deploy-mode scenarios: code deploy auto-selected for Python templates - Flip 2C-code-download from negative to positive path (code deploy is default) - Rewrite 26-endpoint-update pre-hook to inject agentCard into azure.yaml - Fix 2B-endpoint-show goals (command shows protocols/version/auth, not URL) - Update 10-init-flags-agent-name-model with correct azure.yaml URL Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- 10-init-template-python: fix scaffold verification goal (no 'instructions' or literal 'model' in schema; model is via env var). Remove stale Container deploy goal (auto-selects code for templates). - 10-init-template-dotnet: same Container deploy fix (.NET also auto-selects code) - 26-endpoint-update: rewrite pre-hook with sed (embedded Python broke YAML parser) - 00-sample-list-text: relax goal for terminal buffer overflow (JSON scenario covers full data verification) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Location select: use 'search for and select' wording (8 scenarios) to guide text-based matching instead of fragile positional indexing - endpoint-update: replace sed with awk for reliable multi-line agentCard injection into azure.yaml - optimize: add --evaluator coherence flag (required by validate) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- sample-list-json-filters: assert --type agent returns non-empty results - optimize-submit-and-cancel: broaden gating check to any clean HTTP 400 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Other HTTP 400 errors (e.g. train_dataset.version required) are unexpected API errors and should be reported as bugs, not observations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- profile.yaml: fix stale gpt-4.1-mini comment (value is gpt-5.4-mini) - README: fix model default in profile table, soften tier 0 'offline' claim (sample-list needs network), extend tier 2 range to 2D, add 2B/2C/2D rows, add cmd:code and cmd:delete to tag taxonomy, fix eval description - impact-mapping: add code/delete command mappings, add agent_endpoint.go - workflow.md: remove stale eval/optimize coverage gap refs, extend range - running-scenarios.md: extend serial range to 2D - reporting.md: update sample report (eval has scenarios now) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

… registration Previously the script manually copied the extension binary without registering it in azd's config, which meant azd could silently fall back to the published registry version instead of the dev build. Now the script uses the official azd extension tooling: 1. Builds azd core with sudo install to /usr/local/bin 2. Ensures microsoft.azd.extensions is available (auto-installs if needed) 3. azd x build → azd x pack --bundle → azd extension install <bundle.zip> 4. Strict version verification that fails the script if dev build isn't active Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Remove continue_on_error from agentCard injection hook (if injection fails, the scenario cannot succeed — let it fail fast) - Add pre-update version capture goal so the driving agent records the version BEFORE running endpoint update, enabling a concrete comparison Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Update mutually-exclusive validation scenario to use azure.yaml - Remove legacy agent.manifest.yaml mention from init-from-url comment Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add test scenarios for the cli-interactive-tester tool

16db9dd

Signed-off-by: trangevi <trangevi@microsoft.com>

microsoft-github-policy-service Bot assigned Travis Angevine (trangevi) Jun 2, 2026

github-actions Bot added the ext-agents azure.ai.agents extension label Jun 2, 2026

Travis Angevine (trangevi) added 11 commits June 2, 2026 14:14

Some scenario edits

4fbaa67

Signed-off-by: trangevi <trangevi@microsoft.com>

Picking up recent tester tool updates

61fc535

Signed-off-by: trangevi <trangevi@microsoft.com>

Some scenario updates

596826b

Signed-off-by: trangevi <trangevi@microsoft.com>

Some more fixes to the scenarios

71767db

Signed-off-by: trangevi <trangevi@microsoft.com>

Some more improvements

df67a66

Signed-off-by: trangevi <trangevi@microsoft.com>

Remove optimization and evals because I don't understand them yet, wi…

3eac215

…ll add back later Signed-off-by: trangevi <trangevi@microsoft.com>

Add prompt to readme

c18315c

Signed-off-by: trangevi <trangevi@microsoft.com>

Add parameterization support

1d1e5f2

Signed-off-by: trangevi <trangevi@microsoft.com>

Add tags

d3cd9b5

Signed-off-by: trangevi <trangevi@microsoft.com>

Agents.md update, to direct people to the testing

ea2bac4

Signed-off-by: trangevi <trangevi@microsoft.com>

cspell

8a63c1d

Signed-off-by: trangevi <trangevi@microsoft.com>

Travis Angevine (trangevi) marked this pull request as ready for review June 5, 2026 21:32

Copilot AI review requested due to automatic review settings June 5, 2026 21:32

Travis Angevine (trangevi) requested review from JeffreyCA, Glenn Harper (glharper), John Miller (therealjohn) and Travis Wilson (trrwilson) as code owners June 5, 2026 21:32

Copilot started reviewing on behalf of Travis Angevine (trangevi) June 5, 2026 21:33 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Travis Angevine (trangevi) linked an issue Jun 5, 2026 that may be closed by this pull request

Initial pass at scenario testing #8557

Open

PR comments

4793e9f

Signed-off-by: trangevi <trangevi@microsoft.com>

Glenn Harper (glharper) requested review from Rick Winter (RickWinter), Daniel Jurek (danieljurek), tg-msft and Victor Vazquez (vhvb1989) as code owners June 8, 2026 15:58

Copilot started reviewing on behalf of Travis Angevine (trangevi) July 13, 2026 19:58 View session

Jon Gallant (jongio) reviewed Jul 13, 2026

View reviewed changes

Merge branch 'main' into trangevi/test-scenarios

4e64700

# Conflicts: # cli/azd/extensions/azure.ai.agents/cspell.yaml

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 13, 2026 20:09

Copilot started reviewing on behalf of Travis Angevine (trangevi) July 13, 2026 20:12 View session

Jon Gallant (jongio) reviewed Jul 13, 2026

View reviewed changes

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Jon Gallant (jongio) reviewed Jul 13, 2026

View reviewed changes

Travis Angevine (trangevi) and others added 3 commits July 13, 2026 14:02

Merge remote-tracking branch 'origin/main' into trangevi/test-scenarios

d8bfc5d

Travis Angevine (trangevi) marked this pull request as draft July 14, 2026 16:46

auto-merge was automatically disabled July 14, 2026 16:46
Pull request was converted to draft

Travis Angevine (trangevi) and others added 12 commits July 14, 2026 10:43

fix: replace agent.manifest.yaml references with azure.yaml

56f4913

- Update mutually-exclusive validation scenario to use azure.yaml - Remove legacy agent.manifest.yaml mention from init-from-url comment Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

fix: add MMDDH to cspell word list for timestamp format patterns

ac1289d

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Travis Angevine (trangevi) closed this Jul 23, 2026

This was referenced Aug 3, 2026

[evals] Add a starter set of Foundry scenario evals #9414

Closed

[evals] Add a starter set of Foundry scenario evals #9415

Merged

Navigation Menu

Uh oh!

[Agents Extension] Add test scenarios using the cli-interactive-tester tool - #8524

[Agents Extension] Add test scenarios using the cli-interactive-tester tool#8524
Travis Angevine (trangevi) wants to merge 39 commits into
mainfrom
trangevi/test-scenarios

Travis Angevine (trangevi) commented Jun 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 5, 2026 •

edited

Loading

Uh oh!

Jon Gallant (jongio) left a comment

Uh oh!

Jon Gallant (jongio) Jul 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Jon Gallant (jongio) left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Jon Gallant (jongio) left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

		- Same scenario N times in parallel: pass `instance_id="1"`, `"2"`, … See the README's
		parallel-readiness section for which scenarios support it.

		# Copy extension.yaml (azd needs it to discover the extension)
		cp "$EXTENSION_DIR/extension.yaml" "$EXTENSION_INSTALL_DIR/extension.yaml"

Uh oh!

Conversation

Travis Angevine (trangevi) commented Jun 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📋 Prioritization Note

Uh oh!

Jon Gallant (jongio) left a comment

Choose a reason for hiding this comment

Uh oh!

Jon Gallant (jongio) Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Jon Gallant (jongio) left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Jon Gallant (jongio) left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

github-actions Bot commented Jun 5, 2026 •

edited

Loading