You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(ci): Automatically rerun failed main rolling builds for specific failures - #20345
Failed rolling main CI builds can automatically retry once when every failed job matches a narrow infrastructure allowlist. Runs for obsolete commits and runs with unmatched failures are not retried.
The rerun workflow dispatches to separate PR/manual and current-main policies, sharing GitHub job access and reporting helpers. Analysis remains read-only; only the output-gated execution job can request reruns.
Current-main retry rules
After ignoring aggregate result jobs, source attempt 1 must contain one to five failed jobs, each matching one of:
A failed Get or Head request to https://netaspireci.azurecr.io/v2/... with connect: connection refused or Connection reset by peer.
A failed Get or Head request to https://mcr.microsoft.com/v2/... with HTTP 503 or ServiceUnavailable.
Windows process initialization failure (-1073741502 or 0xC0000142) after successful tests, with failures confined to post-test reporting or cleanup steps.
Registry request errors must occur on the same line, within 500 characters of the quoted request. Registry retries require failed-step metadata and are rejected when a known test-execution or startup/smoke-validation step failed. This includes regular and NuGet-dependent tests, extension E2E, polyglot SDK validation, starter startup, native-dashboard tests and packaged CLI smoke tests.
This intentionally skips genuine image-download outages inside failed test steps. A recovered image pull can coexist with an unrelated test failure, and job-wide logs cannot reliably attribute the failure to the registry. The broader registry veto does not expand the Windows successful-test exception.
Immediately before requesting a rerun, the policy rechecks the completed failed attempt, current main SHA, and newer rolling runs. Attempt 2 cannot trigger another retry. Request errors are reported explicitly.
The PR/manual policy retains force mode and its three-attempt cap. Automatic PR retries additionally require the failed SHA to remain the open PR's current head; manual dispatch can target a historical commit while its associated PR remains open.
Validation
223 policy tests passed, with zero failures or skips:
Regression coverage includes 16 execution-step names across both registry families and annotation/log diagnostics, extension whole-run vetoes, and preservation of the Windows exception.
Nine controlled live GitHub scenarios passed in an isolated private repository:
ACR and MCR log-only positives each requested one successful rerun.
Persistent infrastructure failure stopped at attempt 2.
Failed extension tests and mixed product/infrastructure failures skipped the write job.
Stale main and stale open-PR heads were rejected without reruns.
A current PR reran and received one tracking comment; manual dry-run made no writes.
The four policy/shared modules were byte-identical to this revision. Workflow and dispatcher adaptations were limited to three repository/owner guard literals. Registry messages were simulated diagnostics in real GitHub job logs, not induced registry outages. Trial PRs were closed, workflows disabled, and temporary branches removed.
node --check passed for all five runtime modules and the test harness. actionlint .github/workflows/auto-rerun-transient-ci-failures.yml and git diff --check passed.
At head 1b4fbce0d38fb6d12d8cb6eb136b14ac996b7e66, CI passed, including 2,044 Infrastructure tests with zero failures or skips, package generation and all six native CLI archive builds. Native hosting validation passed. All reported checks completed successfully or were intentionally skipped.
Checklist
Is this feature complete?
Yes. Ready to ship.
No. Follow-up changes expected.
Are you including unit tests for the changes and scenario tests if relevant?
Yes
No
Did you add public API?
Yes
If yes, did you have an API Review for it?
Yes
No
Did you add <remarks /> and <code /> elements on your triple slash comments?
Yes
No
No
Does the change make any security assumptions or guarantees?
Yes
If yes, have you done a threat model and had a security review?
The reason will be displayed to describe this comment to others. Learn more.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
Note
This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.
Failed main pushes had no automatic retry path, leaving transient
failures red until another push. Reuse the existing failed-job rerun
workflow with live attempt, main SHA, and supersession checks, a cap
of three retries, and persisted decisions and recovery.
Keep main reruns out of failure analysis and make memory-branch
publication resilient to concurrent updates.
Refs microsoft#20340
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Failure analysis could file a main-breakage issue on attempts 1–3
while retries were pending, but excluded the fourth failed attempt.
Analyze only that terminal attempt. Recheck the live run, main SHA,
and newer CI runs before collecting evidence and publishing issues,
so superseded failures do not create or update issues.
Fixesmicrosoft#20340
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Synchronize with upstream main while retaining the CI retry changes.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Current-main retry decisions were written to the analyzer's memory
branch even though nothing consumes them. A failed write could make
the rerun workflow fail after GitHub accepted a retry, while competing
writes required the analyzer to rebase its analysis records.
Keep the live-run guards and three retry attempts, but report decisions
in the workflow summary and logs. Remove the audit-specific writer,
recovery-only path, conflict handling, and tests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Document when the terminal helper runs and how it handles stale or
unverifiable main runs. Remove the obsolete audit-storage claim and
clarify that the generic CI failure tracker can still report an early
failed push.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Current-run validation occurs after expensive evidence queries
.github/workflows/analyze-ci-failure.md:252
This current-run guard executes only after the main path has already queried the triggering merge, last successful run, and candidate merge history (lines 197–228). That contradicts the stated guarantee that stale attempt-4 analysis is rejected before evidence collection and lets superseded runs perform the expensive attribution calls first. Move this verification to immediately after the run metadata/scope is validated, before the main-specific evidence queries.
A failed main CI attempt could be left without Copilot analysis when
GitHub rejected the request for the next automatic rerun. The analyzer
only admitted the configured final attempt, so a failed request stopped
the sequence before an analyzable attempt existed.
Dispatch fallback analysis when an otherwise eligible rerun request
fails, while retaining current-main, live-attempt, and superseding-run
checks. Keep the retry count in defaultMaxRunAttempt so PR reruns, main
reruns, and final analysis derive from one versioned policy value.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… probe (microsoft#21522)
## Description
Fixesmicrosoft#20017
### Root cause
`GeneratedReferenceExpression_CancellationIsRequestScoped` runs a Java
probe. The probe starts `expression.getValue(token)` on a pool thread
and then calls `requireMessage`, which gave the generated client a
**fixed 1000ms wall-clock budget** (`waitForMessage(input, 1000)`) to
write its first transport message.
Before writing the `invokeCapability` request, the client has to do the
following in a cold JVM:
- load classes;
- marshal the cancellation registration;
- run `token.onCancel`. The test's `SlowPreCancelledToken` sleeps 250ms
here on purpose;
- start the reader loop;
- serialize JSON for the first time.
That leaves about 750ms of slack. A busy CI agent can use that up, and
the probe then fails with `Timed out waiting for transport message` at
`requireMessage` ← `invokeAndComplete` (the first request of the first
invocation). This matches the [CI
failure](https://github.com/microsoft/aspire/actions/runs/34401953654/job/102636354113):
it failed after 2.64s on ubuntu.
The 1s bound measures product latency. Latency is not part of the
contract this test checks; the contract is the request/cancellation
ordering and that cancellation is request-scoped.
### Fix (readiness signal instead of timing)
- `requireMessage(input, invocation)` now waits until the expected
message is **readable**. If the invocation completes without writing the
message, it fails immediately and includes the invocation's error. It
keeps polling `available()` rather than calling a blocking `read`,
because `PipedInputStream.read` reports "Pipe broken" once the writing
pool thread has died.
- `invocation.get(1, TimeUnit.SECONDS)` → `invocation.get()`. This is
safe because the server response has already been written at that point,
so future completion is the readiness signal.
- Hangs are still bounded by the existing 8s probe process timeout. The
negative check (a marshalling failure writes nothing,
`waitForMessage(serverInput, 250)`) is unchanged. No assertions are
weakened and no retries are added.
### Regression test
The test is now a `[Theory]` over the `onCancel` delay:
- `250`: the existing scenario.
- `1100`: client work before the first message exceeds the old fixed 1s
wait. This reproduces what a slow CI agent does, but deterministically.
### Evidence
| Scenario | Old probe | New probe |
|---|---|---|
| `onCancel` delay 1100ms (direct `java` run) | **3/3 fail**,
`IllegalStateException: Timed out waiting for transport message` (same
as CI) | passes |
| `dotnet test` theory (250, 1100) | — | 2/2 passed |
| Full `Aspire.Hosting.CodeGeneration.Java.Tests` | — | 73/73 passed |
| Old probe at 250ms, 2 vCPU + 6 busy loops | 40/40 pass (the flake is
rare; this is the only recorded occurrence) | — |
### Note
Four sibling transport probes use the same fixed `requireMessage` 1s
pattern (`GeneratedTransport_CancelsStreamJsonRpcCallbackRequest`,
`GeneratedTransport_CancelListenerCanReenterClient`,
`GeneratedTransport_ReleasesRemoteTokensAfterCallbackCompletion`,
`GeneratedTransport_CancellationIsDirectionalAndLocalCancelNotifiesHostOnce`).
None of them has a recorded failure, so this PR stays scoped to the
probe that failed. The same change can be applied to them if they flake.
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… -getResultOutputFile (microsoft#21524)
## Description
`Hex1bNativePublishingTests.NativeAssetsKeepTheirLayoutAndContents`
fails intermittently on `Infrastructure (ubuntu-latest)` with:
```
System.Text.Json.JsonReaderException : 'M' is invalid after a single JSON value. Expected end of data. LineNumber: 209 | BytePositionInLine: 0.
```
### Root cause
The MSBuild JSON is **not** truncated. In every failing log I checked
(runs 37668103720, 37782413043 and 35756256834), the complete `-getItem`
JSON is followed on **stdout** by the dotnet CLI's MSBuild-server client
fallback diagnostic:
```
MSBuild server unavailable: could not connect to the server within the timeout window; the server may have failed to start. Falling back to an in-process build.
```
With SDK 11, MSBuild server is used by default for invocations that run
a target (`-t:CopyTestFiles -getItem:...`). When the runner is loaded
(heartbeat showed CPU 97% in run 37668103720), the client can't connect
in time, falls back to an in-process build and prints that line after
the results. The tests parsed raw stdout, so `JsonDocument.Parse` failed
on the trailing `M`.
Other tests in this repo already guard against this exact message:
`MicrosoftTestingPlatformTests` uses `-getResultOutputFile`, and
`TemplateManifestTestEnvironment` uses
`DOTNET_CLI_USE_MSBUILD_SERVER=0`.
### Fix
The test now reads its MSBuild query results from
`-getResultOutputFile:<path>` instead of stdout.
- A new `RunMSBuildQueryAsync` helper appends the switch and returns the
file contents.
- Every `-getProperty`/`-getItem` caller in the class now uses it. That
covers the four JSON parsers and the single-property `Bundle.proj`
`Configuration` check, which compared trimmed stdout and was exposed to
the same noise.
- The result file uses exactly the same format MSBuild would otherwise
print: JSON, or the raw value for a single property.
- MSBuild-server behavior is unchanged, and diagnostics stay in the
captured output for failure messages.
No assertions were weakened, and no sleeps or retries were added.
### Regression test
`NativeAssetsIgnoreMSBuildServerFallbackDiagnostics` runs the real
native-asset query, with all of its existing assertions, through a
`dotnet` wrapper (`.sh` on Unix, `.cmd` on Windows). The wrapper runs
the real CLI and then writes the exact CI fallback line to stdout. The
test also asserts the line was captured, so it cannot pass by skipping
the noisy scenario.
| | Result |
|---|---|
| Before fix (stdout parsing) | **3/3 failed**: `'M' is invalid after a
single JSON value. Expected end of data. LineNumber: 209`, identical to
CI |
| After fix | **10/10 passed** for the regression test, and **4×** the
full `Hex1bNativePublishingTests` class (43 tests) with **43/43
passing** each time |
Local runs used Linux and SDK 11.0.100-rc.2.
### Denominator
5 tracked occurrences across both issues (2026-09-21 to 2026-10-08; 3 on
`main`, 2 on PRs), against about 1,484 `ci.yml` runs from 2026-09-21 to
2026-10-07. Different theory rows failed (`win-x64` and `win-arm64`,
with and without single file), which fits noise from the environment
rather than one data row.
I noticed that `ResourceGeneratorMetadataTests` also parses merged
`msbuild -getItem` output from a script, but I left it unchanged to keep
this PR focused.
Fixesmicrosoft#20799Fixesmicrosoft#20272
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
---------
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…soft#21451)
## Description
`ReleaseScriptShellTests.DryRun_ShowsDownloadAndInstallSteps`
(Acquisition, macOS) fails intermittently. It asserts that the
route-sidecar line appears before the `[DRY RUN] Would run:` line in
`result.Output`.
### Root cause
`get-aspire-cli.sh` prints these lines in a deterministic order, but on
different streams:
- `DRYRUN: would write route sidecar to: ...` is written to **stdout**
(`printf`).
- `[DRY RUN] Would run: ... setup` is written to **stderr** (`say_info`
→ `>&2`) by `setup_cli_bundle`, which runs immediately afterwards.
`ToolCommand` builds `Output` from two independent async readers
(`OutputDataReceived` and `ErrorDataReceived`) that both append to one
list. Lines from different pipes are appended in callback order, not
emission order. Under scheduling pressure the stderr line can be
appended first, so the assertion fails even though the script behaved
correctly.
### Fix
For bash scripts, `ScriptToolCommand` now runs `bash -c 'exec bash "$0"
"$@" 2>&1' <script> <args>`. stderr is redirected into stdout inside the
child, so both streams share one pipe and `Output` matches emission
order. `exec` keeps the exit code and the process tree unchanged. No
assertion was changed, and no sleeps, retries or timeouts were added. No
`ScriptToolCommand` caller consumes stderr separately, and `Output`
already contained both streams.
### Regression test and evidence
New test:
`ScriptToolCommandTests.BashScript_OutputPreservesOrderAcrossStdoutAndStderr`.
A script alternates 500 stdout/stderr lines, and the test asserts that
`Output` keeps the exact order.
| | New regression test | Stress: original dry-run assertion, 2000 runs,
2 busy-loop CPU hogs |
|---|---|---|
| Before (main `a76b57a99`) | **failed 3/3** (e.g. expected `stderr 1`,
actual `stdout 2`) | **1/2000** order inversions (reproduces the CI
failure) |
| After | passed 3/3 | 0/2000 |
All 7 bash `ScriptToolCommand` test classes pass locally: 130 passed, 2
skipped. The 4 `PRScriptInstallerModeTests` Homebrew cask tests fail the
same way on unmodified main because `ruby` isn't installed locally.
### Denominator
- Window: ci.yml runs Oct 16–30, 2026, main and PR CI. I scanned 507
head SHAs from every run that failed or was re-run (193 failed, 270
successful after re-run, plus others).
- First attempts: 213 macOS and 213 Ubuntu first attempts of the
Acquisition job.
- macOS: 0 failures in that set.
- Ubuntu: 1 unrelated failure (af07e33, exit 2, logs expired).
- The only observed failure of this test is attempt 1 of the macOS
Acquisition job on PR microsoft#21448's CI (job 124041318173).
- The failure is rare (about 1 in 200+ first attempts), consistent with
the 1/2000 local reproduction rate under load. The race itself is
deterministic, as shown by the regression test above.
Fixes # (no existing issue; no open issue or PR owns this failure. Draft
microsoft#20976 rewrites this test but keeps the cross-stream `Output` and
doesn't address ordering.)
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…sourcePane tests (microsoft#21480)
## Description
Fixes a flaky race in two Dashboard Playwright outerloop tests added or
changed by microsoft#21412:
-
`DesktopNavMenuTests.MainNavigation_NavigatesBetweenSectionsAndMarksActiveItem`
-
`ResourcePaneInteractionTests.AllResourcesRow_ClearsSelectionAndListsResourcesInOverview`
### Root cause
A resource tab click changes the URL in the browser right away, before
the server renders the destination. Until that render batch reaches the
browser, the tab strip on the page is still the one rendered for the
previous location. The `FluentTabs` in `ResourcesLayout` is `@key`ed by
`renderedLocation`, and `OnTabChanged` deliberately ignores change
events from a strip rendered for an earlier location. The comment in
`ResourcesLayout.razor` explains that this guard stops queued native
change events from a torn-down strip from looking like new user
navigations.
Both tests click a second tab as soon as the URL or `aria-selected`
changes. Both of those update on the client, so neither proves the
server has rendered the new strip. When the second click lands in that
window, it is dropped by design and the test times out waiting for a
navigation that never happens.
Server-side tracing (temporary logging, not in this PR) confirmed the
sequence locally:
1. Click Console logs; `OnParametersSetAsync` runs for `/consolelogs`
(t=.956).
2. The test clicks Structured logs at about .96–.99 on the strip still
keyed to `/resources`.
3. The new strip is rendered at 45.019. The queued event then arrives
with `renderedLocation=/resources` while `_lastLocation=/consolelogs`,
and is ignored.
### Fix (test-only)
Add `PlaywrightTestsBase.ClickResourceTabAndWaitForRenderAsync`. It
takes a handle to the current `.resources-tab-header` element, clicks,
and waits until that element is detached. Because the strip is keyed by
location, detachment means the server's render for the new location has
been applied. The tests then make the same assertions as before. No
sleeps or retries are added and no assertion is weakened.
`AssertActiveItemAsync` now checks the server-rendered active tab rather
than only the client's optimistic selection.
The product guard is intentional, so it is unchanged.
### Evidence
**CI (outerloop, `outerloop_tests / Dashboard.Playwright`; these tests
only run in `tests-outerloop.yml`):**
- Run
[41050874298](https://github.com/microsoft/aspire/actions/runs/41050874298)
is the only outerloop run with Playwright jobs that includes microsoft#21412. On
Ubuntu, both tests failed:
- `MainNavigation`: `Assert.NotNull() Failure` for the next tab's
bounding box.
- `AllResourcesRow`: `Page URL expected to match regex '/resources$' But
was: '.../consolelogs'`.
- On Windows, both passed in the same run.
- Denominator so far: Ubuntu 1/1, Windows 0/1 runs that executed the
tests. `DesktopNavMenuTests` failed 0/28 jobs in the 14 scheduled main
runs from Oct 16–30, before microsoft#21412. `ResourcePaneInteractionTests`
didn't exist yet.
**Local (Linux, 2 cores):**
| Scenario | Before | After |
|---|---|---|
| `MainNavigation`, idle | 8/8 failed | 10/10 passed |
| `MainNavigation`, CPU stress (2× `yes`) | 10/10 failed | 10/10 passed
|
| `AllResourcesRow`, idle and stress | 18/18 passed (window too narrow
locally) | 20/20 passed |
| `AllResourcesRow` with deterministic repro harness ¹ | 5/5 failed with
the exact CI message (`But was: .../consolelogs`) | 5/5 passed |
| Full `Aspire.Dashboard.Playwright.Tests` | n/a | 126/127 ² |
¹ Repro harness (not committed): an init script delays every
server-to-browser Blazor WebSocket message by 300 ms, in order, after
the Console logs click. This widens the window between the client URL
change and the server render:
```csharp
await page.AddInitScriptAsync("""
(() => {
const desc = Object.getOwnPropertyDescriptor(WebSocket.prototype, 'onmessage');
Object.defineProperty(WebSocket.prototype, 'onmessage', {
configurable: true,
get() { return desc.get.call(this); },
set(handler) {
let chain = Promise.resolve();
desc.set.call(this, handler && (e => {
const due = Date.now() + (window.__delayBlazor ? 300 : 0);
chain = chain.then(() => new Promise(r => setTimeout(r, Math.max(0, due - Date.now())))).then(() => handler.call(this, e));
}));
}
});
})();
""");
// ... then before the Console logs click:
await page.EvaluateAsync("() => window.__delayBlazor = true");
```
² The only failure is
`TerminalTests.HeaderMetadata_LiveUpdatesAndFallbackPreserveFocusAndCopyCurrentValues`.
It's a local culture artifact (`'42 %'` vs `'42%'`) in a test that
doesn't use the new helper and isn't touched by this PR.
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…venv timeout) (microsoft#21539)
## Description
Fixesmicrosoft#19542
Also fixes a latent hang in the helper's timeout path.
`AddPythonAppTests.PythonResourceSupportsWithReference` failed once on
`windows-latest` (run
[32375588350](https://github.com/microsoft/aspire/actions/runs/32375588350),
job 96448944641) with `Failed to create virtual environment.` The test
took 2m 06s, and the venv process wrote no stdout or stderr.
### Root cause
- **Unnecessary pip bootstrap.** Every `AddPythonAppTests` test that
uses `CreateTempPythonProject` runs a full `python -m venv .venv` and
then `pip install`, under a fixed 2-minute deadline. Five of those tests
only build the app model, and so does the failing one. They need `.venv`
to exist because `AddPythonApp` adds a `-venv-creator` resource when
it's missing. They never run pip. A full `venv` runs `ensurepip`, which
starts a pip subprocess that installs and byte-compiles about 1,000
files. Without pip, venv creation copies about 5 files. On a cold
Windows runner (Defender, `MoUsoCoreWorker` and svchost were busy in the
heartbeat), this step exceeded 2 minutes. The heartbeat at 14:00:28, 75s
into the wait, shows no `python` process among the top CPU consumers, so
the process was blocked on I/O or the OS rather than computing.
- **Timeout path waits for the process anyway.** On timeout, the helper
read stdout and stderr to EOF *before* calling `Kill`. A process that
truly hung would therefore block the test until the 20-minute MTP hang
timeout instead of failing at 2 minutes. That's why the failure reported
2m 06s: the read waited for python to exit.
### Fix (test-only)
- `CreateTempPythonProject(..., withPip: false)` is the default for
model-only tests. It creates the venv with `python -m venv --without-pip
.venv` and skips `pip install`. The venv interpreter exists, so the
resource model (command path, no venv-creator) is unchanged.
- Tests that actually run pip still bootstrap it.
`PythonResourceFinishesSuccessfully` starts the app, and its installer
resource runs `.venv` pip. The instrumented test installs requirements.
- Both process waits now kill on timeout before draining output.
No assertions were changed, and no retries or sleeps were added.
### Evidence
**Regression test:**
`CreateTempPythonProject_ForModelOnlyTests_CreatesVirtualEnvironmentWithoutBootstrappingPip`
asserts that the model-only venv has an interpreter and no pip.
- Before, with main's `-m venv .venv`: **fails**, with `Model-only test
projects should not bootstrap pip, but found
'/tmp/bBYQbR/.venv/bin/pip'`.
- After: **passes**.
**Per-test duration (TRX, Linux, Python 3.12), main → this PR:**
| Test | main | PR |
|---|---|---|
| PythonResourceSupportsWithReference *(the flaky test)* | 2.51s |
**0.053s** |
| AddPythonAppProducesDockerfileResourceInManifest | 2.53s | 0.037s |
| AddPythonApp_SetsResourcePropertiesCorrectly | 2.65s | 0.039s |
| AddPythonApp_ObsoleteMethod_StillWorks | 2.50s | 0.039s |
| AddPythonAppWithScriptArgs_IncludesTheArguments | 2.50s | 0.044s |
| PythonResourceFinishesSuccessfully (still uses pip; starts the app) |
6.40s | 9.75s (passes) |
| AddInstrumentedPythonProjectProducesDockerfileResourceInManifest
(still uses pip) | 9.12s | 5.88s |
Measured separately: `python -m venv .venv` writes 1,010 files in 4.27s,
while `--without-pip` writes 5 files in 0.05s. Model-only tests no
longer start an `ensurepip`/pip subprocess, which was the step that
exceeded the deadline.
**Recurrence:** this is the only recorded occurrence. A scan of the last
300 `ci.yml` runs (2026-10-30 22:08 to 2026-10-31 06:15 UTC; the CI test
selector skips Hosting.Python on most runs) found 0 failures in 20
`Hosting.Python (windows-latest)` jobs. On main, the Windows suite
usually takes about 75s (for example, job 124463643298: 1m 15s). The
failure is rare and depends on runner conditions, so the evidence above
is the removed exposure rather than a local Windows repro.
### Validation
- `Aspire.Hosting.Python.Tests` (full project, functional
`PythonResourceFinishesSuccessfully` included, which starts the app via
DCP): 119 passed, 0 skipped, 3 failed. The 3 failures are
`WithUv_GeneratesDockerfileInPublishMode*` Verify snapshots. They fail
identically on main in this environment because the run is as root and
Verify scrubs `/root/` to `{UserProfile}`. They're unrelated to this
change, and CI is the authority.
**Windows CI on this PR** (job
[124468350194](https://github.com/microsoft/aspire/actions/runs/41177499661/job/124468350194)):
`Hosting.Python (windows-latest)` reported 122 succeeded, 0 failed, in
**31.6s**. On main this suite takes about 75s, so most of the Windows
suite time was venv/pip bootstrap that model-only tests didn't need.
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- Did you add public API?
- [x] No
- Does the change make any security assumptions or guarantees?
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#21543)
Fixesmicrosoft#19637
## Root cause
Two Infrastructure.Tests classes build `tools/Aspire.TestTools` out of
process, and they used to run in parallel:
| Test class | Build it triggers |
|---|---|
| `GenerateTestSummaryToolTests` | Its class fixture runs `dotnet build
tools/GenerateTestSummary/GenerateTestSummary.csproj --restore`, which
has a ProjectReference to Aspire.TestTools. |
| `DownloadFailingJobLogsToolTests` | Each test runs `dotnet run
tools/scripts/DownloadFailingJobLogs.cs`, a file-based app with
`#:project ../Aspire.TestTools/Aspire.TestTools.csproj`. |
The two classes were in separate xunit collections, so their builds
could overlap. Both builds write the same outputs under
`artifacts/bin/Aspire.TestTools/Debug/net10.0/`. When
`Aspire.TestTools.deps.json` needs regenerating, both MSBuild processes
open it for exclusive write. The loser fails:
```
error MSB4018: The "GenerateDepsFile" task failed unexpectedly.
System.IO.IOException: The process cannot access the file '.../artifacts/bin/Aspire.TestTools/Debug/net10.0/Aspire.TestTools.deps.json' because it is being used by another process.
```
That is the failure in CI job
[97605592624](https://github.com/microsoft/aspire/actions/runs/32781735084/job/97605592624),
which made DownloadFailingJobLogsToolTests' `dotnet run` exit 1.
The other tool fixtures (`CreateFailingTestIssue`,
`ExtractTestPartitions`) don't build Aspire.TestTools as a project.
CreateFailingTestIssue compiles the shared sources into itself.
`GenerateTestSummaryToolTests` invokes the tool with `--no-build`.
## Fix
Put both classes in one new xunit collection, `AspireTestToolsBuild`.
Xunit runs classes in the same collection sequentially, so the fixture
build and the `dotnet run` builds can never overlap. There are no
retries or sleeps, and test behavior is unchanged.
## Evidence
1. **Exact failure reproduced deterministically.**
- **Setup:** a helper process held `Aspire.TestTools.deps.json` open
with `FileShare.None`, as a concurrent GenerateDepsFile writer does. The
file was stale, so MSBuild had to regenerate it.
- **Result:** the fixture's `dotnet build
tools/GenerateTestSummary/GenerateTestSummary.csproj --restore` failed
with the CI error: `error MSB4018: The "GenerateDepsFile" task failed
unexpectedly. System.IO.IOException: The process cannot access the file
'.../Aspire.TestTools.deps.json' because it is being used by another
process.`
- Once the holder released the file, the same build succeeded.
2. **Overlap measured in the real test run.** I ran both classes with
the real test runner (`--filter-class *GenerateTestSummaryToolTests
--filter-class *DownloadFailingJobLogsToolTests`). Before each run, I
forced a TestTools rebuild by touching a source file and deleting
deps.json. A sampler checked `ps` every 100 ms for the fixture's `dotnet
build` and a `dotnet run DownloadFailingJobLogs.cs` alive at the same
time.
| | Concurrent Aspire.TestTools builds observed | Test failures |
|---|---|---|
| Before (main) | **5/5 runs** | 0/5 |
| After (this PR) | **0/8 runs** (5 + 3 after review update) | 0/8 |
The IOException itself is a narrow window (the deps.json write takes
milliseconds). Locally I forced 40 concurrent build/run pairs and none
hit it, which fits the rare CI rate. The overlap that makes it possible
was present on every run before the fix and is gone after.
3. **No metadata-only guard.** Following
`.github/instructions/infrastructure-tests.instructions.md`, this PR
adds no test that asserts collection attributes. Such a test would
restate the source, not observe behavior. Behavioral coverage of the
race isn't practical in a unit test: it needs two real MSBuild processes
and a stale deps.json, and the conflicting write window is milliseconds.
The real-runner evidence above is the verification. Both affected
classes pass after the change (10/10 tests on every run).
**🤖 Posted by AspireAgent (AI Agent)**
---------
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… test-server failures (microsoft#21497)
## Description
Fixes the intermittent
`IntegrationHostProcessTests.StartupExit_FailsPromptlyAndPreservesRuntimeDiagnostics`
timeout in `Aspire.Hosting.RemoteHost.Tests`. No issue tracks it.
### Flake evidence (first attempts, ci.yml, Oct 16–30)
- **PR runs:** 1,086 total, 503 of which were rerun. Attempt-1
failed-test names were parsed from 2,245 failed jobs.
- This test failed attempt 1 on **4 / 503** rerun runs, each on a
different, unrelated branch.
- In each case the same job on the same head passed on rerun (confirmed
via check-runs for 3 of the 4 heads).
- **main:** 0 / 128 runs.
| Date | Branch | OS | Job | createdump time |
|---|---|---|---|---|
| Oct 19 | codex/persistent-dashboard | ubuntu | 118356484862 | 10.7s |
| Oct 20 | darc-main-bddce1ed… | ubuntu | 118451125990 | 9.7s |
| Oct 21 | davidfowl/lean-native-runtime-feasibility | windows |
119032099641 | 16.3s |
| Oct 28 | dependabot/github_actions/github-actions-ebbe822166 | windows
| 122964740478 | 16.4s |
### Root cause
The fixture's generated test server
(`IntegrationHostProcessFixture.HostProgram`) lets expected server
failures escape `Program.Main` as **unhandled exceptions**. This
includes startup failure, missing command, startup deadline, crash-loop
and changed signature, which come from `host.RunAsync()` or
`IntegrationHostLauncher.ThrowIfFailed()`.
On CI the test platform runs with `--crashdump`. The
`DOTNET_DbgEnableMiniDump*` settings are inherited by the child test
server, so every expected failure crashes the runtime and writes a full
dump before the process exits. Each failing log shows the sequence:
```
Unhandled exception. System.InvalidOperationException: Integration host supervisor for 'lifetime-test' ... exited with code 23.
[createdump] Dump successfully written in 16311ms
failed ...StartupExit_FailsPromptlyAndPreservesRuntimeDiagnostics (16s 896ms)
System.TimeoutException : The operation has timed out.
```
`StartupExit` waits only `server.Exit.WaitAsync(15s)`, so whenever dump
writing took about 10–16s the test timed out. The other failure-path
tests use 30s deadlines but pay the same cost, and every one of them
leaves a 250–300MB dump.
### Fix (test infrastructure only)
- The generated host now catches the failure, writes it to stderr
(unwrapping `TargetInvocationException` from the reflected
`ThrowIfFailed`), and sets the dedicated `Environment.ExitCode = 87`.
All diagnostics the tests assert on are kept, including `exited with
code 23` and the recovery messages.
- The failure-path tests (`StartupExit`, `MissingCommand`,
`StartupDeadline` ×2, `CrashLoop`, `ChangedSignature`) now assert
`Assert.Equal(HostFailureExitCode, server.ExitCode)` instead of
`NotEqual(0)`, or in addition to the existing checks for `StartupExit`.
This assertion is stricter and is the regression check: it fails while
the server crashes. On Linux a crashed server exits with 134 (SIGABRT).
No product code changes. No timeouts were changed, and no sleeps or
retries were added.
### Before/after (local, Linux)
Run with `--filter-class *IntegrationHostProcessTests` and crash dumps
enabled (`DOTNET_DbgEnableMiniDump=1`), as on CI:
| | Result | Crash dumps written |
|---|---|---|
| Before (old host program, new assertions) | **6 failed** / 9 passed.
Every failure-path test: `Expected: 87, Actual: 134` | **7** |
| After | **15 / 15 passed** | **0** |
The full `Aspire.Hosting.RemoteHost.Tests` suite passes after the fix:
686 passed, 0 failed, 5 skipped.
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…twork errors (microsoft#21661)
## Description
GitHub artifact service transport failures in `actions/upload-artifact`
/ `actions/download-artifact` steps fail entire CI runs, and the
auto-rerun workflow never retries them.
**Evidence.** The 7-day ci.yml attempt-1 census (2026-10-25 13:15Z →
2026-11-01 11:50Z) covers 840 runs: 80 main and 759 PR. In **7 of those
runs**, the only failed steps were artifact transfer steps, and each
failed with a twirp `NetworkError`. That breaks down to 5 PR runs on 5
unrelated branches (out of 759) and 2 main runs (out of 80).
| Job | Run / event | Failed step | Annotation | Lane denominator
(decided, 7d) |
|---|---|---|---|---|
|
[122578129031](https://github.com/microsoft/aspire/actions/runs/40609188589/job/122578129031)
| push main | `Upload RID-specific packages`, `Upload logs` | `Failed to
CreateArtifact: Unable to make request: ETIMEDOUT` | Build CLI
(osx-x64): 63 main / 453 PR |
|
[124366008186](https://github.com/microsoft/aspire/actions/runs/41144368419/job/124366008186)
| PR | `Upload RID-specific packages` | `Failed to CreateArtifact:
Unable to make request: ENOTFOUND` | same lane |
|
[124324043945](https://github.com/microsoft/aspire/actions/runs/41127599426/job/124324043945)
| PR | `Upload starter validation diagnostics` | `Failed to
CreateArtifact: Unable to make request: ETIMEDOUT` | CLI Starter
Validation (macOS x64): 259 PR |
|
[124151653146](https://github.com/microsoft/aspire/actions/runs/41076051061/job/124151653146)
| PR | `Upload starter validation diagnostics` | `Failed to
CreateArtifact: Unable to make request: ENOTFOUND` | same lane |
|
[124557485092](https://github.com/microsoft/aspire/actions/runs/41208154053/job/124557485092)
| PR | `Upload CLI npm packages` | `Failed to FinalizeArtifact: Unable
to make request: ECONNRESET` | Build CLI (win-x64): 63 main / 453 PR |
|
[122562573241](https://github.com/microsoft/aspire/actions/runs/40608651539/job/122562573241)
| PR | `Download RID-specific NuGet packages` | `Unable to download
artifact(s): Failed to ListArtifacts: Unable to make request: ETIMEDOUT`
| VS Code E2E (Linux, browser-debugger): 61 main / 302 PR |
|
[123248062186](https://github.com/microsoft/aspire/actions/runs/40810912222/job/123248062186)
| push main | `Download VSIX` | `Unable to download artifact(s): Failed
to ListArtifacts: Unable to make request: ETIMEDOUT` | VS Code E2E
(Windows, diagnostic-logs): 61 main / 301 PR |
**Root cause.** `classifyFailedJob` inspects annotations only when a
failed step matches `retryableWithAnnotationStepPatterns` or
`ignoredFailureStepPatterns`. None of these artifact step names match
either list, so `main`'s classifier returns `retryable: false` for all 7
jobs, with the reason *"Failed step '…' is not covered by the retry-safe
rerun rules."* I reproduced this by running `origin/main`'s classifier
against the real job and check-run annotation payloads fetched from the
API. ETIMEDOUT also isn't in `transientAnnotationPatterns`.
`@actions/artifact`'s twirp client throws `Failed to ${method}: Unable
to make request: ${code}` on the first network error and doesn't retry.
**Fix.** A narrow rule in the annotation pass. A job is retryable when
**every** failed step is an `Upload*` or `Download*` step **and** the
annotations match `Failed to
(CreateArtifact|FinalizeArtifact|GetSignedArtifactURL|ListArtifacts):
Unable to make request: (ETIMEDOUT|ENOTFOUND|ECONNRESET)`. These are the
RPC names in `@actions/artifact`'s generated twirp client.
- Test execution steps are never rerun or skipped by this rule. A job
with any non-artifact failed step, such as `Run tests` or a build step,
keeps the existing veto.
- Other upload failures (for example "No files were found") are still
skipped.
- The existing safety rails are unchanged: PR-only, attempt ≤ 3, and at
most 5 retryable jobs.
With this change, the same classifier marks all 7 real payloads as
retryable.
**Tests.** I added 8 cases to `AutoRerunTransientCiFailuresTests`: 5
theory rows using the real step names and annotation strings, a
multi-upload-step case, a non-network upload failure that stays skipped,
and a mixed build+upload failure that stays skipped.
- Before the fix (classifier from `origin/main`): **7 failed**, 97
passed.
- After the fix: **104/104 passed**.
`docs/ci/auto-rerun-transient-ci-failures.md` is updated to describe the
rule.
Fixesmicrosoft#19876 (artifact ETIMEDOUT). Related: microsoft#19252 (artifact ENOTFOUND).
Tracker: microsoft#21600
([claim](microsoft#21600 (comment))).
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
- Does the change require an update in our Aspire docs?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…#21434)
## Description
Fixes a timing-dependent failure in
`ManageDataDialogTests.Render_ClearedSignals_PrunesSelectionsAndSupportsRemovingEmptyResource`
(microsoft#20759).
**Root cause.** `SetupIconCheckboxJs()` registered the
`IconCheckbox.razor.js` handlers with `SetupVoid(...)` but never
configured a result. In bUnit 2.x, an unconfigured invocation stays
pending and is faulted with `JSRuntimeInvocationNotSetException` once
`BunitContext.DefaultWaitTimeout` (10s in `DashboardTestContext`)
elapses. Each `IconCheckbox` awaits `initializeIconCheckboxKeyboard` in
`OnAfterRenderAsync`, so the fault becomes an unhandled renderer
exception. The next `WaitForAssertion` that has to wait then rethrows
it. Any ManageDataDialog test still running 10s after its first checkbox
rendered therefore fails. That happens on a slow runner: the failing CI
run took **20.9s** ([job
112149721345](https://github.com/microsoft/aspire/actions/runs/37427038688/job/112149721345)),
and its stack goes through `IconCheckbox.OnAfterRenderAsync` →
`WaitForAssertion`. Locally the whole class takes about 4s, which is why
it usually passes.
**Fix.** Call `.SetVoidResult()` on both IconCheckbox handlers, matching
every other module setup in this project (for example
`TerminalSetupHelpers`). This only changes test setup; no assertions
changed.
**Regression test.**
`IconCheckbox_CompletesKeyboardInitializationAndCleansUpOnDispose`
renders an `IconCheckbox`, disposes it, and asserts
`disposeIconCheckboxKeyboard` was invoked. `IconCheckbox` only runs
cleanup after initialization completed, so the test fails immediately
without the fix and needs no waiting.
**Validation (Linux, native):**
| | Without fix | With fix |
|---|---|---|
| New regression test | fails in 7ms (`Assert.Contains() Failure: Filter
not matched`) | passes |
| `Render_ClearedSignals_*` (both cases) with a temporary, uncommitted
11s pause after the first render to simulate a slow runner | 2/2 fail
with `JSRuntimeInvocationNotSetException ...
initializeIconCheckboxKeyboard`, matching CI | 2/2 pass |
| Full `Aspire.Dashboard.Components.Tests` suite | — | 1162/1162 pass |
Fixesmicrosoft#20759
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- If yes, did you have an API Review for it?
- [ ] Yes
- [ ] No
- Did you add `<remarks />` and `<code />` elements on your triple slash
comments?
- [ ] Yes
- [ ] No
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- If yes, have you done a threat model and had a security review?
- [ ] Yes
- [ ] No
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#21634)
## Description
Fixes the npm fresh-release E404 part of the `Polyglot SDK Validation`
TypeScript flake tracked in microsoft#21600.
### Root cause
TypeScript AppHost scaffolds and fixtures declare floating npm ranges.
The TypeScript validation container runs `npm install` without a
release-age limit. If a run installs a few minutes after an upstream
publish, npm picks the new version from the packument while its tarball
still returns 404. `aspire init --language typescript` then fails with
`npm error code E404` / `❌ The app could not be created.`
In the last 7 days this caused 5 failed jobs across 3 unrelated branches
(details in microsoft#21633). Each failure came 2–4 minutes after the upstream
publish of `@typescript-eslint/typescript-estree@8.72.1` or
`@types/estree@1.0.10`. These are the same publishes microsoft#21615 fixes for
the CLI E2E Docker shells, but that PR doesn't cover this container.
### Fix
The `Run TypeScript SDK validation` step now passes
`npm_config_before=<now − 1 day>` into the container ([npm `before`
config](https://docs.npmjs.com/cli/v11/using-npm/config#before)). This
matches microsoft#21615. npm range resolution in `aspire init`, the fixture
installs, the JavaScript app installer and the Nuxt fixture only selects
releases that have had time to propagate. The global tool installs at
image build time are unaffected. pnpm, Yarn and Bun don't read this
setting. No retries, sleeps or skips are added.
### Validation
- **Mock registry with npm 11.19.0 (the image's npm):** the packument
lists `1.1.0` (published 2 minutes ago, tarball 404) and `1.0.0` (30
days old).
- Without the cutoff, `npm install pkg@^1.0.0` fails with `npm error
code E404`, the CI signature.
- With the cutoff, it installs `1.0.0`.
- **Real registry with npm 11.19.0 and the cutoff:**
- The TypeScript scaffold's dependency set installs.
- `playground/NuxtApp/web` installs, with 596 packages including
`nuxt@4.5.2`.
- All 58 npm-managed `tests/PolyglotAppHosts/*/TypeScript` and
integration playground `package.json` files resolve. No ETARGET.
- The pnpm fixture resolves.
- **`actionlint`:** passes on `polyglot-validation.yml`.
Other intermittent TypeScript validation failures from the same window
have different causes and are not addressed here: the Nuxt dev-mode
health wait timing out (3), `bun install` `ConnectionRefused` (2), and
an MCR pull failure (1).
Fixesmicrosoft#21633
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [ ] Yes
- [x] No (CI workflow configuration only; validated as described above)
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lation and process exit) (microsoft#21397)
## Description
> **Update after merging `main` (2026-10-30):** microsoft#21360 has since landed
on `main` with its own fix that gives this test a separate `stopCts` for
`StopAsync` and moves it to `LongTimeoutTimeSpan` budgets. The
cancellation-ownership repair this PR originally made is therefore
already on `main`. After resolving the merge conflict, this PR adds
**regression coverage only** on top of `main`'s version of the test:
>
> - The test becomes a `[Theory]` that also covers an explicitly
cancelled startup token after startup has completed
(`cancelStartupAfterStart: true`).
> - The test awaits its actual `StartAsync` task instead of discarding
it.
> - It asserts that the ServiceB root process is running before shutdown
and has exited afterwards, and that `ApplicationStopped` fires.
>
> The timeouts and token ownership come unchanged from `main`. The
before/after evidence below was gathered against the original pre-microsoft#21360
base. It shows why phase-owned shutdown is needed, but it is no longer
this PR's diff. Both theory cases pass locally on the merged head.
Original description (pre-merge): give
`BeforeResourceStartedEvent_NotFiredForExplicitStartOnInitialCreation`
separate startup and shutdown cancellation budgets. A slow startup
previously consumed most of the same 30-second CI token subsequently
passed to `StopAsync`, causing cleanup cancellation even when shutdown
itself completed within the standard bound.
Related: microsoft#21026, microsoft#21328. These issues are intentionally left open for
continued observation.
The test now observes its actual startup task, creates the shutdown
token only when entering shutdown, and runs both natural and explicitly
cancelled completed-startup cases. All existing resource/event
assertions, waits, ports, and timeout values are preserved. Both cases
verify actual host shutdown and ServiceB root-process exit. No product,
shared fixture, workflow, dependency, or policy changes are included.
### Native before/after evidence
| Run | Actual selected execution evidence |
|---|---|
| [Unmodified
baseline](https://github.com/microsoft/aspire/actions/runs/40903532956)
| 8 passed / 1 failed. Failure was cancellation in the original line-365
`StopAsync` after approximately 21 seconds of startup consumed the
shared 30-second budget. |
| [Paired causal
experiment](https://github.com/microsoft/aspire/actions/runs/40908555842)
| Shared-token/cancelled-completed-startup controls failed 9/9;
phase-owned shutdown with the same startup cancellation passed 9/9, with
actual shutdown/root exit within the unchanged bound. Diagnostic
controls and instrumentation are not in this PR. |
| [Clean repair
qualification](https://github.com/microsoft/aspire/actions/runs/40913395606)
| **60 actual TRX executions passed, 0 failed, 0 skipped** on Windows
runners 2 and 3, each completing all three fixed iterations. Includes
both repaired rows six times each, 12 adjacent event cases, and 36
broader lifecycle/cancellation/explicit-start cases. |
The clean repaired source at `d28c36fe8091ae1308fbf5691e36b71c3b51b0e2`
is byte-identical to the test source executed at validation head
`7c6ad6b7d8d99b06eea1224f853ea45f4ad5f822`. Only the separate validation
branch contains workflow selection/TRX-capture configuration.
**Qualification gap:** 90 executions were planned. Runner 1's build
succeeded, but its check reports “The hosted runner lost communication
with the server.” It retained neither logs nor TRX; its 30 planned
outcomes are unknown, not passes or demonstrated test failures. That
qualification workflow remains failed; subsequent normal PR CI does not
fill those missing outcomes. Local native execution was unavailable
after the prescribed restore failed with NU1900/NU1905 resolving
configured NuGet audit metadata; the supported hosted Windows path was
used without policy overrides.
### Normal PR CI status (2026-10-29 18:28 UTC)
[Normal
CI](https://github.com/microsoft/aspire/actions/runs/40922625873)
executed both repaired theory rows and both adjacent event facts
successfully on Windows and Ubuntu. Hosting-1 completed **28 passed / 0
failed / 0 skipped** on Windows and **63 passed / 0 failed / 0 skipped**
on Ubuntu.
Attempt 1 failed in the unchanged Ubuntu
`AzureFunctionsTests.AddAzureFunctionsProject_WorksWithMultipleProjects`
at its line-201 parameterless `host.StopAsync()`
(`TaskCanceledException`, 32.300-second test; suite 2,109 passed / 1
failed / 5 skipped). That separate test does not use this PR's repaired
method or startup token; its root cause remains unproven.
The repository's automatic CI rerun started attempt 2 without a source
change. Its Azure suite passed **2,110 / 0 failed / 5 skipped**,
including that method in 28.566 seconds. The macOS ARM64 build also
succeeded. Latest applicable CI is now green on
`d28c36fe8091ae1308fbf5691e36b71c3b51b0e2` (46 successful checks, 29
skipped); skipped checks are not passing test executions. The first
Azure failure and the separate qualification gap above remain part of
the evidence. Copilot review identified no active issues. This is ready
for human review, not approved for merge.
### Scope and limits
This is not a revival of microsoft#20406's broad
shared-fixture/shutdown-timeout/DCP-log-forwarding changes. That PR was
closed after an out-of-order DCP API-server update bug was identified.
Here the actual selected host path separately demonstrates that passing
an already-cancelled completed-startup token to shutdown causes
cancellation; giving shutdown its own unchanged budget completes host
shutdown and root exit.
The distinct original line-353 resource-startup failure associated with
Docker/WSLc discovery stalls is **not fixed or claimed fixed**.
Root-process exit does not prove that every descendant process exited.
Created using the repository's [fix-flaky-test
skill](https://github.com/microsoft/aspire/blob/main/.agents/skills/fix-flaky-test/SKILL.md),
with existing before/after native evidence and deterministic
cancellation coverage rather than arbitrary delays or larger timeouts.
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- If yes, did you have an API Review for it?
- [ ] Yes
- [ ] No
- Did you add `<remarks />` and `<code />` elements on your triple slash
comments?
- [ ] Yes
- [ ] No
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- If yes, have you done a threat model and had a security review?
- [ ] Yes
- [ ] No
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…#21754)
## Description
Show templates from configured Git and directory sources alongside
built-in templates in interactive `aspire new`, without a separate
source submenu. This builds on microsoft#21329, microsoft#21388, and microsoft#21420; it does not
close those already-merged issues/PRs.
The terminal picker uses nonselectable source headings, retains built-in
ordering and defaults, and selects the exact discovered template object
even when names collide. Leaves use `Template Name (Template
Description)` without repeating the source or short name; missing
descriptions leave just the name. The separate search action has been
removed from the initial picker; explicit template search/invocation
support is unchanged.
Template source objects support optional `displayName` in global and
accepted init-source configuration, for example:
```json
{
"templateSources": {
"company": {
"repository": "https://github.com/example/templates",
"displayName": "Company templates"
}
}
}
```
Missing, null, empty, or space-only display names use the alias. Other
types and control characters fail existing strict configuration
validation. `aspire config set templateSources.company.displayName
"Company templates" --global` edits an existing source; no new flag is
added. Typed load/save, source listing, configuration edits and runtime
inheritance preserve the field. Global-over-inherited alias collisions
replace the entire definition, including the display name. Inherited
values are never persisted to global or generated-project configuration.
Aliases remain the stable group identity, command reference and
source-disambiguating flat label; identical display names do not merge
sources or select a different source. Headings now use bold/default
foreground, following existing CLI section headings such as `aspire
doctor`, rather than Spectre's faded disabled grey. Leaf-only navigation
is unchanged.
Grouped console labels normalize multiline whitespace and truncate the
combined name/description with an ellipsis to the available
terminal-cell width, reserving arrow, indentation and padding. Source
headings and the selected-value echo also fit the width. Spectre's
cell-aware cropping preserves grapheme clusters, wide characters and
combining sequences; metadata is safely escaped. Full template metadata
is retained.
The pinned Spectre.Console 0.57.2 uses the same converter for display
and search and has no separate search predicate. Following the explicit
UX decision to keep Spectre and truncate in the CLI, case-insensitive
search matches only the visible prefix of a truncated label, not its
hidden tail. No dependency update or custom prompt is introduced.
Type-to-search moves to a matching choice; it does not filter the list.
Grouping and terminal display labels are optional interaction-service
metadata. VS Code retains the unchanged flat `string[]` RPC and complete
unique source-qualified labels; extension grouping and description
search remain deferred. Existing inherited-source consent/no-fetch
boundaries, explicit and noninteractive flows, source acquisition
warnings, cached snapshots, authentication protections, and post-action
authorization remain intact.
Validation for the latest friendly-heading revision:
```bash
dotnet test --project tests/Aspire.Cli.Tests/Aspire.Cli.Tests.csproj --no-launch-profile -- --filter-class '*.ConsoleInteractionServiceTests' --filter-class '*.ExtensionInteractionServiceTests' --filter-class '*.NewCommandLocalTemplateTests' --filter-class '*.TemplateSourceConfigurationTests' --filter-class '*.AspireConfigFileTests' --filter-class '*.ConfigSourceServiceTests' --filter-class '*.SettingsSchemaBuilderTests' --filter-class '*.ConfigCommandTests' --filter-not-trait 'quarantined=true' --filter-not-trait 'outerloop=true'
```
Result: **1048 passed, 0 failed, 1 skipped**. Covers typed serialization
and schema, strict field validation, source updates/listing, inherited
display names without persistence, complete global alias precedence,
duplicate friendly headings and exact template generation,
consent/no-fetch, unchanged extension fallback, terminal cell-width
thresholds and narrow terminals, Unicode/combining sequences, escaped
markup, normalized multiline labels and visible-prefix-only search.
Rendering and configuration snapshots were reviewed and the suite rerun
successfully.
```bash
dotnet build /t:UpdateXlf src/Aspire.Cli/Aspire.Cli.csproj -v:q
git diff --check
```
Result: **both passed**. Localized display-name validation and
allowed-field diagnostics regenerated.
A refreshed screenshot of the actual repository-built CLI, with friendly
source names, bold/default headings and a long description cropped with
an ellipsis, is provided in the associated Copilot handoff, not uploaded
GitHub media. Earlier screenshots and the description-search recording
predate these latest formatting changes. No cloud resources or external
account changes were used.
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- If yes, did you have an API Review for it?
- [ ] Yes
- [ ] No
- Did you add `<remarks />` and `<code />` elements on your triple slash
comments?
- [ ] Yes
- [ ] No
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- If yes, have you done a threat model and had a security review?
- [ ] Yes
- [ ] No
- [x] No
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…iliation tests (microsoft#21534)
## Description
`AzureSqlPrincipalReconciliationTests` failed as a class (all 7 tests)
in `Hosting.Azure (ubuntu-latest)` because
`SqlServerContainerFixture.InitializeAsync` could not start SQL Server:
```
failed to set up container networking: driver failed programming external connectivity on endpoint ...: failed to bind host port for 0.0.0.0::172.17.0.3:1433/tcp: address already in use
```
The empty host port in `0.0.0.0::…:1433` shows Docker was picking a
**random** host port, so host port 1433 wasn't involved. Instead, the
daemon's random allocation collided with ports already in use.
### Root cause
- The CI ubuntu runners use **Docker Engine 28.0.4**. I checked the
failing jobs from 2026-09-17 and a current main run from 2026-10-31.
- In Docker before 28.5, random host ports are tracked per host IP. The
engine hands out candidates sequentially from the ephemeral range and
gives up after 10 bind attempts (`maxAllocatePortAttempts`).
- The fixture uses the Testcontainers default: a wildcard (`0.0.0.0`)
random port.
- Containers that DCP starts for the other container tests in this
assembly publish on loopback. That includes the Storage, Cosmos, Service
Bus, Event Hubs, SignalR, WebPubSub and Functions emulator tests, which
run in parallel.
- Ten such loopback publications hold `127.0.0.1:32768…32777`. The
wildcard pool doesn't know about them, so it offers 32768…32777, each
`bind(0.0.0.0:p)` fails with `EADDRINUSE`, and the container start
fails. The upstream report is moby/moby#54079.
### Fix
The fixture now publishes its random port on `127.0.0.1`, using
`WithCreateParameterModifier`, so it shares the allocation pool with
DCP's loopback publications. Docker then skips ports that are already
taken instead of colliding with them.
The loopback binding is only applied when the daemon shares the test
process's loopback, which is what Testcontainers' `Hostname` resolves
to. That means a local `unix`/`npipe` endpoint (not running inside a
container) or a loopback TCP endpoint, with no
`TESTCONTAINERS_HOST_OVERRIDE`. Remote `DOCKER_HOST` and in-container
runs keep the wildcard binding so the port stays reachable.
There are no retries, sleeps or weakened assertions. The seven
`[ActiveIssue(microsoft#19014)]` attributes added in microsoft#20211 are removed, because
the cause is now fixed.
### Regression test
The new `SqlServerContainerPublishesItsPortOnLoopbackWhenReachable` test
inspects the running fixture container through the Docker API. It
asserts that `1433/tcp` has exactly one binding, on `127.0.0.1` whenever
the daemon's loopback is reachable, as it is on CI.
### Evidence
**1. Root-cause reproduction on a fresh `docker:28.0.4-dind` daemon**
This is the same engine version as CI. I first created N DCP-style `-p
127.0.0.1::80` publications, then published `1433` with a random port:
first wildcard (the old fixture), then loopback (the new fixture).
```
--- fresh Docker Engine 28.0.4
[9 loopback publications: 127.0.0.1:32768 … 127.0.0.1:32776]
BEFORE (HostIP="" -> wildcard): 1433/tcp -> 0.0.0.0:32777
AFTER (HostIP=127.0.0.1): 1433/tcp -> 127.0.0.1:32778
--- fresh Docker Engine 28.0.4
[10 loopback publications: 127.0.0.1:32768 … 127.0.0.1:32777]
BEFORE (HostIP="" -> wildcard): failed to bind host port for 0.0.0.0::172.18.0.12:1433/tcp: address already in use
AFTER (HostIP=127.0.0.1): 1433/tcp -> 127.0.0.1:32778
```
- The CI error is reproduced exactly at the 10-collision threshold.
- The loopback publication always succeeds.
- As a control, Docker 29.1.3 succeeds in both modes.
**2. Fail-before / pass-after through the real Testcontainers path**
This used a throwaway test that is not committed. It ran a small image
with `WithPortBinding(1433, true)` and applied the same assertion as the
regression test.
```
failed Binding(withFix: False) Expected: "127.0.0.1" Actual: "" (HostConfig bindings: '':'')
passed Binding(withFix: True) (container started, mapped 127.0.0.1:<port>)
```
**3. Real SQL Server**
On head 7789aa0, CI's `Hosting.Azure (ubuntu-latest)` job ran on Docker
28.0.4. All 8 tests passed (TRX): the 7 re-enabled
`AzureSqlPrincipalReconciliationTests` and the new regression test. The
SQL Server image is too large for the local disk, so this part was only
run in CI.
### Occurrences
- `Hosting.Azure (ubuntu-latest)`, before the tests were disabled on
2026-09-18:
- 2026-09-17: runs 35166494293, 35246832023, 35251762755, 35274272340
and 35275989131 (main and PRs).
- Duplicate issue microsoft#19743 also lists 35262261892 (2026-09-17) and
33086828076 (2026-08-27).
### Scope
The component-test `SqlServerContainerFixture` copies, such as
`Aspire.Microsoft.Data.SqlClient.Tests`, run in jobs without concurrent
DCP containers. They were not observed failing this way, so they are
unchanged.
Fixesmicrosoft#19014
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
---------
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…dress already in use' flake (microsoft#21537)
## Description
Fixesmicrosoft#19449
### Root cause
`DownloadNativeArchivesTests.MockAzdoServer` picked its port with a
probe-and-release pattern:
1. `GetFreeTcpPort()` bound `TcpListener(127.0.0.1, 0)`, read the
assigned port, and **stopped the listener**, which released the port.
2. Later, `Start()` called `HttpListener.Start()` on that port number.
Between steps 1 and 2 the port is free, and any other socket on the
runner can take it. Infrastructure.Tests runs many test classes in
parallel, and they spawn `pwsh`, `dotnet` and `git` processes whose
outgoing connections draw from the same ephemeral range. When another
socket wins, `HttpListener.Start()` throws
`System.Net.HttpListenerException : Address already in use`, which is
exactly the failure in [run
32068234960](https://github.com/microsoft/aspire/actions/runs/32068234960).
The managed `HttpListener` can't close this gap: it rejects port 0
(`HttpListenerException: Invalid port in prefix`), so it can't bind
atomically and report the assigned port.
### Fix
The mock server now binds `TcpListener(IPAddress.Loopback, 0)` in its
constructor and **keeps that socket**, so the OS-assigned port is
reserved from construction until disposal. A minimal HTTP/1.1 responder
replaces the `HttpListener` transport. It reads the request line and
headers, and every response uses `Content-Length` and `Connection:
close`. The mock's endpoints and behaviors are unchanged:
- artifact list JSON
- per-artifact zip download
- permanent 500s and flaky 500s
- corrupt 200 bodies
- truncated responses: `Content-Length` for the full zip, a 16-byte
prefix, then the connection is reset (linger 0), matching the previous
`Response.Abort()`
Requests are still handled one at a time, so the per-artifact counters
still need no synchronization. Only the test file changes. No product
code, csproj or other tests are touched.
### Regression test
`MockAzdoServer_ReservesItsPortBeforeStart` constructs the server and
tries to bind a competing socket to its port **before** `Start()`. It
asserts the bind fails with `AddressAlreadyInUse`, then starts the
server.
### Evidence
- **Before** (original server plus only the new test): fails with
`Assert.Throws() Failure: No exception was thrown`. The port is free
between construction and `Start()`.
- **Exact CI error reproduced** with the original pattern (probe,
release, a competitor binds the port, `HttpListener.Start()`):
`System.Net.HttpListenerException : Address already in use`.
- **After**: all 11 `DownloadNativeArchivesTests` pass with real `pwsh`
7.6 on 13 of 13 runs (3 + 10). The other 10 tests exercise the real
`eng/scripts/download-native-archives.ps1` against the new transport,
including the truncated, corrupt and flaky retry cases.
### Related Infrastructure.Tests issues (separate root causes, not
changed here)
- microsoft#19637
`DownloadFailingJobLogsToolTests.ReportsMissingArtifactWithoutFailing`:
the CI log shows `GenerateDepsFile` failing with `IOException` on
`artifacts/bin/Aspire.TestTools/.../Aspire.TestTools.deps.json`. Two
MSBuild processes write that file at once: `GenerateTestSummaryFixture`
runs `dotnet build GenerateTestSummary.csproj`, which ProjectReferences
Aspire.TestTools, while `DownloadFailingJobLogsToolTests` runs `dotnet
run tools/scripts/DownloadFailingJobLogs.cs`, which pulls in the same
project via `#:project`.
- microsoft#19835
`ExtensionChangelogFinalizedWorkflowTests.ReleaseBranchWithStaleExtensionBaseAdvanceFails`:
a SIGPIPE race in
`.github/workflows/check-extension-changelog-finalized.sh`. Under `set
-o pipefail`, `printf ... | grep -q` returns 141 when `grep -q` exits
before `printf` finishes writing, and the leading `!` turns that into a
false "must contain the finalized release marker" error. That error is
reported at the heading line, `line=3`, matching the CI annotation. The
pattern reproduces 2 times in 20,000 iterations locally.
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ential probing (microsoft#21538)
## Description
The Azure component `TracingEnablesTheRightActivitySource` conformance
tests fail intermittently on CI with `RemoteExecutionException: Half-way
through waiting for remote process` (microsoft#20241 Storage Queues on
windows-latest, run 35373021439; microsoft#18920 Azure OpenAI, run 30416905407).
### Root cause
`ActivitySourceTest` runs in a RemoteExecutor child process, which is
killed after 60s. When the test server isn't reachable (always the case
on CI), that child walks the full `DefaultAzureCredential` chain
**twice**:
1. `CanConnectToServer` is a per-process `Lazy`, so the child re-runs
the parent's `GetCanConnect()` probe (`new DefaultAzureCredential()`
against the real storage account).
2. `ConfigureCredentials` leaves `settings.Credential` null when the
server isn't reachable, so `TriggerActivity` uses the component's
default `DefaultAzureCredential` and walks the chain again.
Each walk probes managed identity (IMDS) and then launches the Azure
CLI, Azure PowerShell and azd, each with its own process timeout. The
microsoft#20241 child dump shows the process killed inside
`AzureCliCredential.GetToken` ← `QueueServiceClient.CreateQueue` ←
`TriggerActivity`, after the Environment, WorkloadIdentity,
ManagedIdentity and VisualStudio credentials had already failed. microsoft#18920
shows the same thing with
`AzureCliCredential`/`AzurePowerShellCredential` timing out.
### Fix (test-only, no assertion changes)
- `ConformanceTests` passes the parent's `CanConnectToServer` value to
the RemoteExecutor child through an environment variable. The new
`GetCanConnectToServer(probe)` helper uses it, so the child doesn't
re-probe.
- A new `ConfigureActivitySourceTestOptions` hook lets the Azure
conformance tests use a fast-failing `UnavailableTokenCredential` in
`ActivitySourceTest` when the server isn't reachable. The client
operation still runs and emits its activity, failing only when it asks
for a token. The other conformance tests (for example, the
`Azure.Identity` log-category tests) still use `DefaultAzureCredential`
as before.
- Applied to: Storage Queues/Blobs/DataLake, Data Tables, KeyVault
(Secret/Key/Certificate), Search and Azure OpenAI (both classes).
ServiceBus isn't touched (another PR owns that area).
### Regression test
`TracingTestDoesNotLaunchDeveloperCredentialTools` (Queues and both
OpenAI classes) runs the activity-source test in a child process with
fake `az`/`azd`/`pwsh`/`powershell` scripts first on `PATH`. The scripts
record that they were launched. The test asserts none of them were
launched and that the child finishes within the RemoteExecutor timeout.
### Evidence (Linux, net11.0; this machine's IMDS endpoint is
unreachable/slow, like CI)
| Test | Before fix | After fix |
|---|---|---|
| Queues `TracingTestDoesNotLaunchDeveloperCredentialTools` | **failed
1/1**: `Half-way through waiting for remote process`; child stuck in
`GetCanConnect()` → `ManagedIdentityCredential` | passed |
| OpenAI `TracingTestDoesNotLaunchDeveloperCredentialTools` (both
classes) | **failed 2/2**: `Half-way through waiting for remote process`
(child in `ManagedIdentityCredential` under `TriggerActivity`) | passed
|
| OpenAI `*Tracing*` (12 tests) | – | 5/5 runs, 12/12 passed |
| Queues `*Tracing*` (6 tests) | – | 3/3 runs, 6/6 passed |
| Queues `ConformanceTests` class (excluding the pre-existing slow
`LoggerFactoryIsUsedByRegisteredClient`) | – | 23 total, 0 failed |
| OpenAI conformance classes (same exclusion) | – | 46 total, 0 failed |
| Blobs / Tables / KeyVault / Search `*Tracing*` | – | 5/5, 5/5, 13/16
(3 skipped), 4/5 (1 skipped); 0 failed |
DataLake (net10.0 only) builds; CI covers its tests.
Fixesmicrosoft#20241Fixesmicrosoft#18920
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- Did you add public API?
- [x] No
- Does the change make any security assumptions or guarantees?
- [x] No
- Does the change require an update in our Aspire docs?
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
---------
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t#21615)
## Description
Fixes the npm part of the
`Aspire.Cli.EndToEnd.Tests.JavaScriptPublishTests.AllPublishMethodsBuildDockerImages`
flake. It is #5 under "Top unowned flakes" in microsoft#21600.
### Root cause
TypeScript AppHost scaffolds declare floating npm ranges
(`TypeScriptLanguageSupport.cs`: `typescript-eslint ^8.57.1`, `eslint
^10.0.3`, `nodemon ^3.1.14`, …). The Docker E2E shells run `npm install`
with no age limit. If a run installs a few minutes after an upstream
publish, npm resolves the new version from the packument while its
tarball is still 404. `aspire init` then fails with `npm error code
E404`, and the test times out waiting for `Created apphost.mts` or hits
an `ERR` prompt.
Publish times compared with failure times:
| Package version (upstream publish) | Failing job (E404 on that
tarball) |
|---|---|
| `@typescript-eslint/typescript-estree@8.72.1`, published
2026-10-26T17:08:21Z |
[121717304181](https://github.com/microsoft/aspire/actions/runs/40362021417/job/121717304181)
at 17:10:58Z (sibling `JavaScriptHostingApisRunFromTypeScriptAppHost`);
[121723096663](https://github.com/microsoft/aspire/actions/runs/40363954680/job/121723096663)
at 17:12:51Z |
| `@types/estree@1.0.10`, published 2026-10-29T04:34:27Z |
[123265152317](https://github.com/microsoft/aspire/actions/runs/40817029451/job/123265152317)
at 04:38:29Z |
All three failures came within 2–5 minutes of the publish, on unrelated
branches.
### 7-day denominator
The job is `Cli.EndToEnd-JavaScriptPublishTests (ubuntu-latest)`.
Window: 2026-10-24T20:00Z → 2026-10-31, 751 head SHAs.
- **849 check runs:** 799 success, 22 failure, 28 cancelled. That leaves
**821 completed** runs.
- **14 failures are deterministic, not flaky.** They are all "Pipeline
failed unexpectedly" on a single dependabot npm branch.
- **8 failures are intermittent** (about 1%):
- **3 are proven npm fresh-release E404s.** They are in the table above;
this PR fixes them.
- **2 are `aspire init` npm installs that stalled without an E404 in the
log**
([122678762903](https://github.com/microsoft/aspire/actions/runs/40645443969/job/122678762903),
[124617913801](https://github.com/microsoft/aspire/actions/runs/41227466919/job/124617913801)).
They have the same symptom, but I can't prove the same cause.
- **3 are not addressed here.** Two are `verify-runtime.sh` giving up
after its 30s budget while `aspire run` still showed "Preparing Aspire
server..." (10-31). One is a 5-minute pipeline timeout.
### Fix
`PrepareDockerEnvironmentAsync` now exports `npm_config_before=<now − 1
day>`, using [npm's `before`
config](https://docs.npmjs.com/cli/v10/using-npm/config#before). The E2E
images ship npm 10; `min-release-age` needs npm ≥ 11.10. With the
cutoff:
- Range resolution in the Docker E2E shells only picks releases that
have had time to propagate.
- Lockfile/pinned installs are unaffected. Dependabot already applies a
7-day cooldown to the E2E npm manifest, so those pins are older than the
cutoff.
- `npm view` ignores `before`. For example, `RepositoryToolUpdateTests`
still sees the real `latest`.
`npm pack` of an exact version newer than the cutoff fails with ETARGET.
The three tests whose subject is acquiring the newest `@playwright/cli`
by exact version therefore opt out with `useSettledNpmReleases: false`:
`PlaywrightCliInstallTests` ×2 and `NewWithAgentInitTests`.
No assertions are weakened, and the change adds no sleeps or retries.
### Regression test
`SettledNpmReleasesTests` starts an in-process npm registry that
reproduces the post-publish state. `1.1.0` is listed in the packument,
published 2 minutes ago, and its tarball returns 404. `1.0.0` is 30 days
old and downloadable.
- **Without the cutoff:** `npm install pkg@^1.0.0` fails with `E404` on
the `1.1.0` tarball. This is the CI signature, and the test asserts this
control case.
- **With `GetSettledNpmReleasesCommand(now)`:** the install succeeds
with `1.0.0`.
Local results:
- With `GetSettledNpmReleasesCommand` stubbed to a no-op, the test
**fails**: `npm error code E404 … GET
http://127.0.0.1:…/aspire-e2e-settle-probe-1.1.0.tgz - Not found`.
- With the fix, it **passes** on npm 10.9.9 (the E2E image version) and
npm 11.20.0.
### E2E evidence
The change only affects the CLI E2E Docker shells, so the evidence is CI
running real CLI/DCP with the cutoff active:
- **Run
[41239786430](https://github.com/microsoft/aspire/actions/runs/41239786430)
on `56c6ec1e`:** all 131 checks passed and 0 failed.
- It includes `Cli.EndToEnd-JavaScriptPublishTests` and every
TypeScript/JavaScript CLI E2E class that runs `aspire init` / `npm
install`.
- `Cli.EndToEnd-SettledNpmReleasesTests` [ran with 2 succeeded and 0
skipped](https://github.com/microsoft/aspire/actions/runs/41239786430/job/124653412094).
- **Follow-up head `33e39428`:** only changes the regression test
(review feedback). Run
[41242524837](https://github.com/microsoft/aspire/actions/runs/41242524837)
passed with 130 checks passing and 0 failing, including
`Cli.EndToEnd-JavaScriptPublishTests`.
`Cli.EndToEnd-SettledNpmReleasesTests` [ran with 2 succeeded and 0
skipped](https://github.com/microsoft/aspire/actions/runs/41242524837/job/124662310776).
`PlaywrightCliInstallTests` and `NewWithAgentInitTests` are
`[OuterloopTest]`, so PR CI doesn't run them. They opt out of the
cutoff, which leaves their npm behavior unchanged.
## Checklist
- Is this feature complete?
- [x] Yes. Ready to ship.
- [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] Yes
- [ ] No
- Did you add public API?
- [ ] Yes
- [x] No
- Does the change make any security assumptions or guarantees?
- [ ] Yes
- [x] No
**🤖 Posted by AspireAgent (AI Agent)**
---------
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#21749)
> Supersedes microsoft#21558. That PR's head is on a fork (`davidfowl/aspire`)
with `maintainer_can_modify=false`, so it couldn't be brought up to date
with `main`. This PR contains the same change, squashed onto current
`main`.
Fixesmicrosoft#21557
## Failure
On `main`,
`Infrastructure.Tests.AnalyzeCiFailureWorkflowTests.LastSuccessfulMainRunFallsBackToOlderWindow`
failed with `Expected: 0 / Actual: 126` ([run 40984409791,
Infrastructure
(ubuntu-latest)](https://github.com/microsoft/aspire/actions/runs/40984409791/job/123832671253)).
**Denominator:** 1 failure in 115 attempt-1 `Infrastructure
(ubuntu-latest)` jobs in the last ~48h of `ci.yml` runs (104 success, 3
failure, 8 cancelled). The other two failures were unrelated,
PR-specific breaks.
## Root cause
This is the `ETXTBSY` ("Text file busy") race that microsoft#21448 fixes for
`Aspire.Acquisition.Tests` (microsoft#18732). This PR fixes it in
`tests/Infrastructure.Tests`, which microsoft#21448 doesn't touch.
1. Several tests write a stub executable (`gh`, `actionlint`, `curl`,
`dotnet.sh`) from the test host with `File.WriteAllText[Async]`, then
`chmod` it and run a script that execs it.
2. xUnit runs test classes in parallel, so other tests in the same host
keep starting processes.
3. A child forked while the host has the stub open for writing inherits
that descriptor until it calls `exec`.
4. Linux refuses to `exec` a file that any process has open for writing,
returning `ETXTBSY`. bash reports that as exit **126**.
Writing to a temp file and renaming it doesn't help, because the
inherited descriptor still refers to the same inode.
## Fix
- New `ExecutableFile.WriteAsync/Write` test helper. On Unix it writes
the contents (from stdin) and sets the mode inside a short-lived
`/bin/sh -c 'cat > "$1" && chmod "$2" "$1"'` child, so the test host
never holds a writable descriptor that another fork could inherit. On
Windows it is a plain `File.WriteAllTextAsync`.
- Every Infrastructure.Tests call site that execs a freshly written stub
now uses the helper:
- `AnalyzeCiFailureWorkflowTests`
- `LintHandwrittenWorkflowsScriptTests`
- `NixCliPackageTests`
- `TemplateManifestGenerationTests`
- `ExtensionReleaseWorkflowTests` (the separate `chmod +x` step is
removed)
- `BuildWarningPolicyTests` is unchanged because it runs `bash build.sh`
rather than exec'ing the file.
- No assertions are changed, and no sleeps or retries are added.
## Regression test
`ExecutableFileTests.WrittenExecutableCanBeExecutedImmediatelyWhileOtherThreadsStartProcesses`
(Unix only):
- Four threads continuously fork `/bin/sh -c "exit 0"`.
- Meanwhile, 1000 stubs are written with the helper and exec'd
immediately.
- The test asserts that no launch fails and that each stub returns its
expected exit code.
`WritesContentsAndExecutableMode` checks that the contents and mode
round-trip.
| Helper's Unix path | Test runs failed | Launches failed |
|---|---|---|
| Before: in-process `File.WriteAllTextAsync` + `File.SetUnixFileMode` |
**20 / 20** | **561 / 20,000** (18–35 per run, all `Text file busy`) |
| After: this PR | **0 / 20** | **0 / 20,000** |
## Change since microsoft#21558
`main` moved `AnalyzeCiFailureWorkflowTests.cs` into the new
`tests/AnalyzeCiFailure.Tests` project. The helper call moved with it,
so `ExecutableFile.cs` is now linked into that project (`<Compile
Include="../Infrastructure.Tests/ExecutableFile.cs"
Link="shared/ExecutableFile.cs" />`). This uses the same pattern the
project already uses for `RepoRoot.cs` and `CommandResult.cs`. Without
the link the project fails to build with CS0103.
## Confidence
- **Behavior and regression coverage:** High. `ExecutableFileTests`
reproduces the race: the old write path failed in 20 of 20 runs, and the
new helper failed in 0 of 20. No assertions changed in the converted
tests.
- **Tests and pipelines run:** Ran locally on Linux against this commit:
- `AnalyzeCiFailure.Tests`: 1162 of 1163 passed. The one failure,
`AnalyzerBuildsFromRealSparseCheckoutWithoutHostileParentImports`, comes
from NuGet's vulnerability check (audit) being unable to reach nuget.org
from the local machine (NU1900/NU1905), not from this change.
- `Infrastructure.Tests` classes this PR touches, all passing:
ExecutableFile 2/2, NixCliPackage 3/3, TemplateManifestGeneration 7/7,
ExtensionReleaseWorkflow 9/9, LintHandwrittenWorkflowsScript 3/3.
- CI is pending.
- **Build artifacts:** Not affected. Only test code and a test `.csproj`
change; no product or packaging changes.
- **E2E:** Not applicable. This is test-infrastructure only.
**How we stop this recurring:** New tests that exec a freshly written
stub should write it with `ExecutableFile.WriteAsync` instead of
`File.WriteAllText` followed by `chmod`. The regression test guards the
helper itself.
**🤖 Posted by AspireAgent (AI Agent)**
Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Registry names near transport errors could classify a job as transient
even when it also contained an unrelated test failure. Match failed
registry requests and reject registry retries whenever a test-execution
step failed or failed-step metadata is absent.
Remove the obsolete Hosting-1/5 exclusion, unused rerun state, and stale
comments. Merge current main while preserving its artifact-transfer
retry classification in the shared PR policy.
Focused rerun-policy and test-selection validation passed all 572 cases.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The supplied diffs omit the described rerun-policy implementation, and the official signing-gate change still requires pipeline 1602 validation against actual artifacts.
Registry diagnostics could make failed extension E2E and SDK validation
jobs eligible for a main retry because the test-step veto recognized
only the regular and NuGet-dependent test runners. A recovered pull
could therefore hide a subsequent real test failure behind a retry.
Recognize the reachable validation and smoke-test steps when vetoing
registry retries. Keep the Windows successful-test matcher separate so
these additional names cannot enable its post-test exception.
Regression coverage failed in 13 cases before the fix. All 223 policy
tests now pass, and controlled live GitHub runs verify the extension
failure veto alongside the existing positive and fail-closed paths.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The signing-gate change requires maintainer verification of definition 1602’s PostBuildSign main path and release/manual signature-verification paths using runtime timeline and artifact evidence unavailable in this review environment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
the
area-engineering-systems
Description
Failed rolling
mainCI builds can automatically retry once when every failed job matches a narrow infrastructure allowlist. Runs for obsolete commits and runs with unmatched failures are not retried.The rerun workflow dispatches to separate PR/manual and current-main policies, sharing GitHub job access and reporting helpers. Analysis remains read-only; only the output-gated execution job can request reruns.
Current-main retry rules
After ignoring aggregate result jobs, source attempt 1 must contain one to five failed jobs, each matching one of:
GetorHeadrequest tohttps://netaspireci.azurecr.io/v2/...withconnect: connection refusedorConnection reset by peer.GetorHeadrequest tohttps://mcr.microsoft.com/v2/...with HTTP 503 orServiceUnavailable.-1073741502or0xC0000142) after successful tests, with failures confined to post-test reporting or cleanup steps.Registry request errors must occur on the same line, within 500 characters of the quoted request. Registry retries require failed-step metadata and are rejected when a known test-execution or startup/smoke-validation step failed. This includes regular and NuGet-dependent tests, extension E2E, polyglot SDK validation, starter startup, native-dashboard tests and packaged CLI smoke tests.
This intentionally skips genuine image-download outages inside failed test steps. A recovered image pull can coexist with an unrelated test failure, and job-wide logs cannot reliably attribute the failure to the registry. The broader registry veto does not expand the Windows successful-test exception.
Immediately before requesting a rerun, the policy rechecks the completed failed attempt, current
mainSHA, and newer rolling runs. Attempt 2 cannot trigger another retry. Request errors are reported explicitly.The PR/manual policy retains force mode and its three-attempt cap. Automatic PR retries additionally require the failed SHA to remain the open PR's current head; manual dispatch can target a historical commit while its associated PR remains open.
Validation
223 policy tests passed, with zero failures or skips:
Regression coverage includes 16 execution-step names across both registry families and annotation/log diagnostics, extension whole-run vetoes, and preservation of the Windows exception.
Nine controlled live GitHub scenarios passed in an isolated private repository:
mainand stale open-PR heads were rejected without reruns.The four policy/shared modules were byte-identical to this revision. Workflow and dispatcher adaptations were limited to three repository/owner guard literals. Registry messages were simulated diagnostics in real GitHub job logs, not induced registry outages. Trial PRs were closed, workflows disabled, and temporary branches removed.
node --checkpassed for all five runtime modules and the test harness.actionlint .github/workflows/auto-rerun-transient-ci-failures.ymlandgit diff --checkpassed.At head
1b4fbce0d38fb6d12d8cb6eb136b14ac996b7e66, CI passed, including 2,044 Infrastructure tests with zero failures or skips, package generation and all six native CLI archive builds. Native hosting validation passed. All reported checks completed successfully or were intentionally skipped.Checklist
<remarks />and<code />elements on your triple slash comments?