Repository navigation
Pin Windows CI container runtime to Docker to stop DCP wslc probes (#21570) - #21608
Merged
Merged
Conversation
DCP 0.27 probes wslc.exe (shipped by the WSL MSI on windows-2025 images) on every AppHost start when no container runtime is configured. Windows runner losses (#21570) occur only in jobs that launch DCP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21608Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21608" |
github-actions
Bot
added
the
area-engineering-systems
Contributor
Tests selectorSelects the full PR test matrix + all PR-gated jobs (ALL) — a rule matching '.github/workflows/run-tests.yml' selects ALL Selection computed for commit |
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Source review confirms configuration forwarding, but Windows-hosted validation is still needed to verify that DCP stops probing wslc.exe and runner-loss rates improve.
No open issues 🎉
What changed in this PR
Pins the container runtime to Docker for Windows CI tests to avoid DCP’s automatic WSLc probes, targeting the hosted-runner communication losses tracked in #21570.
Changes:
- Sets
ASPIRE_CONTAINER_RUNTIME=dockerthroughGITHUB_ENVbefore Windows .NET and VS Code extension E2E tests. - Documents the workaround and links the tracking issue.
- Leaves Linux and macOS runtime selection unchanged.
| File | Description |
|---|---|
.github/workflows/run-tests.yml |
Adds a Windows-only Docker runtime setting for shared test jobs. |
.github/workflows/extension-e2e-tests.yml |
Adds the same setting before Windows extension E2E tests. |
Review effort: Balanced
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Collaborator
Author
|
Attempt 1 of the CI run on head
Repo automation has started a single rerun of the failed jobs (attempt 2). That is the one documented rerun for each of these flakes. After-fix check: the DCP logs in 🤖 Posted by AspireAgent (AI Agent) |
4 of 14 tasks
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Collaborator
Author
|
Rerun batch for loss-rate evidence. I ran a full rerun (attempt 3) of the same head
At the baseline rate, about 3.5 losses would be expected across 288 jobs. Seeing 0 has a Poisson probability of about 3%. Post-merge monitoring on main is the definitive check. Attempt 3 had one unrelated Linux failure: 🤖 Posted by AspireAgent (AI Agent) |
David Fowler (davidfowl)
marked this pull request as ready for review
October 31, 2026 19:58
This was referenced Oct 31, 2026
David Fowler (davidfowl)
added
the
ci-health
Copilot SWE Agent
Collaborator
Author
|
7-day runner-loss census. Scope:
The new findings since the last evidence comment:
🤖 Posted by AspireAgent (AI Agent) |
Ankit Jain (radical)
approved these changes
Nov 3, 2026
Contributor
|
✅ No documentation update needed. docs_optional → build_or_ci_only. Triggered signal: |
Ella Hathaway (ellahathaway)
pushed a commit
that referenced
this pull request
Nov 4, 2026
) ## Description Windows runner loss and hangs are the largest flake family in #21600: Windows extension E2E and .NET test jobs freeze at 47–59 minutes against a 3–9 minute baseline, then lose their runner (#21570). We can't currently answer whether the machines are overloaded on CPU, memory or disk, because nothing records machine load in a form that survives the job. This PR adds a lightweight background resource sampler to: - the Windows .NET test jobs (`run-tests.yml`) - the Windows VS Code extension E2E shards (`extension-e2e-tests.yml`) **What it records** (`eng/scripts/ci-resource-monitor.ps1`): a detached, below-normal-priority `pwsh` process starts right after checkout. Every 20s it writes: - `resources.csv`: CPU %, processor queue length, available MB, commit charge % / committed GB / commit limit, paging pages/s, page file usage %, disk queue length, disk % time, free GB on the system and workspace drives, and process count. - `processes.log`: the top 5 processes by CPU (since the previous sample) and by working set (with private bytes). **Where it goes:** the sampler writes into directories that the existing `if: always()` uploads already collect, so there are no new artifacts: - .NET tests: `artifacts/log/test-logs/resource-monitor/` inside `logs-<test>-windows-latest`. - Extension E2E: `extension/.test-results/resource-monitor/` inside `extension-e2e-diagnostics-win-x64-<shard>-attempt<N>`. A `Stop resource monitor (Windows)` step (`if: always()`) kills the sampler before the upload. It prints a peak/minimum table, including the largest gap between samples (a stall indicator), to the log and the job summary. This covers passing, failing, cancelled and timed-out jobs. **Safety:** - Every mode exits 0 and both steps are `continue-on-error`. - Test commands, timeouts, retries and environment are unchanged. - Linux and macOS are untouched. - Overhead is one `Get-Counter` and one `Get-Process` call every 20s at below-normal priority. **Limitation:** a runner that loses communication uploads nothing and keeps no logs. This telemetry can't capture the final minutes of a lost runner. It can show whether pressure builds on the slow jobs that survive and on the shards that typically get lost. That is still more than we have today: the existing console Heartbeat (60s, .NET tests only) vanishes with the log. Contributes to #21570. It complements #21608 (Docker pin), which adds no telemetry. ### Existing evidence (before this PR) On main, the last 16 completed `ci.yml` push runs had 1,551 Windows jobs. All 20 jobs that ran 45 minutes or longer were runner-lost; job logs return `BlobNotFound`, so there's no Heartbeat data from any of them. Examples: [extension E2E resource-debug-tools, 110m](https://github.com/microsoft/aspire/actions/runs/41283516743/job/124783549300) and [Hosting-4, 59m](https://github.com/microsoft/aspire/actions/runs/41168649739/job/124441989110). Surviving Windows Hosting-6 jobs on main ([example](https://github.com/microsoft/aspire/actions/runs/41215270228/job/124578892630)) show the following in Heartbeat: - memory 3.1–3.7 / 16.0 GB (20–23%) - CPU 60–80% during test execution - C: about 29 GB free, D: about 140 GB free None of this shows exhaustion, but it doesn't cover commit charge, page file or disk queue. ### Validation - `tests/Infrastructure.Tests/PowerShellScripts/CiResourceMonitorTests.cs`: - the summary math (peaks, minimums, averages, stall gap) - Stop with no samples - real Start → background sample → Stop that ends the sampler - `actionlint` is clean on both workflows. - This PR's own Windows CI run: results below. ### Observed in this PR's CI The data comes from three CI runs of this PR on Windows hosted runners: [41289502805](https://github.com/microsoft/aspire/actions/runs/41289502805), [41290605859](https://github.com/microsoft/aspire/actions/runs/41290605859) (both partly cancelled by later pushes) and [41291481771](https://github.com/microsoft/aspire/actions/runs/41291481771) (head `d0358d42`). It covers **331 Windows jobs with 2,923 samples**: 312 .NET test jobs and 19 VS Code extension E2E shards. The sampler emitted no missing-counter warnings. | | .NET test jobs (312) | Extension E2E shards (19) | |---|---|---| | CPU: mean of per-job averages | 54.8% | 44.3% | | CPU: share of samples at 90% or more | 27.6% | 6.5% | | Jobs with at least half their samples at 90% or more CPU | 44 | 0 | | Lowest available memory (of 16 GB) | 9,891 MB | 9,779 MB | | Peak commit charge (limit about 18.9 GB) | 33.4% | 36.6% | | Peak page file usage | 22% | 19.9% | | Peak disk queue length | 146.5 (one job; 4 jobs above 60) | 28.7 | | Lowest free space on C: | 25.6 GB | 23.5 GB | | Largest gap between samples | 37 s | 22 s | Examples: - [Extension E2E shard `discovery-configuration`](https://github.com/microsoft/aspire/actions/runs/41291481771/job/124809523820): CPU 35% average (66% peak), at least 10.9 GB available, commit at most 26.7%, disk queue at most 11.6. The diagnostics artifact `extension-e2e-diagnostics-win-x64-discovery-configuration-attempt1` contains `resource-monitor/resources.csv` and `processes.log`. - [Hosting.Dotnet](https://github.com/microsoft/aspire/actions/runs/41291481771/job/124805517960): CPU 84.5% average with 10 of 16 samples at 90% or more. The `logs-Hosting.Dotnet-windows-latest` artifact contains `resource-monitor/`. Its `processes.log` shows `VBCSCompiler`, `dotnet` and `MSBuild` as the top CPU consumers, i.e. the test-driven project builds. - [Cancelled Hosting-4](https://github.com/microsoft/aspire/actions/runs/41289502805/job/124798187412): Stop and upload still ran (CPU 82% average, disk queue at most 17.9). - The sampler's own `pwsh` uses about 1% CPU in `processes.log`. **Runner loss was reproduced in this run, and those jobs stayed dark.** On head `d0358d42`, three .NET jobs froze in `Run tests (Windows)` at about 09:15Z: [Hosting.Java](https://github.com/microsoft/aspire/actions/runs/41291481771/job/124805518377) (2-minute baseline), [Hosting.Browsers](https://github.com/microsoft/aspire/actions/runs/41291481771/job/124805521155) (3-minute baseline) and [Hosting-4](https://github.com/microsoft/aspire/actions/runs/41291481771/job/124805533340) (8-minute baseline). MTP's `--timeout 20m` never fired. All three were declared lost about 46 minutes after job start: *"The hosted runner lost communication with the server"*, logs `BlobNotFound`, and no artifacts. The [extension E2E `winui-debug` shard](https://github.com/microsoft/aspire/actions/runs/41291481771/job/124809523893) (5–6-minute baseline) froze the same way in `Run extension E2E tests` from 09:34Z and was lost about 47 minutes after job start. This matches main: in 9 of the last 16 `ci.yml` push runs, 1–3 Windows jobs per run were lost at 46–48 minutes. In this PR's two earlier runs the same Java and Browsers jobs, also running the sampler, finished in 1–3 minutes. **What this answers so far:** 1. **Memory and commit:** not under pressure on any surviving Windows job. At least 9.8 GB was always available, commit charge stayed at or below 37%, and the page file at or below 22%. 2. **Disk:** space is fine (at least 23.5 GB free on C:, about 216 GB on D:). The disk queue spikes occasionally (above 60 in 4 of 331 jobs), during builds. 3. **CPU:** the 4-vCPU runners are often saturated during .NET test jobs, mostly by the builds that tests trigger. Extension E2E shards are moderate. No surviving job showed a stall: the largest gap between samples was 37s against a 20s interval. 4. **Frozen jobs:** they freeze within minutes of starting tests, and even the 2-minute Java suite froze. Telemetry can't see inside them, because a lost runner uploads nothing. So the data rules out gradual memory or disk exhaustion on the jobs we can observe, but can't confirm or exclude a sudden in-VM event. Capturing that would need an off-box sink, such as streaming samples to an external store. That is out of scope for this lightweight change. ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [x] Yes - [ ] No - Did you add public API? - [ ] Yes - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - [x] No 🤖 Posted by AspireAgent (AI Agent) --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Mitigates #21570 (Windows hosted runner "lost communication"), tracked in #21600.
Repo lever found: DCP 0.27 (bumped in #20874 on 2026-10-10) added WSLc support. When no container runtime is configured, DCP probes
wslc.exeon every AppHost start. The currentwindows-latestimages (windows-2025andwindows-2025-vs2026) install the latest WSL MSI. Since WSL 3.0, that MSI putsC:\Program Files\WSL\wslc.exeon PATH. So every DCP start on Windows CI now runswslc version,wslc infoandwslc network list, and DCP kills the last one early ("Context expired, stopping process"). actions/runner-images#14938 reports hosted Windows workers freezing during WSL2 provisioning on the same image family.This PR sets
ASPIRE_CONTAINER_RUNTIME=dockerfor Windows test jobs inrun-tests.ymland for the Windows VS Code extension E2E jobs, using aGITHUB_ENVstep that only runs on Windows. With an explicit runtime, DCP probes only that runtime. Docker is the only runtime these runners use, and DCP already chooses it today.Evidence
7-day bounded analysis. Scope: main
ci.ymlattempt-1 jobs from 57 non-cancelled runs, about 19.4k jobs. 97 jobs carried the annotation "The hosted runner lost communication with the server".45 of 57 runs (79%) had at least one lost runner.
All 97 losses were on
windows-latest. None were on Linux or macOS.Every loss was in a test step:
Run extension E2E tests44,Run tests (Windows)52,Run nuget dependent tests (Windows)1.Losses were declared 46–58 minutes into the job, 47 at the median. In a sample of 12 lost jobs, the test step had started 1–4 minutes into the job, so the worker stops about 45 minutes before GitHub declares the loss. These shards normally finish in minutes, and the MTP
--timeout 20mnever fires.Losses happen only in projects that start DCP:
aspire run)No resource pressure appears in Windows heartbeats: memory about 20% of 16 GB, C: about 80% with about 30 GB free.
DCP invokes wslc on CI. The uploaded DCP logs (
logs-Hosting.JavaScript-windows-latest, run 40196754950) contain:actions/runner-imagesinstalls the latest WSL MSI (Install-WSL2.ps1), andmicrosoft/WSL/msipackage/package.wix.ininstallswslc.exeand adds it to PATH.windows-2022does not install WSL2.An explicit runtime stops the other probes. Local harness with DCP 0.27.2, putting fake
docker,podmanandwslcon PATH and logging each call:dcp infoprobesdocker --versionandpodman container ls. The wslc probe is Windows-only.dcp info --container-runtime dockerprobes onlydocker --version.Aspire passes
ASPIRE_CONTAINER_RUNTIMEto DCP as--container-runtime(DcpHost,DcpDependencyCheck).Validation plan: this PR's Windows DCP logs should show no
wslc.exeinvocations, and the loss rate is compared across this PR's CI runs. Results will be posted below.Checklist
🤖 Posted by AspireAgent (AI Agent)