Repository navigation
Description
Describe the bug
The build-and-package job in build-package.yml is failing intermittently across branches, including main. Each failure is a different test, and every one passes on a subsequent run. This makes a green build largely a matter of luck and forces repeated re-runs on unrelated PRs.
Observed while validating #861, which changes five files (three pipeline YAML, two PowerShell scripts), no C# and nothing invoked by the GitHub Actions build. Four consecutive runs produced four distinct failures:
1. EndToEndTests — four tests fail when winapp init cannot restore Microsoft.WindowsAppSDK (job 105005563144)
error: Package 'Microsoft.WindowsAppSDK' is incompatible with 'all' frameworks in project '...\TestApp.csproj'.
error: - Found 0 version(s) in C:\Program Files\dotnet\library-packs
Assert.AreEqual failed. Expected:<0>. Actual:<1>. 'actual' expression: 'initExitCode'. winapp init should succeed
Affected: E2E_DotNetApp_PackageShouldIncludeRuntimeDependency, E2E_DotNetApp_WithWin2D_PackageShouldIncludeActivatableClasses, E2E_DotNetApp_SelfContained_ShouldBundleRuntimeAndEmbedActivatableClassesInExe (all WinAppSDK Stable), and E2E_DotNetProject_InitDetectsCsprojAndAddsPackageReferences_ShouldSucceed (EndToEndTests.cs:206, :296, :369, :508).
2. WindowsSandboxBootstrapTests.WaitForHeartbeat_NeverPublished_SurfacesTheAgentsOwnDiagnostics — seen on main (job 104978506116)
Assert.ThrowsExactlyAsync failed. Expected exception type:<System.Threading.Tasks.TaskCanceledException>.
Actual exception type:<System.OperationCanceledException>.
'action' expression: '() => WindowsSandboxBackend.WaitForHeartbeatAsync(_resultDirectory, Epoch, cancellation.Token)'
at WindowsSandboxBootstrapTests.cs:137
TaskCanceledException derives from OperationCanceledException, and which one surfaces depends on whether the cancellation is observed through a Task or the token directly. ThrowsExactlyAsync makes that an implementation detail the test cannot tolerate.
3. GuestCommandServerTests.Execute_DetachedProcessReturnsAfterStartAndSurvivesTheChannel (job 105012269284)
Assert.IsTrue failed. 'condition' expression:
'SpinWait.SpinUntil(() => process.Disposed, TimeSpan.FromSeconds(1))'.
The agent should release the detached process after it exits.
at GuestCommandServerTests.cs:273
A fixed 1-second budget for a process to exit and be disposed, on a runner already running tests at Workers: 4.
A fourth failure, Mp4SinkWriterEncoder_RealEncoderCoversValidationAndSuccessfulComplete (MF_E_SINK_NO_SAMPLES_PROCESSED), also recurred. That one is already tracked in #834 and is not part of this report.
To Reproduce
- Push any branch, or look at recent
mainruns. - Watch
build-package.yml->build-and-package. - Re-run the failed job. It usually goes green, and a later run fails on a different test.
Recent build-package.yml history showing the pattern across branches:
failure 09-16T21:54 nmetulev-typed-ui-geometry-context
success 09-16T21:54 nmetulev-deterministic-explicit-ui-actions
failure 09-16T21:23 azchohfi-esrp-sign-exes-before-packaging
success 09-16T21:11 main
failure 09-16T21:10 main
success 09-16T21:09 main
failure 09-16T21:08 dependabot/nuget/src/winapp-Analyzer/...
success 09-16T19:55 azchohfi-exercise-release-connections
failure 09-16T19:54 azchohfi-exercise-release-connections
Note main failing at 21:10 between two successes, and the same branch failing then passing a minute apart.
Expected behavior
build-and-package passes deterministically on a commit that has not changed the code under test, so a red build means a real regression.
Suggested directions, roughly in order of value:
- (2) Assert
OperationCanceledExceptioninstead ofThrowsExactlyAsync<TaskCanceledException>, or assert on the message/diagnostics the test actually cares about. This one looks like a straightforward correctness fix rather than infrastructure tuning. - (3) Replace the fixed 1-second
SpinUntilwith a generous timeout, or await an explicit signal that the process was released. - (1) Decide whether these four E2E tests should depend on a live
dotnet add packageagainst nuget.org at all. If they must, they need retry with backoff; if not, pin against a local feed. As written, any nuget.org hiccup turns into a red build on every open PR.
There is a secondary effect worth noting: because build-cli.ps1 aborts at the C# test step, the scripts/tests Pester suite downstream of it never executes. A flake in the C# tests silently skips the PowerShell test coverage entirely, so those suites are not actually being validated in CI on any run that fails this way.
OS Version and details
GitHub-hosted windows-latest runner. Tests run with Test Parallelization enabled (Workers: 4, Scope: MethodLevel).
Activity
azchohfi commentedon Sep 30, 2026
Root cause for flake #1 (
EndToEndTests/winapp initfailing). This is not fixed in the PR for #2 and #3 and needs a separate decision.What happened: it was not a network problem. In job 105005563144,
winapp init --setup-sdks stablegotMicrosoft.WindowsAppSDK 2.5.1as the newest version from nuget.org.dotnet add packagethen failed:This happened while WinAppSDK 2.5.1 was being published. nuget.org already listed the top-level package but not yet its
Microsoft.WindowsAppSDK.Runtimedependency. Both are available now, which is why re-runs pass. The restore log shows the Runtime version list came from NuGet's HTTP cache (CACHE .../microsoft.windowsappsdk.runtime/index.json). That cache lasts 30 minutes, so retrying within the same run would not have helped.Why it matters: the same thing can happen during every future WinAppSDK release, for all four tests that call
initwith the newest SDK.Options for a follow-up:
initfails with NU1102 for aMicrosoft.WindowsAppSDK.*dependency at the exact version it just picked, mark the test Inconclusive. The tests keep checking the newest SDK, and any other failure still fails the build.Secondary effect (Pester skipped when a C# test fails): already fixed on
main. CI now runs theAuxiliarysuite, which owns both Pester suites, as its own job separate from the CLI test shards.Fix two flaky execution-target tests (#950)
Skip a Windows App SDK release until its pinned sub-packages are publ…