Skip to content

[Bug]: build-and-package is flaky across branches, including main (3 undocumented flaky tests) #862

Closed

Description

Describe the bug

The build-and-package job in build-package.yml is failing intermittently across branches, including main. Each failure is a different test, and every one passes on a subsequent run. This makes a green build largely a matter of luck and forces repeated re-runs on unrelated PRs.

Observed while validating #861, which changes five files (three pipeline YAML, two PowerShell scripts), no C# and nothing invoked by the GitHub Actions build. Four consecutive runs produced four distinct failures:

1. EndToEndTests — four tests fail when winapp init cannot restore Microsoft.WindowsAppSDK (job 105005563144)

error: Package 'Microsoft.WindowsAppSDK' is incompatible with 'all' frameworks in project '...\TestApp.csproj'.
error:   - Found 0 version(s) in C:\Program Files\dotnet\library-packs
Assert.AreEqual failed. Expected:<0>. Actual:<1>. 'actual' expression: 'initExitCode'. winapp init should succeed

Affected: E2E_DotNetApp_PackageShouldIncludeRuntimeDependency, E2E_DotNetApp_WithWin2D_PackageShouldIncludeActivatableClasses, E2E_DotNetApp_SelfContained_ShouldBundleRuntimeAndEmbedActivatableClassesInExe (all WinAppSDK Stable), and E2E_DotNetProject_InitDetectsCsprojAndAddsPackageReferences_ShouldSucceed (EndToEndTests.cs:206, :296, :369, :508).

2. WindowsSandboxBootstrapTests.WaitForHeartbeat_NeverPublished_SurfacesTheAgentsOwnDiagnostics — seen on main (job 104978506116)

Assert.ThrowsExactlyAsync failed. Expected exception type:<System.Threading.Tasks.TaskCanceledException>.
Actual exception type:<System.OperationCanceledException>.
'action' expression: '() => WindowsSandboxBackend.WaitForHeartbeatAsync(_resultDirectory, Epoch, cancellation.Token)'
  at WindowsSandboxBootstrapTests.cs:137

TaskCanceledException derives from OperationCanceledException, and which one surfaces depends on whether the cancellation is observed through a Task or the token directly. ThrowsExactlyAsync makes that an implementation detail the test cannot tolerate.

3. GuestCommandServerTests.Execute_DetachedProcessReturnsAfterStartAndSurvivesTheChannel (job 105012269284)

Assert.IsTrue failed. 'condition' expression:
'SpinWait.SpinUntil(() => process.Disposed, TimeSpan.FromSeconds(1))'.
The agent should release the detached process after it exits.
  at GuestCommandServerTests.cs:273

A fixed 1-second budget for a process to exit and be disposed, on a runner already running tests at Workers: 4.

A fourth failure, Mp4SinkWriterEncoder_RealEncoderCoversValidationAndSuccessfulComplete (MF_E_SINK_NO_SAMPLES_PROCESSED), also recurred. That one is already tracked in #834 and is not part of this report.

To Reproduce

  1. Push any branch, or look at recent main runs.
  2. Watch build-package.yml -> build-and-package.
  3. Re-run the failed job. It usually goes green, and a later run fails on a different test.

Recent build-package.yml history showing the pattern across branches:

failure  09-16T21:54  nmetulev-typed-ui-geometry-context
success  09-16T21:54  nmetulev-deterministic-explicit-ui-actions
failure  09-16T21:23  azchohfi-esrp-sign-exes-before-packaging
success  09-16T21:11  main
failure  09-16T21:10  main
success  09-16T21:09  main
failure  09-16T21:08  dependabot/nuget/src/winapp-Analyzer/...
success  09-16T19:55  azchohfi-exercise-release-connections
failure  09-16T19:54  azchohfi-exercise-release-connections

Note main failing at 21:10 between two successes, and the same branch failing then passing a minute apart.

Expected behavior

build-and-package passes deterministically on a commit that has not changed the code under test, so a red build means a real regression.

Suggested directions, roughly in order of value:

  • (2) Assert OperationCanceledException instead of ThrowsExactlyAsync<TaskCanceledException>, or assert on the message/diagnostics the test actually cares about. This one looks like a straightforward correctness fix rather than infrastructure tuning.
  • (3) Replace the fixed 1-second SpinUntil with a generous timeout, or await an explicit signal that the process was released.
  • (1) Decide whether these four E2E tests should depend on a live dotnet add package against nuget.org at all. If they must, they need retry with backoff; if not, pin against a local feed. As written, any nuget.org hiccup turns into a red build on every open PR.

There is a secondary effect worth noting: because build-cli.ps1 aborts at the C# test step, the scripts/tests Pester suite downstream of it never executes. A flake in the C# tests silently skips the PowerShell test coverage entirely, so those suites are not actually being validated in CI on any run that fails this way.

OS Version and details

GitHub-hosted windows-latest runner. Tests run with Test Parallelization enabled (Workers: 4, Scope: MethodLevel).

Activity

  1. azchohfi commented on Sep 30, 2026

    @azchohfi
    CollaboratorAuthor

    Root cause for flake #1 (EndToEndTests / winapp init failing). This is not fixed in the PR for #2 and #3 and needs a separate decision.

    What happened: it was not a network problem. In job 105005563144, winapp init --setup-sdks stable got Microsoft.WindowsAppSDK 2.5.1 as the newest version from nuget.org. dotnet add package then failed:

    error: NU1102: Unable to find package Microsoft.WindowsAppSDK.Runtime with version (= 2.5.1)
    error:   - Found 32 version(s) in nuget.org [ Nearest version: 2.4.1-experimental ]
    

    This happened while WinAppSDK 2.5.1 was being published. nuget.org already listed the top-level package but not yet its Microsoft.WindowsAppSDK.Runtime dependency. Both are available now, which is why re-runs pass. The restore log shows the Runtime version list came from NuGet's HTTP cache (CACHE .../microsoft.windowsappsdk.runtime/index.json). That cache lasts 30 minutes, so retrying within the same run would not have helped.

    Why it matters: the same thing can happen during every future WinAppSDK release, for all four tests that call init with the newest SDK.

    Options for a follow-up:

    • A. When init fails with NU1102 for a Microsoft.WindowsAppSDK.* dependency at the exact version it just picked, mark the test Inconclusive. The tests keep checking the newest SDK, and any other failure still fails the build.
    • B. Use a fixed, known-good WinAppSDK version in these tests. This is fully deterministic, but they would no longer check the newest SDK, and the version needs occasional bumps.
    • C. Retry with backoff and a new NuGet HTTP cache each attempt. This is slower, and a publish window can outlast the retries.

    Secondary effect (Pester skipped when a C# test fails): already fixed on main. CI now runs the Auxiliary suite, which owns both Pester suites, as its own job separate from the CLI test shards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions