You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Wait for an orphaned coordinated build before building again - #21622
Fixes the CLI Starter Validation flake tracked on #21600, where aspire-starter: aspire wait for resource apiservice failed with exit code 18 (WaitResourceFailed).
Root cause
cli-starter-validation.ps1 runs aspire run --detach, then aspire stop right away, then aspire start and aspire wait apiservice. Three things combine to cause the failure:
aspire stop returns as soon as the AppHost process exits. DCP (start-apiserver --monitor <dead pid>) cleans up its executables asynchronously after that.
The coordinated __dotnet-project-build (dotnet build obj/.aspire/build/projects.<hash>.proj) ignores SIGTERM and runs to completion. Locally, kill -TERM did not stop it and it exited 0.
The new AppHost started by aspire start launches a second dotnet build of the same projects in the same build directory while the orphaned build is still running.
The two builds then collide on shared intermediate outputs. The new build fails, WaitForSuccessfulBuildAsync fails apiservice/webfrontend, and aspire wait exits with code 18.
CI evidence (last 7 days, main and PRs):
5 failures out of 4,565 CLI Starter Validation jobs. Runs: 40937292091, 40958253916, 41167202562, 41197867795, 41204248633. Each passed on rerun.
I ran two overlapping dotnet build invocations of the same generated projects.proj across a sweep of start offsets. 2 of 11 runs failed with: error : The process cannot access the file 'Smoke.Web/obj/Debug/net10.0/compressed/<id>.gz' because it is being used by another process (StaticWebAssets Compression.targets(276,5))
Sequential builds never failed.
Fix
The coordinated build now writes an MSBuild file log into its build directory, with a per-AppHost-process name: -flp9:LogFile=<BuildDirectory>/logs/<pid>.<resource>.log;Verbosity=minimal. A build never opens a log that another build holds.
MSBuild's file logger holds that file open for writing for the whole lifetime of the build process. On Unix, .NET backs this with an flock (LOCK_SH). On Windows it's a sharing violation. A probe found the lock held in 62 of 62 checks during a build.
So an open handle on the log reliably means "a build is still running in this directory", even when the AppHost that launched it is gone.
Before launching, a new coordinated build tries to take an exclusive lock on every logs/*.log using the existing FileLock helper. Free logs are deleted when acquired.
If any log is held, it logs Waiting for a coordinated build that is still running in '...' to finish before building. and waits for it to be released.
The wait is bounded at 5 minutes. On timeout it logs a warning and builds anyway with its own log file, so behavior is never worse than today.
Once the previous build finishes, the new build is incremental.
Exclusions:
File-based app builds (dotnet build <file>.cs) are excluded because they reject MSBuild logger switches.
Build directories containing ; or = are excluded because MSBuild's file logger splits its parameters on both and can't express them.
No sleeps, retries or timeouts in tests or scripts were changed.
Tests
New test CoordinatedBuildWaitsForBuildStillRunningInSameBuildDirectory:
Holds another build's log the way MSBuild does (FileShare.Read writer) and leaves a finished log alongside it.
Asserts that argument evaluation for the coordinated build blocks and logs the waiting message.
Releases the file and asserts the build proceeds with its own -flp9 log and the finished log was deleted.
Without the fix, it fails (no wait happens, so the test times out with TaskCanceledException). With the fix, it passes.
New test CoordinatedBuildUsesItsOwnFileLogWhenRunningBuildOutlastsWait: when the wait times out, the build proceeds with its own log, which can be opened while the other build's log is still held.
New theory CoordinatedBuildOmitsFileLogWhenBuildDirectoryCannotBeALoggerParameter (; and =).
Updated coordinated-build argument expectations. File-app build expectations confirm the log is omitted.
Local run: DotnetProjectBuildCoordinatorTests + DotnetProjectResourceTests, 188/188 passed (including the real dotnet build/DCP file-app and restore tests).
E2E evidence
The CLI Starter Validation jobs exercise the real run --detach → stop → start → wait flow, which is the flow this flake hit. They all passed on the exact head 988d7f5579550a731ab9c92aee157c010ab53e20 (CI run 41256233278, 288/288 checks green, no reruns):
`aspire stop` returns once the AppHost exits, but DCP cleans up asynchronously
and `dotnet build` ignores SIGTERM, so a coordinated build launched by the
previous AppHost can keep running. A fresh AppHost (`aspire start`) then
launches a second build of the same projects, and the two collide on shared
intermediate outputs (e.g. static web asset compression sharing violations),
failing the build and every dependent resource. This is the CLI Starter
Validation flake where `aspire wait apiservice` exits with code 18.
The coordinated build now writes an MSBuild file log into its build
directory. MSBuild holds that file open for the lifetime of the build
process, so before launching, a new coordinated build waits (bounded) until
it can take an exclusive lock on the log. File-based app builds are excluded
because `dotnet build <file>.cs` rejects logger switches.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
After the five-minute timeout, the new build still targets the locked build.log, causing logger initialization to fail on Windows instead of allowing the intended fallback to proceed.
This PR addresses the CLI Starter Validation flake tracked in #21600 by waiting for an orphaned coordinated .NET build before starting another build in the same directory.
Changes:
Uses an MSBuild file log to detect an active build, with a five-minute bounded wait.
Excludes file-based apps and build paths containing semicolons from file logging.
Adds waiting and exclusion tests, and updates expected build arguments.
The still-running build owns the log; on Windows a second file logger on it
fails logger initialization instead of building anyway.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cross-ref from #21621 (VS Code Windows browser-debugger standalone-gateway FailedToStart right after __dotnet-project-build-2): each failing run has a single AppHost launch, so the orphaned-AppHost-build overlap fixed here doesn't apply. Instead, in 3/3 failures the C# extension's Roslyn BuildHost ran a design-time load of the same projects during the coordinated -mt build (details: #21621 (comment)). It's the same overlapping-build family, but IDE design-time builds don't take the build.log lock, so I'll keep #21621 separate and start by making coordinated build failures visible in the AppHost logs. No competing change to this PR.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The lock is released before the child build starts, allowing concurrent AppHosts to use the same log file and cause overlapping builds on Unix or logger sharing violations on Windows.
Each AppHost's coordinated build logs to logs/<pid>.<resource>.log and
waits for every log another build still holds, so a build that proceeds
after the wait timeout never opens a file logger on a held log.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Build directories containing = are not excluded, causing MSBuild to truncate the log path, potentially fail logger initialization, and leave the running-build probe without its expected log.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Exclude build directories containing '=' from logger paths
Build directories containing = also need to be excluded: MSBuild's FileLogger.ParseFileLoggerParameters splits on every = and takes only the first value segment, so /work/repo=feature/... becomes /work/repo. This writes to the wrong file or fails logger initialization if that prefix is a directory, and leaves no log where the running-build probe expects it. Treat = like ; and extend the exclusion test to cover it.
MSBuild's file logger splits each parameter on '=' and keeps only the
first value segment, so such a LogFile path would be truncated.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Re: the Copilot review note about build directories containing =: confirmed. MSBuild's FileLogger.ParseFileLoggerParameters splits each parameter on = and keeps only the first value segment. Fixed in 988d7f5: build directories containing ; or = now get no file log (and so no running-build probe), the same as before this PR. The exclusion test is now a theory covering both characters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
area-engineering-systemsinfrastructure helix infra engineering repo stuffci-healthFixes CI flakiness or CI infrastructure (AspireAgent queue)
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
the
area-engineering-systems
Copilot SWE Agent
Description
Fixes the CLI Starter Validation flake tracked on #21600, where
aspire-starter: aspire wait for resource apiservice failed with exit code 18(WaitResourceFailed).Root cause
cli-starter-validation.ps1runsaspire run --detach, thenaspire stopright away, thenaspire startandaspire wait apiservice. Three things combine to cause the failure:aspire stopreturns as soon as the AppHost process exits. DCP (start-apiserver --monitor <dead pid>) cleans up its executables asynchronously after that.__dotnet-project-build(dotnet build obj/.aspire/build/projects.<hash>.proj) ignores SIGTERM and runs to completion. Locally,kill -TERMdid not stop it and it exited 0.aspire startlaunches a seconddotnet buildof the same projects in the same build directory while the orphaned build is still running.The two builds then collide on shared intermediate outputs. The new build fails,
WaitForSuccessfulBuildAsyncfailsapiservice/webfrontend, andaspire waitexits with code 18.CI evidence (last 7 days, main and PRs):
5 failures out of 4,565 CLI Starter Validation jobs. Runs: 40937292091, 40958253916, 41167202562, 41197867795, 41204248633. Each passed on rerun.
Timeline from run 41204248633's CLI logs:
aspire stop).So the two builds overlapped.
Local repro:
aspire stopreturns success,psstill shows AppHost [main] Update dependencies from dotnet/arcade #1's DCP processes and itsdotnet build ... projects.<hash>.proj.dotnet buildinvocations of the same generatedprojects.projacross a sweep of start offsets. 2 of 11 runs failed with:error : The process cannot access the file 'Smoke.Web/obj/Debug/net10.0/compressed/<id>.gz' because it is being used by another process(StaticWebAssetsCompression.targets(276,5))Fix
-flp9:LogFile=<BuildDirectory>/logs/<pid>.<resource>.log;Verbosity=minimal. A build never opens a log that another build holds.flock(LOCK_SH). On Windows it's a sharing violation. A probe found the lock held in 62 of 62 checks during a build.logs/*.logusing the existingFileLockhelper. Free logs are deleted when acquired.Waiting for a coordinated build that is still running in '...' to finish before building.and waits for it to be released.dotnet build <file>.cs) are excluded because they reject MSBuild logger switches.;or=are excluded because MSBuild's file logger splits its parameters on both and can't express them.Tests
New test
CoordinatedBuildWaitsForBuildStillRunningInSameBuildDirectory:FileShare.Readwriter) and leaves a finished log alongside it.-flp9log and the finished log was deleted.Without the fix, it fails (no wait happens, so the test times out with
TaskCanceledException). With the fix, it passes.New test
CoordinatedBuildUsesItsOwnFileLogWhenRunningBuildOutlastsWait: when the wait times out, the build proceeds with its own log, which can be opened while the other build's log is still held.New theory
CoordinatedBuildOmitsFileLogWhenBuildDirectoryCannotBeALoggerParameter(;and=).Updated coordinated-build argument expectations. File-app build expectations confirm the log is omitted.
Local run:
DotnetProjectBuildCoordinatorTests+DotnetProjectResourceTests, 188/188 passed (including the realdotnet build/DCP file-app and restore tests).E2E evidence
The CLI Starter Validation jobs exercise the real
run --detach→stop→start→waitflow, which is the flow this flake hit. They all passed on the exact head988d7f5579550a731ab9c92aee157c010ab53e20(CI run 41256233278, 288/288 checks green, no reruns):