Fix macOS Process hangs caused by stopped children by steveisok · Pull Request #135323 · dotnet/runtime

steveisok · 2026-10-07T01:51:36Z

On macOS, waitid(WEXITED | WNOHANG | WNOWAIT) can report a stopped child even though waitpid(WNOHANG) cannot reap it. The SIGCHLD handler repeatedly observes the same child while holding shared process-management locks, blocking other Process operations.

Fall back to scanning all tracked children when the reported child cannot be reaped, rather than continuing the no-progress loop. This guard lives in the shared Unix implementation; successful reaping retains the existing fast path, and the scan reuses the existing nonblocking fallback for unknown children.

Add isolated regression coverage for synchronous and asynchronous exit waits while another child is stopped.
Restore class-level parallelism for Process tests on macOS by removing the temporary serialization safeguard.
Preserve the existing RemoteExecutor isolation changes and timeouts.

Resolves #135294

Note

This pull request description and code changes were generated with GitHub Copilot assistance.

Fall back to scanning tracked children when waitid reports a child that cannot be reaped, avoiding a SIGCHLD livelock with stopped children on macOS. Add isolated synchronous and asynchronous regression coverage and restore class-level test parallelism on macOS. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-10-07T01:52:21Z

Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-10-07T01:52:45Z

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-10-07T02:12:57Z

cc @tmds

tmds

LGTM!

adamsitnik

The fix is very simple and provides reliable test 👍

Before I hit the approve button, I need to get a final confirmation of how exactly it works (I need to understand risk, as I want to backport it to .NET 11)

Big thanks for your help @steveisok and @tmds !

Document the omitted WSTOPPED option and the SIGCHLD handler's SA_NOCLDSTOP registration without changing behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

adamsitnik

LGTM, big thanks for providing the fix @steveisok !

adamsitnik · 2026-10-08T05:49:17Z

/backport to release/11.0

github-actions · 2026-10-08T05:49:26Z

Started backporting to release/11.0 (link to workflow run)

@adamsitnik

…35409) Backport of #135323 and #135624 to release/11.0 /cc @adamsitnik @steveisok ## Customer Impact - [x] Customer reported - [x] Found internally Initially reported by the SDK (#131944), but other customers have experienced it as well (#131944 (comment)). When user calls `Process.Kill(entireProcessTree: true)` on macOS, the application can hang. ## Regression The changes we have made in #128598 have surfaced a macOS bug we were not aware of. It's a bug in the OS itself that other frameworks need to workaround as well (python/cpython#158962). Full details are explained [here](#135323 (comment)) and [here](#135624 (comment)). ## Testing Failing test was added and it's now passing. ## Risk Low. When we can't reap given process (because it was stopped, not killed) we fall back to checking individual processes rather than keep trying forever. fixes #131944 --------- Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Tom Longhurst <30480171+thomhurst@users.noreply.github.com>

## Description Restore the macOS `Aspire.Cli.Tests` lane disabled by #21385 while retaining the pinned .NET 11 RC2 toolchain. Aspire's process-heavy CLI tests are impacted by the stopped-child process-management livelock described in [dotnet/runtime#135294](dotnet/runtime#135294). The upstream fix is merged in [dotnet/runtime#135323](dotnet/runtime#135323), but we cannot yet consume a build containing it. Related to #21259. ### Evidence and workaround The unmodified suite and a separate parallel process-test control both hung locally. The test host had stopped children and zombie children, and a native sample showed the `.NET SigHandler` repeatedly alternating between `SystemNative_WaitIdAnyExitedNoHangNoWait` and `SystemNative_WaitPidExitedNoHang`. Externally killing the identified stopped children released an unrelated `git --version` process start that had been blocked for approximately 118 seconds. This strongly connects the reproduced Aspire hang to the upstream mechanism; it does not establish that every historical CI hang has the same cause. A usable managed lock-owner dump was not obtained. Re-enable `RunOnGithubActionsMacOS` and select the existing `tests/helix/xunit.runner.json` only when building CLI tests on macOS. That config disables xUnit test-collection parallelism, removing inter-collection process-management concurrency while retaining all existing test cases and assertions. Windows/Linux runner settings, macOS native CLI archives, E2E, other test projects, SDK pins, exclusions, and timeouts are unchanged. **This is test containment, not a runtime repair.** Within-test process races remain possible, and production Aspire CLI processes on the affected runtime remain potentially exposed. No production incident is claimed to have been reproduced or fixed. Remove the macOS serialization override after adopting and validating a runtime containing dotnet/runtime#135323. ### Validation Local environment: macOS 26.7.1 (25G241), ARM64; SDK `11.0.100-rc.2.26504.105`. The test host was observed loading runtime `11.0.0-rc.2.26504.105` from the repository-local `.dotnet` directory. Three sequential full-suite runs, with different test-order seeds and no parallelism command-line override, all passed inside the existing 20-minute CI budget: | Seed | Passed | Failed | Existing skips | Total | Duration | |---|---:|---:|---:|---:|---| | 43 | 8,874 | 0 | 68 | 8,942 | 11m 45.491s | | 44 | 8,874 | 0 | 68 | 8,942 | 11m 27.924s | | 45 | 8,874 | 0 | 68 | 8,942 | 14m 22.506s | `./restore.sh` — passed. Full-suite command, run for each of seeds `43`, `44`, and `45` — passed with the results above: ```bash HOME="$short_home" DOTNET_CLI_HOME="$short_home" \ NUGET_PACKAGES="$package_cache" \ DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 DOTNET_GENERATE_ASPNET_CERTIFICATE=false \ dotnet test --project tests/Aspire.Cli.Tests/Aspire.Cli.Tests.csproj \ --no-launch-profile -- --seed "$run" \ --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true" \ --hangdump --hangdump-timeout 2m --hangdump-type Mini --timeout 20m \ --report-trx --results-directory "$evidence/config-full-$run" ``` `short_home` was a securely created short, canonical `/private/tmp` directory; `package_cache` retained the existing package cache. This isolated unrelated user NuGet configuration and avoided macOS Unix-socket path-length limits. Earlier validation-environment trials failed or were stopped because of user NuGet configuration, an overlong isolated-home socket path, or `/tmp` symlink-relative paths; those are not counted as passes, and no product code or assertions were changed to accommodate them. Ten serialized process-heavy repetitions passed (107 passed / 14 skipped each), as did three expanded process-surface repetitions (211 passed / 17 skipped each). After the final one-file edit, another expanded process check passed (211 passed / 17 skipped, seed 114). The existing serialized config selected by the final change is byte-identical to the candidate used for the full repetitions; the rebuilt output config was also byte-compared successfully. Discovery display names were unchanged. ```bash dotnet test --project tests/Infrastructure.Tests/Infrastructure.Tests.csproj \ --no-launch-profile -- --filter-namespace "Infrastructure.Tests.TestTriggerMap" \ --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true" ``` Passed: all 340 selector infrastructure tests. Final selector tracing attributes the single changed project file to `Aspire.Cli.Tests`, with no unattributed files. `git diff --check` passed. Hosted GitHub macOS CI has not yet been validated. Windows/Linux execution was not rerun on this macOS host. Repeated local success supports this containment workaround, not a guarantee that the defective runtime can no longer hang. ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [ ] Yes - [x] No - Did you add public API? - [ ] Yes - If yes, did you have an API Review for it? - [ ] Yes - [ ] No - Did you add `<remarks />` and `<code />` elements on your triple slash comments? - [ ] Yes - [ ] No - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - If yes, have you done a threat model and had a security review? - [ ] Yes - [ ] No - [x] No Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

dotnet-policy-service Bot assigned steveisok Oct 7, 2026

github-actions Bot added the area-System.Diagnostics.Process label Oct 7, 2026

steveisok requested a review from a team October 7, 2026 01:59

jkotas requested a review from adamsitnik October 7, 2026 02:11

This was referenced Oct 7, 2026

HttpMetricsTest_Http20.TimeInQueue_RecordedForNewConnectionsOnly test failure #122522

Open

System.Net.Sockets.Tests tests fail with ObjectDisposedException #126183

Closed

SslStreamTlsResumeTests.DifferentEncryptionPolicy_NoResume failing #133418

Closed

tmds approved these changes Oct 7, 2026

View reviewed changes

tmds reviewed Oct 7, 2026

View reviewed changes

Comment thread src/libraries/System.Diagnostics.Process/src/System/Diagnostics/ProcessWaitState.Unix.cs Resolved

adamsitnik reviewed Oct 7, 2026

View reviewed changes

Clarify macOS stopped-child reaping comment

6e21914

Document the omitted WSTOPPED option and the SIGCHLD handler's SA_NOCLDSTOP registration without changing behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

adamsitnik approved these changes Oct 8, 2026

View reviewed changes

adamsitnik merged commit 44a22e2 into dotnet:main Oct 8, 2026
86 of 88 checks passed

This was referenced Oct 8, 2026

Fix Process.Kill(entireProcessTree: true) hang on macOS (#131944) #133930

Closed

Fix ptrace stop misclassification and ownership-gated draining in CheckChildren #133948

Closed

github-actions Bot mentioned this pull request Oct 8, 2026

[release/11.0] Fix macOS Process hangs caused by stopped children #135409

Merged

2 tasks done

dotnet-maestro Bot mentioned this pull request Oct 9, 2026

[main] Source code updates from dotnet/runtime dotnet/dotnet#10100

Merged

This was referenced Oct 11, 2026

Test failure: System.Diagnostics.Tests.ProcessWaitingTests.WaitAsyncChain #118977

Closed

Give the nested process chain in ProcessWaitingTests depth-proportional wait budgets #135608

Merged

dotnet-milestone-bot Bot added this to the 12.0-preview1 milestone Oct 13, 2026

adamsitnik mentioned this pull request Oct 13, 2026

[release/10.0] Backport SIGCHLD fixes #135676

Open

2 of 4 tasks

elsa-workflows-codex Bot mentioned this pull request Oct 19, 2026

Task: Qualify native execution hosting with real-process conformance elsa-workflows/elsa-foundation#3056

Closed

10 tasks done

radical mentioned this pull request Oct 21, 2026

Validate the stopped-child runtime fix and contain recurring macOS CLI test-host hangs microsoft/aspire#21259

Open

1 of 6 tasks

mitchdenny mentioned this pull request Oct 29, 2026

Restore macOS CLI tests with serialized collections microsoft/aspire#21389

Merged

4 of 14 tasks

Navigation Menu

Repository navigation

Fix macOS Process hangs caused by stopped children - #135323

Fix macOS Process hangs caused by stopped children#135323
adamsitnik merged 2 commits into
dotnet:mainfrom
steveisok:steveisok-macos-process-hangs

steveisok commented Oct 7, 2026

Uh oh!

azure-pipelines Bot commented Oct 7, 2026

Uh oh!

dotnet-policy-service Bot commented Oct 7, 2026

Uh oh!

jkotas commented Oct 7, 2026

Uh oh!

tmds left a comment

Uh oh!

Uh oh!

adamsitnik left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamsitnik left a comment

Uh oh!

Uh oh!

adamsitnik commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

steveisok commented Oct 7, 2026

Uh oh!

azure-pipelines Bot commented Oct 7, 2026

Uh oh!

dotnet-policy-service Bot commented Oct 7, 2026

Uh oh!

jkotas commented Oct 7, 2026

Uh oh!

tmds left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adamsitnik commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants