Skip to content

[ci-scan] Hang: System.Diagnostics.Process.Tests times out on windows-x64 #131389

New issue
New issue
Closed as not planned
Closed as not planned
[ci-scan] Hang: System.Diagnostics.Process.Tests times out on windows-x64#131389
Labels
Known Build ErrorUse this to report build issues in the .NET Helix tabPriority:1Work that is critical for the release, but we could probably ship withoutagentic-workflowsarea-Infrastructure
Milestone
Future

Description

@github-actions
github-actions
bot
opened on Jul 26, 2026

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1526411
Build error leg or test failing: windows-x64 Release NativeAOT_Libs-System.Diagnostics.Process.Tests
Reproduced in follow-up build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1526686

Error Details

The entire System.Diagnostics.Process.Tests work item hangs on the windows-x64 Release NativeAOT_Libs leg of runtime-nativeaot-outerloop. The assembly starts, runs its [SKIP] set, then the process never completes and the Helix executor kills it at the 900s timeout. No [FAIL] line is emitted — the work item exits via executor timeout (exit code -3).

Running assembly:System.Diagnostics.Process.Tests, Version=11.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
[SKIP] System.Diagnostics.Tests.ProcessCollectionTests.TestThreadCollectionBehavior
...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 900 seconds and was killed
['System.Diagnostics.Process.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Error Message

{
  "ErrorMessage": ["Running assembly:System.Diagnostics.Process.Tests", "Executor timed out after 900 seconds and was killed"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Filed by ci-failure-scan (detection only). ci-failure-fix walks open [ci-scan] KBEs and either opens a small fix PR or comments here to loop in owners — it never disables the test.

Note

This issue was generated by an AI agent (GitHub Copilot / ci-failure-scan).

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by CI Outer-Loop Failure Scanner · 748.4 AIC · ⌖ 17.5 AIC · ⊞ 18.6K · ◷

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1526411
Error message validated: [Running assembly:System.Diagnostics.Process.Tests Executor timed out after 900 seconds and was killed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/28/2026 7:48:05 AM UTC

Activity

  1. github-actions
    added
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    agentic-workflows
    on Jul 26, 2026
  2. dotnet-policy-service
    added
    untriagedNew issue has not been triaged by the area owner
    on Jul 26, 2026
  3. MichalStrehovsky
    added
    area-System.Diagnostics.Process
    on Jul 27, 2026

  4. dotnet-policy-service commented on Jul 27, 2026

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on Jul 27, 2026 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
    See info in area-owners.md if you want to be subscribed.

  5. MichalStrehovsky commented on Jul 27, 2026

    @MichalStrehovsky
    MichalStrehovsky
    on Jul 27, 2026
    Member

    All the hangs are on server core 2019 LTSC:

    Console log: 'System.Diagnostics.Process.Tests' from job 98af1b89-8b1e-4854-889d-7c839a934c37 (windows.10.amd64.serverrs5.open.rt) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2019-helix-amd64@sha256:428be031e034d665d819b5d3309043b5e976c9648c2dfa9f666c628978f768f4 on a00D3CC
running %HELIX_CORRELATION_PAYLOAD%\scripts\ada2e79421634729ac42db2bcb3fdd9b\execute.cmd in C:\h\w\AAA3090F\w\A1B708C9\e max 900 seconds

Console log: 'System.Diagnostics.Process.Tests' from job 2521f281-69ea-48b9-975c-fae9e160cbb1 (windows.10.amd64.serverrs5.open.rt) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2019-helix-amd64@sha256:428be031e034d665d819b5d3309043b5e976c9648c2dfa9f666c628978f768f4 on a00D2KE
running %HELIX_CORRELATION_PAYLOAD%\scripts\f62d1b987a5348ce983920782745d993\execute.cmd in C:\h\w\A517097D\w\AE690992\e max 900 seconds

Console log: 'System.Diagnostics.Process.Tests' from job e12e257d-fd7f-464e-a7dd-0f7bb6d027d7 (windows.10.amd64.serverrs5.open.rt) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2019-helix-amd64@sha256:428be031e034d665d819b5d3309043b5e976c9648c2dfa9f666c628978f768f4 on a00D33B
running %HELIX_CORRELATION_PAYLOAD%\scripts\a401cac4742b463d968fae927b901548\execute.cmd in C:\h\w\B98A0A0D\w\9DE10942\e max 900 seconds

    This is likely not native AOT specific, I just don't know where we run libs tests on obscure Windows SKUs with CoreCLR to double check whether we're seeing it there. It does sound like something "Add Signal, WaitForExitStatus, TryWaitForExitStatus, WaitForExitStatusAsync to Process" PR #131165 could introduce.

  6. github-actions commented on Jul 28, 2026

    @github-actions
    github-actionsBot
    on Jul 28, 2026 – with GitHub Actions
    ContributorAuthor

    Workflow artifact: ci-fix
    Artifact kind: handoff
    Linked KBE: #131389

    Note

    AI/Copilot-generated triage note (ci-failure-fix).

    This Known Build Error has no producible automated code change (reason: non-code/hang failure with no isolable faulting test), so I could not open even a best-effort PR. Looping in owners so it can be fixed forward rather than muted.

    Root cause (best analysis)

    The entire System.Diagnostics.Process.Tests work item hangs on the windows-x64 Release NativeAOT_Libs leg of runtime-nativeaot-outerloop. The assembly starts, runs its [SKIP] set, then never completes and the Helix executor kills it at the 900s timeout (Exit Code:-3, [EXECUTION TIMED OUT]). No [FAIL] line is emitted, so no single test is attributable from the log alone — the hang is between the last emitted test line and executor teardown.

    Because the hang manifests only on the NativeAOT leg (not on the CoreCLR/Mono legs of the same suite), the most likely area is a NativeAOT-specific difference in process/child-process handling or a test that blocks indefinitely waiting on a child process, standard-stream drain, or WaitForExit that behaves differently under NativeAOT. A targeted next step is to bisect the assembly (e.g. run with per-test result reporting / -notrait narrowing or an xunit diagnostic message sink) to identify which Process test stops progressing, then confirm whether the stall is in product code (NativeAOT process interop) or in the test's wait logic. Disabling the whole assembly is not appropriate; the individual hanging test must be identified first.

    Evidence

    Suggested reviewers / area contacts

    • Likely author: none identified with high confidence.
    • Area owners (area-System.Diagnostics.Process): @SamMonoRT, @dotnet/area-system-diagnostics-process

    A fix would likely involve identifying the single hanging Process test (via per-test bisection on the NativeAOT leg) and either correcting a NativeAOT-specific process-interop stall or a test-side unbounded wait, then adding a bounded timeout to the wait rather than muting the test.

    Warning

    Firewall blocked 1 domain

    The following domain was blocked by the firewall during workflow execution:

    • awmgmcpg

    To allow these domains, add them to the network.allowed list in your workflow frontmatter:

    network:
  allowed:
    - defaults
    - "awmgmcpg"

    See Network Configuration for more information.

    Generated by CI Outer-Loop Failure Fixer · 202.4 AIC · ⌖ 17.1 AIC · ⊞ 17.1K · ◷

  7. MichalStrehovsky commented on Jul 28, 2026

    @MichalStrehovsky
    MichalStrehovsky
    on Jul 28, 2026
    Member

    Yep also hangs with CoreCLR, not native AOT specific:

    https://dev.azure.com/dnceng-public/public/_build/results?buildId=1527106&view=logs&j=2686528b-6bb9-52cf-c87c-9a2bbdff7996&t=5c32730d-9d2b-5d94-bfff-01026160003a

  8. MichalStrehovsky
    changed the title [-][ci-scan] Hang: System.Diagnostics.Process.Tests times out on windows-x64 NativeAOT_Libs[/-] [+][ci-scan] Hang: System.Diagnostics.Process.Tests times out on windows-x64[/+] on Jul 28, 2026
  9. JulieLeeMSFT
    added
    Priority:1Work that is critical for the release, but we could probably ship without
    on Aug 4, 2026

  10. JulieLeeMSFT commented on Aug 4, 2026

    @JulieLeeMSFT
    JulieLeeMSFT
    on Aug 4, 2026
    Member

    @steveisok, this test is failing every day. PTAL.

  11. steveisok commented on Aug 4, 2026

    @steveisok
    steveisok
    on Aug 4, 2026
    Member

    @steveisok, this test is failing every day. PTAL.

    I suspect it's an image issue and we need to land #131411

    I triggered the outerloop pipelines there and will see if that solves the problem.

  12. SamMonoRT commented on Aug 18, 2026

    @SamMonoRT
    SamMonoRT
    on Aug 18, 2026
    Member

    can we re-create the hit rates since #129863 was merged?

  13. SamMonoRT
    added
    area-Infrastructure
    on Aug 18, 2026
  14. github-project-automation
    added this to Runtime Infraon Aug 18, 2026
  15. SamMonoRT
    removed
    area-System.Diagnostics.Process
    on Aug 18, 2026

  16. SamMonoRT commented on Aug 18, 2026

    @SamMonoRT
    SamMonoRT
    on Aug 18, 2026
    Member

    Moving to infra area label, not specific to the test.

  17. dotnet-policy-service commented on Aug 18, 2026

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on Aug 18, 2026 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @dotnet/runtime-infrastructure
    See info in area-owners.md if you want to be subscribed.

  18. steveisok commented on Aug 18, 2026

    @steveisok
    steveisok
    on Aug 18, 2026
    Member

    Thanks for the reminder - I have #131907 that I need to clean up the description and elevate. I think there is a infra or windows issue that is causing the timeout.

  19. steveisok
    added this to the Future milestone on Aug 18, 2026
  20. dotnet-policy-service
    removed
    untriagedNew issue has not been triaged by the area owner
    on Aug 19, 2026
  21. jeffschwMSFT
    removed
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    on Sep 21, 2026

  22. jeffschwMSFT commented on Sep 21, 2026

    @jeffschwMSFT
    jeffschwMSFT
    on Sep 21, 2026
    Member

    removing blocking-clean-ci as it has not failed in 30 days

    24-Hour Hit Count 7-Day Hit Count 1-Month Count
    0 0 0

  23. jkotas
    closed this as not plannedon Sep 21, 2026
  24. github-project-automation
    moved this to Done in Runtime Infraon Sep 21, 2026
  25. github-actions
    locked and limited conversation to collaborators on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Known Build ErrorUse this to report build issues in the .NET Helix tabPriority:1Work that is critical for the release, but we could probably ship withoutagentic-workflowsarea-Infrastructure

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions