Skip to content

Add retry to NuGet setup tasks for node24 task-host libuv crash#406

Merged
dcristoloveanu merged 1 commit into
masterfrom
dcristo/retry-nugetauth-node24-crash
May 15, 2026
Merged

Add retry to NuGet setup tasks for node24 task-host libuv crash#406
dcristoloveanu merged 1 commit into
masterfrom
dcristo/retry-nugetauth-node24-crash

Conversation

@dcristoloveanu

Copy link
Copy Markdown
Member

Summary

Add retryCountOnTaskFailure: 3 to the two Node-handler setup tasks in this template (NuGetAuthenticate@1 and NuGetToolInstaller@1).

Why

Node 24 task-host wrappers on Windows ADO agents intermittently abort with Exit code 57005 (Watson bucket FAIL_FAST_FATAL_APP_EXIT_c0000409, faulting in vss-agent\externals\node24\bin\node.exe). Symbolicated against the Node 24.13.0 PDB, the stack ends in:

Assertion failed: !(handle->flags & UV_HANDLE_CLOSING)
file deps/uv/src/win/async.c, line 76

…which is libuv's uv_async_send() invoked from Node's WorkerThreadsTaskRunner::DelayedTaskScheduler shutdown path -- a lifecycle race between the task host's V8 platform tear-down and an in-flight async wakeup.

References:

The agent ships Node 24 (externals/node24) regardless of any pipeline UseNode@1 step -- the task-host runtime is selected purely from each task's target.runtimeVersion. The only currently available mitigation for consumers is to retry the affected wrapper invocations.

In Azure-MessagingStore we measured ~9 NuGetAuthenticate@1 / Npm@1 / UseNode@1 crashes across ~99 builds in the past 7 days (per-task failure rate ~9%). With retryCountOnTaskFailure: 3 the residual rate drops to <0.01%.

Why this is safe

  • NuGetAuthenticate@1 is idempotent (env-var setup + credential-provider probe).
  • NuGetToolInstaller@1 is idempotent (downloads + caches nuget.exe); already had continueOnError: true.
  • Retry triggers on any non-zero exit, but for these setup tasks the only meaningful "real" failures will simply re-fail through all 3 attempts and surface as before.

Rollback

Revert this commit once nodejs/node#61999 merges, Node ships it in a 24.x, the agent bumps NODE24_VERSION to that release, and the 1ES Hosted pool image rolls forward.

Node 24 task-host wrappers on Windows ADO agents intermittently abort with
Exit code 57005 (Watson bucket FAIL_FAST_FATAL_APP_EXIT_c0000409 in
vss-agent\externals\node24\bin\node.exe). Symbolicated stacks point at a
libuv UV_HANDLE_CLOSING race in uv_async_send() called from Node's
WorkerThreadsTaskRunner::DelayedTaskScheduler at shutdown:

    Assertion failed: !(handle->flags & UV_HANDLE_CLOSING)
    file deps/uv/src/win/async.c, line 76

Tracked upstream as nodejs/node#56645 (open; pending fix PR nodejs/node#61999)
and in the agent as microsoft/azure-pipelines-agent#5498. Until either fix
ships and the agent's externals/node24 picks it up, the only mitigation
available to consumers is to retry the affected setup tasks.

retryCountOnTaskFailure: 3 brings the residual per-task failure rate from
~9% (observed) to <0.01%. Both NuGetAuthenticate@1 and NuGetToolInstaller@1
are idempotent setup, so retrying on any failure is safe; legitimate
failures still surface after the retries are exhausted.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dcristoloveanu
dcristoloveanu marked this pull request as ready for review May 14, 2026 22:37
@dcristoloveanu
dcristoloveanu enabled auto-merge (squash) May 15, 2026 00:03
@dcristoloveanu
dcristoloveanu merged commit 67a7e8a into master May 15, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants