Aspire 13.6.0 always errors with "Watch task over Kubernetes ContainerExec resources terminated unexpectedly"

Is there an existing issue for this?

I have searched the existing issues

Describe the bug

After updating to Aspire 13.6.0 from 13.5.3, I get the below error message in the console window every single time I start the Aspire Host. It comes up after a minute of running, and doesn't have any other side effects; the application still runs fine and I can use it as normal.

crit: Aspire.Hosting.Dcp.DcpExecutor[0]
      Watch task over Kubernetes ContainerExec resources terminated unexpectedly.
      Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:01:00'.
       ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
         at System.Net.Http.Http2Connection.Http2Stream.<>c.<WaitForDataAsync>b__88_0(Object s, CancellationToken cancellationToken)
         at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
         at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
         at System.Threading.TimerQueue.FireNextTimers()
         at System.Threading.ThreadPoolWorkQueue.Dispatch()
         at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
         at System.Threading.Thread.StartCallback()
      --- End of stack trace from previous location ---
         at System.Net.Http.Http2Connection.Http2Stream.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
         at k8s.LineSeparatedHttpContent.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
         at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
         at System.IO.StreamReader.ReadLineAsyncInternal(CancellationToken cancellationToken)
         at k8s.LineSeparatedHttpContent.PeekableStreamReader.PeekLineAsync()
         at k8s.LineSeparatedHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context)
         at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, LimitArrayPoolWriteStream tempBuffer)
         at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc)
         at k8s.Kubernetes.CreateResultAsync[T](HttpRequestMessage httpRequest, HttpResponseMessage httpResponse, Nullable`1 watch, CancellationToken cancellationToken)
         at k8s.AbstractKubernetes.ICustomObjectsOperations_ListClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.ListClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass27_1`1.<<WatchAsync>b__1>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 292
      --- End of stack trace from previous location ---
         at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass33_0`1.<<ExecuteWithRetry>b__3>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 501
      --- End of stack trace from previous location ---
         at Polly.ResiliencePipeline.<>c__10`1.<<ExecuteAsync>b__10_0>d.MoveNext()
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
         at Polly.Utils.Pipeline.BridgeComponent`1.<ExecuteCore>g__ConvertValueTaskAsync|4_0[TResult,TState](ValueTask`1 valueTask, ResilienceContext resilienceContext)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
         at Polly.Utils.Pipeline.BridgeComponent`1.<>c__4`2.<<ExecuteCore>b__4_1>d.MoveNext()
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
         at Polly.ResiliencePipeline.<>c__10`1.<<ExecuteAsync>b__10_0>d.MoveNext()
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
         at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass33_0`1.<<ExecuteWithRetry>b__3>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 501
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception)
         at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass27_1`1.<<WatchAsync>b__1>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 292
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception)
         at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.ListClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception)
         at k8s.AbstractKubernetes.ICustomObjectsOperations_ListClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception)
         at k8s.Kubernetes.CreateResultAsync[T](HttpRequestMessage httpRequest, HttpResponseMessage httpResponse, Nullable`1 watch, CancellationToken cancellationToken)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception)
         at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
         at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, LimitArrayPoolWriteStream tempBuffer)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
         at k8s.LineSeparatedHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception)
         at k8s.LineSeparatedHttpContent.PeekableStreamReader.PeekLineAsync()
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception)
         at System.IO.StreamReader.ReadLineAsyncInternal(CancellationToken cancellationToken)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
         at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception)
         at k8s.LineSeparatedHttpContent.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
         at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecuteFromThreadPool(Thread threadPoolThread)
         at System.Threading.ThreadPoolWorkQueue.Dispatch()
         at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
         at System.Threading.Thread.StartCallback()
      --- End of stack trace from previous location ---
         at Polly.Outcome`1.GetResultOrRethrow()
         at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
         at Aspire.Hosting.Dcp.KubernetesService.ExecuteWithRetry[TResult](DcpApiOperationType operationType, String resourceType, Func`3 operation, Func`2 isRetryable, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 497
         at Aspire.Hosting.Utils.PeriodicRestartAsyncEnumerable.CreateAsync[T](Func`3 enumerableFactory, TimeSpan restartInterval, CancellationToken cancellationToken)+MoveNext()
         at Aspire.Hosting.Utils.PeriodicRestartAsyncEnumerable.CreateAsync[T](Func`3 enumerableFactory, TimeSpan restartInterval, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 305
         at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 305
         at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Aspire.Hosting.Dcp.DcpResourceWatcher.<>c__DisplayClass34_1`1.<<Start>b__15>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/DcpResourceWatcher.cs:line 206
      --- End of stack trace from previous location ---
         at Aspire.Hosting.Dcp.DcpResourceWatcher.<>c__DisplayClass34_1`1.<<Start>b__15>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/DcpResourceWatcher.cs:line 206
      --- End of stack trace from previous location ---
         at Polly.ResiliencePipeline.<>c.<<ExecuteAsync>b__3_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Polly.Outcome`1.GetResultOrRethrow()
         at Polly.ResiliencePipeline.ExecuteAsync(Func`2 callback, CancellationToken cancellationToken)
         at Aspire.Hosting.Dcp.DcpResourceWatcher.<>c__DisplayClass34_0.<<Start>g__WatchKubernetesResourceAsync|3>d`1.MoveNext() in /_/src/Aspire.Hosting/Dcp/DcpResourceWatcher.cs:line 204

Expected Behavior

In Aspire 13.5.3, it would start fine without any errors in the console window

Steps To Reproduce

Just starting up Aspire Host app running 13.6.0

Exceptions (if any)

See above

Aspire doctor output

David Negstad (@danegsta)

🤖 Aspire triage pass — 2026-09-30 16:29 UTC

Area: area-orchestrator
Kind: regression
Repro: partial — see lines 208–216 of the issue body
Affected version(s): 13.6.0 (last working: 13.5.3; latest: 13.6.0)
Severity: S4 — a recurring critical console error is noisy, but the reported application remains usable with no other observed effects

Urgency: ⚪ backlog — no milestone
Reason: valid regression report, but the reported impact is limited and the reproduction details are not sufficient to prioritize it for 13.6

Suggested owner(s): David Negstad (@danegsta) (score 28)
Evidence:

danegsta merged Avoid ephemeral executable target port collisions #19822 in area-orchestrator, including DCP executor changes
danegsta closed Track potential macOS NativeAOT service launch failure #20291 and [Aspire CLI] aspire stop --force failing due to race condition #19884 as completed in area-orchestrator
CODEOWNERS lists David Negstad (@danegsta) for src/Aspire.Hosting/**

Confidence: medium
Needs human?: yes — confirm owner availability and obtain fuller reproduction details

We hit this on 13.6.0, and for us it is much more than console noise, so I'd like to add what we found. It also shows that #20664 is needed in 13.6.1.

Impact: project resources stay in `Waiting` forever

The same Polly.Timeout.TimeoutRejectedException ("didn't complete within the allowed timeout of '00:01:00'") and Watch task over Kubernetes … resources terminated unexpectedly message appear for Executable and Endpoint watches as well, not only ContainerExec. Once the Executable watch is dead, the AppHost never observes DCP starting project resources:

DCP does start them. Its logs show the processes Running and Healthy, and their endpoints registered.
The AppHost never logs a state change for them, never runs their health checks, and never requests their logs (zero Kind=Executable log-stream requests in the DCP API server log). They stay Waiting until something times out.
In our case that is DistributedApplicationTestingBuilder-based integration tests timing out after 8 minutes, but any WaitFor chain downstream of a slow container is affected.

When it happens

When no executable exists yet 60 seconds after startup (typically because a container they WaitFor is still pulling or initializing). This is invisible on fast machines. We hit it consistently on GitHub-hosted macOS runners (Docker in a Lima/QEMU VM: image pull takes 3+ min, MySQL/PostgreSQL/RabbitMQ init another 2-3 min), while the same tests pass on 13.5.x and on Windows/Linux runners.

The watch can also die later: even without the 60-second timeout, the 5-minute periodic restart (PeriodicRestartAsyncEnumerable) kills a watch that is still waiting for its first event when the restart fires. Several of our runs created the first executable between 5 and 7 minutes after startup, so this is not theoretical.

Minimal repro (13.6.0)

A container whose health is delayed by 90 seconds, and a project that waits for it:

var start = DateTime.UtcNow;
builder.Services.AddHealthChecks().AddCheck("slow-start", () =>
    DateTime.UtcNow - start > TimeSpan.FromSeconds(90)
        ? HealthCheckResult.Healthy()
        : HealthCheckResult.Unhealthy());

var valkey = builder.AddValkey("cache").WithHealthCheck("slow-start");
builder.AddProject<Projects.Api>("api").WithReference(valkey).WaitFor(valkey);

About a minute in, the crit messages are logged, and api never leaves Waiting even though the process is running.

Verified: #20664 fixes it

We built v13.6.0 plus only the two src/Aspire.Hosting hunks of #20664 (Aspire.Hosting built with AssemblyVersion=13.6.0.0, swapped into the test output) and ran the same repro with the official assembly as a control:

Assembly	Health delay	Result
official 13.6.0	90 s	fails: `crit` messages, `api` stuck in `Waiting`
13.6.0 + #20664	90 s	passes, no `crit` messages
13.6.0 + #20664	390 s (crosses the 5-minute restart)	passes, no `crit` messages

So both parts of the PR matter: the per-attempt timeout and the factory handling in PeriodicRestartAsyncEnumerable. (Tested on Windows; the build used -p:DefaultTargetFramework=net10.0 because the repo requires an SDK 11 preview.)

We also see the effect of an equivalent workaround on the real macOS runners (see below): with the watch kept alive, executables created 5 min 25 s after startup (past both the 60-second timeout and the 5-minute restart) are observed by the AppHost and reach Running (Healthy).

Request

Could this be reconsidered for 13.6.1? The triage assessment (S4, "noisy but usable") does not match what we see, and the Customer Impact section of #20664 only mentions container exec and terminal updates. The release/13.6 branch is already versioned 13.6.1, and neither this issue nor #20664 has a milestone yet.

Two more problems we found while working around this

With the watches kept alive, our macOS runs got further and then failed on the next resource, which is a different problem from this issue. We filed them separately:

A single timed-out or 504 DCP Create call permanently fails the resource (FailedToStart); not retried since 13.6.0 #20691: a single timed-out or 504 Create call permanently fails the resource (not retried).
API server: creating an Executable hangs 30-45 s (504) while the log storage restarts its parent resource watcher for Executables dcp#286: DCP API server blocks creating an Executable for 30-45 s while its log storage restarts the Executable parent watcher.

Neither is fixed by #20664, so even with this PR our macOS runs can still fail occasionally. They are not a reason to hold the PR back.

Workaround we use until a fixed version ships (reflection against internals, 13.6.0 only)

The API timeouts below cannot be configured today (see #20691), hence the reflection. Called after building the DistributedApplicationTestingBuilder app and before starting it, it extends the API timeouts and replaces the watcher's retry pipeline with one that restarts a terminated watch (a new watch receives the existing resources again):

private static void WorkAroundDcpTimeouts(IServiceProvider services, TimeSpan timeout)
{
    const BindingFlags Flags = BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public;

    Type? kubernetesServiceType = typeof(DistributedApplication).Assembly.GetType("Aspire.Hosting.Dcp.IKubernetesService");
    object? kubernetesService = kubernetesServiceType == null ? null : services.GetService(kubernetesServiceType);

    if (kubernetesService != null)
    {
        foreach (string propertyName in new[] { "MaxRetryDuration", "KubernetesInitializationTimeout" })
        {
            kubernetesService.GetType().GetProperty(propertyName)?.SetValue(kubernetesService, timeout);
        }
    }

    Type? executorType = typeof(DistributedApplication).Assembly.GetType("Aspire.Hosting.Dcp.IDcpExecutor");
    object? executor = executorType == null ? null : services.GetService(executorType);
    object? watcher = executor?.GetType().GetProperty("ResourceWatcher", Flags)?.GetValue(executor);
    PropertyInfo? pipelineProperty = watcher?.GetType().GetProperty("WatchResourceRetryPipeline", Flags);

    if (watcher != null && pipelineProperty != null)
    {
        ResiliencePipeline pipeline = new ResiliencePipelineBuilder().AddRetry(new RetryStrategyOptions
        {
            ShouldHandle = new PredicateBuilder().Handle<Exception>(),
            BackoffType = DelayBackoffType.Constant,
            Delay = TimeSpan.FromSeconds(1),
            MaxRetryAttempts = int.MaxValue
        }).Build();

        pipelineProperty.SetValue(watcher, pipeline);
    }
}

I believe microsoft/dcp#286 is tracking the root cause of this. I've got an understanding of what's causing the issue; a fix for an out-of-order events issue made another pre-existing bug go from somewhat recoverable only because background threads were allowed to accumulate to an immediate error. I'll be submitting a PR with a fix shortly.

Update from our side, in case it helps readers of this issue.

With 13.6.1 (which contains #20664, with DCP 0.26.5) we no longer see the Watch task over Kubernetes ΓÇª terminated unexpectedly messages in our CI runs. We still see project resources stuck in Waiting on slow GitHub-hosted macOS runners, and the cause is the DCP API server stall tracked in microsoft/dcp#286: a watch that falls behind blocks the server from publishing further events of that kind, so creates and updates hang. The maintainers have confirmed the cause there and have a fix in progress in tilt-dev/tilt-apiserver#117 (a bounded queue per watcher that evicts a stalled consumer).

On the Aspire side: in healthy runs the AppHost's connection to the DCP API server carries 5 watches, one per kind. In the goroutine dumps of four stalled runs the blocked watch is on a connection with 7 or 8 watches, i.e. 2 or 3 more than the AppHost opens, and the extra ones appear after the AppHost's periodic 5-minute watch restart. So it is the AppHost's consumer that stops reading, and some watches from before the restart seem to stay open. That is inferred from the number of watches per HTTP/2 connection, not from a process ID. The details and a reproduction that only needs the API server are in the microsoft/dcp#286 comments.

One fact from the source that may be relevant, without my having verified that it is the cause: DcpResourceWatcher handles the events of all kinds one at a time, under a single _outputSemaphore that is held while the handler runs, so one slow handler leaves the watches of the other kinds unread as well. A small AppHost whose resource takes 5:45 minutes to become healthy did not reproduce the stall, so I cannot say what makes the consumer fall behind.

Keep DCP watches draining and close abandoned connections (#20972)

Thanks for #20972. Its description (a watch left open without an active reader when watch setup is canceled) matches what I see on 13.6.1 with DCP 0.26.5, so here is data from CI runs that may help verify it, and that supports backporting it to release/13.6.

What I see. In healthy runs the AppHost's HTTP/2 connection to the DCP API server carries exactly 5 watches, one per kind it watches (lsof shows the AppHost process has a single connection). When no Executable exists yet (the test host disables the dashboard, and the first project waits for a container that needs about 2 minutes to become healthy), the number of watches on that connection grows in steps of 60 seconds. In one failing run, measured from the start of the watches: 5 watches, then 7 at about 65 s, 9 at about 125 s, 10 at about 185 s. In a second failing run it went from 5 to 7 at about 70 s. apiserver_longrunning_requests{verb="WATCH"} shows the extra ones belong to executables and containerexecs, the two kinds that had no objects. This is not the 5-minute restart: the steps are 60 seconds apart, which matches the per-attempt timeout in KubernetesService.WatchAsync.

What it leads to. When the first executables are created, their events are also sent to those extra streams. In two failed runs the blocked watch in the API server's goroutine dump is on this connection, with one Create/Update blocked in WatchSet.notifyWatchers [chan send] (the symptom in microsoft/dcp#286). The create then times out after 20 seconds and the resource goes to FailedToStart.

What prevents it, today. Raising KubernetesService.MaxRetryDuration and KubernetesInitializationTimeout to 8 minutes (through reflection) keeps the connection at 5 watches. In 23 consecutive runs of the slowest sample (first executable 80 to 153 seconds after DCP started, so past the 60 seconds) the AppHost's connection never carried more than 5 watches and no dump showed a blocked publisher. Without it, two consecutive runs of that sample each failed with the pattern above.

Not specific to slow image pulls. Pre-pulling the images in the build removes the 5-minute case we first saw, but not this one: a MySQL container needs about 2 minutes to become healthy by itself, so the executables still appear after the 60 seconds. Anything that keeps a kind without objects for a minute triggers it.

What I do not know. Whether #20972 removes the leak is not verified on our side yet; I only know the behavior it is meant to fix. A bare AppHost with a resource that takes 5:45 minutes to become healthy did not stall, so the amount of event data matters too: the stall needs enough events on the extra streams to exhaust their flow-control window.

[release/13.6] Keep DCP watches draining and close abandoned connecti…

13.6.3 includes fixes that should resolve this issue. Let me know anyone is still hitting it after upgrading.

Navigation Menu

Repository navigation

Aspire 13.6.0 always errors with "Watch task over Kubernetes ContainerExec resources terminated unexpectedly" #20609

Is there an existing issue for this?

Describe the bug

Expected Behavior

Steps To Reproduce

Exceptions (if any)

Aspire doctor output

Impact: project resources stay in `Waiting` forever

When it happens

Minimal repro (13.6.0)

Verified: #20664 fixes it

Request

Two more problems we found while working around this

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Aspire 13.6.0 always errors with "Watch task over Kubernetes ContainerExec resources terminated unexpectedly" #20609

Description

Is there an existing issue for this?

Describe the bug

Expected Behavior

Steps To Reproduce

Exceptions (if any)

Aspire doctor output

Activity

joperezr commented on Sep 30, 2026

bart-vmware commented on Oct 2, 2026

Impact: project resources stay in Waiting forever

When it happens

Minimal repro (13.6.0)

Verified: #20664 fixes it

Request

Two more problems we found while working around this

danegsta commented on Oct 11, 2026

bart-vmware commented on Oct 13, 2026

bart-vmware commented on Oct 14, 2026

danegsta commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Impact: project resources stay in `Waiting` forever