Skip to content

Fix subchannel permanently stuck in TransientFailure after ConnectTimeout - #2736

Merged
JamesNK merged 3 commits into
grpc:masterfrom
JamesNK:fix/subchannel-stuck-transient-failure
May 20, 2026
Merged

Fix subchannel permanently stuck in TransientFailure after ConnectTimeout#2736
JamesNK merged 3 commits into
grpc:masterfrom
JamesNK:fix/subchannel-stuck-transient-failure

Conversation

@JamesNK

@JamesNK JamesNK commented May 14, 2026

Copy link
Copy Markdown
Member

Problem

Fixes #2734

When ConnectTimeout fires while TryConnectAsync is running and the transport returns ConnectResult.Failure (e.g. from a SocketException), the ThrowIfCancellationRequested() at line 395 throws OperationCanceledException. The same can happen at line 425 when a backoff delay is interrupted after the connect timeout has already fired.

The catch (OperationCanceledException) block logs the cancellation but does not transition the subchannel state. Since the subchannel is in TransientFailure at this point (set by the failed TryConnectAsync), the connect loop exits and the subchannel is permanently stuck — RequestConnection() only starts a new connect loop from Idle state.

Fix

Transition to Idle in the catch (OperationCanceledException) block. This restores the invariant that RequestConnection() relies on: if the retry loop is not running, the subchannel must be in Idle so a new loop can be started.

Test

Added ConnectTimeout_FiresDuringFailedConnect_SubchannelRecoverable which:

  1. Configures a short ConnectTimeout (100ms)
  2. Has the transport take longer than the timeout then return Failure
  3. Verifies the OperationCanceledException is caught (via ConnectCanceled log)
  4. Asserts that RequestConnection() can restart the connect loop and reach Ready

JamesNK added 3 commits May 14, 2026 09:35
@JamesNK
Fix subchannel permanently stuck in TransientFailure after ConnectTim…
1afeb60 
…eout

When ConnectTimeout fires and TryConnectAsync returns Failure (not Timeout),
ThrowIfCancellationRequested() throws OperationCanceledException. The catch
block did not transition the subchannel state, leaving it stuck in
TransientFailure with no running connect loop. RequestConnection() could
never restart the loop because it only starts from Idle state.

Transition to Idle in the OperationCanceledException catch block so that
RequestConnection() can start a new connect loop.

Fixes grpc#2734
@JamesNK
Apply suggestion from @JamesNK
690305b
@JamesNK
Fix test?
c625ec6
@JamesNK
JamesNK merged commit c042687 into grpc:master May 20, 2026
12 of 13 checks passed
@andyvans

andyvans commented Jul 5, 2026

Copy link
Copy Markdown

@JamesNK was a new version of Grpc.Net.Client ever released with this fix? https://www.nuget.org/packages/Grpc.Net.Client

This was referenced Aug 3, 2026
Closed
Merged
github-actions Bot pushed a commit to sloweyyy/cloud-native-ecommerce-platform that referenced this pull request Aug 3, 2026
@dependabot
chore: Bump the dotnet-minor-and-patch group with 6 updates (#509)
4b7b502 
Updated
[Asp.Versioning.Mvc](https://github.com/dotnet/aspnet-api-versioning)
from 10.0.0 to 10.0.1.

<details>
<summary>Release notes</summary>

_Sourced from [Asp.Versioning.Mvc's
releases](https://github.com/dotnet/aspnet-api-versioning/releases)._

No release notes found for this version range.

Commits viewable in [compare
view](https://github.com/dotnet/aspnet-api-versioning/commits).
</details>

Updated
[Asp.Versioning.Mvc.ApiExplorer](https://github.com/dotnet/aspnet-api-versioning)
from 10.0.0 to 10.0.1.

<details>
<summary>Release notes</summary>

_Sourced from [Asp.Versioning.Mvc.ApiExplorer's
releases](https://github.com/dotnet/aspnet-api-versioning/releases)._

No release notes found for this version range.

Commits viewable in [compare
view](https://github.com/dotnet/aspnet-api-versioning/commits).
</details>

Updated [Grpc.AspNetCore](https://github.com/grpc/grpc-dotnet) from
2.80.0 to 2.83.0.

<details>
<summary>Release notes</summary>

_Sourced from [Grpc.AspNetCore's
releases](https://github.com/grpc/grpc-dotnet/releases)._

## 2.83.0

## What's Changed
* [Release v2.80.x] Fix the version number - Step 3 by @​asheshvidyut in
grpc/grpc-dotnet#2709
* Revert "[Release v2.80.x] Fix the version number - Step 3" by
@​asheshvidyut in grpc/grpc-dotnet#2710
* Bump lodash from 4.17.23 to 4.18.1 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2712
* Bump basic-ftp from 5.2.0 to 5.2.1 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2713
* Fix typo in UnaryServerHandler method description by @​HamzaWahed in
grpc/grpc-dotnet#2714
* Bump axios from 1.13.6 to 1.15.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2715
* Bump basic-ftp from 5.2.1 to 5.2.2 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2716
* Bump follow-redirects from 1.15.11 to 1.16.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2717
* Resolver: fix 'retreive'/'retreiveing' typos in XML doc comments by
@​SAY-5 in grpc/grpc-dotnet#2718
* Bump basic-ftp from 5.2.2 to 5.3.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2719
* Bump protobufjs from 7.5.3 to 7.5.5 in
/examples/Browser/Server/wwwroot by @​dependabot[bot] in
grpc/grpc-dotnet#2721
* Fix flaky DuplexStream_SendToUnimplementedMethod_ThrowError test by
@​JamesNK in grpc/grpc-dotnet#2720
* Add SECURITY.md by @​JamesNK in
grpc/grpc-dotnet#2722
* Update OpenTelemetry packages to latest stable by @​JamesNK in
grpc/grpc-dotnet#2729
* Bump fast-uri from 3.1.0 to 3.1.2 in /examples/Browser/Server/wwwroot
by @​dependabot[bot] in grpc/grpc-dotnet#2730
* Bump basic-ftp from 5.3.0 to 5.3.1 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2727
* Bump axios from 1.15.0 to 1.16.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2728
* Fix XML documentation typo by @​martincostello in
grpc/grpc-dotnet#2724
* Bump @​protobufjs/utf8 from 1.1.0 to 1.1.1 in
/examples/Browser/Server/wwwroot by @​dependabot[bot] in
grpc/grpc-dotnet#2733
* Bump protobufjs from 7.5.5 to 7.5.8 in
/examples/Browser/Server/wwwroot by @​dependabot[bot] in
grpc/grpc-dotnet#2735
* Fix typo in comment about socket ping scheduling by @​yasoob in
grpc/grpc-dotnet#2671
* Bump ip-address and socks in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2726
* Fix subchannel permanently stuck in TransientFailure after
ConnectTimeout by @​JamesNK in
grpc/grpc-dotnet#2736
* Bump ws from 8.18.3 to 8.20.1 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2737
* _moveNextTask may be overwritten, after we release the lock, but befo…
by @​osexpert in grpc/grpc-dotnet#2738
* Bump form-data from 4.0.5 to 4.0.6 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2743
* Bump joi from 17.13.3 to 17.13.4 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2741
* Bump ws from 8.20.1 to 8.21.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2742
* Using Grpc.Tools 2.83.0-pre1 by @​asheshvidyut in
grpc/grpc-dotnet#2746
* Update version for v2.83.x by @​asheshvidyut in
grpc/grpc-dotnet#2747
* Update versions for stable release 2.83.x by @​asheshvidyut in
grpc/grpc-dotnet#2750

## New Contributors
* @​HamzaWahed made their first contribution in
grpc/grpc-dotnet#2714
* @​SAY-5 made their first contribution in
grpc/grpc-dotnet#2718
* @​martincostello made their first contribution in
grpc/grpc-dotnet#2724
* @​yasoob made their first contribution in
grpc/grpc-dotnet#2671
* @​osexpert made their first contribution in
grpc/grpc-dotnet#2738

**Full Changelog**:
grpc/grpc-dotnet@v2.80.0...v2.83.0

## 2.83.0-pre1

## What's Changed
* [Release v2.80.x] Fix the version number - Step 3 by @​asheshvidyut in
grpc/grpc-dotnet#2709
* Revert "[Release v2.80.x] Fix the version number - Step 3" by
@​asheshvidyut in grpc/grpc-dotnet#2710
* Bump lodash from 4.17.23 to 4.18.1 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2712
* Bump basic-ftp from 5.2.0 to 5.2.1 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2713
* Fix typo in UnaryServerHandler method description by @​HamzaWahed in
grpc/grpc-dotnet#2714
* Bump axios from 1.13.6 to 1.15.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2715
* Bump basic-ftp from 5.2.1 to 5.2.2 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2716
* Bump follow-redirects from 1.15.11 to 1.16.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2717
* Resolver: fix 'retreive'/'retreiveing' typos in XML doc comments by
@​SAY-5 in grpc/grpc-dotnet#2718
* Bump basic-ftp from 5.2.2 to 5.3.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2719
* Bump protobufjs from 7.5.3 to 7.5.5 in
/examples/Browser/Server/wwwroot by @​dependabot[bot] in
grpc/grpc-dotnet#2721
* Fix flaky DuplexStream_SendToUnimplementedMethod_ThrowError test by
@​JamesNK in grpc/grpc-dotnet#2720
* Add SECURITY.md by @​JamesNK in
grpc/grpc-dotnet#2722
* Update OpenTelemetry packages to latest stable by @​JamesNK in
grpc/grpc-dotnet#2729
* Bump fast-uri from 3.1.0 to 3.1.2 in /examples/Browser/Server/wwwroot
by @​dependabot[bot] in grpc/grpc-dotnet#2730
* Bump basic-ftp from 5.3.0 to 5.3.1 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2727
* Bump axios from 1.15.0 to 1.16.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2728
* Fix XML documentation typo by @​martincostello in
grpc/grpc-dotnet#2724
* Bump @​protobufjs/utf8 from 1.1.0 to 1.1.1 in
/examples/Browser/Server/wwwroot by @​dependabot[bot] in
grpc/grpc-dotnet#2733
* Bump protobufjs from 7.5.5 to 7.5.8 in
/examples/Browser/Server/wwwroot by @​dependabot[bot] in
grpc/grpc-dotnet#2735
* Fix typo in comment about socket ping scheduling by @​yasoob in
grpc/grpc-dotnet#2671
* Bump ip-address and socks in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2726
* Fix subchannel permanently stuck in TransientFailure after
ConnectTimeout by @​JamesNK in
grpc/grpc-dotnet#2736
* Bump ws from 8.18.3 to 8.20.1 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2737
* _moveNextTask may be overwritten, after we release the lock, but befo…
by @​osexpert in grpc/grpc-dotnet#2738
* Bump form-data from 4.0.5 to 4.0.6 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2743
* Bump joi from 17.13.3 to 17.13.4 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2741
* Bump ws from 8.20.1 to 8.21.0 in
/testassets/InteropTestsGrpcWebWebsite/Tests by @​dependabot[bot] in
grpc/grpc-dotnet#2742
* Using Grpc.Tools 2.83.0-pre1 by @​asheshvidyut in
grpc/grpc-dotnet#2746
* Update version for v2.83.x by @​asheshvidyut in
grpc/grpc-dotnet#2747

## New Contributors
* @​HamzaWahed made their first contribution in
grpc/grpc-dotnet#2714
* @​SAY-5 made their first contribution in
grpc/grpc-dotnet#2718
* @​martincostello made their first contribution in
grpc/grpc-dotnet#2724
* @​yasoob made their first contribution in
grpc/grpc-dotnet#2671
* @​osexpert made their first contribution in
grpc/grpc-dotnet#2738

**Full Changelog**:
grpc/grpc-dotnet@v2.80.0...v2.83.0-pre1

Commits viewable in [compare
view](grpc/grpc-dotnet@v2.80.0...v2.83.0).
</details>

Updated
[Microsoft.EntityFrameworkCore.Design](https://github.com/dotnet/dotnet)
from 10.0.9 to 10.0.10.

<details>
<summary>Release notes</summary>

_Sourced from [Microsoft.EntityFrameworkCore.Design's
releases](https://github.com/dotnet/dotnet/releases)._

No release notes found for this version range.

Commits viewable in [compare
view](https://github.com/dotnet/dotnet/commits).
</details>

Updated
[Microsoft.EntityFrameworkCore.SqlServer](https://github.com/dotnet/dotnet)
from 10.0.9 to 10.0.10.

<details>
<summary>Release notes</summary>

_Sourced from [Microsoft.EntityFrameworkCore.SqlServer's
releases](https://github.com/dotnet/dotnet/releases)._

No release notes found for this version range.

Commits viewable in [compare
view](https://github.com/dotnet/dotnet/commits).
</details>

Updated
[Microsoft.EntityFrameworkCore.Tools](https://github.com/dotnet/dotnet)
from 10.0.9 to 10.0.10.

<details>
<summary>Release notes</summary>

_Sourced from [Microsoft.EntityFrameworkCore.Tools's
releases](https://github.com/dotnet/dotnet/releases)._

No release notes found for this version range.

Commits viewable in [compare
view](https://github.com/dotnet/dotnet/commits).
</details>

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Aug 3, 2026
Merged
Merged
Merged
Closed
Merged
Closed
Open
Open
Closed
Open
Merged
Merged
Closed
Closed
Closed
Merged
Merged
Closed
Merged
This was referenced Aug 6, 2026
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BrennanConroy BrennanConroy BrennanConroy approved these changes
@asheshvidyut asheshvidyut Awaiting requested review from asheshvidyut

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Subchannel permanently stuck in TransientFailure after ConnectTimeout fires

3 participants

@JamesNK @andyvans @BrennanConroy