Skip to content

SslSessionsCache growth on Linux with MTLS using many client certificates #71401

Closed

Description

@RobinsonWM

Description

If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.

If I run the same code on Windows and Linux, I only see the memory growth on Linux.

This cache appears to grow without bound:

private static readonly ConcurrentDictionary<SslCredKey, SafeCredentialReference> s_cachedCreds =

There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.

Reproduction Steps

https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba

The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.

The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.

This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.

Expected behavior

Over time, if load is steady, the application's resident memory usage is also steady.

Actual behavior

The application works fine for a long time, but its resident memory usage grows linearly.

Regression?

I don't know if this is a regression

Known Workarounds

The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.

private static void ClearCache()
{
    var sslAssembly = Assembly.GetAssembly(typeof(SslStream));
    var sslSessionCacheClass = sslAssembly.GetType("System.Net.Security.SslSessionsCache");
    var cachedCredsInfo = sslSessionCacheClass.GetField("s_cachedCreds", BindingFlags.NonPublic | BindingFlags.Static);
    var cachedCreds = cachedCredsInfo.GetValue(null);
    cachedCreds.GetType().GetMethod("Clear", BindingFlags.Public | BindingFlags.Instance).Invoke(cachedCreds, null);
}

Configuration

.NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.

Other information

I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.

Activity

  1. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Jun 28, 2022
  2. wfurt commented on Jun 29, 2022

    @wfurt
    Member

    There should be cap on the SslSessionsCache but I don't think we have any tests to verify that. That part changed a bit in 7.0 with TLS resume and it will change even more with #69527. I think we should look into it for 7.0

  3. added this to the 7.0.0 milestone on Jun 30, 2022
  4. added and removed
    untriagedNew issue has not been triaged by the area owner
    on Jun 30, 2022
  5. karelz commented on Jun 30, 2022

    @karelz
    Member

    Triage: We should confirm if we grow indefinitely.

  6. self-assigned this
    on Jul 12, 2022
  7. wfurt commented on Jul 17, 2022

    @wfurt
    Member

    Can you please try it with 7.0 preview build @RobinsonWM? I did little bit of digging and it seems like this comes from client side in 6.0. I don't see the growth with 7.0 but it would be nice to confirm it with your setup.

  8. added
    needs-author-actionAn issue or pull request that requires more info or actions from the author.
    on Jul 24, 2022
  9. karelz commented on Aug 2, 2022

    @karelz
    Member

    Triage: We do not have 7.0 repro - it seems to be solved there. Closing as Fixed in 7.0.

  10. ghost locked as resolved and limited conversation to collaborators on Sep 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions