Repository navigation
SslSessionsCache growth on Linux with MTLS using many client certificates #71401
Description
Description
If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.
If I run the same code on Windows and Linux, I only see the memory growth on Linux.
This cache appears to grow without bound:
runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs
Line 16 in dbc7d8c
| private static readonly ConcurrentDictionary<SslCredKey, SafeCredentialReference> s_cachedCreds = |
There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.
Reproduction Steps
https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba
The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.
The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.
This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.
Expected behavior
Over time, if load is steady, the application's resident memory usage is also steady.
Actual behavior
The application works fine for a long time, but its resident memory usage grows linearly.
Regression?
I don't know if this is a regression
Known Workarounds
The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.
private static void ClearCache()
{
var sslAssembly = Assembly.GetAssembly(typeof(SslStream));
var sslSessionCacheClass = sslAssembly.GetType("System.Net.Security.SslSessionsCache");
var cachedCredsInfo = sslSessionCacheClass.GetField("s_cachedCreds", BindingFlags.NonPublic | BindingFlags.Static);
var cachedCreds = cachedCredsInfo.GetValue(null);
cachedCreds.GetType().GetMethod("Clear", BindingFlags.Public | BindingFlags.Instance).Invoke(cachedCreds, null);
}Configuration
.NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.
Other information
I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.
Activity
wfurt commentedon Jun 29, 2022
There should be cap on the
SslSessionsCachebut I don't think we have any tests to verify that. That part changed a bit in 7.0 with TLS resume and it will change even more with #69527. I think we should look into it for 7.0karelz commentedon Jun 30, 2022
Triage: We should confirm if we grow indefinitely.
wfurt commentedon Jul 17, 2022
Can you please try it with 7.0 preview build @RobinsonWM? I did little bit of digging and it seems like this comes from client side in 6.0. I don't see the growth with 7.0 but it would be nice to confirm it with your setup.
karelz commentedon Aug 2, 2022
Triage: We do not have 7.0 repro - it seems to be solved there. Closing as Fixed in 7.0.