Repository navigation
Async opening of connections in parallel is slow/blocking #601
Description
Activity
@cheenamalhotra @OQO FWIW this serial opening of concurrent connections for the same pool, even in case of multithreaded scenario is intentional and by design of the Connection Pool.
From the past owners of SqlClient I understand that the idea was to not overwhelm a SQL server because of a flood of connections from the single Connection Pool which needs many connections as well as many clients trying to connect to a single SQL server when failover is in progress.
The recommendation is to use MinPoolSize in Connection String for connection pool and allow the pool to warm up to the MinPoolSize after the first connection attempt is made. This is something that can be tried. After that the connection pool will try to maintain MinPoolSize count of connections in the pool.As of whether the behavior should be changed or not, it may be OK to relax the requirement for Azure SQL DBs, but it may be good to keep this behavior for on-prem SQL servers.
There are scenarios where a server fails over and multiple clients with connection pools are trying to login to the same server. In those cases, we may not want the server overwhelmed by a slew of connections coming in.@saurabh500 Thank you for your comment on issue #408, I replied here to be able to refer to the code above. I do understand your point, there definitely are some valid concerns that warrant rate limiting connections.
But I don't think they fully apply for our case. The main problem for us is that we are opening connections over a high latency connection to SQL server (in this case SQL Azure with a round trip latency of over 100ms between SQL Azure and our web server). So the huge majority of the waiting is done due to the latency of the network connection and not due to any load on the SQL server. In our scenario opening a single connection takes nearly one second (see above).
This means that while our web server has been optimized to start (or restart) in less than 3 seconds, it takes several minutes till it is able to open sufficient connections to serve the users who have connected in the meantime. During this time only roughly a single connection per second is being opened!
Of course SQL Azure is happy to open easily 10-100 times more connections per second, as is quickly proven by connecting to SQL Azure from within the same datacenter.
Maybe the rate limiting should be on the number of connections opened, and should be less strict on the number of connections being in the process of being opened concurrently.
Note that even in non-pooled mode only 8 connections at a time seem to be getting opened so we did not find a good workaround. So in our scenario in non-pooled mode we can only open about 8 connections per second!
This is another case where a user is asking to create n (where n>30) connections instantly. Unless those are n connections to n different database servers it seems like something of an antipattern. What can a single process be doing where it's capable of feeding 50 queries at once?
@Wraith2 I saw your comment on #408. I answered it here to be able to relate to the example above.
In our application we have hundreds of users on a single server and we are able to feed 10-20 queries at once.
However, for our servers that are in locations with high network latency between SQL Azure and our web servers we need many more connections to keep the same throughput.
Let's assume a network round trip network latency of 2ms in a local data center and a query time of 2ms. So total time from the client perspective is 4ms. Let's now assume a location with a round trip network latency of 78ms. In this case the total time from the client perspective for the query is 80ms. So in order to achieve the same amount of queries executed per second on SQL server we need 20 times as many connections because the huge majority of the time is spent waiting due to network latency.
So in order to get the same throughput in high latency environments one needs a significant larger number of connections between the application server and database server. So where in a low latency environment 10 connections might suffice, one might need 200 connections in a higher latency environment to achieve the same throughput. (Note that these number of open connections do definitely not overwhelm SQL Azure.)
So that is why it would be great: 1. open new connections at a much faster rate. 2. open the connection in a truly async way (no thread blocking).
@OQO I was able to reproduce the issue. I took the 4 seconds task delay out and replace it with actual query the result was better after, but as you mentioned the results of comparison between pooled connections and non pooled ones are not as we expect them to be. We will look into the issue and will response back soon.
Reacted by OQO@OQO I have a question to understand better,
Why are we clearing the pool each time?
clearing the pool forces the driver to create another pool each time and that is time consuming. If we only clear the pool before the non pooled connection as below:
this will make a huge difference in re using pooled connection.
Is there any specific reason behind clearing the pool?
In this test we want to compare two scenarios of opening multiple connections with high latency network at application startup: 1. pooled and 2. non-pooled.
In order to compare these two scenarios we reset the connection pools between the experiments so that we can have a fair comparison. If you would like you could just make separate executables to run the two different scenarios (Results will be the same)
The WARM UP scenario just serves to measure a baseline so that we know how long it takes to set up a single connection. Opening a single connection is done multiple times, since the first time is always slower due to the loading of the required assemblies etc. doing it then twice more shows a stable time so that proves that our measurements are no longer influenced by loading assemblies etc.
Hope this clarifies it!
@OQO I made some changes and it seems the code runs fine. Can you kindly make below changes and test your application again:
- Add a MinPoolSize of 100 to your connection string. If you add any number greater than 100 you need to change
MaxPoolSize as well. Its default value is 100 and cannot be less than the value of MinPoolSize. - For pooled connection please add Pooling=true to your connection string and Pooling=false to your non-pooled connections
- Replace the Task.Delay(4000) with an actual sqlcommand. I believe this is the main reason of the issue. Each call is waiting for four seconds and no pool is created in that period of time. Each connection tries to create a new pool and causes the delay.
Here is the result I got with this run:
WARM UP ALL POOLS CLEARED 00:00:00.5658719 8d64ca61-e6c0-40bc-bfff-e4065c04441a False Single open time 00:00:00.0002643 8d64ca61-e6c0-40bc-bfff-e4065c04441a True Single open time with one previously opened connection ALL POOLS CLEARED 00:00:00.2208180 159fd79a-fa22-49d1-9221-ecbaab3aa1e7 False Single open time 00:00:00.0000798 159fd79a-fa22-49d1-9221-ecbaab3aa1e7 True Single open time with one previously opened connection ALL POOLS CLEARED 00:00:00.2175882 ee30b57f-4d74-44e2-a646-3b91fb907a85 False Single open time 00:00:00.0000638 ee30b57f-4d74-44e2-a646-3b91fb907a85 True Single open time with one previously opened connection CONCURRENT POOLED CONNECTIONS ALL POOLS CLEARED Start delay OpenAsync time Connection ID ReusedFromPool 00:00:00.0009645 00:00:00.2125541 0 a44afca6-84ee-4739-8257-841eb336eeac False 00:00:00.0017354 00:00:00.2848127 1 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0018036 00:00:00.3237833 2 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0018484 00:00:00.3504944 3 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0018931 00:00:00.3738208 4 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0019455 00:00:00.3961843 5 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0019828 00:00:00.4185117 6 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0020194 00:00:00.4248015 7 970418af-0377-4baa-9e43-8fbb7ec057db False 00:00:00.0020619 00:00:00.4423684 8 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0020953 00:00:00.4667225 9 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0021357 00:00:00.4712881 10 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0021622 00:00:00.4897947 11 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0022053 00:00:00.4959876 12 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0022395 00:00:00.5156536 13 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0022780 00:00:00.5196160 14 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0023019 00:00:00.5383633 15 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0023279 00:00:00.5430932 16 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0023658 00:00:00.5626749 17 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0023884 00:00:00.5665512 18 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0024116 00:00:00.5885240 19 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0024405 00:00:00.5936663 20 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0024678 00:00:00.6114261 21 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0024879 00:00:00.6146811 22 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0025107 00:00:00.6344081 23 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0025318 00:00:00.6348700 24 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0025567 00:00:00.6463720 25 d0132087-2ff8-47d0-a4ea-c9637237352e False 00:00:00.0025823 00:00:00.6579774 26 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0026027 00:00:00.6583110 27 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0026106 00:00:00.6817343 28 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0026181 00:00:00.6817323 29 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0026230 00:00:00.6969113 30 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0026322 00:00:00.7081931 32 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0026276 00:00:00.7081975 31 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0026390 00:00:00.7218787 33 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0026626 00:00:00.7320723 34 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0026719 00:00:00.7323766 35 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0026768 00:00:00.7468168 36 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0026813 00:00:00.7527362 37 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0026898 00:00:00.7527289 38 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0026948 00:00:00.7720170 39 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0026995 00:00:00.7720194 40 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0027043 00:00:00.7769340 41 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0027124 00:00:00.7910900 42 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0027169 00:00:00.7969304 43 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0027212 00:00:00.8008876 44 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0027285 00:00:00.8155257 45 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0027334 00:00:00.8204605 46 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0027379 00:00:00.8252616 47 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0027448 00:00:00.8348439 48 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0027497 00:00:00.8453721 49 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0027543 00:00:00.8456617 50 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0027639 00:00:00.8542243 51 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0027686 00:00:00.8685864 52 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0027730 00:00:00.8896550 53 8d19776e-66f4-4c27-bb2d-1ae783aa2298 False 00:00:00.0027797 00:00:00.8902260 54 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0027844 00:00:00.8902921 55 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0027889 00:00:00.8931239 56 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0027945 00:00:00.9141225 57 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0028015 00:00:00.9181820 58 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0028059 00:00:00.9238160 59 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0028102 00:00:00.9365150 60 8d19776e-66f4-4c27-bb2d-1ae783aa2298 True 00:00:00.0028169 00:00:00.9369906 61 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0028213 00:00:00.9413880 62 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0028257 00:00:00.9464264 63 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0028345 00:00:00.9601610 64 8d19776e-66f4-4c27-bb2d-1ae783aa2298 True 00:00:00.0028392 00:00:00.9603352 65 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0028435 00:00:00.9643761 66 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0028501 00:00:00.9696277 67 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0028547 00:00:00.9835216 68 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0028592 00:00:00.9836422 69 8d19776e-66f4-4c27-bb2d-1ae783aa2298 True 00:00:00.0028636 00:00:00.9877402 70 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0028701 00:00:00.9932701 71 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0028744 00:00:01.0080811 72 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0028790 00:00:01.0080823 73 8d19776e-66f4-4c27-bb2d-1ae783aa2298 True 00:00:00.0028862 00:00:01.0129749 74 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0028908 00:00:01.0129850 75 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0028951 00:00:01.0315222 76 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0029020 00:00:01.0388836 77 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0029066 00:00:01.0389158 78 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0029109 00:00:01.0389803 79 8d19776e-66f4-4c27-bb2d-1ae783aa2298 True 00:00:00.0029177 00:00:01.0549180 80 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0029221 00:00:01.0601843 81 8d19776e-66f4-4c27-bb2d-1ae783aa2298 True 00:00:00.0029264 00:00:01.0601964 82 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0029329 00:00:01.0604757 83 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0029376 00:00:01.0782453 84 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0029421 00:00:01.0840301 85 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0029465 00:00:01.0841778 86 8d19776e-66f4-4c27-bb2d-1ae783aa2298 True 00:00:00.0029532 00:00:01.0842424 87 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0029576 00:00:01.1016888 88 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0029618 00:00:01.1079855 89 8d19776e-66f4-4c27-bb2d-1ae783aa2298 True 00:00:00.0029703 00:00:01.1080267 90 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0029747 00:00:01.1083378 91 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0029792 00:00:01.1098207 92 43aeffaa-e3e7-4043-b447-bb25dc788010 False 00:00:00.0029887 00:00:01.1296268 93 a44afca6-84ee-4739-8257-841eb336eeac True 00:00:00.0029936 00:00:01.1309511 94 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0029980 00:00:01.1309694 95 d0132087-2ff8-47d0-a4ea-c9637237352e True 00:00:00.0030047 00:00:01.1311000 96 8d19776e-66f4-4c27-bb2d-1ae783aa2298 True 00:00:00.0030112 00:00:01.1599673 97 43aeffaa-e3e7-4043-b447-bb25dc788010 True 00:00:00.0030199 00:00:01.1603473 98 970418af-0377-4baa-9e43-8fbb7ec057db True 00:00:00.0030275 00:00:01.1603798 99 d0132087-2ff8-47d0-a4ea-c9637237352e True **00:00:01.1877216 100 connections opened in paralel** CONCURRENT NON-POOLED CONNECTIONS ALL POOLS CLEARED Start delay OpenAsync time Connection ID ReusedFromPool 00:00:00.0015674 00:00:00.2354028 5 d92f7623-d11c-4756-8b61-0c711fdf4975 False 00:00:00.0009907 00:00:00.2382715 1 bb7b5599-60ce-4033-8111-1861ff79d672 False 00:00:00.0020465 00:00:00.2372214 7 003ab253-e0e5-43e1-bccf-8c5133363531 False 00:00:00.0013089 00:00:00.2538747 3 e6df8b89-442b-4556-bbaf-0c7ccb1499eb False 00:00:00.0000114 00:00:00.2581702 0 ebc4b086-dafb-40ef-a887-318523adab84 False 00:00:00.0014228 00:00:00.2552560 4 82eabb5f-bd9e-4e9b-8a76-3b570d3bc650 False 00:00:00.0011820 00:00:00.2571401 2 96e79b10-b3e8-4c6c-b553-554f45618091 False 00:00:00.0018006 00:00:00.2593064 6 45306a7c-5b39-43dd-b324-6b3264e23c04 False 00:00:00.0023857 00:00:00.4769152 15 06c3a19e-3b77-48f7-9faa-760b3dbcff63 False 00:00:00.0023181 00:00:00.4769820 9 1a863657-57c9-4e69-a8af-538ae328530f False 00:00:00.0023703 00:00:00.4769531 13 a6a6b455-d98a-4dad-bd19-76485b524545 False 00:00:00.0023606 00:00:00.4940845 12 0507227b-0329-4dc6-a06c-185deabce77d False 00:00:00.0022457 00:00:00.5097288 8 1bb0f6cc-cbad-4efc-9e3f-0e01d16c308f False 00:00:00.0023550 00:00:00.5096994 11 0b02b77e-f986-42b6-8426-e5b9303b53f2 False 00:00:00.0023755 00:00:00.5096468 14 1936a29d-f247-47bb-8035-831a106b3c7b False 00:00:00.0023310 00:00:00.5096935 10 a7f258af-f472-4be0-ade0-d62575f4aae0 False 00:00:00.0024017 00:00:00.7236330 17 24a34a68-101d-4303-ae33-d14b97838c24 False 00:00:00.0024706 00:00:00.7235693 23 d453b0df-1c49-49ef-b161-a5111c4011b4 False 00:00:00.0024483 00:00:00.7305378 21 b51018ac-adfc-4c61-91a6-d916bea8e527 False 00:00:00.0024390 00:00:00.7305450 20 6b423024-a9fb-4f11-bcfb-f569358a0be4 False 00:00:00.0024614 00:00:00.7351283 22 d865acf7-6ebf-4812-86fc-c19dd9d41762 False 00:00:00.0023940 00:00:00.7487760 16 28e4e049-d093-40ba-8da7-ee02725cc422 False 00:00:00.0024251 00:00:00.7702248 19 2a428dba-8ee1-416a-aeaa-70aa026baf0b False 00:00:00.0024160 00:00:00.7702481 18 61bc0db2-56a1-491b-b2cd-cc3ab51a76e7 False 00:00:00.0024939 00:00:00.9610591 25 78c8060c-29cb-4acc-bd3f-b3c547268b10 False 00:00:00.0024792 00:00:00.9974713 24 eb253aa2-fea8-4de7-8814-c2d408eb8c69 False 00:00:00.0025433 00:00:00.9974117 29 5ae87421-2cd9-4fd6-9fba-1be5bf315098 False 00:00:00.0025530 00:00:00.9974126 30 4739909f-6746-43df-8fbe-5b37b5390fa7 False 00:00:00.0025031 00:00:01.0001629 26 ef72c499-e79c-45fe-9f26-5e1d240595c4 False 00:00:00.0025615 00:00:01.0001019 31 45336018-44f6-4504-9199-9f93188d7b93 False 00:00:00.0025252 00:00:01.0001413 28 9de54941-ae39-4feb-ad28-7f4a8999b3de False 00:00:00.0025165 00:00:01.0103940 27 a6ab2921-f1c7-4bfe-aceb-2d799f3137bc False 00:00:00.0045902 00:00:01.1909512 33 79aa6621-c586-4061-8235-dc7ac7ebe6d6 False 00:00:00.0025758 00:00:01.2261350 32 a6165f5f-c104-4e5d-9104-03813f720780 False 00:00:00.0046033 00:00:01.2303122 34 6d9e3125-ffac-4ab5-a354-de331ce9e977 False 00:00:00.0046173 00:00:01.2363577 38 72da110e-56b5-40e4-8afe-07819566ea16 False 00:00:00.0046107 00:00:01.2395185 36 2cc8323c-c257-4fc2-bf40-56369e988bfa False 00:00:00.0046137 00:00:01.2396061 37 6f0ae447-c035-4942-bf9a-38e57aa4b5dc False 00:00:00.0046075 00:00:01.2396379 35 bb667ac4-28c1-4de6-ab41-d0af57e1007b False 00:00:00.0046203 00:00:01.2413661 39 7cc3c4a6-fcf8-4bab-b782-49ab3df03b01 False 00:00:00.0046382 00:00:01.4420154 41 e437847b-3b5d-497c-b507-a7f567974cb4 False 00:00:00.0046452 00:00:01.4873462 43 9c15de44-8527-49e5-aaa9-20025f96a029 False 00:00:00.0046545 00:00:01.5010750 46 786d5cee-c6c3-4973-97b5-19c0fed29ce7 False 00:00:00.0046516 00:00:01.5010777 45 c38c2445-7440-4e91-bb13-70cdd803f097 False 00:00:00.0046578 00:00:01.5203254 47 894a1316-4c6a-4850-9c2f-adf3f0537022 False 00:00:00.0046348 00:00:01.5203843 40 30a9d3a6-8e8d-4581-a642-e3f14381f848 False 00:00:00.0046482 00:00:01.5203790 44 3944eee8-e15b-4e88-a34e-720a051ffeeb False 00:00:00.0046416 00:00:01.5204084 42 4eb494d6-ee6c-42d5-b757-956f4f2ca738 False 00:00:00.0046640 00:00:01.6672600 49 4399663a-b8d8-4f57-a459-7a35b7461058 False 00:00:00.0046761 00:00:01.7286690 53 f2fb7fe6-e48d-447c-8c03-e01d35f984aa False 00:00:00.0046699 00:00:01.7331905 51 8fd6f99f-a940-49aa-bdce-3cdcb06afc9a False 00:00:00.0046799 00:00:01.7425553 54 061b8f50-1982-48a0-85b2-dab312239720 False 00:00:00.0046608 00:00:01.7425760 48 696583b9-32ad-4cc0-b81e-dc05554f9082 False 00:00:00.0046670 00:00:01.7573964 50 7752cef0-e845-40d1-8156-153f606c7e22 False 00:00:00.0046731 00:00:01.7574359 52 9d7f1003-1219-410b-af8a-2819fe17524e False 00:00:00.0046827 00:00:01.7590414 55 9d662432-076c-4bed-9d44-caf64730b69b False 00:00:00.0046890 00:00:01.9044821 57 294bf930-b718-4f1e-9ddd-5fe27c5c1755 False 00:00:00.0046861 00:00:01.9502193 56 61eb9896-0866-49bc-854b-72b00b26ab12 False 00:00:00.0046955 00:00:01.9502205 59 9169839e-798d-4d59-bf1a-90d638284d8a False 00:00:00.0047021 00:00:01.9548800 61 342f55be-db4c-4536-b06b-0312e7e20e42 False 00:00:00.0046987 00:00:01.9880930 60 acbbe8a4-1198-4f0a-b6b1-72495979c0d9 False 00:00:00.0047051 00:00:01.9894790 62 8dde242e-7374-43ff-bc38-78ba0ae81b51 False 00:00:00.0046921 00:00:01.9894926 58 37304eca-cfdb-4632-8a41-19e4ee1e7d1e False 00:00:00.0047084 00:00:02.0105511 63 ef2c4809-e7fc-4519-bef1-df956ed90d5c False 00:00:00.0047145 00:00:02.1298016 65 0ccb1fa8-ff69-4aa0-9f8e-858b966def6a False 00:00:00.0047270 00:00:02.1762375 69 864af172-f2a2-451e-9577-8f7f65abda60 False 00:00:00.0047207 00:00:02.1762405 67 4ca92397-9645-4ded-abdf-d7df0948b993 False 00:00:00.0047115 00:00:02.1781056 64 5ef06898-fc92-4e16-a71b-09098544cb15 False 00:00:00.0047241 00:00:02.2222019 68 5746a3aa-905d-478d-8356-3eaa1434e5c9 False 00:00:00.0047178 00:00:02.2246495 66 cabaefb4-cdfd-4a97-a8cd-25c12ecb54fd False 00:00:00.0047334 00:00:02.2247876 70 5d8234bb-2512-492c-a73b-d671a9c5defe False 00:00:00.0047366 00:00:02.2444297 71 a22fc356-c720-4e19-bbeb-eff4b509c688 False 00:00:00.0047430 00:00:02.3842036 73 66c58acf-3d8b-420e-8be5-890ebfee31eb False 00:00:00.0047395 00:00:02.4170934 72 6226c851-0d7e-46ec-8c70-59b28cd12d84 False 00:00:00.0047492 00:00:02.4170869 75 76ab8982-54e3-47c6-9d8f-3696f7af240b False 00:00:00.0047556 00:00:02.4299743 77 c82e3e18-b172-4ba4-ac52-9d31e976817a False 00:00:00.0047522 00:00:02.4900592 76 b5cea1e3-e446-438d-9814-a4434e568724 False 00:00:00.0047459 00:00:02.4900696 74 da3ca4b5-4ee7-4450-b4e9-bbf9968cba1f False 00:00:00.0047587 00:00:02.4943586 78 c39960cd-9fc8-4e18-a8cc-7bc6e3da27b9 False 00:00:00.0047616 00:00:02.4961259 79 abcc79da-8323-4a61-adae-860045aaee78 False 00:00:00.0047684 00:00:02.6059913 81 2b45005c-c76a-43a3-af8c-2cdf85169388 False 00:00:00.0047654 00:00:02.6463052 80 6120f7f6-c536-472a-b22d-75c7517d3dfb False 00:00:00.0047748 00:00:02.6462959 83 35514aa4-7904-4983-93d0-92a99234320a False 00:00:00.0047812 00:00:02.6500396 85 bdb0ea1e-cf3b-40c5-82ee-086f2d37ecbd False 00:00:00.0047718 00:00:02.7420578 82 373b4495-f496-4e0f-95a5-331c5491826f False 00:00:00.0047843 00:00:02.7420438 86 aec4b195-6bc8-46e0-9803-894d31aafbe1 False 00:00:00.0047878 00:00:02.7424678 87 97678582-f555-49fa-8e8c-f8b7613c4b0b False 00:00:00.0047781 00:00:02.7424993 84 7c4225df-8595-4415-88e3-e31adb2136dc False 00:00:00.0047953 00:00:02.8467297 89 cad5d133-d07d-44f8-871b-1bfdfe1f1d3e False 00:00:00.0048016 00:00:02.9017044 91 71852685-b78f-4164-8163-e6d081761432 False 00:00:00.0047908 00:00:02.9194366 88 a4762e2d-28c5-46f1-a92d-96f1b7493c65 False 00:00:00.0048078 00:00:02.9379883 93 5088e61a-02e7-460a-873a-004be744f13a False 00:00:00.0048108 00:00:02.9667535 94 ef8481e9-cf5b-42b3-9c04-f6556827e8a9 False 00:00:00.0047982 00:00:02.9667635 90 cba61b80-9aea-44d7-901f-0b872fd3d83f False 00:00:00.0048045 00:00:02.9927552 92 4b9ae25c-68d3-4b4e-b83a-cba66dd1120b False 00:00:00.0048137 00:00:02.9927616 95 47958f45-b577-475d-8091-a2f7a397a9f4 False 00:00:00.0048201 00:00:03.0725529 97 84671af9-a0bc-4ef0-b2e9-81db210336d6 False 00:00:00.0048173 00:00:03.1463661 96 173161e1-c3bc-4c10-b5b5-eea77e25d6b2 False 00:00:00.0048284 00:00:03.1463578 99 a8e0b072-1946-41c5-af7a-7ff1e2b1d12f False 00:00:00.0048256 00:00:03.1864995 98 7fe96a38-4865-493d-8534-a77a013c8cc7 False **00:00:03.2365981 100 connections opened in paralel** Testing finished- Add a MinPoolSize of 100 to your connection string. If you add any number greater than 100 you need to change
This is the updated version of the test program based on your feedback. Min and max pool sizes have been set. And a query is executed on the database. The query consists of a wait so that we can reliably test the connection issues. Tests are run with query time between .5 seconds and 3.5 seconds.
Please set
lowLatencyConnectionStringto a database close to you, and sethighLatencyConnectionStringto a database far away. In the experiments run below opening a single connection to the close Azure database takes around 200ms, and opening a single new connection to an Azure database further away takes a little over 1000ms.Note that in your last experiment above you are connecting to a database over a network connection with a low latency. If you are in the USA, try connecting to a database in Asia East for example to check the high latency scenario.
using Microsoft.Data.SqlClient; using System; using System.Collections.Concurrent; using System.Diagnostics; using System.IO; using System.Threading.Tasks; using System.Transactions; namespace ConsoleExperiments { class Program { static void Main(string[] args) { TestConnections().Wait(); } static async Task TestConnections() { // Low latency connection string lowLatencyConnectionString = "Server=tcp:XXXX.database.windows.net,1433;Initial Catalog=XXXXs;Persist Security Info=False;User ID=XXXX;Password=XXXX!;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;"; // High latency connection string highLatencyConnectionString = "Server=tcp:XXXX.database.windows.net,1433;Initial Catalog=XXXX;Persist Security Info=False;User ID=XXXXr;Password=XXXX;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;"; string[] connectionStrings = new string[] { lowLatencyConnectionString, highLatencyConnectionString }; string connectionType = "LowLatency"; string csv = ""; foreach (string connectionString in connectionStrings) { TimeSpan queryTime = TimeSpan.FromSeconds(0.5); for (int i = 0; i < 7; i++) { (TimeSpan pooledTime, TimeSpan nonPooledTime) = await SingleRun(connectionString, queryTime); csv += $"{connectionType},{queryTime.TotalSeconds},{pooledTime.TotalSeconds},{nonPooledTime.TotalSeconds}{Environment.NewLine}"; queryTime += TimeSpan.FromSeconds(0.5); } connectionType = "HighLatency"; } File.WriteAllText("ConnectionTest.csv", csv); Console.WriteLine("\nTesting finished"); Console.ReadLine(); } private static async Task<(TimeSpan pooledTime, TimeSpan nonPooledTime)> SingleRun(string connectionString, TimeSpan queryTime) { connectionString += "Min Pool Size=200;Max Pool Size=500;"; Console.WriteLine("WARM UP"); await MeasureSingleConnectionAndReuse(connectionString); ClearPools(); await MeasureSingleConnectionAndReuse(connectionString); ClearPools(); await MeasureSingleConnectionAndReuse(connectionString); ClearPools(); Console.WriteLine("\n\nCONCURRENT POOLED CONNECTIONS"); TimeSpan pooledTime = MeasureParallelConnections(connectionString + "Pooling=true;", queryTime); ClearPools(); Console.WriteLine("\n\nCONCURRENT NON-POOLED CONNECTIONS"); TimeSpan nonPooledTime = MeasureParallelConnections(connectionString + "Pooling=false;", queryTime); ClearPools(); return (pooledTime, nonPooledTime); } private static void ClearPools() { SqlConnection.ClearAllPools(); Console.WriteLine("ALL POOLS CLEARED"); } static ConcurrentDictionary<Guid, object> _connectionIDs = new ConcurrentDictionary<Guid, object>(); private static TimeSpan MeasureParallelConnections(string connectionString, TimeSpan queryTime) { Console.WriteLine("Start delay OpenAsync time Connection ID ReusedFromPool"); Stopwatch sw = new Stopwatch(); sw.Start(); int numOpens = 100; Task[] tasks = new Task[numOpens]; Stopwatch start = new Stopwatch(); start.Start(); for (int i = 0; i < numOpens; i++) { tasks[i] = MeasureSingleConnection(i, start, connectionString, queryTime); } Task.WaitAll(tasks); start.Stop(); Console.WriteLine($"{sw.Elapsed} {numOpens} connections opened in paralel"); return start.Elapsed; } private static async Task MeasureSingleConnection(int index, Stopwatch start, string connectionString, TimeSpan queryTime) { TimeSpan startDelay = start.Elapsed; Stopwatch sw = new Stopwatch(); using (SqlConnection connection = new SqlConnection(connectionString)) { sw.Start(); await connection.OpenAsync(); Console.WriteLine($"{startDelay} {sw.Elapsed} {index} {connection.ClientConnectionId} {IsReuse(connection)}"); //await Task.Delay(4000); ExecuteQuery(connection, queryTime); } } private static async Task MeasureSingleConnectionAndReuse(string connectionString) { Stopwatch sw = new Stopwatch(); using (SqlConnection connection = new SqlConnection(connectionString)) { sw.Start(); await connection.OpenAsync(); Console.WriteLine($"{sw.Elapsed} {connection.ClientConnectionId} {IsReuse(connection)} Single open time "); } using (SqlConnection connection = new SqlConnection(connectionString)) { sw.Restart(); await connection.OpenAsync(); Console.WriteLine($"{sw.Elapsed} {connection.ClientConnectionId} {IsReuse(connection)} Single open time with one previously opened connection"); } } private static bool IsReuse(SqlConnection connection) { return !_connectionIDs.TryAdd(connection.ClientConnectionId, null); } private static void ExecuteQuery(SqlConnection connection, TimeSpan queryTime) { SqlCommand command = connection.CreateCommand(); command.CommandText = $"WAITFOR DELAY '{queryTime:hh\\:mm\\:ss\\:fff}';"; command.ExecuteNonQuery(); } } }
Here are the results of a single run (Please run it multiple times to get smoother curves). (Low latency network connection opening of single connection is around 200ms and High latency network connection opening of single connection is around 1000ms)
It still shows that the non pooled approach is still faster in almost all scenarios. Only opening one connection at a time when in pooled mode when connecting to a database over a high latency connection is shown to be significantly slower than using the non pooled mode!
As a separate second issue note also that in both modes the opening of connections is done by blocking threads, and not by awaiting the network traffic! This needlessly blocks threads instead of awaiting them internally. Let me know if you would like me to file a separate issue for this.
Reacted by Jan Klass@OQO Thanks for the update. Just a quick test recommendation:
By design, when pooling is enabled the best practice is to create one single connection first. That will create the pool and after go for the rest. By doing this you create a pool based on MinPoolSize first and then it goes through all other connections. we call thispool warm up. Can you try this scenario please
Thanks.There is another interesting issue in here. This only happens on delays. Right now the server Waits for the queryTime. When I change the query to a buffer, for example
var buffer = new object[100]; command.CommandText = $"SELECT * FROM <tableeName>"; using SqlDataReader reader = command.ExecuteReader(); while (reader.Read()) { reader.GetValues(buffer); }code runs again as expected. I will look into the part that a single connection is made and pool is created properly, but a clarifying question, we have to put the delay either in the code task or in the server?
I will take a deeper look tomorrow.
You can also change the query time to 50 milliseconds or similar value. This will lead to the same results as the query you are putting in instead of waiting. I prefer to put wait times in so that we get more reliable query times during testing.
The problem observed in the graphs only occur if we keep the connections busy. If the connections are only busy with queries for very short amount of times, then they can be reused more quickly, and there will be no problems. However, in many real life scenarios (Like in ours)queries take more than 500ms to complete.
Thanks to @OQO for raising this issue and the comment
As a separate second issue note also that in both modes the opening of connections is done by blocking threads, and not by awaiting the network traffic! This needlessly blocks threads instead of awaiting them internally. Let me know if you would like me to file a separate issue for this.
We ran into an issue in production when switching from System.Data.SqlClient to Microsoft.Data.SqlClient with upgrading to netcore3.1 where we had some sync over async code that we have not had the time to switch over completely yet (bad I know). The end result was that when a new server came up under load, a whole bunch of threads got blocked waiting on the connections to be created for the pool and then our the threads consumed continued to increase on our servers and never recovered (partly also because our sync over async increased the threads blocked). We are working on async all the way, but even as we're making progress can still see this issue. We used the dotnet counters to look at the sql connections and the active connections ended up pegged at the max and not freed up. The retrieved/released number dropped really low when we saw this problem as well.
Thanks to @JRahnama for the suggestion to warm up the connection pools with the MinPoolSize - that resolved this issue for us. We're now testing out what those numbers should be in our multi-tenant environment and how it affects our SQL servers.
27 remaining items
- added a commit that references this issue
on Aug 24, 2026 The new pool has reached feature parity and will be available behind an app context switch starting in 7.1.0 (
UseConnectionPoolV2). Look out for more details in the coming release notes. It will stay gated behind the context switch until the 8.0.0 release. I will leave this issue open until 8.0.0.Reacted by Wraith, Cheena Malhotra, Martin Gasser, David Rettenbacher, Anton Egorov, Justin Adler, Erik Ejlskov Jensen and Rune MobergReacted by Kyle Wascher, Giorgi Chakhidze and Justin Adler
🎉🎉🎉 WOOT! this is awesome! Can't wait to get this into our apps!
Reacted by Anton Egorov and Thiago Oliveira Santos/triage
We've pushed the 8.0 release date forward and as a result, we now won't have enough time to gather enough data to enable Pool V2 by default in that release. Pool V2 will still be available using the app context switch. We'll now evaluate making it the default in 8.1.
Reacted by Cheena Malhotra
Metadata
Metadata
Assignees
Type
Projects
- Status
Describe the bug
Multiple opens of connections are very slow and seem to be bocking. In pooled mode it seems only a very limited amount of new connections is being opened concurrently (in example below it seems only 1 seems to be opened up at a time). In non-pooled mode more connections are opened concurrently (in example below 8 connections seem to be opened at a time).
Issue has big impact when connecting to SQL Azure from location with higher latency connection. Issue does not have measurable impact when connecting to SQL Azure when in the same datacenter due to super fast connection times.
This issue has a great impact on startup time of our servers, since it takes up to several minutes before our web servers can handle all the requests due to the slow opening of the SQL connections.
To reproduce
Expected behavior
The code first connects sequentially three times to get a good base line of the OpenAsync times of a new connection and a reused connection from the pool.
Then it opens 100 connections in parallel first in pooled mode, then in non-pooled mode. It keeps every connection open for 4 seconds to simulate a slow query (note that even with lower query times the same problems are observed). Note that number of threads of application stays low the whole time, so this is not a thread pool issue.
In the parallel runs we log the time since the start of the experiment to ensure that delays are not due to slow thread pool scheduling. All these numbers are very low and show that all OpenAsync calls are more or less done at the same time. Note that CPU usage is also extremely low during these tests.
The second displayed time shows the time it took till for OpenAsync to return.
Note that doing the total test of 100 queries in non-pooled mode is on average twice as fast as in pooled mode which is very unexpected.
Even in non-pooled mode looking at the times it seems connections are created in batches (in this case of eight) and block the creation of new connections.
In pooled mode only one new connection to SQL server seems to get opened at a time.
In pooled mode we observe one worker thread at a time working (and mostly waiting/blocking) opening new connections. In pooled mode we observe 8 threads at a time working on new connections (and mostly waiting/blocking). It seems that the code internally for OpenAsync for opening connections seem to be blocking worker threads instead of using async to wait for the network. Note however, that this seems not to be the root cause of the slowness in opening new connections. Otherwise we would see an attempt at using more threads to open connections.
Further technical details
Microsoft.Data.SqlClient version: 2.0.0-preview4.20142.4 (same problem on 1.1.3 and on System.Data.SqlClient)
.NET target: .net Core 3.1 (Same issue on regular .net)
SQL Server version: SQL Azure
Operating system: Windows 10 (But also on Windows Server editions and on Ubuntu)
Additional context
Results obtained from test run: