Repository navigation
System.Net tests failing with System.TimeoutException #54778
Description
For example:
- Test: System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection
System.TimeoutException : The operation has timed out.
Stack Trace:
/_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs(196,0): at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout)
/_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(1441,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection(String timeoutPropertyName)
--- End of stack trace from previous location ---
Activity
[-]Random System.Net failing with System.TimeoutException[/-][+]System.Net tests failing with System.TimeoutException[/+]jkotas commentedon Jun 26, 2021
Failed in #54656
Examples:
jkotas commentedon Jun 26, 2021
Failed in #54774
Examples:
karelz commentedon Jun 28, 2021
Failures 4/28-6/27 (incl. PRs):
karelz commentedon Jun 28, 2021
Looks like a regression around 6/18-6/23 ... @dotnet/ncl any idea what might have caused the regression?
karelz commentedon Jun 28, 2021
@geoffkizer 2 of the tests are HTTP/3 Mock cookies tests -- is it perhaps caused by your PR #54727?
geoffkizer commentedon Jun 28, 2021
Yes, these tests were disabled until that PR.
Looks like these are still flaky, and apparently much flakier on Linux than Windows for whatever reason.
We should disable these again. I'll put up a PR.
karelz commentedon Jun 28, 2021
Thanks @geoffkizer! We still have 3 tests not covered by your PR ... we need to figure out what to do about them and likely disable them as well.
geoffkizer commentedon Jun 28, 2021
I'm seeing a ton of TimeoutExceptions on my PR to disable these tests. For example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-54843-merge-5fea1e54014e4d049e/System.Net.Http.Functional.Tests/console.1c5cee98.log?sv=2019-07-07&se=2021-07-18T17%3A51%3A18Z&sr=c&sp=rl&sig=eMv7cbmp3wfyHjUEuXsbeRsXa%2FtVrWqCy%2FPhbk8pVPQ%3D
Not sure what is going on here, but it doesn't look isolated to any particular test(s).
MihaZupan commentedon Jun 29, 2021
Top 27 methods by number of failures caused by timeout are all happening on
ubuntu.1604.amd64.open.rt.karelz commentedon Jun 30, 2021
Here's breakdown of main branch per date, demonstrating that something happened on 6/23:
Failures 3/12-6/29 (incl. PRs) - main branch:
Total 6/23-6/29: 1970 ... out of that 362 are Http3_Mock tests
Top hitter tests breakdown:
geoffkizer commentedon Jun 30, 2021
Some of these tests are pretty basic -- single request, no delays or timing issue, etc. Strange.
karelz commentedon Jun 30, 2021
Yes, it almost feels like something changed in infra ...
karelz commentedon Jun 30, 2021
@geoffkizer I updated the table -- there are 4 tests in *_Http3_Mock category. Could they have regressed due to your recent changes?
MattGal commentedon Jun 30, 2021
@karelz it's quite easy for me to re-send a job from > 4 days ago to the current machines and see if it still has the problem, even repeatedly if that's useful. Just pick a "blessed" one and I can send it around.
Some more thoughts:
MihaZupan commentedon Jun 30, 2021
The mass failures across all PRs started happening somewhere around 9pm on the 24th (+- a few h) - that is why I am suspicious of c88da29).
@MattGal Would you be able to run CI on the runtime at 1d84e01 (parent of the above commit)?
I've been running CI on a revert of the commit in #54939 and haven't hit a timeout failure yet.
MattGal commentedon Jun 30, 2021
Yes but I need some details about what you're interested in, please ping me on Teams?
geoffkizer commentedon Jul 1, 2021
GetAsync_CancelDuringResponseHeadersReceived_TaskCanceledQuickly is a test I enabled for HTTP3 in my PR. so it's possible that this is a separate regression from whatever else is going on here. But it seems like we should figure out the general issue here first.
MihaZupan commentedon Jul 1, 2021
Looks like the failures stopped with #55006.
karelz commentedon Jul 20, 2021
There are still some occurrences of test
Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnectionfailures after 7/1 when the above was supposed to be addressed, though they seem to be on old PRs likely without the change: