Bump Helix workitem timeout for libraries outerloop runs by akoeplinger · Pull Request #49876 · dotnet/runtime

akoeplinger · 2021-03-19T16:00:31Z

System.IO.Compression.Brotli.Tests in debug configuration (which we use in PR builds) takes 870 seconds already which is quite close to the existing 900s timeout.

=== TEST EXECUTION SUMMARY ===
   System.IO.Compression.Brotli.Tests  Total: 121, Errors: 0, Failed: 0, Skipped: 0, Time: 870.888s

This caused the timeout seen in https://helix.dot.net/api/2019-06-17/jobs/4f8db33f-41d8-43c4-97b9-a65dfd229437/workitems/System.IO.Compression.Brotli.Tests/console

ghost · 2021-03-19T16:00:40Z

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

System.IO.Compression.Brotli.Tests takes 870 seconds already which is quite close to the 900s timeout.

This caused the timeout seen in https://helix.dot.net/api/2019-06-17/jobs/4f8db33f-41d8-43c4-97b9-a65dfd229437/workitems/System.IO.Compression.Brotli.Tests/console

Author:	akoeplinger
Assignees:	-
Labels:	`area-Infrastructure-libraries`
Milestone:	-

akoeplinger · 2021-03-19T16:00:57Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2021-03-19T16:01:12Z

Azure Pipelines successfully started running 1 pipeline(s).

System.IO.Compression.Brotli.Tests takes 870 seconds already which is quite close to the 900s timeout.

akoeplinger · 2021-03-19T16:03:32Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2021-03-19T16:03:45Z

Azure Pipelines successfully started running 1 pipeline(s).

ViktorHofer · 2021-03-19T17:08:33Z

cc @jeffhandley @adamsitnik @carlossanlop @jozkee

Any idea why the Brotli tests take that long on Outerloop?

akoeplinger · 2021-03-19T23:15:37Z

Any idea why the Brotli tests take that long on Outerloop

I think the reason is that in PR builds we run with the Debug config while on rolling builds it uses Release (which takes about 230sec)

ViktorHofer · 2021-03-23T16:19:04Z

My concern was more an overall question about if Brotli tests are intended to take that long unrelated to the configuration used.

akoeplinger · 2021-03-23T20:42:54Z

Can we move forward with this? The timeout bump is needed right now and the failed tests in this PR are already known/tracked by #44352.

adamsitnik · 2021-04-30T07:46:00Z

The Brotli tests predate me, could you please tell me how I can check the execution times of given tests using our infra? Once I have at least a single troublemaking test name I can take a look

ViktorHofer · 2021-04-30T12:56:16Z

That should i.e. be possible via a Kusto query. @safern @wfurt do you have such a query handy?

wfurt · 2021-04-30T18:03:16Z

 Jobs
| project JobId, Name, Source, Type, Build, Started, Finished, QueueName
| where Started >= ago(10d) 
| where Source ==  "ci/public/dotnet/runtime/refs/heads/main" and Type == "test/functional/cli/outerloop/"
| join kind=inner
(WorkItems | project FriendlyName, JobId, Status, Finished - Started, ExitCode, FailCount
| where FriendlyName == "System.IO.Compression.Brotli.Tests") on JobId

ARM seems 2x-3x slower than x64.

akoeplinger · 2021-05-12T09:48:31Z

@wfurt I think that's a bit misleading since from what I can see we run different build configs in main vs. PR outerloop runs.

E.g. if you just look at PR outerloop runs then all of the System.IO.Compression.Brotli.Tests runs that took longer than 12mins in the last 10 days were on amd64:

 Jobs
| project JobId, Name, Source, Type, Build, Started, Finished, QueueName, Repository
| where Started >= ago(10d) 
| where Source startswith "pr/public/dotnet/runtime" and Type == "test/functional/cli/outerloop/"
| join kind=inner
(WorkItems | project FriendlyName, JobId, Status, Duration=Finished - Started, ExitCode, FailCount
| where FriendlyName == "System.IO.Compression.Brotli.Tests") on JobId
| where Duration > 12m

QueueName	Status	Duration
windows.10.amd64.server19h1.es.open.rt	Pass	00:12:04.0480000
windows.10.amd64.serverrs5.open.rt	Pass	00:13:16.4310000
windows.81.amd64.open.rt	Pass	00:12:16.1240000
windows.10.amd64.server19h1.es.open.rt	Pass	00:13:18.4820000
windows.10.amd64.server20h1.open.rt	Pass	00:13:51.4430000
windows.10.amd64.serverrs5.open.rt	Pass	00:12:13.1330000
windows.81.amd64.open.rt	Pass	00:12:23.2540000
windows.10.amd64.server20h1.open.rt	Pass	00:12:41.5990000
windows.81.amd64.open.rt	Pass	00:13:10.5980000
windows.10.amd64.server20h1.open.rt	Pass	00:13:11.5580000
ubuntu.1804.amd64.open.rt	Pass	00:12:07.4310000
windows.10.amd64.server19h1.es.open.rt	Pass	00:12:24.9460000
windows.10.amd64.server20h1.open.rt	Pass	00:13:30.9770000
windows.10.amd64.serverrs5.open.rt	Pass	00:13:18.4470000
windows.81.amd64.open.rt	Pass	00:12:47.8390000
windows.10.amd64.server19h1.es.open.rt	Pass	00:12:42.3510000
windows.10.amd64.server20h1.open.rt	Pass	00:12:57.0060000
windows.10.amd64.serverrs5.open.rt	Pass	00:12:41.2640000
windows.10.amd64.server19h1.es.open.rt	Pass	00:12:30.0630000

@adamsitnik unfortunately we don't track data about passing tests in Kusto, only in AzDO (and they're not quite easy to extract from there)

ericstj · 2021-06-07T20:41:56Z

@akoeplinger are you still interested in this change?

akoeplinger · 2021-06-09T10:17:06Z

@ericstj Yeah I still think this is a good idea since the outerloop runs tend to take longer by design.

ericstj · 2021-06-09T17:28:59Z

OK, what are next steps here? My original comment was due to the PR age. Do you just need reviewers to approve?

akoeplinger · 2021-06-10T12:40:59Z

@ericstj yeah, just need someone to approve.

ViktorHofer · 2021-06-10T14:37:25Z

@adamsitnik does the data that @akoeplinger provided help?

ViktorHofer · 2021-06-10T14:38:23Z

This is good to go in but our policy says that before merging the CI build should be recent so let me retrigger CI.

ViktorHofer · 2021-06-10T14:38:39Z

/azp run runtime-libraries-coreclr outerloop

ViktorHofer · 2021-06-10T14:38:45Z

/azp run runtime

ViktorHofer · 2021-06-10T14:38:52Z

/azp run dotnet-linker-tests

azure-pipelines · 2021-06-10T14:38:54Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2021-06-10T14:39:03Z

Azure Pipelines successfully started running 1 pipeline(s).

ViktorHofer · 2021-06-10T14:39:05Z

/azp run runtime-dev-innerloop

azure-pipelines · 2021-06-10T14:39:14Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2021-06-10T14:39:15Z

Azure Pipelines successfully started running 1 pipeline(s).

ViktorHofer · 2021-06-10T14:39:28Z

/azp run runtime-staging

azure-pipelines · 2021-06-10T14:39:41Z

Azure Pipelines successfully started running 1 pipeline(s).

akoeplinger added the area-Infrastructure-libraries label Mar 19, 2021

akoeplinger requested a review from ViktorHofer March 19, 2021 16:00

akoeplinger mentioned this pull request Mar 19, 2021

Big-endian fixes: networking stack #48398

Merged

akoeplinger force-pushed the bump-outerloop-workitem-timeout branch from eca8036 to 0868a90 Compare March 19, 2021 16:02

Bump Helix workitem timeout for libraries outerloop runs

e3e8f84

System.IO.Compression.Brotli.Tests takes 870 seconds already which is quite close to the 900s timeout.

akoeplinger force-pushed the bump-outerloop-workitem-timeout branch from 0868a90 to e3e8f84 Compare March 19, 2021 16:03

karelz mentioned this pull request May 10, 2021

Test failure System.Net.Sockets.Tests.LoggingTest.EventSource_EventsRaisedAsExpected #50639

Closed

ViktorHofer approved these changes Jun 10, 2021

View reviewed changes

akoeplinger merged commit 5bc8807 into dotnet:main Jun 15, 2021

akoeplinger deleted the bump-outerloop-workitem-timeout branch June 15, 2021 12:10

karelz mentioned this pull request Jun 17, 2021

[wasm] SendReceive_VaryingLengthBuffers_Success failing in CI #53957

Closed

ghost locked as resolved and limited conversation to collaborators Jul 15, 2021

Conversation

akoeplinger commented Mar 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Mar 19, 2021

Uh oh!

akoeplinger commented Mar 19, 2021

Uh oh!

azure-pipelines Bot commented Mar 19, 2021

Uh oh!

akoeplinger commented Mar 19, 2021

Uh oh!

azure-pipelines Bot commented Mar 19, 2021

Uh oh!

ViktorHofer commented Mar 19, 2021

Uh oh!

akoeplinger commented Mar 19, 2021

Uh oh!

ViktorHofer commented Mar 23, 2021

Uh oh!

akoeplinger commented Mar 23, 2021

Uh oh!

adamsitnik commented Apr 30, 2021

Uh oh!

ViktorHofer commented Apr 30, 2021

Uh oh!

wfurt commented Apr 30, 2021

Uh oh!

akoeplinger commented May 12, 2021

Uh oh!

ericstj commented Jun 7, 2021

Uh oh!

akoeplinger commented Jun 9, 2021

Uh oh!

ericstj commented Jun 9, 2021

Uh oh!

akoeplinger commented Jun 10, 2021

Uh oh!

ViktorHofer commented Jun 10, 2021

Uh oh!

ViktorHofer commented Jun 10, 2021

Uh oh!

ViktorHofer commented Jun 10, 2021

Uh oh!

ViktorHofer commented Jun 10, 2021

Uh oh!

ViktorHofer commented Jun 10, 2021

Uh oh!

azure-pipelines Bot commented Jun 10, 2021

Uh oh!

azure-pipelines Bot commented Jun 10, 2021

Uh oh!

ViktorHofer commented Jun 10, 2021

Uh oh!

azure-pipelines Bot commented Jun 10, 2021

Uh oh!

azure-pipelines Bot commented Jun 10, 2021

Uh oh!

ViktorHofer commented Jun 10, 2021

Uh oh!

azure-pipelines Bot commented Jun 10, 2021

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

akoeplinger commented Mar 19, 2021 •

edited

Loading