Repository navigation
Problems with Azure Devops Reporter #7371
Description
- This issue is blocking
- This issue is causing unreasonable pain
As I've written up in https://github.com/dotnet/core-eng/issues/13026, we introduced a threading bug in https://github.com/dotnet/arcade/pull/7310/files that can crash the reporter. It may be difficult to intentionally reproduce this problem since it kind of relies on actually doing the reporting, but we can inspect the code or just revert the lock part of the change and instead just do the "don't let it pass if it doesn't finish" part of the change.
Activity
alexperovich commentedon May 11, 2021
The error stack trace in https://github.com/dotnet/core-eng/issues/13026 doesn't have anything to do with the lock I added. This error is happening because the interpreter can't get the lock for "stdout" and none of my change affects locking of stdout. This error is just a very rare race condition where one of the worker threads (Thread 0x000070000f354000 in that stack trace) happens to be inside a _print call at the exact time that the process is trying to exit.
Looking at what it was trying to print. One of the worker threads never got a chance to start, before the work was all completed and the process exited.
MattGal commentedon May 11, 2021
Cool, I am mistaken. We should still perhaps use this or another issue to have the conversation about dealing with failure better.
[-]Mutex bug in Azure Devops Reporter can cause crashes (regression)[/-][+]Add Retries in Azure Devops Reporter can cause crashes (regression)[/+][-]Add Retries in Azure Devops Reporter can cause crashes (regression)[/-][+]Add Retries in Azure Devops Reporter (regression)[/+][-]Add Retries in Azure Devops Reporter (regression)[/-][+]Add Retries in Azure Devops Reporter[/+]MattGal commentedon May 12, 2021
Just had a great conversation with @safern and he made a very keen insight. Specifically, if we just update the arcade reporter behavior to just return the real, not lying exit code when Azure Devops reporter fails, it's easy to implement and gives us the best of both things; when reporting fails, the work item still can pass, and when the work item fails and reporting fails, we still fail the check, e.g.:
MattGal commentedon May 17, 2021
Unfortunately while interesting, this idea doesn't work as simply as I'd hoped, because there's no way to get the decision made in generating RunTests.cmd (example template).
Without removing these lines, there's no way to make the above pseudocode work since the exit code of the test is coerced in code not owned by the Helix Arcade SDK. I will confer with some runtime folks and get an idea of what they'd like to do.
dotnet#7371 - retry in the case of HTTP 503 (10x, 3 seconds between a…
MattGal commentedon May 17, 2021
Going with the "just retry for 503s" approach, #7399
Add retries to Azure Devops Reporter scripts (#7399)
MattGal commentedon May 18, 2021
This is merged and should begin flowing to main-branch Arcade consumers soon, I'll keep this in validate until I see some uptake.
ericstj commentedon May 18, 2021
Another hit on the threading issue: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-52793-merge-d1edb0fae6604797b1/Microsoft.Extensions.Hosting.Functional.Tests/console.f1e93568.log?sv=2019-07-07&se=2021-06-07T20%3A07%3A07Z&sr=c&sp=rl&sig=fZdoSW%2Fcu7ZkOzwiZiYlIOE6rA4GERWu6EbZ98P3QFY%3D
MattGal commentedon May 19, 2021
@ericstj I may take some stabs at this one tomorrow but it's much harder than simply adding retries to AzDO calls.
[-]Add Retries in Azure Devops Reporter[/-][+]Problems with Azure Devops Reporter[/+]Workaround for dotnet#7371. Have azure-pipelines reporter parse XML f…
Intentionally always fail to exercise dotnet#7371
Workaround for #7371 (#7421)
MattGal commentedon May 27, 2021
Added 65 unique test run instances to the IcM, since they seem to be unable to find them.
MattGal commentedon May 27, 2021
THere's not much to do here and the mitigations seem to be helping so putting in Tracking/blocked.
Work around dotnet#7371
MattGal commentedon May 28, 2021
new PR up, we can easily avoid the race condition by eliminating the race entirely.
MattGal commentedon May 28, 2021
I accidentally made what I intended to be a 2s sleep 33 minutes, and oddly pushing an updated version didn't seem to fix this; will pick this back up on Tuesday.
Work around dotnet#7371
Work around #7371 (#7457)
MattGal commentedon Jun 2, 2021
Merged the PR with attempted workaround, will leave this in Validate til it makes it to Runtime.
hoyosjs commentedon Jun 2, 2021
Couple more instances just to track them for CI council:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-f5fbb294940e4bc1be/System.Net.Http.Functional.Tests/console.b785c3d1.log?sv=2019-07-07&se=2021-06-22T05%3A46%3A10Z&sr=c&sp=rl&sig=D6AaihkkUI602TYpBT79V6J3IeBLGUmT39zzC0I7yMU%3D
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-2cc6c8d9f5d44da09c/System.Net.Http.Enterprise.Tests/console.05500c86.log?sv=2019-07-07&se=2021-06-22T14%3A08%3A35Z&sr=c&sp=rl&sig=svdRWf1iYAeYHEDReo9eLUgtNLOnDFsrb67eTlnnnwQ%3D
MattGal commentedon Jun 9, 2021
Current workaround is in Runtime, and seems to be holding, closing (somehow I thought I already had)