Repository navigation
Segmentation fault in LibraryImportGenerator.Unit.Tests #67031
Description
Observed in https://dev.azure.com/dnceng/public/_build/results?buildId=1676597&view=results:
Discovering: LibraryImportGenerator.Unit.Tests (method display = ClassAndMethod, method display options = None)
Discovered: LibraryImportGenerator.Unit.Tests (found 117 test cases)
Starting: LibraryImportGenerator.Unit.Tests (parallel test collections = on, max threads = 4)
LibraryImportGenerator.UnitTests.Compiles.ValidateSnippetsWithMarshalType [SKIP]
No current scenarios to test.
./RunTests.sh: line 168: 23 Segmentation fault (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig LibraryImportGenerator.Unit.Tests.runtimeconfig.json --depsfile LibraryImportGenerator.Unit.Tests.deps.json xunit.console.dll LibraryImportGenerator.Unit.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Tue Mar 22 21:09:57 UTC 2022 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
For more details please go to the console logs
@dotnet/interop-contrib
Activity
danmoseley commentedon Apr 22, 2022
Looking at the last few they're either
or
They seemed to stop on 4/15 though (the 1st kind was last). @jkotas do you recall either of these getting fixed? I can't find them in some quick searching.
Execute: Web | Desktop | Web (Lens) | Desktop (SAW)
https://engsrvprod.kusto.windows.net/engineeringdata
danmoseley commentedon Apr 22, 2022
There is also a Windows dump (no Unix dumps for some reason). It may be unrelated. This was 4/16 on Windows 10 x64. @AndyAyersMS is this something already fixed?
jkotas commentedon Apr 22, 2022
m_alignpad == 0is a classic GC hole type of crash. I do not recall a fix going in in last week that would be fixing a GC hole.I do not see this dump. Also, the stacktrace does not look like anything else mentioned in this issue so far.
The source generator tests are running orders of magnitude more code than other libraries tests, so they will tend to hit intermittent runtime stress bugs more frequently.
The next action for this one should be to figure out why we are not getting dumps. We are not going to make progress of intermittent crashes like this without dumps.
AndyAyersMS commentedon Apr 22, 2022
I don't know -- from the stack this seems to be related to the TC mechanism.
Looks like there is a windows dump for this merged PR: #67184 under https://dev.azure.com/dnceng/public/_build/results?buildId=1701571&view=ms.vss-test-web.build-test-results-tab
AndyAyersMS commentedon Apr 22, 2022
Updated query to see windows failures (
-1234is a cancelled job so excluded those)danmoseley commentedon Apr 22, 2022
There are in fact Linux dumps - query error.
Execute: Web | Desktop | Web (Lens) | Desktop (SAW)
https://engsrvprod.kusto.windows.net/engineeringdata
Gives URL's to get "how-to-debug-dump.md" for these.
jkotas commentedon Apr 22, 2022
I have looked at number of these and they are all from JIT stress runs, typically hitting asserts in the JIT, not something that runs in standard CI by default.
danmoseley commentedon Apr 22, 2022
Non JIT stress ones too, but on checked CLR, and only up to 4/9.
let wi =
WorkItems
| join kind=leftsemi (Jobs | where Queued > ago (30d) | where Source == "ci/public/dotnet/runtime/refs/heads/main") on $left.JobName == $right.Name
| where ExitCode != 0
| where FriendlyName == "LibraryImportGenerator.Unit.Tests";
Files
| lookup kind=inner wi on $left.WorkItemName == $right.Name
| where ExitCode ==134
| where FileName == "how-to-debug-dump.md"
| join WorkItems on $left.WorkItemName == $right.Name
| join Jobs on $left.JobName == $right.Name
| extend PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
Pipeline = tostring(parse_json(Properties).DefinitionName),
BuildId = tostring(parse_json(Properties).BuildId)
| where Pipeline !contains("jitstress")
| project Timestamp, QueueName, ExitCode, Uri, ConsoleUri, PhaseName, Pipeline, BuildId
danmoseley commentedon Apr 22, 2022
It seems to me we can put aside the Linux issues and use this to track the Windows issues - @AndyAyersMS query works for those.
I probably can't debug further myself. Just trying to figure out next actions on these.
AndyAyersMS commentedon Apr 22, 2022
Seems like this crash is perhaps related to #67144 which was fixed on 4/15 by #67922.
Perhaps these tests are also stressing collectable assemblies?
jkotas commentedon Apr 22, 2022
Yes, that sounds plausible.
jkotas commentedon Apr 23, 2022
This failed on linux-arm64 again in #68436. I have opened a fresh issue that is explicitly about the Segmentation fault on linux-arm64 only.