Repository navigation
Test failure JIT\\opt\\ObjectStackAllocation\\ObjectStackAllocationTests\\ObjectStackAllocationTests.cmd #81103
Description
Failed in Run: runtime-coreclr outerloop 20230124.1 and many ohter outerloop runs
Failed tests:
R2R-CG2 windows x64 Checked @ Windows.10.Amd64.Open
R2R-CG2 windows x64 Checked no_tiered_compilation @ Windows.10.Amd64.Open
R2R-CG2 windows arm64 Checked no_tiered_compilation @ Windows.11.Arm64.Open
R2R-CG2 windows arm64 Checked @ Windows.11.Arm64.Open
- JIT\\opt\\ObjectStackAllocation\\ObjectStackAllocationTests\\ObjectStackAllocationTests.cmd
Error message:
Fatal error. Internal CLR error. (0x80131506)
Return code: 1
Raw output file: C:\h\w\950D07E7\w\B73309EE\uploads\Reports\JIT.opt\ObjectStackAllocation\ObjectStackAllocationTests\ObjectStackAllocationTests.output.txt
Raw output:
BEGIN EXECUTION
ObjectStackAllocationTests.dll
1 file(s) copied.
11:01:24.34
Response file: C:\h\w\950D07E7\w\B73309EE\e\JIT\opt\ObjectStackAllocation\ObjectStackAllocationTests\ObjectStackAllocationTests.dll.rsp
C:\h\w\950D07E7\w\B73309EE\e\JIT\opt\ObjectStackAllocation\ObjectStackAllocationTests\IL-CG2\ObjectStackAllocationTests.dll
-o:C:\h\w\950D07E7\w\B73309EE\e\JIT\opt\ObjectStackAllocation\ObjectStackAllocationTests\ObjectStackAllocationTests.dll
--targetarch:arm64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\950D07E7\p\System.*.dll
-r:C:\h\w\950D07E7\p\Microsoft.*.dll
-r:C:\h\w\950D07E7\p\mscorlib.dll
-r:C:\h\w\950D07E7\p\netstandard.dll
-O
" "dotnet" "C:\h\w\950D07E7\p\crossgen2\crossgen2.dll" @"C:\h\w\950D07E7\w\B73309EE\e\JIT\opt\ObjectStackAllocation\ObjectStackAllocationTests\ObjectStackAllocationTests.dll.rsp" --codegenopt JitObjectStackAllocation=1 -r:C:\h\w\950D07E7\w\B73309EE\e\JIT\opt\ObjectStackAllocation\ObjectStackAllocationTests\IL-CG2\*.dll"
11:01:25.66
Crossgen2 failed with exitcode - -1073741819
Stack trace:
at JIT_opt._ObjectStackAllocation_ObjectStackAllocationTests_ObjectStackAllocationTests_._ObjectStackAllocation_ObjectStackAllocationTests_ObjectStackAllocationTests_cmd()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Activity
JulieLeeMSFT commentedon Jan 24, 2023
@AndyAyersMS, it is blocking outerloop for many days. Please take a look.
AndyAyersMS commentedon Jan 24, 2023
Looks like this started around 1/12:
AndyAyersMS commentedon Jan 25, 2023
Failure history above seems to implicate this commit range but nothing there looks all that likely:
[Note we have since determined the issue is with the crossgen 2 clr host, which is using older bits]
2038514 Fix Configuration to ensure calling the property setters. (#80438)
00596e5 Adding the Vector512 and Vector512 types (#76642)
5a48c62 Invalidate buffered data when bypassing the FileStream cache (#80432)
59649c3 Get entry assembly name from host for GetProcessInfo2 command during suspension (#80301)
8960ab9 Fix small buffer handling with the Platform Crypto Provider
1d0a69f Fix timeout in regex source generated test (#80459)
9084883 Fix NetFramework name in Directory.Build.targets (#80440)
b0eee49 Address feedback from #79828. (#80509)
3f03037 Fix build invocation on Windows without os param (#80544)
a40d411 [main] Update dependencies from dotnet/arcade (#80234)
25e84d3 [mono] Fixed getting namespace names of array types. (#80426)
d4a59b3 Remove two string.Replace calls from CreditCardAttribute.IsValid (#80523)
d87f7b2 (upstream/dnceng-wasm-test-reporting-experiment) Disable
ExposedLocalsNumbering(#80522)AndyAyersMS commentedon Jan 25, 2023
Can't repro this locally or with runfo bits. From a recent core dump it looks like it is a crash in the dotnet used to run crossgen2, which is
8.0.0-alpha.1.23058.2\coreclr.dll from hash 5da4a9e
I can't find symbols for this yet...
AndyAyersMS commentedon Jan 25, 2023
Ok, found them (thanks
dotnet-symbols). This seems to be crashing in the runtime.@mangod9 @janvorli does this seem at all familiar? I can look at a few more dumps to see if they're all consistent.
janvorli commentedon Jan 25, 2023
I am looking into it.
AndyAyersMS commentedon Jan 25, 2023
Added more stack trace to the above -- this is happening right at startup:
janvorli commentedon Jan 25, 2023
There is something weird going on in this loop:
runtime/src/coreclr/vm/precode.cpp
Lines 479 to 488 in 9b76c28
The
temporaryEntryPointsis 0x00007ffd86116000, thecountshould be 0xaa (that I have dug out in the coreclr!MethodTableBuilder::SetupMethodTable2 frame from thepChunk) and theoneSizeis 0x18. In theFixupPrecode::Init, we are accessing the data of those entry points that are one page after that address, so they start at 0x00007ffd86117000. That means that at the end of the loop, we should reach 0x00007ffd86117ff0. But we are crashing since we went to 0x00007ffd86118000, which is the next page and it is not mapped (the fact that it is not mapped is expected).janvorli commentedon Jan 25, 2023
The issue is not in the source code tree state that is being tested but rather in the source code tree state of the runtime used by crossgen2, which is the 8.0.0-alpha.1.23058.2\coreclr.dll from hash 5da4a9e as Andy has mentioned.
AndyAyersMS commentedon Jan 25, 2023
Not sure if this is what you meant -- in the dump I'm looking at the faulting address is mapped/reserved but not committed:
From what I can see all the failures are on windows, mostly x64 but some on arm64. Maybe some kind of off by one setting up initial allocations that (for reasons unknown) only hits this particular test? Seems quite odd because crossgen2 has to run hundreds of times during this test pass, and only this invocation fails, and (I suspect) doesn't always fail.
12 remaining items
AndyAyersMS commentedon Jan 25, 2023
Seems like we'll also need to disable this test and wait for a dotnet update to re-enable, right...? Fixing this in main won't completely resolve the issue as crossgen2 will still be using an older runtime.
janvorli commentedon Jan 25, 2023
Ah, that's right.
Disable the /JIT/opt/ObjectStackAllocation/ObjectStackAllocationTests
MichalStrehovsky commentedon Jan 25, 2023
Alternative fix to just disabling the test would be to add the problematic environment variables to the adhoc list crossgen2 testing already suppresses:
runtime/src/tests/Common/CLRTest.CrossGen.targets
Lines 273 to 278 in eaf0bb2
janvorli commentedon Jan 26, 2023
That's something that we should probably do too, it doesn't seem to make sense to run crossgen with the settings that are meant for the actual test run. But let's keep it as is for now, it will allow us to double check that the issue is gone after my fix propagates to the dotnet runtime used for build / crossgen execution.
MichalStrehovsky commentedon Jan 26, 2023
I don't know for how long we're going to run crossgen2 on top of the JIT-based CoreCLR in testing. There have been thoughts to use the shipping configuration, which is crossgen2 compiled with NativeAOT. So we're going to lose this as a "regression test" eventually.
Fix max chunk size limiting (#81192)
Disable the /JIT/opt/ObjectStackAllocation/ObjectStackAllocationTests (…
Fix max chunk size limiting
Fix max chunk size limiting
Fix max chunk size limiting (#81607)