Repository navigation
Assert failure: !CREATE_CHECK_STRING(pMT && pMT->Validate()) #67046
Description
+ grep COMPlus
COMPlus_TieredCompilation=1
COMPlus_JitStress=2
COMPlus_DbgMiniDumpName=/home/helixbot/dotnetbuild/dumps/coredump.%d.dmp
COMPlus_DbgEnableMiniDump=1
+ ./RunTests.sh --runtime-path /root/helix/work/correlation
----- start Tue Mar 22 07:59:07 UTC 2022 =============== To repro directly: =====================================================
pushd .
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Text.Encodings.Web.Tests.runtimeconfig.json --depsfile System.Text.Encodings.Web.Tests.deps.json xunit.console.dll System.Text.Encodings.Web.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
Discovering: System.Text.Encodings.Web.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Text.Encodings.Web.Tests (found 212 test cases)
Starting: System.Text.Encodings.Web.Tests (parallel test collections = on, max threads = 4)
Assert failure(PID 62 [0x0000003e], Thread: 86 [0x0056]): !CREATE_CHECK_STRING(pMT && pMT->Validate())
File: /__w/1/s/src/coreclr/vm/object.cpp Line: 514
Image: /root/helix/work/correlation/dotnet
Gathering state for process 62 dotnet
Crashing thread 00000056 signal 00000006
Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.62.dmp
Written 164274176 bytes (40106 pages) to core file
GCStress wasn't set, but this is typically a GC info bug. JitStress is probably irrelevant.
Activity
BruceForstall commentedon Apr 15, 2022
After configuring a Linux/arm32 environment (using Docker on Linux/arm64), with Release libraries and Checked coreclr, I can easily reproduce this failure, with
COMPlus_TieredCompilation=1andCOMPlus_JitStress=2. However, it won't repro with:COMPlus_TieredCompilation=0COMPlus_JitStressunsetCOMPlus_GCStress=4COMPlus_ReadyToRun=0-parallel none -maxthreads 1The stack is:
andn it looks like we're reporting slot SP offset 0x20 in
:
but I'm less sure of that. This does match the JitStress=2 codegen from TryFormat, but the
m_codeAddressfield of acodeInfostruct I found doesn't point to the first address of the function, which makes me think this might not be the correct function, or maybe there's some corruption?I don't currently have a working LLDB in my Linux/arm32 Docker setup, so no SOS either.
BruceForstall commentedon Apr 15, 2022
I set up a Raspberry Pi 4 with a 32-bit server Ubuntu installation (Ubuntu 20.04.4 LTS), and I have a working LLDB 10 there, but I can't get the bug to repro in that environment.
BruceForstall commentedon Apr 15, 2022
@janvorli @AndyAyersMS I probably need your help to make any more progress on this (or maybe @janvorli I can hand this off to you?)
BruceForstall commentedon Apr 16, 2022
According to the CI, these are the changes where the issue was introduced:
https://dev.azure.com/dnceng/public/_traceability/runview/changes?currentRunId=1670086
(assuming it's consistently failing, which I believe it is).
There are 189 changes. Not sure if it makes sense trying to identify a culprit. Some notable changes:
BruceForstall commentedon Apr 16, 2022
Ok, I've figured this out.
The problem was introduced by the enabling of fast tail call optimization for arm32 (#66282).
The code that fails is:
which does a fast tail call. When it is setting up the tail call arguments, it turns off GC reporting, since the incoming argument space GC info might be incompatible with the GC info needed by the tail call arguments put in the same location. However, one of the IR nodes in the argument setup is:
When we generate this, we disable GC tracking (again) then re-enable GC tracking (which we shouldn't do). Thus, we end up reporting a range of the tailcall argument setup as interruptible. Most of the slots end up being marked untracked, and gc pointers match with gc pointers. However, one slot is a gcref incoming, and is a byref outgoing, but the untracked slot is marked gcref, not byref. So the byref value is not a legal object pointer, leading to a GC assert.
I think the solution will be to have a "ref count" of "disable gc reporting" requests, not just a true/false.
cc @jakobbotsch
Enable multiple nested "no GC" region requests (#68105)
Enable multiple nested "no GC" region requests (dotnet#68105)