Assert failure: m_alignpad == 0 in libraries tests

Hit in #70226

Dump and logs: https://dev.azure.com/dnceng/public/_build/results?buildId=1806089&view=ms.vss-test-web.build-test-results-tab&runId=48090090&resultId=197319&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Stacktrace:

coreclr!DbgAssertDialog+0x1af [D:\a\_work\1\s\src\coreclr\utilcode\debug.cpp @ 594] 
coreclr!ObjHeader::IllegalAlignPad+0x33 [D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp @ 2952] 
coreclr!ObjHeader::GetBits+0x41 [D:\a\_work\1\s\src\coreclr\vm\syncblk.h @ 1545] 
coreclr!ObjHeader::Validate+0x50 [D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp @ 2042] 
coreclr!Object::ValidateInner+0x493 [D:\a\_work\1\s\src\coreclr\vm\object.cpp @ 581] 
coreclr!Object::Validate+0xa1 [D:\a\_work\1\s\src\coreclr\vm\object.cpp @ 498] 
coreclr!OBJECTREF::operator->+0x21 [D:\a\_work\1\s\src\coreclr\vm\object.cpp @ 1242] 
coreclr!MarshalNative::GCHandleInternalAlloc+0x171 [D:\a\_work\1\s\src\coreclr\vm\marshalnative.cpp @ 492] 
System_Private_CoreLib!System.ReadOnlyMemory`1[[System.Byte, System.Private.CoreLib]].Pin()+0xffffffff`a4114a41

This is failing in the next-object validation.

m_alignpad is 0 in the dump. It means that it got cleared between the assert condition was checked and the crash dump was taken.

The most likely explanation is that there is a race condition between GCHeap::NextObj and the memory clearing for newly allocated objects in the GC that causes GCHeap::NextObj to return next object that was not cleared yet. @dotnet/gc Could you please take a look?

Same assert here: #68511

The same assert happened for jitminopts in System.Text.RegularExpressions.Unit.Tests on win-x64:

https://dev.azure.com/dnceng/public/_build/results?buildId=1807559&view=ms.vss-test-web.build-test-results-tab&runId=48105794&resultId=197597&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Same assert failed on System.Buffers.Tests https://dev.azure.com/dnceng/public/_build/results?buildId=1809197&view=ms.vss-test-web.build-test-results-tab&runId=48130508&resultId=197594&paneView=debug in this unrelated PR #70194

note: doesn't repro when I change optimization level from -O2 to -O0 for cee_wks_core (Checked config)

@Maoni0

@Maoni0 has a fix already. I also found a reasonably frequent repro last week - the readytorun\coreroot_determinism\coreroot_determinism coreclr test and run it with her fix over the weekend. Before the fix, it reproed about every 80 iterations. With the fix, 1000 iterations have passed without any repro.

@janvorli

thanks so much @janvorli for verifying. @mrsharm will be submitting a PR.

@EgorBo

and thanks @EgorBo for finding another test that repros consistently.

@EgorBo

Thank you very much, @EgorBo! Based on your list of instructions, I was able to fairly quickly repro the assertion failure on my Windows desktop and have confirmed that after:

Removing the demotion check:

//  if ((g == 0) && hp->settings.demotion)
//        return NULL;//could be racing with another core allocating

Adding a null check on the GCSafeMethodTable:

   Object * nextObj = GCHeapUtilities::GetGCHeap ()->NextObj (this);
            if ((nextObj != NULL) &&
                (nextObj->GetGCSafeMethodTable() != nullptr) &&
                (nextObj->GetGCSafeMethodTable() != g_pFreeObjectMethodTable))

the assertion failure doesn't occur based on running the tests overnight.

The main question I had was: how do I find out which test did the assertion failure occur for?

Right now, I get a pretty general message that there was an assertion failure for a System.Collections.Test but no indication as to which test it failed on:

=== TEST EXECUTION SUMMARY ===
   System.Collections.Tests  Total: 32703, Errors: 0, Failed: 0, Skipped: 0, Time: 26.645s
  Discovering: System.Collections.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Collections.Tests (found 5562 of 7453 test cases)
  Starting:    System.Collections.Tests (parallel test collections = on, max threads = 20)

Assert failure(PID 24604 [0x0000601c], Thread: 18352 [0x47b0]): m_alignpad == 0

CORECLR! ObjHeader::Validate + 0x43 (0x00007ffa`66f8bcc3)
CORECLR! Object::ValidateInner + 0x493 (0x00007ffa`66f12ed3)
CORECLR! Object::Validate + 0xA1 (0x00007ffa`66f12a01)
CORECLR! OBJECTREF::operator-> + 0x21 (0x00007ffa`66f0c0b1)
CORECLR! JIT_ClassProfile32 + 0xC7 (0x00007ffa`671310b7)
<no module>! <no symbol> + 0x0 (0x00007ffa`0a65e527)
<no module>! <no symbol> + 0x0 (0x000000dd`6cafb030)
<no module>! <no symbol> + 0x0 (0x000000dd`6cafac00)
<no module>! <no symbol> + 0x0 (0x00000276`e4fd77f0)
<no module>! <no symbol> + 0x0 (0x00000276`e543c740)
    File: C:\runtime\src\coreclr\vm\syncblk.cpp Line: 2955
    Image: C:\runtime\artifacts\bin\testhost\net7.0-windows-Release-x64\dotnet.exe

  Discovering: System.Collections.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Collections.Tests (found 5562 of 7453 test cases)
  Starting:    System.Collections.Tests (parallel test collections = on, max threads = 20)
  Finished:    System.Collections.Tests

If we know the exact test, we'll have a quicker repro with a more targeted run.

@mrsharm

@mrsharm in my repro you can append -verbose after -notrait category=failing and xunit will print test names
I think it were diffrient tests each run and I wasn't able to reproduce the issue when I was asking xunit to run just one test specifically (via -method xx)

This is still failing in libraries-pgo leg on linux-arm

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-50c328df6da7462690/System.Numerics.Vectors.Tests/1/console.1ae6f61a.log?helixlogtype=result

https://dev.azure.com/dnceng/public/_build/results?buildId=1927773&view=results

This assert can be hit for many different GC holes, GC heap corruptions and GC bugs. It is important to at least capture the stacktrace where the assert is hit, and track different stack traces by separate issues.

I think it is unlikely that the Linux arm crash you have seen is same root cause as this issue. I expect that it is going to have a different stacktrace. Unfortunately, we cannot tell for sure since no dump was collected.

I am closing this as non-actionable. If you see a test failing with this assert, do not re-activate this issue. Instead open a new issue and capture stack trace where the assert is hit in the issue description.

It seems that it is inconsistent. Tried reproing it but doesn't repro. I will keep an eye and open new issue.

Navigation Menu

Repository navigation

Assert failure: m_alignpad == 0 in libraries tests #70231

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Assert failure: m_alignpad == 0 in libraries tests #70231

Description

Activity

jkotas commented on Jun 4, 2022

BruceForstall commented on Jun 5, 2022

BruceForstall commented on Jun 5, 2022

mdh1418 commented on Jun 6, 2022

17 remaining items

EgorBo commented on Jul 17, 2022

janvorli commented on Jul 18, 2022

Maoni0 commented on Jul 18, 2022

Maoni0 commented on Jul 18, 2022

mrsharm commented on Jul 19, 2022

EgorBo commented on Jul 19, 2022

kunalspathak commented on Aug 9, 2022

jkotas commented on Aug 9, 2022

kunalspathak commented on Aug 10, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions