System.Collections.Concurrent.Tests crashing in CI #45517

Activity

1. ViktorHofer

   added

   blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

   area-System.Collections

   on Dec 3, 2020
3. ViktorHofer commented on Dec 3, 2020

   

   ViktorHofer

   on Dec 3, 2020

   MemberAuthor

   More actions

   cc @jkotas in case you are interested in the dump
4. jkotas commented on Dec 3, 2020

   

   jkotas

   on Dec 3, 2020

   Member

   More actions

   Crash during background GC handle table scan:

   (lldb) bt
   * thread #1, name = 'dotnet', stop reason = signal SIGSEGV
     * frame #0: 0x00007f0e1aef3ed0 libcoreclr.so`WKS::gc_heap::background_mark_simple1(unsigned char*) [inlined] MethodTable::GetFlag(this=<unavailable>, flag=enum_flag_HasComponentSize) const at methodtable.h:3705:16
       frame #1: 0x00007f0e1aef3ed0 libcoreclr.so`WKS::gc_heap::background_mark_simple1(unsigned char*) [inlined] MethodTable::HasComponentSize(this=<unavailable>) const at methodtable.h:1688
       frame #2: 0x00007f0e1aef3ed0 libcoreclr.so`WKS::gc_heap::background_mark_simple1(unsigned char*) [inlined] WKS::my_get_size(ob=<unavailable>) at gc.cpp:9693
       frame #3: 0x00007f0e1aef3ec9 libcoreclr.so`WKS::gc_heap::background_mark_simple1(oo="i�\xa2\r\x7f") at gc.cpp:20333
       frame #4: 0x00007f0e1aef4882 libcoreclr.so`WKS::gc_heap::background_promote(Object**, ScanContext*, unsigned int) at gc.cpp:20528:17
       frame #5: 0x00007f0e1aef482e libcoreclr.so`WKS::gc_heap::background_promote(ppObject=0x00007f0e1c966110, sc=<unavailable>, flags=<unavailable>) at gc.cpp:20608
       frame #6: 0x00007f0e1ae8245d libcoreclr.so`BlockScanBlocksWithoutUserData(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*) [inlined] ScanConsecutiveHandlesWithoutUserData(pValue=0x00007f0e1c966110, pLast=0x00007f0e1c966200, pInfo=0x00007f0d427fbb70, (null)=<unavailable>) at handletablescan.cpp:443:13
       frame #7: 0x00007f0e1ae82424 libcoreclr.so`BlockScanBlocksWithoutUserData(pSegment=0x00007f0e1c960000, uBlock=40, uCount=1, pInfo=0x00007f0d427fbb70) at handletablescan.cpp:555
       frame #8: 0x00007f0e1ae834b0 libcoreclr.so`xxxTableScanQueuedBlocksAsync(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) [inlined] ProcessScanQNode(pAsyncInfo=<unavailable>, pQNode=<unavailable>) at handletablescan.cpp:1231:9
       frame #9: 0x00007f0e1ae83482 libcoreclr.so`xxxTableScanQueuedBlocksAsync(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1201
       frame #10: 0x00007f0e1ae83436 libcoreclr.so`xxxTableScanQueuedBlocksAsync(pTable=<unavailable>, pSegment=<unavailable>, pCrstHolder=0x00007f0d427fbba0)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1312
       frame #11: 0x00007f0e1ae842ca libcoreclr.so`xxxTableScanHandlesAsync(HandleTable*, unsigned int const*, unsigned int, TableSegment* (*)(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*), void (*)(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*), ScanCallbackInfo*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) [inlined] xxxAsyncSegmentIterator(pTable=0x00005636d227bea0, pPrevSegment=<unavailable>, pCrstHolder=0x00007f0d427fbba0)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1537:9
       frame #12: 0x00007f0e1ae842a6 libcoreclr.so`xxxTableScanHandlesAsync(HandleTable*, unsigned int const*, unsigned int, TableSegment* (*)(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*), void (*)(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*), ScanCallbackInfo*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) [inlined] TableScanHandles(pTable=<unavailable>, puType=<unavailable>, uTypeCount=<unavailable>, pfnSegmentIterator=<unavailable>, pfnBlockHandler=<unavailable>, pInfo=<unavailable>, pCrstHolder=<unavailable>)(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*), void (*)(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*), ScanCallbackInfo*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1699
       frame #13: 0x00007f0e1ae83e32 libcoreclr.so`xxxTableScanHandlesAsync(pTable=0x00005636d227bea0, puType=<unavailable>, uTypeCount=<unavailable>, pfnSegmentIterator=<unavailable>, pfnBlockHandler=<unavailable>, pInfo=<unavailable>, pCrstHolder=0x00007f0d427fbba0)(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*), void (*)(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*), ScanCallbackInfo*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1781
       frame #14: 0x00007f0e1ae7ee71 libcoreclr.so`HndScanHandlesForGC(hTable=0x00005636d227bea0, scanProc=(libcoreclr.so`PromoteObject(Object**, unsigned long*, unsigned long, unsigned long) at objecthandle.cpp:305:5), param1=139694926969976, param2=139698558158720, types=0x00007f0d427fbc28, typeCount=<unavailable>, condemned=2, maxgen=2, flags=2)(Object**, unsigned long*, unsigned long, unsigned long), unsigned long, unsigned long, unsigned int const*, unsigned int, unsigned int, unsigned int, unsigned int) at handletable.cpp:789:9
       frame #15: 0x00007f0e1ae85dfd libcoreclr.so`Ref_TraceNormalRoots(condemned=2, maxgen=2, sc=0x00007f0d427fbc78, fn=(libcoreclr.so`WKS::gc_heap::background_promote(Object**, ScanContext*, unsigned int) at gc.cpp:20553))(Object**, ScanContext*, unsigned int)) at objecthandle.cpp:1059:21
       frame #16: 0x00007f0e1aee7504 libcoreclr.so`WKS::gc_heap::background_mark_phase() at gc.cpp:27932:9
       frame #17: 0x00007f0e1aee6400 libcoreclr.so`WKS::gc_heap::gc1() at gc.cpp:17500:13
       frame #18: 0x00007f0e1af013ac libcoreclr.so`WKS::gc_heap::bgc_thread_function() at gc.cpp:29072:9
       frame #19: 0x00007f0e1adac366 libcoreclr.so`(anonymous namespace)::CreateSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::__invoke(void*) [inlined] (anonymous namespace)::CreateSuspendableThread(this=<unavailable>, argument=<unavailable>)(void*), void*, char16_t const*)::$_0::operator()(void*) const at gcenv.ee.cpp:1376:17
       frame #20: 0x00007f0e1adac301 libcoreclr.so`(anonymous namespace)::CreateSuspendableThread(argument=<unavailable>)(void*), void*, char16_t const*)::$_0::__invoke(void*) at gcenv.ee.cpp:1356
   
5. jkotas commented on Dec 3, 2020

   

   jkotas

   on Dec 3, 2020

   Member

   More actions

   GC heap is corrupted. The corrupted objects are System.Xml.Linq.XAttributes used by xunit harness for report.

   (lldb) verifyheap
   object 00007f0d6c2bffc0 contains free object 00007F0D6C2BFE10 at 00007F0D6C2BFFD8
   Last good object: 00007F0D6C2BFE10.
   object 00007f0d6c2c7d20 contains free object 00007F0D6C2C7D58 at 00007F0D6C2C7D48
   Last good object: 00007F0D6C2C7CE0.
   object 00007f0d6c2c83c8 contains free object 00007F0D6C2C8400 at 00007F0D6C2C83E0
   Last good object: 00007F0D6C2C8390.
   object 00007f0d6c2c89a0 contains free object 00007F0D6C2C89D8 at 00007F0D6C2C89B8
   Last good object: 00007F0D6C2C8818.
   

   (lldb) dumpobj 00007f0d6c2bffc0
   Name:        System.Xml.Linq.XAttribute
   MethodTable: 00007f0da202d968
   EEClass:     00007f0da20354e8
   Size:        56(0x38) bytes
   File:        /home/helixbot/work/A19B08C0/p/shared/Microsoft.NETCore.App/6.0.0/System.Private.Xml.Linq.dll
   Fields:
                 MT    Field   Offset                 Type VT     Attr            Value Name
   00007f0da1cdeb58  40001cf        8 ...l.Linq.XContainer  0 instance 00007f0d6c2bc4a8 parent
   00007f0da19d0c18  40001d0       10        System.Object  0 instance 0000000000000000 annotations
   00007f0da202d968  400006a       18 ...l.Linq.XAttribute  0 instance 00007f0d6c2bfe10 next
   00007f0da202d418  400006b       20 ...em.Xml.Linq.XName  0 instance 00007f0d6c1c9820 name
   00007f0da1a97c28  400006c       28        System.String  0 instance 00007f0d6c2bfff8 value
   (lldb) gcroot 00007f0d6c2bffc0
   Thread 5608:
       00007FFD9F287900 00007F0DA1F58113 Xunit.ConsoleClient.ConsoleRunner.ExecuteAssembly(System.Object, Xunit.XunitProjectAssembly, Boolean, Boolean, System.Nullable`1<Boolean>, System.Nullable`1<Int32>, Boolean, Boolean, System.Nullable`1<Xunit.AppDomainSupport>, Boolean, Boolean, Xunit.XunitFilters, Boolean)
           r14:
               ->  00007F0D6C1B0560 Xunit.DelegatingLongRunningTestDetectionSink
               ->  00007F0D6C1B03F0 Xunit.DelegatingXmlCreationSink
               ->  00007F0D6C1B0480 System.Collections.Generic.Dictionary`2[[System.Guid, System.Private.CoreLib],[System.Xml.Linq.XElement, System.Private.Xml.Linq]]
               ->  00007F0D6C3D9240 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, System.Private.CoreLib],[System.Xml.Linq.XElement, System.Private.Xml.Linq]][]
               ->  00007F0D6C2BC4A8 System.Xml.Linq.XElement
               ->  00007F0D6C2BFFC0 System.Xml.Linq.XAttribute
   
   Found 1 unique roots (run 'gcroot -all' to see all roots).
   (lldb)
   
6. jkotas commented on Dec 3, 2020

   

   jkotas

   on Dec 3, 2020

   Member

   More actions

   This needs more dumps to develop pattern. I do not see any more clues.
7. jkotas

   added

   area-VM-coreclr

   tenet-reliabilityReliability/stability related issue (stress, load problems, etc.)Reliability/stability related issue (stress, load problems, etc.)

   needs more info

   and removed

   area-System.Collections

   on Dec 3, 2020
8. stephentoub

   removed

   blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

   on Mar 5, 2021
9. mangod9 commented on Jul 6, 2021

   

   mangod9

   on Jul 6, 2021

   Member

   More actions

   Doesnt look like the test was disabled, so assume this is no longer reproing?
10. mangod9

    added this to the 6.0.0 milestone on Jul 6, 2021
11. mangod9 commented on Jul 10, 2021

    

    mangod9

    on Jul 10, 2021

    Member

    More actions

    @ViktorHofer ok to close assuming this hasnt reproed in a while?
12. mangod9 commented on Jul 21, 2021

    

    mangod9

    on Jul 21, 2021

    Member

    More actions

    Closing for now
13. mangod9

    closed this as completedon Jul 21, 2021
14. ghost locked as resolved and limited conversation to collaborators on Aug 20, 2021