Skip to content

Crash in MethodTable::MethodDataObject::FillEntryDataForAncestor #119761

Closed

Description

@rolfbjarne

Description

This is a crash I saw once on our bots, I have no way to reproduce it, nor do I know what the process in question was doing (probably 'dotnet build', but it can be any number of things we do during our build).

However, I have a crash report:

dotnet-2025-09-12-093407.ips.txt

This is what the crash report claims the crashing thread was doing:

Thread 12 Crashed:: .NET TP Worker
0   libcoreclr.dylib              	       0x101887444 MethodTable::MethodDataObject::FillEntryDataForAncestor(MethodTable*) + 396
1   libcoreclr.dylib              	       0x1018874e8 MethodTable::MethodDataObject::GetDeclMethodDesc(unsigned int) + 132
2   libcoreclr.dylib              	       0x1018741e0 MemberLoader::FindMethod(MethodTable*, char const*, unsigned char const*, unsigned int, ModuleBase*, MemberLoader::FM_Flags, Substitution const*) + 392
3   libcoreclr.dylib              	       0x1018736f0 MemberLoader::GetDescFromMemberRef(ModuleBase*, unsigned int, MethodDesc**, FieldDesc**, SigTypeContext const*, int, TypeHandle*, int, unsigned char const**, unsigned int*) + 1252
4   libcoreclr.dylib              	       0x1018e0e60 ZapSig::DecodeMethod(ModuleBase*, unsigned char const*, SigTypeContext*, ZapSig::Context*, TypeHandle*, unsigned char const**, unsigned char const**, unsigned char const**, int) + 940
5   libcoreclr.dylib              	       0x1018e0aa8 ZapSig::DecodeMethod(Module*, ModuleBase*, unsigned char const*, TypeHandle*) + 84
6   libcoreclr.dylib              	       0x1018a1d90 ExternalMethodFixupWorker + 544
7   libcoreclr.dylib              	       0x101a6f514 DelayLoad_MethodCall + 88
[...]

I also have a minidump (~6gb), can be found here: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=12376959&view=artifacts&pathAsName=false&type=publishedArtifacts (expand the "minidumps" folder).

If the build has been deleted, or get an access denied error, I can provide a download link to the minidump (note to self: it's in the 'dotnet-runtime-issue-119761' directory in OneDrive).

Configuration

macOS 15.6, arm64

.NET 9.0.110-servicing.25418.5 (probably)

Activity

  1. added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Sep 16, 2025
  2. removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Sep 16, 2025
  3. agocke commented on Sep 19, 2025

    @agocke
    Member

    @davidwrighton Can you take a quick look at this?

  4. added this to the 10.0.0 milestone on Sep 19, 2025
  5. removed
    untriagedNew issue has not been triaged by the area owner
    on Sep 19, 2025
  6. rolfbjarne commented on Oct 22, 2025

    @rolfbjarne
    MemberAuthor
  7. jkotas commented on Nov 19, 2025

    @jkotas
    Member

    I looked at the dump. I am not able to figure out the context of the crash since the context pointer was optimized out. #121773 should make that better.

  8. rolfbjarne commented on Dec 8, 2025

    @rolfbjarne
    MemberAuthor

    This just happened locally for me, I got coredumps:

    @jkotas this is from the latest .NET 11 builds, so I believe it should contain the PR you made a few weeks ago.


    It happened while compiling https://github.com/dotnet/macios/blob/2b7db63a5e7657b684c7bd869db1c28288baaf89/tests/rgen/Microsoft.Macios.Bindings.Analyzer.Tests/Microsoft.Macios.Bindings.Analyzer.Tests.csproj:

    $ ps aux|grep 90241
    rolf             90241   0.0  1.9 413003872 652992 s007  SX+  10:58AM   0:00.77 /Users/rolf/work/dotnet/macios/net11.0/macios/builds/downloads/dotnet-sdk-11.0.100-alpha.1.25605.110/dotnet build Microsoft.Macios.Bindings.Analyzer.Tests/Microsoft.Macios.Bindings.Analyzer.Tests.csproj --verbosity quiet --nologo -consoleLoggerParameters:NoSummary
  9. jkotas commented on Dec 18, 2025

    @jkotas
    Member

    Analysis of the crash dump at https://microsofteur-my.sharepoint.com/:u:/g/personal/rokvin_microsoft_com/IQBQZcpqCDEFQomqbOt7Lwu3AQqlysuAZ0-0XzpaKmYvmtI?e=hC1QPd :

    We crash when running R2R fixups for NuGet.Commands.MSBuildRestoreUtility.ReplayWarningsAndErrorsAsync at:

        frame #0: 0x00000001033157e8 libcoreclr.dylib`MethodTable::MethodDataObject::FillEntryDataForAncestor(MethodTable*) + 400
        frame #1: 0x0000000103315894 libcoreclr.dylib`MethodTable::MethodDataObject::GetDeclMethodDesc(unsigned int) + 140
        frame #2: 0x0000000103301b4c libcoreclr.dylib`MemberLoader::FindMethod(MethodTable*, char const*, unsigned char const*, unsigned int, ModuleBase*, MemberLoader::FM_Flags, Substitution const*) + 364
        frame #3: 0x00000001033014c0 libcoreclr.dylib`MemberLoader::GetDescFromMemberRef(ModuleBase*, unsigned int, MethodDesc**, FieldDesc**, SigTypeContext const*, int, TypeHandle*, int, unsigned char const**, unsigned int*) + 2284
        frame #4: 0x00000001033692c0 libcoreclr.dylib`ZapSig::DecodeMethod(ModuleBase*, unsigned char const*, SigTypeContext*, ZapSig::Context*, TypeHandle*, unsigned char const**, unsigned char const**, unsigned char const**, int) + 944
        frame #5: 0x0000000103368f04 libcoreclr.dylib`ZapSig::DecodeMethod(Module*, ModuleBase*, unsigned char const*, TypeHandle*) + 84
        frame #6: 0x00000001032f8a18 libcoreclr.dylib`LoadDynamicInfoEntry(Module*, unsigned int, unsigned long*, int) + 360
        frame #7: 0x0000000103296358 libcoreclr.dylib`Module::FixupNativeEntry(READYTORUN_IMPORT_SECTION*, unsigned long, unsigned long*, int) + 56
        frame #8: 0x000000010336f404 libcoreclr.dylib`int Module::FixupDelayListAux<Module*, int (Module::*)(READYTORUN_IMPORT_SECTION*, unsigned long, unsigned long*, int)>(unsigned long, Module*, int (Module::*)(READYTORUN_IMPORT_SECTION*, unsigned long, unsigned long*, int), READYTORUN_IMPORT_SECTION*, unsigned int, ReadyToRunLoadedImage*, int) + 316
        frame #9: 0x000000010336d86c libcoreclr.dylib`ReadyToRunInfo::GetEntryPoint(MethodDesc*, PrepareCodeConfig*, int) + 872
        frame #10: 0x000000010332a494 libcoreclr.dylib`MethodDesc::GetPrecompiledR2RCode(PrepareCodeConfig*) + 60
        frame #11: 0x0000000103329db4 libcoreclr.dylib`MethodDesc::PrepareILBasedCode(PrepareCodeConfig*) + 332
        frame #12: 0x00000001032a9d64 libcoreclr.dylib`CodeVersionManager::PublishVersionableCodeIfNecessary(MethodDesc*, CallerGCMode, bool*, bool*) + 1224
        frame #13: 0x000000010332cf28 libcoreclr.dylib`MethodDesc::DoPrestub(MethodTable*, CallerGCMode) + 260
        frame #14: 0x000000010332c974 libcoreclr.dylib`PreStubWorker + 628
        frame #15: 0x00000001034e1a90 libcoreclr.dylib`ThePreStub + 80
    

    Managed stack:

    0000000170A9E088 00000001033157e8 [PrestubMethodFrame: 0000000170a9e088] NuGet.Commands.MSBuildRestoreUtility.ReplayWarningsAndErrorsAsync(System.Collections.Generic.IEnumerable`1<NuGet.ProjectModel.IAssetsLogMessage>, NuGet.Common.ILogger)
    0000000170A9E280 000000010EF0F51C NuGet.Commands.RestoreCommand+<EvaluateNoOpAsync>d__104.MoveNext()
    0000000170A9E350 0000000110D18EC8 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[NuGet.Commands.RestoreCommand+<EvaluateNoOpAsync>d__104, NuGet.Commands]](<EvaluateNoOpAsync>d__104 ByRef)
    0000000170A9E3D0 0000000110D18E1C System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.ValueTuple`3[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]], System.Private.CoreLib]].Start[[NuGet.Commands.RestoreCommand+<EvaluateNoOpAsync>d__104, NuGet.Commands]](<EvaluateNoOpAsync>d__104 ByRef)
    0000000170A9E400 000000010EEB8CD0 NuGet.Commands.RestoreCommand.EvaluateNoOpAsync(NuGet.Common.TelemetryActivity, NuGet.ProjectModel.CacheFile, System.Diagnostics.Stopwatch)
    0000000170A9E480 000000010EF10D34 NuGet.Commands.RestoreCommand+<ExecuteAsync>d__100.MoveNext()
    0000000170A9E650 0000000110D1540C System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[NuGet.Commands.RestoreCommand+<ExecuteAsync>d__100, NuGet.Commands]](<ExecuteAsync>d__100 ByRef)
    0000000170A9E740 0000000110D151FC System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[NuGet.Commands.RestoreCommand+<ExecuteAsync>d__100, NuGet.Commands]](<ExecuteAsync>d__100 ByRef)
    0000000170A9E770 000000010EEB7DF0 NuGet.Commands.RestoreCommand.ExecuteAsync(System.Threading.CancellationToken)
    0000000170A9E850 000000010EF1B948 NuGet.Commands.RestoreRunner+<ExecuteAsync>d__8.MoveNext()
    0000000170A9E880 0000000110D138EC System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[NuGet.Commands.RestoreRunner+<ExecuteAsync>d__8, NuGet.Commands]](<ExecuteAsync>d__8 ByRef)
    0000000170A9E8F0 0000000110D1343C System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[NuGet.Commands.RestoreRunner+<ExecuteAsync>d__8, NuGet.Commands]](<ExecuteAsync>d__8 ByRef)
    0000000170A9E920 000000010EEC1D10 NuGet.Commands.RestoreRunner.ExecuteAsync(NuGet.Commands.RestoreSummaryRequest, System.Threading.CancellationToken)
    0000000170A9E980 000000010EF1B630 NuGet.Commands.RestoreRunner+<ExecuteAndCommitAsync>d__7.MoveNext()
    0000000170A9E9B0 0000000110D13250 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[NuGet.Commands.RestoreRunner+<ExecuteAndCommitAsync>d__7, NuGet.Commands]](<ExecuteAndCommitAsync>d__7 ByRef)
    0000000170A9EA20 0000000110D13154 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[NuGet.Commands.RestoreRunner+<ExecuteAndCommitAsync>d__7, NuGet.Commands]](<ExecuteAndCommitAsync>d__7 ByRef)
    0000000170A9EA50 000000010EEC1C68 NuGet.Commands.RestoreRunner.ExecuteAndCommitAsync(NuGet.Commands.RestoreSummaryRequest, NuGet.Commands.IRestoreProgressReporter, System.Threading.CancellationToken)
    0000000170A9EAC0 000000010585F95C System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].InnerInvoke()
    0000000170A9EAE0 0000000105439ED0 System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
    0000000170A9EB10 000000010545E6F4 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
    0000000170A9EB80 00000001054490F8 System.Threading.ThreadPoolWorkQueue.Dispatch()
    

    We are running fixups for System.Linq.Enumerable Empty method. The problem is that the MethodTable that we are trying to lookup the Empty method in is corrupted.

    • MethodTable itself looks reasonable:
    - 0:009> dt MethodTable 0x00000001074d88e0 
    libcoreclr!MethodTable
       +0x000 m_dwFlags        : 0
       +0x004 m_BaseSize       : 0x18
       +0x008 m_dwFlags2       : 0x742
       +0x00c m_wNumVirtuals   : 4
       +0x00e m_wNumInterfaces : 0
       +0x010 m_pParentMethodTable : 0x00000001`05250878 MethodTable
       +0x018 m_pModule        : 0x00000001`074d48c8 Module
       +0x020 m_pAuxiliaryData : 0x00000001`074d91e8 MethodTableAuxiliaryData
       +0x028 m_pEEClass       : 0x00000001`05283ac0 EEClass
       +0x028 m_pCanonMT       : 0x00000001`05283ac0
       +0x030 m_pPerInstInfo   : (null) 
    
    • EEClass linked from the MethodTable is either bogus or corrupted:
    0:009> dt EEClass 0x00000001`05283ac0
    libcoreclr!EEClass
       +0x000 m_pGuidInfo      : 0x00000001`10d2df48 GuidInfo
       +0x008 m_rpOptionalFields : 0x00000001`10d2df80 EEClassOptionalFields
       +0x010 m_pMethodTable   : 0x00000001`10d2dfb8 MethodTable
       +0x018 m_pFieldDescList : 0x00000001`10d2e028 FieldDesc
       +0x020 m_pChunks        : 0x00000001`10d2dff0 MethodDescChunk
       +0x028 m_dwAttrClass    : 0x10d32038
       +0x02c m_VMFlags        : 1
       +0x030 m_NormType       : 0x70 'p'
       +0x031 m_cbBaseSizePadding : 0x20 ' '
       +0x032 m_NumInstanceFields : 0x10d3 <= obviously bogus value
       +0x034 m_NumMethods     : 1
       +0x036 m_NumStaticFields : 0
       +0x038 m_NumHandleStatics : 0x5f8
       +0x03a m_NumThreadStaticFields : 0x654
    

    Unclear how the EEClass got corrupted.

  10. jkotas commented on Dec 18, 2025

    @jkotas
    Member

    EEClass is overwritten by VirtualStubDispatch cells:

    0:009> dq 0x00000001`05283ac0
    00000001`05283ac0  00000001`10d2df48 00000001`10d2df80
    00000001`05283ad0  00000001`10d2dfb8 00000001`10d2e028
    00000001`05283ae0  00000001`10d2dff0 00000001`10d32038
    00000001`05283af0  00000001`10d32070 00000001`065405f8
    00000001`05283b00  00000001`065405f8 00000001`10d2d180
    00000001`05283b10  00000001`10d2d180 00000001`10d321f8
    00000001`05283b20  00000001`065405f8 00000001`06530700
    00000001`05283b30  00000001`10d2d1c0 00000001`10d2d1c0
    0:009> u 00000001`10d2df48
    00000001`10d2df48 f940000d ldr         x13,[x0]
    00000001`10d2df4c 100000e9 adr         x9,00000001`10d2df68
    00000001`10d2df50 a940312a ldp         x10,x12,[x9]
    00000001`10d2df54 eb0a01bf cmp         x13,x10
    00000001`10d2df58 54000041 bne         00000001`10d2df60
    00000001`10d2df5c d61f0180 br          x12
    00000001`10d2df60 580000c9 ldr         x9,00000001`10d2df78
    00000001`10d2df64 d61f0120 br          x9
    0:009> u  00000001`10d2df80
    00000001`10d2df80 f940000d ldr         x13,[x0]
    00000001`10d2df84 100000e9 adr         x9,00000001`10d2dfa0
    00000001`10d2df88 a940312a ldp         x10,x12,[x9]
    00000001`10d2df8c eb0a01bf cmp         x13,x10
    00000001`10d2df90 54000041 bne         00000001`10d2df98
    00000001`10d2df94 d61f0180 br          x12
    00000001`10d2df98 580000c9 ldr         x9,00000001`10d2dfb0
    00000001`10d2df9c d61f0120 br          x9
    
  11. 30 remaining items

  12. AndyAyersMS commented on Jun 19, 2026

    @AndyAyersMS
    Member

    Note

    This comment was generated by Copilot.

    Have a crash on 10.0.9 with a dump, similar to the one above:

    Repro stats:

    • Crash 1: iteration 962 / ~4h (SIGBUS, .NET 10.0.3, no dump)
    • Crash 2: iteration 4755 / ~17h (SIGSEGV, .NET 10.0.9, full heap dump captured)

    Both crashes are in Dictionary::PopulateEntry → GenericHandleWorkerCore → GenericHandleWorker on a TP worker thread.

    Crash 2 details (with dump)

    Signal: SIGSEGV — KERN_INVALID_ADDRESS at 0x20b0

    Faulting code — the backpatching loop in genericdict.cpp lines 1293–1308:

    while (dictionarySize > sizeOfInitialDictionary)
    {
        pDictionary = *pDictionary->GetBackPointerSlot(numGenericArgs);
        if (pDictionary == nullptr)
            break;
        dictionarySize = pDictionary->GetDictionarySlotsSize(numGenericArgs); // CRASH
        ...
    }

    Assembly at crash site (PopulateEntry+5220):

    +5208: ldr w11, [x23, x10]   // dictionarySize = GetDictionarySlotsSize (OK)
    +5212: ldr x23, [x23, x11]   // pDictionary = *GetBackPointerSlot → 0x20a0
    +5216: cbz x23, exit          // null check passes (0x20a0 != 0)
    +5220: ldr w11, [x23, x10]   // CRASH: load from 0x20a0+0x10 = 0x20b0
    

    The back pointer loaded at +5212 is 0x20a0 — a corrupt near-null value. It passes the null check but segfaults on the next access.

    Speculation: possible race: GetTypeDictionaryWithSizeCheck (line 575) expands the dictionary under lock, writing a new back pointer into the old dictionary. But PopulateEntry's backpatch loop walks the chain without holding that lock. If another thread is mid-expansion, the back-pointer slot can contain a stale value.

    The dump does not show racing threads, so this is a guess.

    Dump is here

  13. AndyAyersMS commented on Jun 20, 2026

    @AndyAyersMS
    Member

    After another 1053 runs, I have a second crash; this one is in TLS access: tls crash.

    Likely this is #127776 -- I'll try and verfiy that.

    Guess I'll keep running this and see if I can get a second repro for either of these, or something else turns up.

  14. AndyAyersMS commented on Jun 22, 2026

    @AndyAyersMS
    Member

    I have 6 or so crashes now, seeing if I can glean anything from them as a whole. Hampered a bit because MacOS crash dumps must be analyzed on MacOS currently.

    It takes 1000-5000 repetitions of the above to get a crash, so I can generate one or two per day.

  15. AndyAyersMS commented on Jun 23, 2026

    @AndyAyersMS
    Member

    After trying to pin down the problem seen in several of the crash dumps (including using arm's interleaving modeling) decided to try an instrumented build. Root cause appears to be a publication race on EEClass's dictionary layout pointer.

    fyi @jkotas @davidwrighton

    I don't know if this is the same issue originally seen on .NET 9.

    Here is AI's take on this.

    Repro summary

    Built libcoreclr.dylib from main with one block of added instrumentation
    in Dictionary::PopulateEntry's backpatch loop: detect a back‑pointer load
    that returns a non‑NULL value smaller than 0x10000, STRESS_LOG the chain
    state, and call EEPOLICY_HANDLE_FATAL_ERROR(COR_E_EXECUTIONENGINE). The
    overhead vs. shipping libcoreclr.dylib is ~2%.

    Looped the macios tests/rgen make build-bindings-analyzer-tests target
    on M4 macOS arm64 (full parallelism, no inter-run sleep). The
    instrumentation fired on iteration 1367, ~4h 20m in. Got a full minidump,
    StressLog intact.

    What the dump shows

    Enumerator<String> (non‑canonical instantiation of an inner type in
    NuGet.NoAllocEnumerateExtensions). The captured chain:

    field value meaning
    pDictionary 0x12905dcd8 embedded initial dict (lives at MT+0x68)
    dictionarySize (from size_slot) 40 = (1+1+3)·8, no back-pointer slot allocated
    sizeOfInitialDictionary 16 = (1+1+0)·8, ⇒ m_numInitialSlots == 0
    back-pointer slot address 0x12905dd00 = &MT->m_pAuxiliaryData[0]
    back-pointer value loaded 0x2080 = AuxiliaryData.m_dwFlags

    Decoding 0x2080: enum_flag_DependenciesLoaded | enum_flag_EnsuredInstanceActive.

    The embedded initial dict has no back-pointer slot in its allocation
    (extraAllocBytes == 0 because at MT creation m_numSlots == m_numInitialSlots).
    The read at pDict + 40 lands on the next struct in the MT block —
    MethodTableAuxiliaryData. The bogus "back-pointer" gets dereferenced two
    instructions later → SIGSEGV at offset 0x40. (This is the same crash
    signature as the six original Dictionary::PopulateEntry+5220 dumps from
    the macios CI harness.)

    Why does the loop enter?

    The loop condition is dictionarySize > sizeOfInitialDictionary. We
    captured 40 > 16 → enters. Inspecting the live DictionaryLayout in the
    dump at the time the dump was taken shows m_numSlots = 12, m_numInitialSlots = 3, which would have given sizeOfInit = 40 and the
    loop would NOT have entered. So at log time, the layout pointer that
    PopulateEntry captured was either a different (older or unpublished)
    layout, or the same layout observed in a partially-initialized state.

    Root cause hypothesis

    EEClass's m_pDictLayout is published with a plain pointer store on the
    writer side and read with a plain load on the reader side; no acquire/
    release pairing.

    Writer path (DictionaryLayout::FindToken → ExpandDictionaryLayout →
    SetDictionaryLayout):

    // ExpandDictionaryLayout
    pNewDictionaryLayout = Allocate(newSize, pAllocator, NULL);
    //   Allocate writes m_numSlots = newSize, m_numInitialSlots = newSize
    //   (memory is zero-init from LoaderHeap)
    pNewDictionaryLayout->m_numInitialSlots = pCurrentDictLayout->m_numInitialSlots;
    // ... copies m_slots[i] ...
    return pNewDictionaryLayout;
    
    // FindToken (under expansion Crst):
    pMT->GetClass()->SetDictionaryLayout(pNewLayout);
    // SetDictionaryLayout: GetOptionalFields()->m_pDictLayout = pLayout; // plain store

    Reader path (Dictionary::PopulateEntry):

    pDictLayout = pMT->GetClass()->GetDictionaryLayout(); // plain load
    DWORD sizeOfInitialDictionary =
        (numGenericArgs + 1 + pDictLayout->GetNumInitialSlots()) * sizeof(DictionaryEntry *);

    The reader does not take the expansion lock, so the Crst release on the
    writer side does not establish synchronization with this reader. On ARM64
    the writer's pointer-publication store can be observed before the writes
    that initialized the dictionary fields propagate to the reader's core. The
    reader then sees the zero-initialized state of m_numInitialSlots
    returning 0, computes sizeOfInit = 16, and the backpatch loop iterates
    into adjacent memory.

    This explains every observed property of the failure:

    • Why we always see the loop step past the embedded initial dict.
    • Why the loaded "back-pointer" is some small structurally-significant
      DWORD value (it is whatever the adjacent field in the MT block happens to
      hold — m_dwFlags here; MethodTableAuxiliaryData::m_dwFlags for this
      MT, but other adjacent fields for other layouts).
    • Why all six historic crashes have the same offset (+5220) but different
      small "back-pointer" values (0x2080, 0x20a0, etc.) — these are the
      contents of adjacent AuxiliaryData/optional-fields storage for different
      MT shapes.
    • Why we couldn't reproduce the symptom with herd7/litmus7 modeling the
      dict expansion alone — the missing barrier is on the LAYOUT pointer,
      not on the per-instance dictionary pointer (which is properly
      InterlockedExchange'd in GetTypeDictionaryWithSizeCheck).

    Suggested fix

    Make the dictionary-layout publication match the same release/acquire
    discipline used for the per-instance dictionary pointer. Either of the
    following alone is sufficient; both is cleaner:

    • Writer-side release. Change EEClass::SetDictionaryLayout to use a
      release store (e.g. VolatileStore or InterlockedExchangeT).
    • Reader-side acquire. Change EEClass::GetDictionaryLayout to use
      VolatileLoad.

    The same pattern exists for method dictionary layouts on
    InstantiatedMethodDesc::IMD_SetDictionaryLayout /
    GetDictLayoutRaw — should be fixed identically.

    Notes / caveats

    • The Crst was helpful at the writer but only synchronizes with threads
      that also take the Crst. PopulateEntry's layout read intentionally
      doesn't take the lock.
    • Allocate setting m_numInitialSlots = numSlots and ExpandDictionaryLayout
      immediately overwriting it is a fragile pattern: it leaves a tiny window
      where the field has the "wrong" value even from the writer's point of
      view. Not strictly part of this bug (writer's own writes are
      program-ordered), but worth tightening up.
    • I have not yet verified on x64 — the layout pointer publication is a
      bug on any weak-memory architecture but x64's TSO would hide it under
      most conditions.
  16. davidwrighton commented on Jun 23, 2026

    @davidwrighton
    Member

    @AndyAyersMS Well, its definitely a bug that you've found. Let's get it fixed, and hopefully that actually fixes the problem you're seeing.

  17. AndyAyersMS commented on Jun 23, 2026

    @AndyAyersMS
    Member

    Note this was using 10.0.9. Presumably we'll see this same failure crop up in 11.0 too. #127776 is also causing the occasional crash, and that is fixed in main.

    I can "verify" a fix by first reproducing the failure in 11.0 without a fix and then getting enough crash-free runs with it. Current longest stretch without a repro was ~5000, so say we can get ~15,000 without a crash, that seems like fairly good indication the issue is fixed (using the infamous "rule of 3"). Would take around 2 days.

  18. AndyAyersMS commented on Jun 24, 2026

    @AndyAyersMS
    Member

    Trying to repro the .NET 9 failure now. Will let it run overnight and switch over to .NET 11 tomorrow to repro the failure there and then validate the fix(es).

  19. AndyAyersMS commented on Jun 25, 2026

    @AndyAyersMS
    Member

    I haven't been able to repro the crash on 9.0, even with similar vintage SDKs. Also tried this on an M1 which is more like the M1 Mini from the original report.

    Am ready to validate #129763 once it shows up in a VMR build (wasn't there as of this morning).

  20. AndyAyersMS commented on Jun 26, 2026

    @AndyAyersMS
    Member

    Quick A/B update on the dictionary-publication race fix from #129763.

    Without the fix — 11.0.100-preview.7.26325.102: the macios Bindings.Analyzer.Tests restore loop (with --force) crashed in Dictionary::PopulateEntry after 331 builds on M4 Max, with the GenericsHelpers.ClassWithSlotAndModule → OptimisticallyNonAllocatingEnumerable<>+Enumerator..ctor stack from the earlier root-cause analysis.

    With the fix — 11.0.100-preview.7.26325.110: same machine, same workload, 6,250 clean builds and counting. That's past the upper bound of the documented 1,000–5,000 builds-per-crash interval.

    Will let it run to ~15K and post a final tally.

  21. AndyAyersMS commented on Jun 28, 2026

    @AndyAyersMS
    Member

    Stopped the 11.0 loop after 16,125 successful runs.

    I've never been able to repro the issue in .NET 9, so perhaps something in the 9.0.x servicing fixed that problem.

    Going to close this, please open a new issue if anything else pops up.

  22. rolfbjarne commented on Jun 29, 2026

    @rolfbjarne
    MemberAuthor

    Thanks!

    Will this be backported to .NET 10? (no big deal if it isn't, I'm just trying to know when I should ignore this crash if I see it again)

  23. locked and limited conversation to collaborators on Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMItenet-reliabilityReliability/stability related issue (stress, load problems, etc.)

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions