You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Crash in MethodTable::MethodDataObject::FillEntryDataForAncestor #119761
This is a crash I saw once on our bots, I have no way to reproduce it, nor do I know what the process in question was doing (probably 'dotnet build', but it can be any number of things we do during our build).
If the build has been deleted, or get an access denied error, I can provide a download link to the minidump (note to self: it's in the 'dotnet-runtime-issue-119761' directory in OneDrive).
I looked at the dump. I am not able to figure out the context of the crash since the context pointer was optimized out. #121773 should make that better.
We are running fixups for System.Linq.EnumerableEmpty method. The problem is that the MethodTable that we are trying to lookup the Empty method in is corrupted.
The back pointer loaded at +5212 is 0x20a0 — a corrupt near-null value. It passes the null check but segfaults on the next access.
Speculation: possible race:GetTypeDictionaryWithSizeCheck (line 575) expands the dictionary under lock, writing a new back pointer into the old dictionary. But PopulateEntry's backpatch loop walks the chain without holding that lock. If another thread is mid-expansion, the back-pointer slot can contain a stale value.
The dump does not show racing threads, so this is a guess.
I have 6 or so crashes now, seeing if I can glean anything from them as a whole. Hampered a bit because MacOS crash dumps must be analyzed on MacOS currently.
It takes 1000-5000 repetitions of the above to get a crash, so I can generate one or two per day.
After trying to pin down the problem seen in several of the crash dumps (including using arm's interleaving modeling) decided to try an instrumented build. Root cause appears to be a publication race on EEClass's dictionary layout pointer.
I don't know if this is the same issue originally seen on .NET 9.
Here is AI's take on this.
Repro summary
Built libcoreclr.dylib from main with one block of added instrumentation
in Dictionary::PopulateEntry's backpatch loop: detect a back‑pointer load
that returns a non‑NULL value smaller than 0x10000, STRESS_LOG the chain
state, and call EEPOLICY_HANDLE_FATAL_ERROR(COR_E_EXECUTIONENGINE). The
overhead vs. shipping libcoreclr.dylib is ~2%.
Looped the macios tests/rgenmake build-bindings-analyzer-tests target
on M4 macOS arm64 (full parallelism, no inter-run sleep). The
instrumentation fired on iteration 1367, ~4h 20m in. Got a full minidump,
StressLog intact.
What the dump shows
Enumerator<String> (non‑canonical instantiation of an inner type in NuGet.NoAllocEnumerateExtensions). The captured chain:
The embedded initial dict has no back-pointer slot in its allocation
(extraAllocBytes == 0 because at MT creation m_numSlots == m_numInitialSlots).
The read at pDict + 40 lands on the next struct in the MT block — MethodTableAuxiliaryData. The bogus "back-pointer" gets dereferenced two
instructions later → SIGSEGV at offset 0x40. (This is the same crash
signature as the six original Dictionary::PopulateEntry+5220 dumps from
the macios CI harness.)
Why does the loop enter?
The loop condition is dictionarySize > sizeOfInitialDictionary. We
captured 40 > 16 → enters. Inspecting the live DictionaryLayout in the
dump at the time the dump was taken shows m_numSlots = 12, m_numInitialSlots = 3, which would have given sizeOfInit = 40 and the
loop would NOT have entered. So at log time, the layout pointer that PopulateEntry captured was either a different (older or unpublished)
layout, or the same layout observed in a partially-initialized state.
Root cause hypothesis
EEClass's m_pDictLayout is published with a plain pointer store on the
writer side and read with a plain load on the reader side; no acquire/
release pairing.
The reader does not take the expansion lock, so the Crst release on the
writer side does not establish synchronization with this reader. On ARM64
the writer's pointer-publication store can be observed before the writes
that initialized the dictionary fields propagate to the reader's core. The
reader then sees the zero-initialized state of m_numInitialSlots
returning 0, computes sizeOfInit = 16, and the backpatch loop iterates
into adjacent memory.
This explains every observed property of the failure:
Why we always see the loop step past the embedded initial dict.
Why the loaded "back-pointer" is some small structurally-significant
DWORD value (it is whatever the adjacent field in the MT block happens to
hold — m_dwFlags here; MethodTableAuxiliaryData::m_dwFlags for this
MT, but other adjacent fields for other layouts).
Why all six historic crashes have the same offset (+5220) but different
small "back-pointer" values (0x2080, 0x20a0, etc.) — these are the
contents of adjacent AuxiliaryData/optional-fields storage for different
MT shapes.
Why we couldn't reproduce the symptom with herd7/litmus7 modeling the
dict expansion alone — the missing barrier is on the LAYOUT pointer,
not on the per-instance dictionary pointer (which is properly InterlockedExchange'd in GetTypeDictionaryWithSizeCheck).
Suggested fix
Make the dictionary-layout publication match the same release/acquire
discipline used for the per-instance dictionary pointer. Either of the
following alone is sufficient; both is cleaner:
Writer-side release. Change EEClass::SetDictionaryLayout to use a
release store (e.g. VolatileStore or InterlockedExchangeT).
Reader-side acquire. Change EEClass::GetDictionaryLayout to use VolatileLoad.
The same pattern exists for method dictionary layouts on InstantiatedMethodDesc::IMD_SetDictionaryLayout / GetDictLayoutRaw — should be fixed identically.
Notes / caveats
The Crst was helpful at the writer but only synchronizes with threads
that also take the Crst. PopulateEntry's layout read intentionally
doesn't take the lock.
Allocate setting m_numInitialSlots = numSlots and ExpandDictionaryLayout
immediately overwriting it is a fragile pattern: it leaves a tiny window
where the field has the "wrong" value even from the writer's point of
view. Not strictly part of this bug (writer's own writes are
program-ordered), but worth tightening up.
I have not yet verified on x64 — the layout pointer publication is a
bug on any weak-memory architecture but x64's TSO would hide it under
most conditions.
Note this was using 10.0.9. Presumably we'll see this same failure crop up in 11.0 too. #127776 is also causing the occasional crash, and that is fixed in main.
I can "verify" a fix by first reproducing the failure in 11.0 without a fix and then getting enough crash-free runs with it. Current longest stretch without a repro was ~5000, so say we can get ~15,000 without a crash, that seems like fairly good indication the issue is fixed (using the infamous "rule of 3"). Would take around 2 days.
Trying to repro the .NET 9 failure now. Will let it run overnight and switch over to .NET 11 tomorrow to repro the failure there and then validate the fix(es).
I haven't been able to repro the crash on 9.0, even with similar vintage SDKs. Also tried this on an M1 which is more like the M1 Mini from the original report.
Am ready to validate #129763 once it shows up in a VMR build (wasn't there as of this morning).
Quick A/B update on the dictionary-publication race fix from #129763.
Without the fix — 11.0.100-preview.7.26325.102: the macios Bindings.Analyzer.Tests restore loop (with --force) crashed in Dictionary::PopulateEntry after 331 builds on M4 Max, with the GenericsHelpers.ClassWithSlotAndModule → OptimisticallyNonAllocatingEnumerable<>+Enumerator..ctor stack from the earlier root-cause analysis.
With the fix — 11.0.100-preview.7.26325.110: same machine, same workload, 6,250 clean builds and counting. That's past the upper bound of the documented 1,000–5,000 builds-per-crash interval.
Activity
agocke commentedon Sep 19, 2025
@davidwrighton Can you take a quick look at this?
rolfbjarne commentedon Oct 22, 2025
Just got another one of this crash in CI: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=12630631&view=artifacts&pathAsName=false&type=publishedArtifacts (the dump file is in the "minidumps" folder).
Crash report: https://gist.github.com/rolfbjarne/372a35458f5a56eed409173def872103
rolfbjarne commentedon Nov 13, 2025
Another one: https://gist.github.com/rolfbjarne/d69c1586d793012b24f935698da188e9 - https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=12774594&view=artifacts&pathAsName=false&type=publishedArtifacts
jkotas commentedon Nov 19, 2025
I looked at the dump. I am not able to figure out the context of the crash since the context pointer was optimized out. #121773 should make that better.
Add context argument to crash dump functions to preserve register sta…
rolfbjarne commentedon Dec 8, 2025
This just happened locally for me, I got coredumps:
createdump -full 90241: https://microsofteur-my.sharepoint.com/:u:/g/personal/rokvin_microsoft_com/IQAVBG5J8P_ESJKGdpoqBj_hAcDHhM5vHWzxoht0pi_2YN8?e=nq1BHM@jkotas this is from the latest .NET 11 builds, so I believe it should contain the PR you made a few weeks ago.
It happened while compiling https://github.com/dotnet/macios/blob/2b7db63a5e7657b684c7bd869db1c28288baaf89/tests/rgen/Microsoft.Macios.Bindings.Analyzer.Tests/Microsoft.Macios.Bindings.Analyzer.Tests.csproj:
$ ps aux|grep 90241 rolf 90241 0.0 1.9 413003872 652992 s007 SX+ 10:58AM 0:00.77 /Users/rolf/work/dotnet/macios/net11.0/macios/builds/downloads/dotnet-sdk-11.0.100-alpha.1.25605.110/dotnet build Microsoft.Macios.Bindings.Analyzer.Tests/Microsoft.Macios.Bindings.Analyzer.Tests.csproj --verbosity quiet --nologo -consoleLoggerParameters:NoSummaryjkotas commentedon Dec 18, 2025
Analysis of the crash dump at https://microsofteur-my.sharepoint.com/:u:/g/personal/rokvin_microsoft_com/IQBQZcpqCDEFQomqbOt7Lwu3AQqlysuAZ0-0XzpaKmYvmtI?e=hC1QPd :
We crash when running R2R fixups for
NuGet.Commands.MSBuildRestoreUtility.ReplayWarningsAndErrorsAsyncat:Managed stack:
We are running fixups for
System.Linq.EnumerableEmptymethod. The problem is that the MethodTable that we are trying to lookup theEmptymethod in is corrupted.Unclear how the EEClass got corrupted.
jkotas commentedon Dec 18, 2025
EEClass is overwritten by VirtualStubDispatch cells:
30 remaining items
AndyAyersMS commentedon Jun 19, 2026
Note
This comment was generated by Copilot.
Have a crash on 10.0.9 with a dump, similar to the one above:
Repro stats:
Both crashes are in
Dictionary::PopulateEntry→GenericHandleWorkerCore→GenericHandleWorkeron a TP worker thread.Crash 2 details (with dump)
Signal: SIGSEGV —
KERN_INVALID_ADDRESS at 0x20b0Faulting code — the backpatching loop in
genericdict.cpplines 1293–1308:Assembly at crash site (
PopulateEntry+5220):The back pointer loaded at +5212 is
0x20a0— a corrupt near-null value. It passes the null check but segfaults on the next access.Speculation: possible race:
GetTypeDictionaryWithSizeCheck(line 575) expands the dictionary under lock, writing a new back pointer into the old dictionary. ButPopulateEntry's backpatch loop walks the chain without holding that lock. If another thread is mid-expansion, the back-pointer slot can contain a stale value.The dump does not show racing threads, so this is a guess.
Dump is here
AndyAyersMS commentedon Jun 20, 2026
After another 1053 runs, I have a second crash; this one is in TLS access: tls crash.
Likely this is #127776 -- I'll try and verfiy that.
Guess I'll keep running this and see if I can get a second repro for either of these, or something else turns up.
AndyAyersMS commentedon Jun 22, 2026
I have 6 or so crashes now, seeing if I can glean anything from them as a whole. Hampered a bit because MacOS crash dumps must be analyzed on MacOS currently.
It takes 1000-5000 repetitions of the above to get a crash, so I can generate one or two per day.
AndyAyersMS commentedon Jun 23, 2026
After trying to pin down the problem seen in several of the crash dumps (including using arm's interleaving modeling) decided to try an instrumented build. Root cause appears to be a publication race on
EEClass's dictionary layout pointer.fyi @jkotas @davidwrighton
I don't know if this is the same issue originally seen on .NET 9.
Here is AI's take on this.
Repro summary
Built
libcoreclr.dylibfrom main with one block of added instrumentationin
Dictionary::PopulateEntry's backpatch loop: detect a back‑pointer loadthat returns a non‑NULL value smaller than
0x10000, STRESS_LOG the chainstate, and call
EEPOLICY_HANDLE_FATAL_ERROR(COR_E_EXECUTIONENGINE). Theoverhead vs. shipping
libcoreclr.dylibis ~2%.Looped the macios
tests/rgenmake build-bindings-analyzer-teststargeton M4 macOS arm64 (full parallelism, no inter-run sleep). The
instrumentation fired on iteration 1367, ~4h 20m in. Got a full minidump,
StressLog intact.
What the dump shows
Enumerator<String>(non‑canonical instantiation of an inner type inNuGet.NoAllocEnumerateExtensions). The captured chain:pDictionary0x12905dcd8MT+0x68)dictionarySize(from size_slot)40(1+1+3)·8, no back-pointer slot allocatedsizeOfInitialDictionary16(1+1+0)·8, ⇒m_numInitialSlots == 00x12905dd00&MT->m_pAuxiliaryData[0]0x2080AuxiliaryData.m_dwFlagsDecoding
0x2080:enum_flag_DependenciesLoaded | enum_flag_EnsuredInstanceActive.The embedded initial dict has no back-pointer slot in its allocation
(
extraAllocBytes == 0because at MT creationm_numSlots == m_numInitialSlots).The read at
pDict + 40lands on the next struct in the MT block —MethodTableAuxiliaryData. The bogus "back-pointer" gets dereferenced twoinstructions later → SIGSEGV at offset
0x40. (This is the same crashsignature as the six original
Dictionary::PopulateEntry+5220dumps fromthe macios CI harness.)
Why does the loop enter?
The loop condition is
dictionarySize > sizeOfInitialDictionary. Wecaptured
40 > 16→ enters. Inspecting the liveDictionaryLayoutin thedump at the time the dump was taken shows
m_numSlots = 12, m_numInitialSlots = 3, which would have givensizeOfInit = 40and theloop would NOT have entered. So at log time, the layout pointer that
PopulateEntrycaptured was either a different (older or unpublished)layout, or the same layout observed in a partially-initialized state.
Root cause hypothesis
EEClass'sm_pDictLayoutis published with a plain pointer store on thewriter side and read with a plain load on the reader side; no acquire/
release pairing.
Writer path (
DictionaryLayout::FindToken→ExpandDictionaryLayout→SetDictionaryLayout):Reader path (
Dictionary::PopulateEntry):The reader does not take the expansion lock, so the Crst release on the
writer side does not establish synchronization with this reader. On ARM64
the writer's pointer-publication store can be observed before the writes
that initialized the dictionary fields propagate to the reader's core. The
reader then sees the zero-initialized state of
m_numInitialSlotsreturning
0, computessizeOfInit = 16, and the backpatch loop iteratesinto adjacent memory.
This explains every observed property of the failure:
DWORD value (it is whatever the adjacent field in the MT block happens to
hold —
m_dwFlagshere;MethodTableAuxiliaryData::m_dwFlagsfor thisMT, but other adjacent fields for other layouts).
+5220) but differentsmall "back-pointer" values (
0x2080,0x20a0, etc.) — these are thecontents of adjacent AuxiliaryData/optional-fields storage for different
MT shapes.
dict expansion alone — the missing barrier is on the LAYOUT pointer,
not on the per-instance dictionary pointer (which is properly
InterlockedExchange'd inGetTypeDictionaryWithSizeCheck).Suggested fix
Make the dictionary-layout publication match the same release/acquire
discipline used for the per-instance dictionary pointer. Either of the
following alone is sufficient; both is cleaner:
EEClass::SetDictionaryLayoutto use arelease store (e.g.
VolatileStoreorInterlockedExchangeT).EEClass::GetDictionaryLayoutto useVolatileLoad.The same pattern exists for method dictionary layouts on
InstantiatedMethodDesc::IMD_SetDictionaryLayout/GetDictLayoutRaw— should be fixed identically.Notes / caveats
that also take the Crst.
PopulateEntry's layout read intentionallydoesn't take the lock.
Allocatesettingm_numInitialSlots = numSlotsandExpandDictionaryLayoutimmediately overwriting it is a fragile pattern: it leaves a tiny window
where the field has the "wrong" value even from the writer's point of
view. Not strictly part of this bug (writer's own writes are
program-ordered), but worth tightening up.
bug on any weak-memory architecture but x64's TSO would hide it under
most conditions.
davidwrighton commentedon Jun 23, 2026
@AndyAyersMS Well, its definitely a bug that you've found. Let's get it fixed, and hopefully that actually fixes the problem you're seeing.
AndyAyersMS commentedon Jun 23, 2026
Note this was using 10.0.9. Presumably we'll see this same failure crop up in 11.0 too. #127776 is also causing the occasional crash, and that is fixed in main.
I can "verify" a fix by first reproducing the failure in 11.0 without a fix and then getting enough crash-free runs with it. Current longest stretch without a repro was ~5000, so say we can get ~15,000 without a crash, that seems like fairly good indication the issue is fixed (using the infamous "rule of 3"). Would take around 2 days.
AndyAyersMS commentedon Jun 24, 2026
Trying to repro the .NET 9 failure now. Will let it run overnight and switch over to .NET 11 tomorrow to repro the failure there and then validate the fix(es).
AndyAyersMS commentedon Jun 25, 2026
I haven't been able to repro the crash on 9.0, even with similar vintage SDKs. Also tried this on an M1 which is more like the M1 Mini from the original report.
Am ready to validate #129763 once it shows up in a VMR build (wasn't there as of this morning).
AndyAyersMS commentedon Jun 26, 2026
Quick A/B update on the dictionary-publication race fix from #129763.
Without the fix —
11.0.100-preview.7.26325.102: the maciosBindings.Analyzer.Testsrestore loop (with--force) crashed inDictionary::PopulateEntryafter 331 builds on M4 Max, with theGenericsHelpers.ClassWithSlotAndModule→OptimisticallyNonAllocatingEnumerable<>+Enumerator..ctorstack from the earlier root-cause analysis.With the fix —
11.0.100-preview.7.26325.110: same machine, same workload, 6,250 clean builds and counting. That's past the upper bound of the documented 1,000–5,000 builds-per-crash interval.Will let it run to ~15K and post a final tally.
AndyAyersMS commentedon Jun 28, 2026
Stopped the 11.0 loop after 16,125 successful runs.
I've never been able to repro the issue in .NET 9, so perhaps something in the 9.0.x servicing fixed that problem.
Going to close this, please open a new issue if anything else pops up.
rolfbjarne commentedon Jun 29, 2026
Thanks!
Will this be backported to .NET 10? (no big deal if it isn't, I'm just trying to know when I should ignore this crash if I see it again)