Skip to content

VSD DispatchCache ResolveCacheElem self-loop (pNext == self) hangs the process on osx-x64 — #128868 present but insufficient #128955

Closed

Description

@wtgodbe

The bug

A ResolveCacheElem in the Virtual Stub Dispatch DispatchCache ends up with pNext pointing
at itself. The next DispatchCache::Insert walks that collision chain forever while
holding the cache Crst
, so every other thread that needs a VSD resolve blocks on that lock →
whole-process hang. The runtime under test already contains #128868, so #128868 does not fix it.

Evidence

Confirmed at instruction level in two independent, symbolized x86_64 cores from separate CI
builds and two different managed workloads.

Spinning thread (the only thread running CLR code):

DispatchCache::Insert + 208           ; dedup pre-scan, looping on rax = rax->pNext (offset 0x18)
VirtualCallStubManager::ResolveWorker + 2526
VSD_ResolveWorker + 752
ResolveWorkerAsmStub + 113

Corrupt node 0x10d71a2e0: pMT=0x1131be660, token=0x21a00000006, target=0x4227a0e1c8,
pNext=0x10d71a2e0 (== self). The end sentinel is never reached and the inserting thread's
key never matches, so the walk loops forever. Mapping (pMT, token -> target) is valid —
only pNext is corrupt. No crash.

Lock cascade: the spinning thread holds the cache Crst; many other threads are stuck at

__psynch_mutexwait -> CrstBase::Enter -> DispatchCache::Insert + 32
  <- VirtualCallStubManager::ResolveWorker            (dispatch miss)
  <- MethodTable::TryResolveConstraintMethodApprox <- CEEInfo::getCallInfo   (JIT)

A full per-bucket census of the other core showed exactly one cycle among 4096 buckets — a
single isolated 1-node self-loop at a bucket head — with 147/164 threads blocked on the lock.

The only writer of pNext is Insert's prepend (elem->pNext = entries[hash]; entries[hash] = elem), so a self-loop means entries[hash] already equaled the elem being prepended — i.e.
ResolveCacheElem address reuse / double-publish at the bucket head. The producing store is not
captured in-the-act (the dumps show the aftermath only).

Suggestion

Bounded walk + cycle guard in DispatchCache::Insert/Lookup and the asm chain-walk: detect
a self-reference / over-length chain during the pNext traversal and treat it as a miss
(re-resolve) or reset the bucket. This eliminates the hang regardless of the root producer.

To pinpoint the producer, add a checked-build assert in Insert that entries[hash] != elem
before the prepend; a checked CI run would then trap the producing store.

Caveats

  • Both dumps are x86_64 (the arm64 leg's dotnet ran under Rosetta 2). No native-arm64 hang
    observed
    — no cross-arch claim.
  • Source: vm/virtualcallstub.cpp (DispatchCache::Insert/Lookup), VirtualCallStubAMD64.asm.
  • Cores, full per-bucket census, symbolized stacks, and node decode available on request.

Known Issue Error Message

DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0

Activity

  1. dotnet-policy-service commented on Jun 3, 2026

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @JulieLeeMSFT, @VSadov
    See info in area-owners.md if you want to be subscribed.

  2. added
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    on Jun 3, 2026
  3. dotnet-policy-service commented on Jun 3, 2026

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @JulieLeeMSFT, @VSadov
    See info in area-owners.md if you want to be subscribed.

  4. dotnet-policy-service commented on Jun 3, 2026

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @agocke
    See info in area-owners.md if you want to be subscribed.

  5. wtgodbe commented on Jun 3, 2026

    @wtgodbe
    MemberAuthor
  6. AaronRobinsonMSFT commented on Jun 3, 2026

    @AaronRobinsonMSFT
    Member

    @wtgodbe Do you have a DMP for this issue?

  7. wtgodbe commented on Jun 3, 2026

    @wtgodbe
    MemberAuthor
  8. jkotas commented on Jun 3, 2026

    @jkotas
    Member

    @davidwrighton is investigating

  9. Youssef1313 commented on Jun 5, 2026

    @Youssef1313
    Member

    @davidwrighton Any updates please? This is a severe issue for aspnetcore (and potentially other repos might hit that)

  10. davidwrighton commented on Jun 5, 2026

    @davidwrighton
    Member

    I'm looking at it, and there isn't a clear cause for these failures that I've found yet.

  11. added
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    and removed
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    on Jun 8, 2026
  12. SamMonoRT commented on Jun 16, 2026

    @SamMonoRT
    Member

    @davidwrighton - any updates on this, do you need more logs etc from the team?

  13. added
    blockingMarks issues that we want to fast track in order to unblock other important work
    on Jun 16, 2026
  14. wtgodbe commented on Jun 17, 2026

    @wtgodbe
    MemberAuthor

    I believe I have it narrowed down to this commit range: bb4ef64...ab6aa47. I'm still bisecting, I'll share the more specific ranges as I continue to narrow it down

  15. Youssef1313 commented on Jun 18, 2026

    @Youssef1313
    Member

    @davidwrighton Should this be closed now that #129531 is merged or is there anything else remaining here?

    Also, is the fix linked to #128859 needed or should it be reverted?

  16. davidwrighton commented on Jun 22, 2026

    @davidwrighton
    Member

    The fix in #128859 is probably unnecessary, but its not harmful, and the lack of a lock there is only unnecessary due to OTHER very, very unusual code paths, so I would prefer to leave it in place. (Notably, we only call that code on the finalizer thread, so its effectively locked, but that isn't something we document as a requirement for that code, so its better just to leave the locking in place.) I think we can close this bug. Hopefully its gone now.

  17. locked and limited conversation to collaborators on Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-VM-coreclrblockingMarks issues that we want to fast track in order to unblock other important workblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'os-macosmacOS aka OSX

Type

No type

Projects

  • Status
    No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions