Repository navigation
GetOrSetAsync for a cacheKey hangs for an infinite time #281
Description
Activity
dotnet-dump analyze=>dumpasyncof my stuck application seems to confirm the assumption... ... ... STACK 47 00007f82a6e7d0f8 00007f85935ebab8 ( ) System.Threading.Tasks.Task<CacheTower.ICacheEntry> 00007f82a6e7d140 00007f8593608078 (7) CacheTower.CacheStack+<RefreshValueAsync>d__19<Zyfra.Udl.Cmn.Models.DataSources.DsConfigExtDto> 00007f82a6e7c060 00007f85935e34a8 (1) CacheTower.CacheStack+<GetOrSetAsync>d__17<Zyfra.Udl.Cmn.Models.DataSources.DsConfigExtDto> 00007f82a6e7c130 00007f85935e4878 (0) Zyfra.Udl.DataSources.Providers.PropertyConfigProvider+<GetPropertyConfigFromCacheAsync>d__16 00007f82a6e7d018 00007f85935ec040 (4) Zyfra.Udl.Rtd.Core.RabbitMqSubscriptionManager+<AddConsumerToPropertySubscriptionAsync>d__35 00007f82a68c99c8 00007f8592c5ea28 (0) Zyfra.Udl.Rtd.Core.RabbitMqSubscriptionManager+<HandleRtdCommandsAsync>d__33 00007f82a68c7c28 00007f85927fbb30 ( ) System.Threading.Tasks.UnwrapPromise<System.Threading.Tasks.VoidTaskResult> ... ... ...We are experiencing the same issue in a similar environment
We are experiencing the same issue in a similar environment
We ended up switching to FusionCache, satisfied so far
Reacted by Adem Tas, Masih Akbari and nawfalhasan
What does the bug do?
GetOrSetAsynccall for particularcacheKeysometimes hangs for an infinite time.I think the reason of the behaviour is this
TaskCompletionSourcewhich can be created, for intance, when 2 threads getting value for a first time in parallel (cache miss), and thread A goes to this branch of code , thread B goes to this branch of code, sleeps for a while, and then callsKeyLock.WaitAsync(cacheKey)when theKeyLockis already released by thread A.Then, Thread B waits for the
TaskCompletionSourcejust created by itself, and from the thread perspective it looks like infinite call ofGetOrSetAsyncmethod, and only another thread callingGetOrSetAsyncwith the samecacheKeycan release Thread B from this block.Not sure, but it looks like even this another thread call won't help if it happens after
cacheEntry(created by thread A) is expired.How can it be reproduced?
In my case it could only be reproduced under (relatively) heavy load.
Environment (if applicable)