Repository navigation
New memory cache implementation #48567
Description
Background and Motivation
The MemoryCache implementation and interface leaves much to be desired. At the core of it, we ideally want to expose something more akin to ConcurrentDictionary<TKey, TValue> that supports expiration and can handle memory pressure. What we have right now has issues:
- ICacheEntry uses an unintuitive pattern for adding new entries (Revisit MemoryCache design #36556)
- It's non generic which means it boxes value types (keys and values) (Make IMemoryCache fully generic to avoid boxing #48455)
- Since there's no TryAdd, it's hard to implement GetOrAdd/Async in an extension method in a way that prevents the "cache stampede problem" (GetOrCreate is not atomic which is a serious issue for a cache ( eg poor perf and memory issues) #36499)
- It's impossible to enumerate keys or entries (Getting all cache keys Microsoft.Extensions.Caching.Memory #36026)
- There's no way to clear the cache (Add .Clear() method to MemoryCache #45593)
Proposed API
The APIs are still TBD but I'm thinking a generic memory cache.
namespace Microsoft.Extensons.Caching
{
public class MemoryCache<TKey, TValue>
{
public TValue this[TKey key] { get; set; }
public bool IsEmpty { get; }
public int Count { get; }
public ICollection<TKey> Keys { get; }
public ICollection<TValue> Values { get; }
public void Clear();
public bool ContainsKey(TKey key);
public IEnumerator<KeyValuePair<TKey, TValue>> GetEnumerator();
public KeyValuePair<TKey, TValue>[] ToArray();
public bool TryAdd(TKey key, CacheEntry<TValue> value);
public bool TryGetValue(TKey key, [MaybeNullWhen(false)] out TValue value);
public bool TryRemove(TKey key, [MaybeNullWhen(false)] out TValue value);
}
public class CacheEntry<TValue>
{
TValue Value { get; set; }
DateTimeOffset? AbsoluteExpiration { get; set; }
TimeSpan? AbsoluteExpirationRelativeToNow { get; set; }
TimeSpan? SlidingExpiration { get; set; }
IList<IChangeToken> ExpirationTokens { get; }
IList<PostEvictionCallbackRegistration> PostEvictionCallbacks { get; }
CacheItemPriority Priority { get; set; }
long? Size { get; set; }
}
}I'm convinced now that this shouldn't be an interface or an abstract class but I'm open to discussion.
Usage Examples
TBD
Alternative Designs
TBD
Risks
Having 3 implementations.
Activity
adamsitnik commentedon Feb 25, 2021
How is it being typically used by the end-users: to cache multiple instances of the same type or different types? If we make it generic but users end up having more cache instances we might get a memory overhead bigger from just the boxing in the non-generic version.
adamsitnik commentedon Feb 25, 2021
Would it be possible to make these properties have setter only, to keep the internal representation of these fields an implementation detail? So we could for example translate
AbsoluteExpirationandAbsoluteExpirationRelativeToNowto ticks like StackOverflow did in their implementationWe could also take @edevoogd experiment under the consideration: #45842 (comment)
adamsitnik commentedon Feb 25, 2021
The StackOverflow implementation also introduced an
AccessCountfield to the cache entry: #45592 (comment)adamsitnik commentedon Feb 25, 2021
cc @NickCraver @mgravell
davidfowl commentedon Feb 25, 2021
Libraries end up with their own private caches. Apps usually have a single cache.
AndersMad commentedon Feb 27, 2021
Wish: Pre- vs. PostEvictionCallbacks (instead of or additional)
Short: Called before eviction with an option to skip eviction / keep the cache entry.
Long: Skip could be:
And all topic main 👍 !.. That could reduce my code removing reflection to get the main collection etc. + make cache a lot faster with this suggestion. Think the hardest part of cache is estimating the memory. Came here to clone - hope above will be a thing ~:)
davidfowl commentedon Mar 1, 2021
Can you file an issue for this? With details and usage patterns etc.
alastairtree commentedon Mar 15, 2021
There is a long running discussion on why I have not implemented Clear() in LazyCache over at alastairtree/LazyCache#67 which largely is due to the use of MemoryCache and it's current API. If this went ahead I suspect I would rewrite LazyCache to use MemoryCache<TKey, TValue> as David's proposal would make that much easier.
Worth considering usage scenarios - are you suggesting one cache per item-type, or one cache per app? Having TValue in the definition would encourage me to have one cache per item-type, which makes doing Clear on all cached data (a common use case) more difficult as there are many caches. Typically up to now apps usually have one cache, with the TValue know at time of Get, but that does cause the boxing issue.
davidfowl commentedon Mar 15, 2021
I think having lots of caches is fine (we basically have this today all over the framework). It's should be treated like a dictionary with expiration.
41 remaining items
ericsampson commentedon Jun 26, 2021
That's a fair point :) If the .NET/ASP docs for caching can list these community packages, that would go a long way.
"straight forward" is an important qualifier :)
Maybe the theoretical MemoryCache extension library could have
GetOrAddLazy*alongside the current re-entrant factory version, to help discoverability in the IDE etc. Because that's the biggest current footgun for people IME. Cheers
ohroy commentedon Jul 23, 2021
+1
current cache is hard to use, the cache stampede is very serious problem
davidfowl commentedon Jul 23, 2021
Its possible to work around manually.
jodydonetti commentedon Jul 23, 2021
If you'd like to avoid the cache stampede problem you can take a look at some alternatives (in alphabetical order):
Hope this helps.
ohroy commentedon Jul 23, 2021
sure, it can work around by lazy and lock etc, but not everyone can realize that it may have the problem of cache stampede , which will lead to serious consequences.
whatever, it is not easy to use.
I very hope to have a built-in solution , thank you for this proposal
molinch commentedon Jan 20, 2022
I know it's a fairly broad question where implementation may depend on uses cases, but what solution would you suggest when the cache key is dynamic ? (for example when you want to cache fetched user permissions so cache key could be user-17, user-18, ...). Having a
SemaphoreSlimper key seems complicated and in the long run implies a memory leak. Relying onLazy<T>semantics may work too depending on how the current MemoryCache is implemented.jodydonetti commentedon Jan 20, 2022
One of these #48567 (comment) ?
molinch commentedon Jan 21, 2022
Thanks @jodydonetti, since our need is simply an IMemoryCache without cache stampede issue, nothing more, nothing less. We went with @StephenCleary solution in the end: https://gist.github.com/StephenCleary/39a2cd0aa3c705a984a4dbbea8275fe9
I like this solution, it's a slim wrapper on top of IMemoryCache and you can easily follow the code.
robertbaumann commentedon Jul 17, 2022
Consider something that can be extensible to support Azure Caching Guidance
davidfowl commentedon Nov 29, 2023
There's a proposal for a new cache implementation here dotnet/extensions#4766. Can those interested review it and leave comments?
jodydonetti commentedon Nov 30, 2023
Thanks David, will do!
austinmfb commentedon Dec 5, 2023
Interesting that there is no mention of managing cache dependencies in this thread. Not that it would have to be baked into a new cache class, but still seems like a relevant design consideration. Is nobody really using a consistent pattern for this that they want their cache class to handle? Is everyone just opting for inline code in each application component requiring this, and managing its specific set of dependent caches?
JoelDavidLang commentedon Dec 7, 2023
I was bitten by this just recently. I did not see any documentation in CreateEntry() nor on ICacheEntry that stated that the object needed to be disposed to commit it to the cache. As a result, the code I initially wrote didn't have working caching!
I would like to see this documentation clarified for the current version of the cache at least.