[API Proposal]: Soft References

Background and motivation

MemoryCache is the "standard" way to cache things in .NET, but its behavior is unintuitive and it does not guarantee that it will evict cache entries quickly enough to prevent OutOfMemoryExceptions, among other issues such as badly bloated cache entries. Plus, it apparently relies on some kind of black magic (the documentation for which I have not been able to locate) to detect the size in bytes of objects in the cache, so that using it correctly is difficult: even if I am using it correctly, it's difficult to be confident of that! I would like to be able to put objects in a cache that contain two kinds of references: (1) references to "owned" subobjects that should be counted as part of the parent object, and (2) references to (large) shared objects that can never be evicted. I can't imagine how anything except the garbage collector would be able to detect that (1) is only reachable via the cache and so should be counted for "eviction" purposes, while (2) cannot be GC'd.

Finally, if the goal is to prevent memory exhaustion, MemoryCache is problematic because multiple cache instances can exist that do not coordinate with one another.

Weak references tend to be collected far too quickly to be used in caches. Soft references would solve this problem. Soft references are like weak references, but garbage-collected much less aggressively.

API Proposal

An obvious interface would be to replicate WeakReference<T>:

namespace System;

public sealed class SoftReference<T> : IWeakReference<T>
{
    public SoftReference(T target);
    ~SoftReference();

    public void SetTarget(T target);
    public bool TryGetTarget([MaybeNullWhen(false)][NotNullWhen(true)] out T target);
}

// An interface implemented by WeakReference<T> and SoftReference<T>
public interface IWeakReference<T>
{
    bool TryGetTarget([MaybeNullWhen(false)][NotNullWhen(true)] out T target);
    void SetTarget(T target);
}

API Usage

SoftReference<MyEntity>> _entity;

void Cache(MyEntity entity) => _entity = new SoftReference<MyEntity>(entity);

// Later on...
if (_entity != null && _entity.TryGetTarget(out MyEntity e))
  Console.WriteLine("We've still got it!");
else
  Console.WriteLine("Ain't got it!");

Alternative Designs

Another obvious design is to define SoftReference<T> as a derived class of WeakReference<T>. A third possibility is to add "softness" as a feature of the existing WeakReference class.

WeakReferences have a "track resurrection" feature. It's not clear to me that this would add value to a soft reference, but IMO this feature should be supported if it does not add significant complexity to the GC.

It should be kept in mind that after this feature is introduced, many applications could be dominated by soft references (i.e. at any given time, most objects are reachable only through soft references). Therefore, perhaps there should be a property of GC to control the preferred total memory usage of the process, which would affect the aggressiveness of soft-reference collection.

    // Sets a limit for memory usage that the GC should attempt to enforce 
    // by collecting more aggressively near the limit. This particularly affects 
    // the degree to which objects referenced via soft references are collected.
    public long SoftMemoryLimit { get; set; }

(I would very much like a hard memory limit too, but that's another story.)

Ideally, the GC would not collect all soft-referenced object when memory pressure is encountered, but instead prioritize which objects to collect first according to some kind of "priority". I believe the most commonly-desired way to prioritize would be by recency: to first get rid of objects that have not been used recently. To that end there could be a LastUsed property:

    // Controls garbage collection priority; unreachable objects with
    // lower values for LastUsed tend to be collected first.
    // - To artificially delay GC for a soft reference, increase it (eg add 24 hours)
    // - To artificially encourage GC for a soft reference, decrease it (eg subtract 
    //   24 hours; or use DateTime.MinValue to treat soft ref like WeakReference)
    // - Setter could convert all dates to UTC so that the GC can directly compare 
    //   LastUsed.Ticks of different soft references.
    public DateTime LastUsed { get; set; }

    // Variant of TryGetTarget that sets LastUsed = DateTime.UtcNow if target is alive
    public bool TryGetTargetAndSetLastUsed([MaybeNullWhen(false)][NotNullWhen(true)] out T target) {
        if (TryGetTarget(out target)) {
            LastUsed = DateTime.UtcNow;
            return true;
        }
        return false;
    }

Rather than adding bool updateLastUsed as a parameter on TryGetTarget, it could be a separate boolean property so that it is possible to configure IWeakReference.TryGetTarget() to update LastUsed.

It is possible that there are multiple soft references to the same object. It is tempting to put the last-used timestamp on the object itself so that there can only be a single timestamp:

// Any class could implement this interface in order to control GC priority
public interface IGCSoftReferencePriority
{
    DateTime LastUsed { get; set; }
}

However this approach would have major disadvantages:

Users may certainly wish to hold soft references to objects that don't implement IGCSoftReferencePriority
The very act of checking whether IGCSoftReferencePriority is implemented might be too expensive inside the GC
I expect that user-defined code cannot be called during GC's stop-the-world phase, and the property could do strange things like allocate memory, loop indefinitely, return a different value each time it is called, etc.

Risks

I have no idea how difficult it would be to implement this in the GC.

Should it be a variant of WeakReference, with an option to control it?

Soft references are like weak references, but garbage-collected much less aggressively.

How do you propose to implement "much less aggressively"?

Soft references could be considered as strong references until memory pressure gets high where they would be converted to weak references. Under the hood they could (maybe I'm wrong I have no idea) be implemented as a new type of GCHandle and stored in a separate table that would be scanned by the GC's mark phase only when memory pressure is too high. The transition from a strong to a weak reference would be one-way to keep things simple.

There are two major questions. The first is what does "memory pressure gets high" mean, and it's kind of solved as evidenced in the shared ArrayPool (still would need tweaking though). The second question is how many and which soft references to release on high memory pressure. All? The earliest N(%) created? The top N(%) by object size?

In the meanwhile, using constructs like the Gen2GcCallback and checking memory pressure could help implement something similar to what I proposed in managed code, albeit with a bigger overhead.

The first is what does "memory pressure gets high" mean, and it's kind of solved as evidenced in the shared ArrayPool

ArrayPool uses number of strategies. In addition to checking for memory pressure, it also uses timestamp to track when the pooled array was used last time and releases the pooled array once it was not used for a while.

checking memory pressure could help implement something similar to what I proposed in managed code, albeit with a bigger overhead.

It is not obvious to me that the soft references have lower overhead than checking memory pressure. For example, I expect that we would see regression in ArrayPool if it was switched to use the soft references proposed here.

#53895 has a lot of additional discussion of this problem space.

@jkotas

@jkotas I'm no GC expert and don't even know how weak references work, so I'd leave it up to the GC team to choose an implementation. I like how @teo-tsirpanis frames it, as soft refs should act like strong refs until some memory pressure threshold is crossed (such as the SoftMemoryLimit I suggested, or OS-specific heuristics).

... you're asking for some sort of SoftReference, but your actual problem seems to be caching things. If you just want to cache things, it's a good bet there are third-party libraries with slimmer (or tuned in other ways) cache implementations.

it does not guarantee that it will evict cache entries quickly enough to prevent OutOfMemoryExceptions

It doesn't guarantee that cache entries will ever be evicted (ie, they can all be held).

Finally, if the goal is to prevent memory exhaustion, MemoryCache is problematic because multiple cache instances can exist that do not coordinate with one another.

Why would they? Being independent is usually a benefit.

(1) references to "owned" subobjects that should be counted as part of the parent object, and (2) references to (large) shared objects that can never be evicted.

It sounds like you want two caches tuned differently, or a regular cache and a dictionary. If necessary, implement a wrapper that divides them appropriately for you.

@Clockwork-Muse

@Clockwork-Muse I want a cache that uses as much memory as possible with (for all practical purposes) no chance of OutOfMemoryException (and also avoiding using swap space, if available). If you can tell me how to do that without soft references, please do.

What I was getting at was - are you planning on writing one, or would a different implementation of an in-memory cache be sufficient (provided by a third-party library, or due to changes to MemoryCache itself)?

and also avoiding using swap space, if available

Side note: this may be outside application notice/control, depending on OS/settings.

I do not want to write another cache, and I do not know how a cache could (even in principle) have the characteristics that soft references would provide.

this may be outside application notice/control,

Yes, but to the extent it is possible, it would involve OS-specific metrics/mechanisms that maybe the CLR already uses to help choose the GC interval. I expect it is possible to ask the OS how much physical memory there is, and how much memory is being used by other apps and by the current app; this seems like all that is needed to choose a soft cap.

Soft reference is actively used in JVM for caching but personally I'm not a fan of this mechanism because SoftMemoryLimit is very speculative. Probably, its better to use generations here and add a new configuration to WeakReference and GCHandle. For instance, we can allow to collect the object if it is not reachable through any strong reference and located in Gen2. This analysis can be done during full/background GC. I think it is more predictable behavior. Many users will not be able to calculate SoftMemoryLimit correctly. Normally, Gen2 should not grow constantly in normal applications so this policy has the same effect as the memory limit.

Soft reference is actively used in JVM for caching

... When I did some cursory research for this issue earlier, I found documentation that for at least one JVM implementation, SoftReference is/was only an alias for WeakReference, but I can't find it now. So that might not work the way you expect.

Btw, this implementation does not guarantee the target will live until a gen 2 collection, it only guarantees it will live until it is promoted to gen 2. To guarantee life until a gen 2 collection will require internal APIs. I believe there is a Gen2Callback internally of some sort.

[Edit] Do resurrected objects get promoted to higher generations? If so, we could check the generation of the tracker object in its finalizer before calling parent.OnGC()to guarantee survival until gen 2 collection.

[Edit2] I just reread the GC documentation, and it seems I was incorrect here. Objects that are promoted to gen 2 will only be collected in a gen 2 collection, even if they are eligible for collection during a gen 0 or gen 1 collection.

There is another problem - the implementation is not thread safe.

There is another problem - the implementation is not thread safe.

Which part? SetTarget and TryGetTarget are as thread safe as WeakReference is. [Edit] Actually I take that back. TryGetTarget should cache the target in a local before returning instead of overwriting the out variable.

I thought about thread safety for the OnGC, but I wasn't sure if it really matters. Isn't the GC usually stop-the-world and single threaded? Is a concurrent GC really an issue to be concerned about?

Ok, here's a thread-safer version. I think there's no need to try to synchronize SetTarget, because if the user is calling that on 2 separate threads, that's a race condition where you wouldn't be able to tell which one sticks anyway.

I also don't believe WeakReference is thread-safe anyway, so this may be an effort in futility (no point in making SoftReference thread safe if WeakReference isn't).

public class SoftReference<T> where T : class
{
    private sealed class Tracker
    {
        private readonly SoftReference<T> parent;

        internal Tracker(SoftReference<T> parent)
        {
            this.parent = parent;
        }

        ~Tracker()
        {
            parent.OnGC();
            GC.ReRegisterForFinalize(this);
        }

        internal void StopTracking()
        {
            GC.SuppressFinalize(this);
        }
    }

    volatile private T? target;
    private readonly WeakReference<T?> targetReference;
    private readonly WeakReference<Tracker> callbackReference; // WeakReference allows finalizer to run, but it always resurrects itself until this is finalized.

    public SoftReference(T? target, bool trackResurrection)
    {
        this.target = target;
        targetReference = new(target, trackResurrection);
        var tracker = new Tracker(this);
        callbackReference = new(tracker, trackResurrection: true);
        GC.KeepAlive(tracker);
    }

    ~SoftReference()
    {
        callbackReference.TryGetTarget(out Tracker tracker);
        tracker.StopTracking();
    }

    private void OnGC()
    {
        T? _target = target;
        if (_target != null && GC.GetGeneration(_target) == GC.MaxGeneration)
        {
            Interlocked.CompareExchange(ref target, null, _target); // downgrade from soft to weak reference
        }
    }

    internal void SetTarget(T? target)
    {
        this.target = target;
        targetReference.SetTarget(target);
    }

    internal bool TryGetTarget(out T? target)
    {
        return targetReference.TryGetTarget(out target);
    }
}

But if you really wanted, you could just lock (targetReference).

[Edit] I removed the strong reference read in TryGetTarget because I realized it's unnecessary. Always reading from the WeakReference is perfectly fine.

One bit of feedback on this and WeakReference from someone much like the proposer that uses WeakReference extensively for caching/GC control applications.
Please provide a common Interface type IReference that both WeakReference, SoftReference and a hypothetical but obvious implementation HardReference (it's an Object!). Currently I'm either forced to juggle different lists, coordinate different locks when promoting/demoting or (as I've done) wrap WeakReference in another class and accept the additional object cost. If WeakReference implemented any interface I could have avoid several annoying things.

@timcassell

@timcassell One lesson I learned previously from playing with resurrection is to never use it.

One issue of your code is, when the SoftReference object is no longer needed, execution order of the finalizers of SoftReference and its tracker will be unknown, meaning that you may have StopTracking() called first, and ~Tracker() called after it, which resurrects everything again forever.

Also I don't want to assume that checking the referenced object's generation can reflect the overall memory pressure. As jkotas said, it's probably much easier to explicitly check memory usage. The worst thing you can do is to add a separate background thread and check periodically. Even though, it will still be better, because GC no longer needs to handle those weak references and resurrections repeatedly, especially when the number of tracked objects increases.

Please provide a common Interface type IReference that both WeakReference, SoftReference and a hypothetical but obvious implementation HardReference

You can implement weak references and strong references yourself and add whatever interfaces to it. The standard WeakReference<T> internally uses GCHandle. I am not sure about soft reference though.

@timcassell

@timcassell One lesson I learned previously from playing with resurrection is to never use it.

One issue of your code is, when the SoftReference object is no longer needed, execution order of the finalizers of SoftReference and its tracker will be unknown, meaning that you may have StopTracking() called first, and ~Tracker() called after it, which resurrects everything again forever.

Does GC.SuppressFinalize not remove it from the finalizer queue? That would be surprising behavior.

According to the documentation, it will not be called.

This method sets a bit in the object header of obj, which the runtime checks when calling finalizers.

Also I don't want to assume that checking the referenced object's generation can reflect the overall memory pressure.

I agree. I don't particularly like this approach, I was just piggy-backing off @sakno's idea.

@timcassell

@timcassell @acaly Even if would not be removed from finalizer's queue, why cannot we just apply dispoable pattern? Code in finalizer will be not evaluated further.

Also, shouldn't tracker inherit from CriticalFinalizerObject?

using System.Runtime.ConstrainedExecution;

    public class SoftReference<T>
        where T : class
    {
        private sealed class Tracker : CriticalFinalizerObject, IDisposable
        {
            private bool disposed;
            private readonly SoftReference<T> parent;

            internal Tracker(SoftReference<T> parent)
            {
                this.parent = parent;
            }

            ~Tracker()
            {
                if (disposed)
                {
                    return;
                }

                parent.OnGC();
                GC.ReRegisterForFinalize(this);
            }

            public void Dispose()
            {
                disposed = true;
                GC.SuppressFinalize(this);
            }
        }

        private volatile T? target;
        private readonly WeakReference<T?> targetReference;
        private readonly WeakReference<Tracker> callbackReference;

        public SoftReference(T? target, bool trackResurrection)
        {
            this.target = target;
            targetReference = new WeakReference<T?>(target, trackResurrection);
            var tracker = new Tracker(this);
            callbackReference = new WeakReference<Tracker>(tracker, trackResurrection: true);
            GC.KeepAlive(tracker);
        }

        ~SoftReference()
        {
            callbackReference.TryGetTarget(out Tracker tracker);
            tracker.Dispose();
        }

        private void OnGC()
        {
            T? _target = target;
            if (_target != null && GC.GetGeneration(_target) == GC.MaxGeneration)
            {
                Interlocked.CompareExchange(ref target, null, _target);
            }
        }

        internal void SetTarget(T? target)
        {
            this.target = target;
            targetReference.SetTarget(target);
        }

        internal bool TryGetTarget(out T? target)
        {
            return targetReference.TryGetTarget(out target);
        }
    }

@timcassell

@timcassell @acaly Even if would not be removed from finalizer's queue, why cannot we just apply dispoable pattern? Code in finalizer will be not evaluated further.

Yes, you could do that.

Also, shouldn't tracker inherit from CriticalFinalizerObject?

There is nothing critical about this. If the framework wants to clean up the domain without calling the finalizer, we don't care.

Could be that in my perspective soft reference should be more proactive. However, CriticalFinalizerObject seems to be gurantee for freeing memory even under pressure. Isn't it a goal of SoftReference to be cleaned up under pressure?

Could be that in my perspective soft reference should be more proactive. However, CriticalFinalizerObject seems to be gurantee for freeing memory even under pressure. Isn't it a goal of SoftReference to be cleaned up under pressure?

CriticalFinalizerObject is used to guarantee the finalizer runs even while domains are being forcibly unloaded or threads aborted. Those aren't supported in .Net Core, so it only really applies to .Net Framework. In the case of domain unload, the memory is already being forcibly cleaned up, so there's no need for us to do anything. We're not holding any unmanaged memory here.

Navigation Menu

Repository navigation

[API Proposal]: Soft References #63113

Background and motivation

API Proposal

API Usage

Alternative Designs

Risks

18 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[API Proposal]: Soft References #63113

Description

Background and motivation

API Proposal

API Usage

Alternative Designs

Risks

Activity

huoyaoyuan commented on Dec 24, 2021

jkotas commented on Dec 24, 2021

teo-tsirpanis commented on Dec 24, 2021

jkotas commented on Dec 24, 2021

qwertie commented on Dec 24, 2021

Clockwork-Muse commented on Dec 25, 2021

qwertie commented on Dec 27, 2021

Clockwork-Muse commented on Dec 27, 2021

qwertie commented on Dec 28, 2021

sakno commented on Jan 3, 2022

Clockwork-Muse commented on Jan 3, 2022

18 remaining items

timcassell commented on Jan 13, 2022

sakno commented on Jan 13, 2022

timcassell commented on Jan 13, 2022

timcassell commented on Jan 13, 2022

s-michael-telford commented on Jun 16, 2023

acaly commented on Jun 16, 2023

acaly commented on Jun 16, 2023

timcassell commented on Jun 16, 2023

Hau-Hau commented on Sep 3, 2024

timcassell commented on Sep 3, 2024

Hau-Hau commented on Sep 3, 2024

timcassell commented on Sep 3, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions