Proposal: Add mechanism to handle circular references when serializing

See initial proposal with extended comments here:
https://github.com/dotnet/runtime/pull/354/files

See proposal extension for ReferenceResolver on #30820 (comment).

Rationale and Usage

Currently there is no mechanism to prevent infinite looping in circular objects nor to preserve references when using System.Text.Json.

Community is heavily requesting this feature since is consider by many as a very common scenario, specially when serializing POCOs that came from an ORM Framework, such as Entity Framework; even though JSON specifiacation does not support reference loops by default. Therefore this will be shipped as an opt-in feature.

The current solution to deal with reference loops is to rely in MaxDepth and throw a JsonException after it is exceeded. Now, this is a decent and cheap solution but we will also offer other not-so-cheap options to deal with this problem while keeping the current one in order to not affect the out-of-the-box performance.

Proposed API

namespace System.Text.Json
{
    public class JsonSerializerOptions
    {
        public ReferenceHandling ReferenceHandling { get; set; } = ReferenceHandling.Default;
    }
}

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// This enumeration defines the various ways the <see cref="JsonSerializer"/> 
    /// can deal with references on serialization and deserialization.
    /// </summary>
    public enum ReferenceHandling
    {
        Default,
        Preserve
    }
}

EDIT:

We considered having ReferenceHandlig.Ignore but it was cut out of the final design due the lack of scenarios where you would really need Ignore over Preserve.

Although is not part of the shipping API, the samples and definitions of Ignore remain in this description for their informative value.

In depth

Default:
- On Serialize: Throw a JsonException when MaxDepth is exceeded, this may occur by either a Reference Loop or by passing a very deep object. This option will not affect the performance of the serializer.
- On Deserialize: No effect.
Ignore:
- On Serialize: Ignores (skips writing) the property/element where the reference loop is detected.
- On Deserialize: No effect.
Preserve:
- On Serialize: When writing complex types, the serializer also writes them metadata ($id, $values and $ref) properties in order re-use them by writing a reference to the object or array.
- On Deserialize: While the other options show no effect on Deserialization, Preserve does affect its behavior with the following: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it.

Feature Parity (Examples of System.Text.Json vs Newtonsoft's Json.Net)

Having the following class:

class Employee 
{ 
    string Name { get; set; }
    Employee Manager { get; set; }
    List<Employee> Subordinates { get; set; }
}

Using Ignore on Serialize

On System.Text.Json:

public static void WriteIgnoringReferenceLoops()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var options = new JsonSerializerOptions
    {
        ReferenceHandling = ReferenceHandling.Ignore
        WriteIndented = true,
    };

    string json = JsonSerializer.Serialize(angela, options);
    Console.Write(json);
}

On Newtonsoft's Json.Net:

public static void WriteIgnoringReferenceLoops()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var settings = new JsonSerializerSettings
    {
        ReferenceLoopHandling = ReferenceLoopHandling.Ignore
        Formatting = Formatting.Indented 
    };

    string json = JsonConvert.SerializeObject(angela, settings);
    Console.Write(json);
}

Output:

{
    "Name": "Angela",
    "Manager": {
        "Name": "Bob",
        "Subordinates": [] //Note how subordinates is empty due Angela is being ignored.
    }
}

Using Preserve on Serialize

On System.Text.Json:

public static void WritePreservingReference()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var options = new JsonSerializerOptions
    {
        ReferenceHandling = ReferenceHandling.Preserve
        WriteIndented = true,
    };

    string json = JsonSerializer.Serialize(angela, options);
    Console.Write(json);
}

On Newtonsoft's Json.Net:

public static void WritePreservingReference()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var settings = new JsonSerializerSettings
    {
        PreserveReferencesHandling = PreserveReferencesHandling.All
        Formatting = Formatting.Indented 
    };

    string json = JsonConvert.SerializeObject(angela, settings);
    Console.Write(json);
}

Output:

{
    "$id": "1",
    "Name": "Angela",
    "Manager": {
        "$id": "2",
        "Name": "Bob",
        "Subordinates": { //Note how the Subordinates' square braces are replaced with curly braces in order to include $id and $values properties, $values will now hold whatever value was meant for the Subordinates list.
            "$id": "3",
            "$values": [
                {  //Note how this object denotes reference to Angela that was previously serialized.
                    "$ref": "1"
                }
            ]
        }            
    }
}

Using Preserve on Deserialize

On System.Text.Json:

public static void ReadJsonWithPreservedReferences(){
    string json = 
    @"{
        ""$id"": ""1"",
        ""Name"": ""Angela"",
        ""Manager"": {
            ""$id"": ""2"",
            ""Name"": ""Bob"",
            ""Subordinates"": {
                ""$id"": ""3"",
                ""$values"": [
                    { 
                        ""$ref"": ""1"" 
                    }
                ]
            }            
        }
    }";

    var options = new JsonSerializerOptions
    {
        ReferenceHandling = ReferenceHandling.Preserve
    };

    Employee angela = JsonSerializer.Deserialize<Employee>(json, options);
    Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true.
}

On Newtonsoft's Json.Net:

public static void ReadJsonWithPreservedReferences(){
    string json = 
    @"{
        ""$id"": ""1"",
        ""Name"": ""Angela"",
        ""Manager"": {
            ""$id"": ""2"",
            ""Name"": ""Bob"",
            ""Subordinates"": {
                ""$id"": ""3"",
                ""$values"": [
                    { 
                        ""$ref"": ""1"" 
                    }
                ]
            }            
        }
    }";

    var options = new JsonSerializerSettings
    {
        MetadataPropertyHanding = MetadataPropertyHandling.Default //Json.Net reads metadata by default, just setting the option for ilustrative purposes.
    };

    Employee angela = JsonConvert.DeserializeObject<Employee>(json, settings);
    Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true.
}

Notes:

MaxDepth validation will not be affected by ReferenceHandling.Ignore or ReferenceHandling.Preserve.
We are merging the Json.Net types ReferenceLoopHandling and PreserveReferencesHandling (we are also not including the granularity on this one) into one single enum; ReferenceHandling.
While Immutable types and System.Arrays can be Serialized with Preserve semantics, they will not be supported when trying to Deserialize them as a reference.
Value types, such as structs, will not be supported when Deserializing as well.
Additional features, such as Converter support, ReferenceResolver, JsonPropertyAttribute.IsReference and JsonPropertyAttribute.ReferenceLoopHandling, that build on top of ReferenceLoopHandling and PreserveReferencesHandling were considered but they will not be included in this first effort.
We are still looking for evidence that backs up supporting ReferenceHandling.Ignore.

Issues related:

@ahsonkhan

@ahsonkhan @JamesNK thoughts?

Consider changing "values" in the array metadata object to "$values" to indicate it isn't a real property.

{
  "$id": "1",
  "Name": "Angela",
  "Subordinates": {
    "$id": "2",
    "$values": [
      {
        "$id": "3",
        "Name": "Bob",
        "Manager": {
            "$ref": "1"
        }
      }
    ]
  }
}

Instead of a dedicated property, what about adding ReferenceHandling as a property to JsonPropertyAttribute (is that the right name? I don't recall off the top of my head).

[JsonReferenceHandling(ReferenceHandling.Preserve)]

// to

[JsonProperty(ReferenceHandling = ReferenceHandling.Preserve)]

Consider changing "values" in the array metadata object to "$values" to indicate it isn't a real property.

Agree.

Instead of a dedicated property, what about adding ReferenceHandling as a property to JsonPropertyAttribute (is that the right name? I don't recall off the top of my head).

JsonPropertyAttribute does not exist in System.Text.Json as far as I know.

@steveharter

JsonPropertyAttribute does not exist in System.Text.Json as far as I know.

Correct, we have JsonPropertyNameAttribute
https://github.com/dotnet/corefx/blob/b41b09eadd5eb3f5575845a982f2c177c37f7ce7/src/System.Text.Json/ref/System.Text.Json.cs#L748-L753

cc @steveharter

Is this proposal compatible with Json.Net?

ReferenceHandling: byte

Why byte?

What determines if objects are equal? object.ReferenceEquals or object.Equals? Newtonsoft.Json uses object.Equals.

Is there a way to override equality? Newtonsoft.Json has https://www.newtonsoft.com/json/help/html/P_Newtonsoft_Json_JsonSerializerSettings_EqualityComparer.htm

What happens if there are additional properties in a JSON reference object? (has $ref) What about if $ref isn't the first property in an object?

What happens if $id isn't the first property?

Is this proposal compatible with Json.Net?

Yes for the metadata semantics that I am trying to follow; no for a feature parity, however we are not tied to follow the first one, just come up with what is already there since I have not found value in going on a different direction.

Why byte?

I was following existing enums in the namespace that also inherits from byte, thing that is made to reduce their size in the Utf8JsonReader struct. but since the Serializer is a class, you are right, there is no much value in changing the enum's default value, I will fix that. Thanks.

What determines if objects are equal? object.ReferenceEquals or object.Equals? Newtonsoft.Json uses object.Equals.

Currently the implementation relies on default equality of Hashset/Dictionary, which uses the Object.GetHashCode() and Object.Equals(Object obj).
If someone wants to provide its own equals mechanism, he can override these methods and/or implement IEquatable on its class.

What happens if there are additional properties in a JSON reference object? (has $ref) What about if $ref isn't the first property in an object?

Json.Net throws when finds that there are properties other than $ref in a JSON reference object, I was planning on sticking to that behavior.
I see no point in allowing other properties to co-exist with a $ref, unless the user relies on the $ref property for another purpose; for that maybe in a future we could add a mechanism to disable this functionality similar to MetadataPropertyHandling.Ignore

@JamesNK

@JamesNK on "Ignore" - is this mode necessary and what scenarios would prefer that over "Preserve"?

"Ignore" has potnential inconsistentcy w.r.t. what is serialized vs. what is ignored which would make me question using it in certain scenarios including:

Serialization order currently depends on reflection order which is not deterministic, so if property A and B both serialize the same reference, sometimes A will be serialized and sometimes B will be serialized.
The root of the graph may change causing a given type to sometimes serialize a reference, and other times not.

In many scenarios the [JsonIgnore] could be used instead to prevent cycles which has the advantage of being deterministic.

However if:

"Throws" isn't desired
and "Preserve" isn't desired because the metadata ($id $ref) or other reasons (performance?)
and [JsonIgnore] can't be used (perhaps attributes can't be used)
and potential non-deterministic behavior is OK

then "Ignore" makes sense.

Video

We seem to lean towards not having Ignore -- it results in payloads that nobody can reason about. The right fix for those scenarios to modify the C# types to exclude back pointers from serialization.

We'd like to see more detailed spec for the behavior, specifically error cases.

Ok, I don't feel strongly that ignore is necessary. And it should never be the default setting so you can always add it in the future if desired.

@jozkee

Hi @jozkee , is there a way to try PreserveReferencesHandling, like in a nuget preview?

It is not a question of happiness, but of functionality. This is telling users to implement custom code for basic functionality, and I'd say having a global way to pass options to a serializer/deserializer also on the client is not asking too much for a framework which heavily relies on those operations.

Frankly speaking, looking at all the threads and on StackOverflow here on this issue, there has been a lot of energy wasted arguing that "Ignore is not needed". Probably the same energy could have been channelled into implementing the functionality, and resolve the issue.

@BrunoBlanes

@BrunoBlanes Here are the issue numbers I have read and/or commented on regarding this issue:

#29900

dotnet/aspnetcore#28286

#42584

It looked to me like the fix for the problem was to update System.Net.Http.Json to version 5.0.0 in the client project and then
implement one of the following two code changes (as described in the referenced articles above) in the ConfigureServices method
of Startup.cs in the Server project:

services.AddControllersWithViews().AddJsonOptions(options =>
{
options.JsonSerializerOptions.ReferenceHandler = System.Text.Json.Serialization.ReferenceHandler.Preserve;
});

or

services.AddControllersWithViews(options =>
{
options.OutputFormatters.RemoveType();
options.OutputFormatters.Add(new SystemTextJsonOutputFormatter(new JsonSerializerOptions(JsonSerializerDefaults.Web)
{
ReferenceHandler = ReferenceHandler.Preserve
}));
});

Am i on the right track here? Is there anything else that needs to be done? Do i also need to update my Server project
to change it from targeting .NET Core 3.1 to targeting .NET 5.0?

@ryantgray

@ryantgray I thionk you are on the right track. You shouldn't need to update to .Net 5, as long as you're running STJ 5 then you should be good. (I believe the ReferenceHandler flag didn't work at all in v3) Preserve is an excellent choice for retaining the full object tree, although it absolutely has drawbacks, as well as advantages:

Drawbacks

It puts a load of additional metadata into the JSON, which as far as I'm aware only STJ knows how to parse correctly.
It kinda breaks the original data structure - This is related to the metadata, but a lot of the actual data gets moved around and it's not as human readable as normally serialised data.
As a result of the above, you essentially need complete control over the clients that will be reading the data. For example (at least in .net) you can pretty much only properly deserialise with STJ - Newtonsoft can serialise the tokens but doesn't seem capable of deserialising properly.
Preserve deserialisation may be a little unreliable - See Should we add support to ignore cycles on serialization? #40099 (comment) (although this is anecdotal).

Advantages

Compared to the output of JSON.Net with Ignore, the payload (depending on the complexity and other attributes of the object tree) can be SIGNIFICANTLY SMALLER. This is because with preserved references each isntance is only serialised once, and essentially stored as a variable in the json, then when it's referred to it's just done as a reference to the variable, instead of serialised again like Json.net:

It's much faster - STJ utlises new features of .net and types like the Span types and all sorts of other magic of .Net core 3, whereas Newtonsoft is obviously a lot older so only really uses the String types that were available at the time.

The conversation continues over on #40999

This issue was closed a while ago so you're best posting in there.

@butler1233

@butler1233 Thanks Lee but your comment did not address my issue. I don't really care about the advantages and disadvantages. I only care about getting it to work. It seems like it should be easy to have someone confirm that you guys implemented changes in JSON 5.0 to address the scenario I'm mentioning and, if so, what needs to be done in a standard default implementation of Blazor WebAssembly that Visual Studio creates to implement it. Also, you comment "this issue was closed a while ago so you're best posting in 40999" but that is closed as well and seems to have nothing to do with my issue.

Can someone please just give me the specific steps including the code changes that need to be made, or point me to them, for what needs to be done in a default implementation of Blazor WebAssembly to set JSON Reference Handling to Preserve? Then, if i have done everything you guys say needs to be done and am still having issues I will take the necessary steps to address them at that point but the first thing i want to do is make sure I have all the necessary changes implemented correctly. Thanks.

@ryantgray

@ryantgray Again, as mentioned so many times in this thread, ReferenceHandling.Ignore is not supported in S.T.Json, do you wish to implement it using Newtonsoft.Json? I thought you already had it up and working with ReferenceHandling.Preserve.
Keep in mind that we are all doing you a favor, this is not the place to learn to code, questions like yours should be opened at Stackoverflow.

@BrunoBlanes

@BrunoBlanes It feels like there's a communication disconnect here. Maybe your getting me confused with AlbertoPa? I have never inquired about Ignore. My inquiry was always about Preserve and no, i have never gotten Preserve working. I understand this is not the place to learn code and did start out posting on places like Stackoverflow. However, noone has been able to provide any insight, possibly because you just released your changes to JSON within the last month and the combination of that and Blazor being so new...? If there is a better place/way to address my issue I'd be more than happy to pursue it if i knew what it was. However, I'm trying to implement some functionality released by Microsoft that is not working and so far, noone else seems to have any answers and i'm not even 100% sure the changes you guys made will address the situation in a Blazor WebAssembly application so I feel Microsoft may be the only entity that can address this.

@ryantgray

It looked to me like the fix for the problem was to update System.Net.Http.Json to version 5.0.0 in the client project and then
implement one of the following two code changes (as described in the referenced articles above) in the ConfigureServices method
of Startup.cs in the Server project:
services.AddControllersWithViews().AddJsonOptions(options =>
{
options.JsonSerializerOptions.ReferenceHandler = System.Text.Json.Serialization.ReferenceHandler.Preserve;
});

@ryantgray Didn't this fix it for you?

That's the way it looked to me to based on the links I provided but no it did not fix it.

If you are using Blazor, do also set PropertyNamingPolicy = JsonNamingPolicy.CamelCase or null in your AddJsonOptions method.

@BrunoBlanes

@BrunoBlanes Thanks, I tried implementing your suggestion with both of the code options I posted and I still get the same errors.

Server

services.AddControllersWithViews().AddJsonOptions(options =>
{
    options.JsonSerializerOptions.ReferenceHandler = System.Text.Json.Serialization.ReferenceHandler.Preserve;
    options.JsonSerializerOptions.PropertyNamingPolicy = null;
});

Client

var response = await Http.GetFromJsonAsync<T>("{Address}", new JsonSerializerOptions
{
    ReferenceHandler =  System.Text.Json.Serialization.ReferenceHandler.Preserve,
    PropertyNamingPolicy = null
});

Try this and let me know what error are you getting.

Advantages

Compared to the output of JSON.Net with Ignore, the payload (depending on the complexity and other attributes of the object tree) can be SIGNIFICANTLY SMALLER. This is because with preserved references each isntance is only serialised once, and essentially stored as a variable in the json, then when it's referred to it's just done as a reference to the variable, instead of serialised again like Json.net:

I observe a reduction in size in relatively small models or when the EF navigation properties are not fully populated. When pulling complex models, that does not seem to be the case. One example: if I pull an incomplete object with a query, to list objects for CRUD operations in a table, then System.Text.Json gives a smaller payload than Newtonsoft.Json (up to 11 Bytes smaller in my tests over 766 Bytes of objects: 1.4% difference). However, when populating all related data, the situation inverts: System.Text.Json generates a payload 8.27% larger than Newtonsoft. This may be OK if the data size is modest, but it isn't negligible for larger models.

It's much faster - STJ utlises new features of .net and types like the Span types and all sorts of other magic of .Net core 3, whereas Newtonsoft is obviously a lot older so only really uses the String types that were available at the time.

This is absolutely the case, hence why it would be good to be able to use it without struggling 😆 . The advantage on the server side is clearly visible when serializing large amounts of data.

Edit: Bruno's latest response works, also with

PropertyNamingPolicy = System.Text.Json.JsonNamingPolicy.CamelCase

at least in my cases. I seem to have some issue when the objects are self-referencing, but I need to troubleshoot more to find what is causing it, because it is not happening systematically 🤷‍♂️

Ah, my apologies, it looks like I linked to the wrong issue. Should have been #40099. They didnt make the changes to add something like Ignore in 5, although as discussed in that (hopefully correctly linked) issue it's being considered for 6.

…

________________________________ From: Ryan Gray <notifications@github.com> Sent: Sunday, 13 December 2020, 17:01 To: dotnet/runtime Cc: Lee Butler; Mention Subject: Re: [dotnet/runtime] Proposal: Add mechanism to handle circular references when serializing (#30820) @butler1233<https://github.com/butler1233> Thanks Lee but your comment did not address my issue. I don't really care about the advantages and disadvantages. I only care about getting it to work. It seems like it should be easy to have someone confirm that you guys implemented changes in JSON 5.0 to address the scenario I'm mentioning and, if so, what needs to be done in a standard default implementation of Blazor WebAssembly that Visual Studio creates to implement it. Also, you comment "this issue was closed a while ago so you're best posting in 40999" but that is closed as well and seems to have nothing to do with my issue. Can someone please just give me the specific steps including the code changes that need to be made, or point me to them, for what needs to be done in a default implementation of Blazor WebAssembly to set JSON Reference Handling to Preserve? Then, if i have done everything you guys say needs to be done and am still having issues I will take the necessary steps to address them at that point but the first thing i want to do is make sure I have all the necessary changes implemented correctly. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#30820 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADZLJXK4TPBN7VEBZIYGF6DSUTXONANCNFSM4KY4YW5Q>.

@ryantgray

@ryantgray, it seems to me that @BrunoBlanes' suggestion should cover your problem.

Do I also need to update my Server project to change it from targeting .NET Core 3.1 to targeting .NET 5.0?

You need to use at least version 5.0.0 of System.Text.Json in order to be able to use the ReferenceHandler.Preserve feature, this can be achieved by either changing your target .NET version to 5.0 or by installing the latest standalone version of S.T.Json from nuget.

in case you are still having issues, kindly can you please share it with us here and/or in a new issue tagging me. A standalone repro (using the Blazor template for example) would also be very helpful for us.

Navigation Menu

Repository navigation

Proposal: Add mechanism to handle circular references when serializing #30820

Rationale and Usage

Proposed API

EDIT:

In depth

Feature Parity (Examples of System.Text.Json vs Newtonsoft's Json.Net)

Using Ignore on Serialize

Using Preserve on Serialize

Using Preserve on Deserialize

Notes:

Issues related:

104 remaining items

Drawbacks

Advantages

The conversation continues over on #40999

Server

Client

Advantages

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal: Add mechanism to handle circular references when serializing #30820

Description

Rationale and Usage

Proposed API

EDIT:

In depth

Feature Parity (Examples of System.Text.Json vs Newtonsoft's Json.Net)

Using Ignore on Serialize

Using Preserve on Serialize

Using Preserve on Deserialize

Notes:

Issues related:

Activity

jozkee commented on Sep 10, 2019

JamesNK commented on Sep 10, 2019

jozkee commented on Sep 10, 2019

ahsonkhan commented on Sep 11, 2019

scalablecory commented on Sep 11, 2019

scalablecory commented on Sep 11, 2019

JamesNK commented on Sep 11, 2019

JamesNK commented on Sep 11, 2019

jozkee commented on Sep 11, 2019

jozkee commented on Sep 16, 2019

steveharter commented on Sep 17, 2019

terrajobst commented on Sep 17, 2019

JamesNK commented on Sep 17, 2019

paulovila commented on Sep 27, 2019

104 remaining items

AlbertoPa commented on Dec 13, 2020

ryantgray commented on Dec 13, 2020

butler1233 commented on Dec 13, 2020

Drawbacks

Advantages

The conversation continues over on #40999

ryantgray commented on Dec 13, 2020

BrunoBlanes commented on Dec 13, 2020

ryantgray commented on Dec 13, 2020

BrunoBlanes commented on Dec 13, 2020

ryantgray commented on Dec 13, 2020

BrunoBlanes commented on Dec 13, 2020

ryantgray commented on Dec 13, 2020

BrunoBlanes commented on Dec 13, 2020

Server

Client

AlbertoPa commented on Dec 13, 2020

Advantages

butler1233 commented on Dec 14, 2020

jozkee commented on Dec 15, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions