JsonSerializer polymorphic serialization and deserialization support

@Symbai

EDIT see #30083 (comment) for a finalized proposal.

Original proposal by @Symbai (click to view)

public class FooBase
public class FooA : FooBase
public class FooB : FooBase
List<FooBase> SerializationObject = new List<FooBase> { new FooA(), new FooB() }

Serialized with Newtonsoft.JSON

JsonSerializerSettings Settings = new JsonSerializerSettings
{
    SerializationBinder = new KnownTypesBinder
    {
        KnownTypes = new List<Type> { typeof(FooA), typeof(FooB) }
    },
    TypeNameHandling = TypeNameHandling.Objects
};
List<FooBase> FooBases = new List<FooBase>() { new FooA(), new FooB() };
var json = JsonConvert.SerializeObject(FooBases, Settings);

will be:
[{"$type":"FooA","NameA":"FooA","Name":"FooBase"},{"$type":"FooB","NameB":"FooB","Name":"FooBase"}]

When using System.Text.Json.JsonSerializer to deserialize the json, FooA and FooB are both type of FooBase. Is there a way that JsonSerializer supports inheirited classes? How can I make sure the type will be the same and not the base class it inherits from?

Figured it's somehow possible using JsonConverter, which is for some reason completely undocumented https://docs.microsoft.com/en-us/dotnet/api/system.text.json?view=netcore-3.0 but I have no idea how to deserialize without copying each property by hand like it's shown in this example https://github.com/dotnet/corefx/issues/36639. Copying properties by hand seems extremely odd as classes may have lots of them and maintainability isn't given in such a case anymore. Is there a way I can simply deserialize into a specific type in the converter?

Or am I missing something and there is a simple solution like in Newtonsoft.Json ?

public class FooAConverter : System.Text.Json.Serialization.JsonConverter<FooBase>
{
    public override bool CanConvert(Type type)
    {
        return typeof(FooBase).IsAssignableFrom(type);
    }
    public override FooBase Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        reader.Read();
        var typeProperty = reader.GetString();
        if (typeProperty != "$type")
            throw new NotSupportedException();
        reader.Read();
        var typeValue = reader.GetString();
        FooBase value;
        switch (typeValue)
        {
            case "FooA":
                value = new FooA(); //<-- return deserialized FooA?
                break;
            case "FooB":
                value = new FooB(); //<-- return deserialized FooB?
                break;
            default:
                throw new System.Text.Json.JsonException();
        }
        while (reader.Read())
        {
            //How to deserialize without copying every property at this point?
            if (reader.TokenType == JsonTokenType.EndObject)
            {
                return value;
            }
        }
        throw new NotSupportedException();
    }

    public override void Write(Utf8JsonWriter writer, FooBase value, JsonSerializerOptions options)    {    }
}

@steveharter

cc @steveharter

It's pretty awkward that JsonConverter doesn't give you a way to recurse back into the "regular" flow (i.e. "not handled by this particular converter") for deserializing subcomponents.

This seems to make the whole design only useful for very simple cases, with few properties and no nested custom types.

Figured it's somehow possible using JsonConverter, which is for some reason completely undocumented https://docs.microsoft.com/en-us/dotnet/api/system.text.json?view=netcore-3.0

It's in the sub-namespace Serialization: https://docs.microsoft.com/en-us/dotnet/api/system.text.json.serialization?view=netcore-3.0

I have no idea how to deserialize without copying each property by hand like it's shown in this example

and

t's pretty awkward that JsonConverter doesn't give you a way to recurse back into the "regular" flow (i.e. "not handled by this particular converter") for deserializing subcomponents

Yes that is known - the existing converters are bare-bones converters that let you do anything you want, but also require you do essentially do everything for a given type (but you can recurse to handle other types). They were primarily intended for data-type converters, not for objects and collections.

We are working on making this better in 5.0 timeframe by providing object- and collection- based specific converters that would make this much easier, as well as improve performance in certain cases.

@steveharter

but you can recurse to handle other types

@steveharter That seems like it could help. Can you give an example of how to do this?

We are using custom converters like the following for inherited class support. I'm pretty sure deserialization is abysmal from a perf point of view, but it works and doesn't need much code and serialization works as it should. So as a Workaround you can use somthing like this:

public class ParameterTransferObjectConverter : JsonConverter<ParameterTransferObject>
{
    public override ParameterTransferObject Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        using (var doc = JsonDocument.ParseValue(ref reader))
        {
            ParamType typeDiscriminator = (ParamType)doc.RootElement.GetProperty(@"parameterType").GetInt32();
            var type = ParameterManager.GetTransferObjectType(typeDiscriminator);
            // Enhance: doc.RootElement.GetRawText() has likely bad perf characteristics
            return (ParameterTransferObject)JsonSerializer.Deserialize(doc.RootElement.GetRawText(), type, options);
        }
    }

    public override void Write(Utf8JsonWriter writer, ParameterTransferObject value, JsonSerializerOptions options)
    {
        var type = ParameterManager.GetTransferObjectType(value.ParameterType);
        JsonSerializer.Serialize(writer, value, type, options);
    }
}

@ANahr

Thanks @ANahr, indeed performance must be abysmal in comparison, but this allowed me to at least get deserialization to work till System.Text.Json matures, in my opinion it's a mistake for Microsoft to encourage people to use it, it should still be flagged as preview

Marking this as the issue to track polymorphic deserialization for 5.0.
Serialization is tracked in https://github.com/dotnet/corefx/issues/38650.

@Milkitic

From @Milkitic in dotnet/corefx#41305

For some reasons that Json for configuration use, I did a test on how the System.Text.Json handles the interface deserialization. I read the API and can't find anything about this function. First I tried IEnumerable<T>. Here is the code:

using System.Collections.Generic;
using System.Text.Json;

namespace dotnet30test
{
    class Program2
    {
        static void Main(string[] args)
        {
            var obj = new TestSerializationObj2
            {
                Names = new HashSet<string> { "1", "2" }
            };
            var sysContent = JsonSerializer.Serialize(obj, new JsonSerializerOptions { WriteIndented = true, });
            var sysNewObj = JsonSerializer.Deserialize<TestSerializationObj>(sysContent);
        }
    }

    public class TestSerializationObj2
    {
        public IEnumerable<string> Names { get; set; }
    }
}

The Names property is forced to convert from Hashset<string> to List<string>, but it works successfully indeedly.
And then I try the custom interface, it throws JsonException. What I think that needed is something like Json.NET's auto type name handling. Here is the code:

using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using NewtonJson = Newtonsoft.Json;
using SysJson = System.Text.Json;

namespace dotnet30test
{
    class Program
    {
        static void Main(string[] args)
        {
            var obj = new TestSerializationObj
            {
                CustomObj = new DefaultCustom()
            };

            var newtonContent = NewtonJson.JsonConvert.SerializeObject(obj,
                new NewtonJson.JsonSerializerSettings
                {
                    TypeNameHandling = NewtonJson.TypeNameHandling.Auto,
                    Formatting = NewtonJson.Formatting.Indented
                });
            var newtonNewObj = NewtonJson.JsonConvert.DeserializeObject<TestSerializationObj>(newtonContent,
                new NewtonJson.JsonSerializerSettings { TypeNameHandling = NewtonJson.TypeNameHandling.Auto }); // works

            var sysContent = SysJson.JsonSerializer.Serialize(obj, new SysJson.JsonSerializerOptions { WriteIndented = true, });
            var sysNewObj = SysJson.JsonSerializer.Deserialize<TestSerializationObj>(sysContent); // throws exception
        }
    }


    public sealed class TestSerializationObj
    {
        public Dictionary<string, int> Dictionary { get; set; } = new Dictionary<string, int>() { ["a"] = 0, ["b"] = 1 };
        public ICustom CustomObj { get; set; }
    }

    public class DefaultCustom : ICustom
    {
        public string Name { get; set; } = "Default Implementation";
        public int Id { get; set; } = 300;
    }

    public interface ICustom
    {
        int Id { get; set; }
    }
}

Thanks!

Please try this library I wrote as an extension to System.Text.Json to offer polymorphism: https://github.com/dahomey-technologies/Dahomey.Json

What about letting the user provide their own Func<string, Type> and Func<Type, string> functions?

The design intentionally avoids this type of functionality, primarily since admitting arbitrary Func<string, Type> conversions would invariably expose STJ to the same class of security issues that are impacting BinaryFormatter.

Please make sure to support configuring the type discriminator property name. "$type" is very common but not universal.

Thanks for picking this up. I wrote my own one with a bad performance pretty much like this one but with the caveat of the lacking infrastructure.

While this change looks great for a ~~round-trip~~ full polymorphic solution, does it mean that there's no chance for a simple option on the serializer so that serialized output resolves polymorphic values when there's no need for additional type information?

I want to nit pick on "roundtrip": polymorphic deserialization is about receiving input which has a type attribute. Roundtrip or .net to .net is a tiny percentage of the demand.

@IFYates

@IFYates you can write your own converter factory to handle polymorphic serialization. It is not good enough for default implementation but works.

[AttributeUsage(AttributeTargets.Class, AllowMultiple = true)]
public class JsonKnownTypeAttribute : Attribute
{
    public Type NestedType { get; }
    public string Discriminator { get; }

    public JsonKnownTypeAttribute(Type nestedType, string discriminator)
    {
        NestedType = nestedType;
        Discriminator = discriminator;
    }
}

public class PolyJsonConverter : JsonConverterFactory
{
    private static readonly Type converterType = typeof(PolyJsonConverter<>);

    public override bool CanConvert(Type typeToConvert)
    {
        var attr = typeToConvert.GetCustomAttributes<JsonKnownTypeAttribute>(false);
        return attr.Any();
    }

    public override JsonConverter CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        var attr = typeToConvert.GetCustomAttributes<JsonKnownTypeAttribute>();
        var concreteConverterType = converterType.MakeGenericType(typeToConvert);
        return (JsonConverter)Activator.CreateInstance(concreteConverterType, attr);
    }
}

internal class PolyJsonConverter<T> : JsonConverter<T>
{
    private readonly Dictionary<string, Type> _discriminatorCache;
    private readonly Dictionary<Type, string> _typeCache;

    private static readonly JsonEncodedText TypeProperty = JsonEncodedText.Encode("$type");

    public PolyJsonConverter(IEnumerable<JsonKnownTypeAttribute> resolvers)
    {
        _discriminatorCache = resolvers.ToDictionary(p => p.Discriminator, p => p.NestedType);
        _typeCache = resolvers.ToDictionary(p => p.NestedType, p => p.Discriminator);
    }

    public override T Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        using var doc = JsonDocument.ParseValue(ref reader);
        if (!doc.RootElement.TryGetProperty(TypeProperty.EncodedUtf8Bytes, out var typeElement))
            throw new JsonException();

        var discriminator = typeElement.GetString();
        if (discriminator is null || !_discriminatorCache.TryGetValue(discriminator, out var type))
            throw new JsonException();


        return (T)doc.Deserialize(type, options);
    }

    public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
    {
        var type = value.GetType();
        if (!_typeCache.TryGetValue(type, out var discriminator))
            throw new JsonException();

        writer.WriteStartObject();
        writer.WritePropertyName(TypeProperty.EncodedUtf8Bytes);
        writer.WriteStringValue(discriminator);
        using var doc = JsonSerializer.SerializeToDocument(value, type, options);
        foreach (var prop in doc.RootElement.EnumerateObject())
            prop.WriteTo(writer);

        writer.WriteEndObject();
    }
}

Example of usage

var data = new A[] { new B { PropA = "A-B", PropB = "B-B" }, new C { PropA = "A-C", PropC = "C-C" } };
var options = new JsonSerializerOptions
{
    Converters = { new PolyJsonConverter() }
};
var json = JsonSerializer.Serialize(data, options);
// 
var l = JsonSerializer.Deserialize<A[]>(json, options);

[JsonKnownType(typeof(B), "TypeB")]
[JsonKnownType(typeof(C), "TypeC")]
abstract class A
{
    public string PropA { get; set; }
}

class B : A
{
    public string PropB { get; set; }
}

class C : A
{
    public string PropC { get; set; }
}

This code produces following json:

[
  {
    "$type": "TypeB",
    "PropB": "B-B",
    "PropA": "A-B"
  },
  {
    "$type": "TypeC",
    "PropC": "C-C",
    "PropA": "A-C"
  }
]

I would prefer an official solution rather than custom "workaround" as this might break in the future. If you are in control over both serialization and deserialization this could work. If you don't you risk that whatever you released into the world, will break your application sooner or later. I have the feeling many developers might agree with me. This "feature request" has been delayed over and over again. With 76 upvotes I really hope the .NET team finally gives it the attention it deserves.

Indeed. There are enough samples on the internet. We need a canon solution. Polymorphic deserialization is a basic feature.

Another topic: Using attribute based metadata is hopefully not the only way. Not everyone is in control of the target types. I understand the security implications of providing callback lookups or list based inputs which can be misused by a stack overflow sample but attributes only a not a way (see EF Core and friends)

Given that the scope of this feature has changed substantially since the issue was originally opened, I've created a new user story which should hopefully serve to better establish expectations wrt the brand of polymorphism we are planning to deliver. I'm going to close this one, feel free to continue the conversation in the new story if you have any feedback. Thanks!

Navigation Menu

Repository navigation

JsonSerializer polymorphic serialization and deserialization support #30083

51 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

JsonSerializer polymorphic serialization and deserialization support #30083

Description

Activity

Symbai commented on Jun 28, 2019

ahsonkhan commented on Jun 28, 2019

Timovzl commented on Sep 23, 2019

steveharter commented on Sep 24, 2019

steveharter commented on Sep 24, 2019

Timovzl commented on Sep 25, 2019

ANahr commented on Oct 21, 2019

lfr commented on Oct 24, 2019

layomia commented on Nov 23, 2019

layomia commented on Nov 23, 2019

mcatanzariti commented on Dec 4, 2019

51 remaining items

eiriktsarpalis commented on Aug 26, 2021

tthiery commented on Dec 4, 2021

IFYates commented on Dec 15, 2021

tthiery commented on Dec 15, 2021

Ilchert commented on Dec 30, 2021

Symbai commented on Dec 30, 2021

tthiery commented on Dec 30, 2021

eiriktsarpalis commented on Jan 13, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions