JsonSerializer: Allow out-of-order reading of metadata properties.

EDIT See #72604 (comment) for an API proposal.

Description

Attempting to deserialize a polymorphic structure with System.Text.Json that doesn't feature $type at start of the object results in an exception.

For my scenario, this currently prevents me from using the built-in polymorphism support with jsonb columns in Postgres, as object properties have no guaranteed order.

Developer comment on this issue: #63747 (comment)

Reproduction Steps

var databaseState = @"{
  ""BaseDictionary"": {
    ""6ed6e524-2ca4-4fea-8e21-7245ccb61863"": {
      ""Id"": ""6ed6e524-2ca4-4fea-8e21-7245ccb61863"",
      ""Name"": ""Something"",
      ""$type"": ""Derived"",
      ""OtherGuid"": ""d471c77d-5412-4e7a-a98d-8304e87792ed""
    }
  }
}";

JsonSerializer.Deserialize<WrapperType>(databaseState);

public record WrapperType(Dictionary<Guid, WrapperType.Base> BaseDictionary)
{
    [JsonDerivedType(typeof(Derived), nameof(Derived))]
    [JsonDerivedType(typeof(AlsoDerived), nameof(AlsoDerived))]
    public abstract record Base(Guid Id, string Name);
    public record Derived(Guid Id, string Name, Guid OtherGuid): Base(Id, Name);
    public record AlsoDerived(Guid Id, string Name) : Base(Id, Name);
}

Expected behavior

Deserialization should work.

Actual behavior

System.NotSupportedException: 'Deserialization of types without a parameterless constructor, a singular parameterized constructor, or a parameterized constructor annotated with 'JsonConstructorAttribute' is not supported.

Regression?

Limitation of initial implementation

Known Workarounds

I currently use PolyJson as an alternative, as the implementation reads ahead to find the discriminator.

Configuration

Impacts any version of STJ 7.0

Other information

No response

Per #63747 (comment) this is a known limitation of the metadata reader. In the future we might consider adding read-ahead support specifically for type discriminators (I don't believe reference preservation to have such requirements), but this would be at the expense of potentially buffering excessive data in the case of async serialization.

Is there any known workaround for this? I want to deserialize an object that does not come from System.Text.Json but has a discriminator that can be used that is of course not guaranteed to be first. The only thing I managed so far is to just deserialized first that one field and then specify type explicitly (i.e. not using the new polymorphic deserialization at all).

Maybe there is something else that can be done, like e.g. explicit reordering or something like that?

No workaround that doesn't involve writing a custom converter, unfortunately. In the future we might consider exposing a flag that reads ahead for type discriminators, but obviously that would come at the expense of performance so it would be turned off by default.

Thanks for the answer! I assume the "future" will not be in .Net 7, so not coming anytime soon, right?

Correct, .NET 7 is currently in RC so feature development has been concluded.

Why can't I use my own property for the type discriminator?

My object has a string Type property already. According to this documentation
it says:

Avoid a JsonPolymorphicAttribute.TypeDiscriminatorPropertyName if it conflicts with a property in your type hierarchy.

I would like to tell System.Text.Json that there is a property (the first one) named Type, just use it.

[JsonPolymorphic(TypeDiscriminatorPropertyName = "type")]
    [JsonDerivedType(typeof(ChatbotSettings), typeDiscriminator: ChatbotSettings.TypeName)]
    [JsonDerivedType(typeof(UnattendedSettings), typeDiscriminator: UnattendedSettings.TypeName)]
    [JsonDerivedType(typeof(AttendedSettings), typeDiscriminator: AttendedSettings.TypeName)]
    internal abstract class RobotSettings
    {
        /// <summary>
        /// Do not use. This is for Json serialization.
        /// </summary>
        internal RobotSettings() { }

        internal RobotSettings(string type) => Type = type;

        internal required string Type { get; init; }
    }

Why can't I use my own property for the type discriminator?

The value of a string property can be arbitrary and might not necessarily reflect the runtime type of the object you are trying to serialize. In your example I would simply annotate the Type property with JsonIgnoreAttribute.

I have this issue as well. When serializing JSON to MySQL using ef core, it seems that the ordering is not preserved so the data cannot be deserialized anymore.

There is a bit of a hack to work around this since MySQL seems to do deterministic ordering:

[JsonPolymorphic(TypeDiscriminatorPropertyName = "!")]

You might be able to get away with " " (space) as well, but basically a very low value ascii character that's valid will cause it to be sorted first. Definitely would not recommend doing this in production, but I am using this for myself.

With .NET 7 I have a solution that should be at least fast.

I have a custom converter (in this case it is abstract because I have different ways to resolve type and discriminator) to handle writing and reading:

public abstract class InheritanceConverterBase<T> : JsonConverter<T>, IInheritanceConverter where T : notnull
{
    private readonly JsonEncodedText discriminatorProperty;

    public string DiscriminatorName { get; }

    protected InheritanceConverterBase(string discriminatorName)
    {
        discriminatorProperty = JsonEncodedText.Encode(discriminatorName);
        DiscriminatorName = discriminatorName;
    }

    public abstract Type GetDiscriminatorType(string name, Type typeToConvert);

    public abstract string GetDiscriminatorValue(Type type);

    public override T? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        // Creating a copy of the reader (The derived deserialisation has to be done from the start)
        Utf8JsonReader typeReader = reader;

        if (typeReader.TokenType != JsonTokenType.StartObject)
        {
            throw new JsonException();
        }

        while (typeReader.Read())
        {
            if (typeReader.TokenType == JsonTokenType.PropertyName &&
                typeReader.ValueTextEquals(discriminatorProperty.EncodedUtf8Bytes))
            {
                // Advance the reader to the property value
                typeReader.Read();

                if (typeReader.TokenType != JsonTokenType.String)
                {
                    ThrowHelper.JsonException($"Expected string discriminator value, got '{reader.TokenType}'");
                    return default!;
                }

                // Resolve the type from the discriminator value.
                var type = GetDiscriminatorType(typeReader.GetString()!, typeToConvert);

                // Perform the actual deserialization with the original reader
                return (T)JsonSerializer.Deserialize(ref reader, type, options)!;
            }
            else if (typeReader.TokenType == JsonTokenType.StartObject || typeReader.TokenType == JsonTokenType.StartArray)
            {
                if (!typeReader.TrySkip())
                {
                    typeReader.Skip();
                }
            }
        }

        ThrowHelper.JsonException($"Object has no discriminator '{DiscriminatorName}.");
        return default!;
    }

    public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
    {
        JsonSerializer.Serialize<object>(writer, value!, options);
    }
}

public interface IInheritanceConverter
{
    string DiscriminatorName { get; }

    Type GetDiscriminatorType(string name, Type typeToConvert);

    string GetDiscriminatorValue(Type type);
}

The problem was always the writing part, because you have to buffer the resulting object:

public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
{
	var name = GetDiscriminatorValue(value.GetType());

	writer.WriteStartObject();
	writer.WriteString(discriminatorProperty, name);

	using (var document = JsonSerializer.SerializeToDocument(value, value.GetType(), options))
	{
		foreach (var property in document.RootElement.EnumerateObject())
		{
			property.WriteTo(writer);
		}
	}

	writer.WriteEndObject();
}

So you effectively write the object, then you deserialize it and then you serialize it again. So my idea with .NET 7 was to add a custom property to the contract with a custom resolver. You cannot use modifiers, because you need the type info of the base class.

public sealed class PolymorphicTypeResolver : DefaultJsonTypeInfoResolver
{
	public override JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options)
	{
		var typeInfo = base.GetTypeInfo(type, options);

		var baseType = type.BaseType;

		if (baseType == null || baseType == typeof(object))
		{
			return typeInfo;
		}

		var baseInfo = GetTypeInfo(baseType, options);

		if (baseInfo.Converter is IInheritanceConverter inheritanceConverter)
		{
			var discriminatorField = typeInfo.CreateJsonPropertyInfo(typeof(string), inheritanceConverter.DiscriminatorName);
			var discriminatorValue = inheritanceConverter.GetDiscriminatorValue(type);

			discriminatorField.Get = x =>
			{
				return discriminatorValue;
			};

			typeInfo.Properties.Insert(0, discriminatorField);
		}

		return typeInfo;
	}
}

I don't like that you need the type resolver and the converter right now, but I do not see another option.

Yes, the metadata reader checks for a number of properties that need to be validated (invalid combinations, duplicates, etc.) so it can't just stop on the first occurrence of $type.

Candidate implementation: #97474

Video

Looks good as proposed. The default may need to change from false to true in the future; but we'll let that be based on data/feedback.

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        public bool AllowOutOfOrderMetadataProperties { get; set; } = false;
    }
}

namespace System.Text.Json.Serialization
{
    public partial class JsonSourceGenerationOptionsAttribute
    {
        public bool AllowOutOfOrderMetadataProperties { get; set; } = false;
    }
}

The new flag should become available with .NET 9 Preview 2.

I know there were a few workarounds posted already, but none of them supported my use case. I don't want to use source generation, and I need it to support nested polymorphic objects as well as always including the discriminator when serializing subclasses (even if we're not casting them to the base class). So, this is what I came up with (by modifying things I saw in this thread and others), hopefully it helps anyone else who needs this behaviour before .NET 9!

Custom attribute and converter

/// <summary>
/// Same as <see cref="JsonDerivedTypeAttribute"/> but used for the hack below. Necessary because using the built-in
/// attribute will lead to NotSupportedExceptions.
/// </summary>
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Interface, AllowMultiple = true, Inherited = false)]
public class HackyJsonDerivedAttribute(Type subtype, string discriminator) : Attribute
{
    public Type Subtype { get; set; } = subtype;
    public string Discriminator { get; set; } = discriminator;
}

public sealed class PolymorphicJsonConverterFactory : JsonConverterFactory
{
    public override bool CanConvert(Type typeToConvert)
    {
        return typeToConvert.IsAbstract && typeToConvert.GetCustomAttributes<HackyJsonDerivedAttribute>().Any();
    }

    public override JsonConverter? CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        return (JsonConverter?) Activator.CreateInstance(
            typeof(PolymorphicJsonConverter<>).MakeGenericType(typeToConvert), options);
    }
}

/// <summary>
/// A temporary hack to support deserializing JSON payloads that use polymorphism but don't specify $type as the first field.
/// Modified from https://github.com/dotnet/runtime/issues/72604#issuecomment-1440708052.
/// </summary>
public sealed class PolymorphicJsonConverter<T> : JsonConverter<T>
{
    private readonly string _discriminatorPropName;
    private readonly Dictionary<string, Type> _discriminatorToSubtype = [];

    public PolymorphicJsonConverter(JsonSerializerOptions options)
    {
        _discriminatorPropName = options.PropertyNamingPolicy?.ConvertName("$type") ?? "$type";
        foreach (var subtype in typeof(T).GetCustomAttributes<HackyJsonDerivedAttribute>())
        {
            _discriminatorToSubtype.Add(subtype.Discriminator, subtype.Subtype);
        }
    }

    public override bool CanConvert(Type typeToConvert) => typeof(T) == typeToConvert;

    public override T Read(
        ref Utf8JsonReader reader, Type objectType, JsonSerializerOptions options)
    {
        var reader2 = reader;
        using var doc = JsonDocument.ParseValue(ref reader2);

        var root = doc.RootElement;
        var typeField = root.GetProperty(_discriminatorPropName);

        if (typeField.GetString() is not { } typeName)
        {
            throw new JsonException(
                $"Could not find string property {_discriminatorPropName} " +
                $"when trying to deserialize {typeof(T).Name}");
        }

        if (!_discriminatorToSubtype.TryGetValue(typeName, out var type))
        {
            throw new JsonException($"Unknown type: {typeName}");
        }

        return (T) JsonSerializer.Deserialize(ref reader, type, options)!;
    }

    public override void Write(
        Utf8JsonWriter writer, T? value, JsonSerializerOptions options)
    {
        var type = value!.GetType();
        JsonSerializer.Serialize(writer, value, type, options);
    }
}

JsonTypeInfo modifier to always include discriminator when writing JSON

options.TypeInfoResolver = new DefaultJsonTypeInfoResolver
{
    Modifiers =
    {
        static typeInfo =>
        {
            var propertyNamingPolicy = typeInfo.Options.PropertyNamingPolicy;

            // Temporary hack to ensure subclasses of abstract classes will always include the $type field
            if (typeInfo.Type.BaseType is {IsAbstract: true} &&
                typeInfo.Type.BaseType.GetCustomAttributes<HackyJsonDerivedAttribute>().Any())
            {
                var discriminatorPropertyName = propertyNamingPolicy?.ConvertName("$type") ?? "$type";
                if (!typeInfo.Properties.Any(p => p.Name == discriminatorPropertyName))
                {
                    var discriminatorValue = typeInfo.Type.BaseType
                        .GetCustomAttributes<HackyJsonDerivedAttribute>()
                        .First(attr => attr.Subtype == typeInfo.Type).Discriminator;
                    var propInfo = typeInfo.CreateJsonPropertyInfo(typeof(string), discriminatorPropertyName);
                    propInfo.Get = _ => discriminatorValue;
                    typeInfo.Properties.Insert(0, propInfo);
                }
            }
        },
    },
};

Navigation Menu

Repository navigation

JsonSerializer: Allow out-of-order reading of metadata properties. #72604

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

53 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

JsonSerializer: Allow out-of-order reading of metadata properties. #72604

Description

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

Activity

eiriktsarpalis commented on Jul 21, 2022

ilya-scale commented on Oct 3, 2022

eiriktsarpalis commented on Oct 3, 2022

ilya-scale commented on Oct 3, 2022

eiriktsarpalis commented on Oct 3, 2022

JobaDiniz commented on Oct 15, 2022

eiriktsarpalis commented on Oct 17, 2022

kOchirasu commented on Nov 8, 2022

SebastianStehle commented on Nov 9, 2022

53 remaining items

eiriktsarpalis commented on Jan 22, 2024

eiriktsarpalis commented on Jan 24, 2024

bartonjs commented on Jan 30, 2024

eiriktsarpalis commented on Feb 2, 2024

tyler-boyd commented on Feb 7, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions