Repository navigation
[API Proposal]: Add support for nested data streaming in JsonSerializer.DeserializeAsyncEnumerable. #114265
Description
Background and motivation
Currently IAsyncEnumerable is only supported for top level (root) arrays. The problem is many APIs return json data structured with a wrapper object. For example: { "data": [/* Huge array here */] }. This kind of jsons eliminate the ability to use IAsyncEnumerable without manual parsing.
The expected support is parsing the array inside the data property as if it was a root array.
API Proposal
There are actually two cases for the API - one for simple cases and one for more control albeit more low level.
For the high level extend deserialize of IAsyncEnumerable to accept a root json path or even a property name.
JsonSerializer.DeserializeAsyncEnumerable(stream, "data");This should also apply to http client methods regarding IAsyncEnumerable.
For the low level - the idea is giving an easier json token parser that allows control on how to deserialize.
For example given the json { "data1": [/* Huge array here */], "data2": [/* Huge array here */]} and the requirement to create an IAsyncEnumerable from data1 concatenated with data2 the api could allow:
using var jsonStreamReader = new JsonStreamReader(stream, bufferSize /* Maybe? */);
await jsonStreamReader.ReadAsync(); // skip object start
await jsonStreamReader.ReadAsync(); // skip data1
var data1AsyncEnumerable = jsonStreamReader.DeserializeAsyncEnumerable<DataItem>(options);
await foreach (var item in data1AsyncEnumerable)
{
yield return item;
}
await jsonStreamReader.ReadAsync(); // skip data2
var data2AsyncEnumerable = jsonStreamReader.DeserializeAsyncEnumerable<DataItem>(options);
await foreach (var item in data2AsyncEnumerable)
{
yield return item;
}Here JsonStreamReader is the new API that allows much more control.
API Usage
Mentioned above in API proposal
Alternative Designs
Currently the only way is writing my own parser with low level APIs which isn't trivial at all
Risks
None (probably?)
Activity
dotnet-policy-service commentedon Apr 4, 2025
Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.
eiriktsarpalis commentedon Apr 4, 2025
It's unlikely we would add a "JsonStreamReader" just to enable this feature. The more likely outcome is introducing an overload to the existing
JsonSerializer.DeserializeAsyncEnumerablemethods that takes a JSON pointer/JSON path parameter pinpointing the nested array that requires streaming.[-][API Proposal]: Support non root IAsyncEnumerable parsing[/-][+][API Proposal]: Add support for nested data streaming in `JsonSerializer.DeserializeAsyncEnumerable`.[/+]Harpush commentedon Apr 4, 2025
That's fine too. The idea of the reader is to allow any combination of read tokens and deserialize. I would expect a regular deserialize (non IAsyncEnumerable) method too
brantburnett commentedon Apr 4, 2025
I have a nested array streaming use case in the Couchbase SDK which requires maintaining the other context. It's implemented using a custom wrapper around
StreamusingUtf8JsonReaderandJsonReaderState. Unfortunately, the example is complicated a bit because it's abstracted to also support Newtonsoft. I share it in case it gives insight into other use cases.https://github.com/couchbase/couchbase-net-client/blob/master/src/Couchbase/Query/StreamingQueryResult.cs
https://github.com/couchbase/couchbase-net-client/blob/master/src/Couchbase/Core/IO/Serializers/SystemTextJson/SystemTextJsonStreamReader.cs
Harpush commentedon Apr 5, 2025
Just to add some more examples to why
JsonStreamReaderis a good idea:{metadata: {...}, data: [/*huge json*/]}):IAsyncEnumerableof dictionary ({a: {...}, b: {...}, c: {...}}) withKeyValuePair: