System.Text.Json failing some large file tests

https://dev.azure.com/dnceng/public/_build/results?buildId=1387876&view=ms.vss-test-web.build-test-results-tab&runId=40401306&resultId=194463&paneView=debug

If I didn't know any better, I'd say "memory corruption".

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-59626-merge-ff748885001649f2ad/System.Text.Json.Tests/1/console.55b96e63.log?sv=2019-07-07&se=2021-10-17T15%3A49%3A24Z&sr=c&sp=rl&sig=jkQ9vlfitTdm9LAJkyCb4ZnZxrI5Gb%2F8qvt%2FSn0S27M%3D

Console log: 'System.Text.Json.Tests' from job ff748885-0016-49f2-ad26-20905cd95f3d workitem 0045d34b-f5a3-424a-8789-b58e57023e81 (osx.1015.amd64.open) executed on machine dci-mac-build-195
+ ./RunTests.sh --runtime-path /tmp/helix/working/A15608F4/p
----- start Mon Sep 27 11:49:52 EDT 2021 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/A15608F4/p/dotnet exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/A15608F4/w/A1BD08B8/e /private/tmp/helix/working/A15608F4/w/A1BD08B8/e
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 2776 of 2815 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 12)
    System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(formatted: False, skipValidation: False) [FAIL]
      Assert.Equal() Failure
                                          â†“ (pos 1102164)
      Expected: Â·Â·Â·qKioqKioqKioqKioqKiosKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqÂ·Â·Â·
      Actual:   Â·Â·Â·qKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqÂ·Â·Â·
                                          â†‘ (pos 1102164)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(738,0): at System.Text.Json.JsonTestHelper.AssertContentsAgainstJsonNet(String expectedValue, String value, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(714,0): at System.Text.Json.JsonTestHelper.AssertContents(String expectedValue, ArrayBufferWriter`1 buffer, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Utf8JsonWriterTests.cs(3202,0): at System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(Boolean formatted, Boolean skipValidation)
    System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(formatted: False, skipValidation: False, keyString: ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>") [FAIL]
      Assert.Equal() Failure
                                          â†“ (pos 484474)
      Expected: Â·Â·Â·e\u003e\u003e\u003e\w003e\u003e\u003e\u003e\u003e\u003e\u003eÂ·Â·Â·
      Actual:   Â·Â·Â·e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003eÂ·Â·Â·
                                          â†‘ (pos 484474)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(738,0): at System.Text.Json.JsonTestHelper.AssertContentsAgainstJsonNet(String expectedValue, String value, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(714,0): at System.Text.Json.JsonTestHelper.AssertContents(String expectedValue, ArrayBufferWriter`1 buffer, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Utf8JsonWriterTests.cs(5617,0): at System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(Boolean formatted, Boolean skipValidation, String keyString)
    System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync [FAIL]
      Assert.Equal() Failure
                                          â†“ (pos 12025)
      Expected: Â·Â·Â·eeeeeeeeeeeeeeeeeeeegeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeÂ·Â·Â·
      Actual:   Â·Â·Â·eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeÂ·Â·Â·
                                          â†‘ (pos 12025)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(61,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(20,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(String expected, String actual)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(102,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TElement](Stream memoryStream, String expectedJson, Type type, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(71,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TElement](Object obj, Type type, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(53,0): at System.Text.Json.Serialization.Tests.StreamTests.RunTestAsync[TElement]()
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(25,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync()
        --- End of stack trace from previous location ---
  Finished:    System.Text.Json.Tests
=== TEST EXECUTION SUMMARY ===
   System.Text.Json.Tests  Total: 20247, Errors: 0, Failed: 3, Skipped: 0, Time: 58.695s

System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync
System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(formatted: False, skipValidation: False)
System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(formatted: False, skipValidation: False, keyString: ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>")

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers",
  "BuildRetry": false
}

Report

Build	Definition	Test	Pull Request
646735	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100141
646752	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101172
646629	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100685
646704	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101117
646679	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101178
646615	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100094
646642	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100824
646471	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101171
646414	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101168
646500	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101140
646478	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101133
646303	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101106
646425	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101053
646408	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100951
646378	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101122
646397	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
646351	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101162
646235	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101161
646220	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101158
646197	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101037
646202	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101156
646116	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101141
645817	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100966
646046	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101137
645902	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101143
645931	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100946
645895	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100697
645856	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101141
645850	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101140
645791	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101138
645825	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101037
645838	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
645804	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101139
645757	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101133
645807	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#97041
645780	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101137
645795	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100951
645684	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101120
645607	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101087
645580	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101095
645577	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101129
645573	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100946
645473	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101122
645505	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100966
645489	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#101123
645396	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100141
645217	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101106
645199	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101110
645086	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100954
645056	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100949
645041	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100951
645012	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution
644951	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#94970
644913	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101101
644773	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#98838
644550	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101017
644343	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100898
644301	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101078
644318	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution
644104	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#101017
644152	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100587
644038	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100094
643951	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100765
642449	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution
642419	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100664
642197	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100765
642193	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101009
642160	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101008
642028	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution
641917	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#101002
641896	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100821
641548	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
641471	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100664
641482	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100938
641521	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution
641439	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100989
641128	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
641052	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100266
640998	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100890
640958	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100765
640906	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100520
640860	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100954
640857	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100954
640844	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100951
640748	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#99199
640730	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100729
640726	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100669
640634	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100743
640505	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
640242	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100812
639773	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100890
639582	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100890
639339	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100890
639263	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100907
638748	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100497
638373	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#100890
638196	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100881
638143	dotnet/runtime	WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution	#100877
637953	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#100812
633731	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#99278
Displaying 100 of 132 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
45	99	132

Known issue validation

Build: 🔎
Result validation: ⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.

Interesting observation: In all three failures the expected character vs the actual char had a difference of 2. (q => s, u => w, e => g).

Maybe it really is some sort of memory corruption (something has a bad pointer somewhere, and added 2).

Secondary observation: They all feel like they have expected and actual backwards 😄.

Seems related to #45464

The runtime-extra-platforms rolling build also hit the failure in Writing3MBBase64Bytes (on Windows Server 2022): https://dev.azure.com/dnceng/public/_build/results?buildId=1612202&view=ms.vss-test-web.build-test-results-tab&runId=44896660&resultId=189854&paneView=debug

Not sure if this is exactly related, but build https://dev.azure.com/dnceng/public/_build/results?buildId=1697303&view=logs&j=05e92ac1-194e-59cf-664a-fa72d1cdd19b&t=caea1d4b-c90c-5be1-e57d-c4635079c333 failed with HandleCollectionsAsync test failing in a very similar way. Here are the logs: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-50e485d6cc7b461f9a/System.Text.Json.Tests/1/console.6fad303f.log?sv=2019-07-07&se=2022-04-23T08%3A48%3A09Z&sr=c&sp=rl&sig=SNdOcc025qirLoyBCvPN7U%2BHl1uIgU94Zo64yUp0CLk%3D

@bartonjs

In @bartonjs case, the second least significant bit is flipping 0->1 in every case. If it was a bad pointer I think we'd expect 1->0 also, but it's a sample of three.

In @joperezr case (again as @bartonjs points out, Expected and Actual are reversed - cc @eiriktsarpalis ) it is quite different, several bits flipped, and it's then repeated.

      Assert.Equal() Failure
                                       ↓ (pos 13390)
      Expected: ···xxxxxxxxxxxxxxxxxxxxooooooooooooooooooooooooooooooooooooooooo···
      Actual:   ···xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx···
                                       ↑ (pos 13390)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(61,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(55,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(20,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(String expected, String actual)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(87,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TCollection,TElement](String json, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(50,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TCollection,TElement](TCollection collection, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(30,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync[TCollection,TElement](TCollection collection, Int32 bufferSize, Witness`1 elementType)
        --- End of stack trace from previous location ---
    System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync<Collection<StreamTests+ImmutableStructWithStrings>, StreamTests+ImmutableStructWithStrings>(collection: [System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, ...], bufferSize: 16384, elementType: Witness`1 { }) [FAIL]
      Assert.Equal() Failure
                                       ↓ (pos 8986)
      Expected: ···ppppppppppppppppppppwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww···
      Actual:   ···ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp···
                                       ↑ (pos 8986)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(61,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)

It's important to record machine names in such cases. Last one above was dci-mac-build-365.local. Looks like we lost the other two? We have in the past send a box to the trash because it was regularly flipping 1 bit.

@bartonjs

Oh, @bartonjs case was dci-mac-build-195.

Might be related to this other issue? #65021

In the same CI job that Joe shared, there's another log file with a slightly different callstack, in case it's helpful:

Pipeline: Libraries Test Run release coreclr OSX x64 Release
Test method:

System.Text.Json.Serialization.Tests.StreamTests_Async.HandleCollectionsAsync
<ConcurrentStack<StreamTests+ImmutableStructWithStrings>, StreamTests+ImmutableStructWithStrings>
(collection: [System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings,
System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings,
System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings,
System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings,
System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, ...],
bufferSize: 16384, elementType: Witness`1 { })
[FAIL]

Callstack:

      System.Text.Json.JsonException : '0x00' is invalid within a JSON string. The string should be correctly escaped. Path: $[14].MySecondString | LineNumber: 0 | BytePositionInLine: 439315.
      ---- System.Text.Json.JsonReaderException : '0x00' is invalid within a JSON string. The string should be correctly escaped. LineNumber: 0 | BytePositionInLine: 439315.
      Stack Trace:
        /_/src/libraries/System.Text.Json/src/System/Text/Json/ThrowHelper.Serialization.cs(265,0): at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.ReadCore.cs(79,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Helpers.cs(43,0): at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.String.cs(324,0): at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.String.cs(48,0): at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(75,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TCollection,TElement](String json, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(50,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TCollection,TElement](TCollection collection, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(30,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync[TCollection,TElement](TCollection collection, Int32 bufferSize, Witness`1 elementType)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----
        /_/src/libraries/System.Text.Json/src/System/Text/Json/ThrowHelper.cs(279,0): at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1365,0): at System.Text.Json.Utf8JsonReader.ConsumeStringAndValidate(ReadOnlySpan`1 data, Int32 idx)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1271,0): at System.Text.Json.Utf8JsonReader.ConsumeString()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1032,0): at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(800,0): at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(278,0): at System.Text.Json.Utf8JsonReader.Read()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Object/ObjectWithParameterizedConstructorConverter.cs(40,0): at System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs(221,0): at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Collection/JsonCollectionConverter.cs(116,0): at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs(246,0): at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.ReadCore.cs(62,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)

Even if this has morphed from the original issue, we should take a look at this during 9.0.0 since it causes CI failures.

I think there are other better checks that are more specific that are catching the things being caught there now like #100111 and #98578 so I'm closing this

@lewing

@lewing / @bartonjs this is our top reported known issue right now, I think that the search text is too board.

System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers

To Larry's point above, is there a more specific search text for the failure that this issue is tracking?

Yeah I'm closing it again and I'll make mroe specific tests if any aren't caught by the others

the problem is there are failures of this specific test and the failures look like:

/datadisks/disk1/work/9B2B0894/w/B43409F0/e /datadisks/disk1/work/9B2B0894/w/B43409F0/e
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 7405 of 7466 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on [2 threads], stop on fail = off)
    System.Text.Json.Tests.Utf8JsonReaderTests.TestDepth(depth: 256) [SKIP]
      Takes very long to run on interpreter.
    System.Text.Json.Tests.Utf8JsonReaderTests.TestDepth(depth: 512) [SKIP]
      Takes very long to run on interpreter.
   System.Text.Json.Tests: [Long Running Test] 'System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes', Elapsed: 00:05:42
   System.Text.Json.Tests: [Long Running Test] 'System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers', Elapsed: 00:02:01
   System.Text.Json.Tests: [Long Running Test] 'System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers', Elapsed: 00:04:01
Killed
['System.Text.Json.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-101117-merge-608176b4786749e2b4/System.Text.Json.Tests/1/console.6c31de45.log?helixlogtype=result
which isn't easy to distinguish with build analysis

we should probably just skip the test on the interpreter

opened #101193

It is looking increasingly like the ooms were indeed only part of the problem and now we're hitting a bunch of different ways to timeout in the interpreter on desktop and wasm that this catches

Navigation Menu

Repository navigation

System.Text.Json failing some large file tests #59678

Error Message

Report

Summary

Known issue validation

28 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

System.Text.Json failing some large file tests #59678

Description

Error Message

Report

Summary

Known issue validation

Activity

bartonjs commented on Sep 27, 2021

eiriktsarpalis commented on Sep 28, 2021

elinor-fung commented on Feb 15, 2022

joperezr commented on Apr 4, 2022

danmoseley commented on Apr 4, 2022

danmoseley commented on Apr 4, 2022

carlossanlop commented on Apr 4, 2022

carlossanlop commented on Apr 4, 2022

28 remaining items

jeffhandley commented on Nov 17, 2023

lewing commented on Apr 13, 2024

jeffschwMSFT commented on Apr 17, 2024

lewing commented on Apr 17, 2024

lewing commented on Apr 17, 2024

lewing commented on Apr 17, 2024

lewing commented on Apr 17, 2024

lewing commented on Apr 17, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions