[wasm] AOT: System.Globalization.Calendars.Hybrid.WASM.Tests and System.Globalization.Hybrid.WASM.Tests getting terminated

@kouvel

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=456551
Build error leg or test failing: normal-System.Globalization.Calendars.Hybrid.WASM.Tests.WorkItemExecution
Pull request: #94217

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "MONO.*src/mono/mono/metadata/icall.c:618. <disabled>",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

This is failing on rolling builds too, and is unrelated to the linked PR. Changes since the last passing rolling build - 8d59c90...920cd6e which includes #87672 .

The assertion is

runtime/src/mono/mono/metadata/icall.c

Lines 6176 to 6180 in ba65769

    
           void 
        
           ves_icall_System_Environment_FailFast (MonoStringHandle message, MonoExceptionHandle exception, MonoStringHandle errorSource, MonoError *error) 
        
           { 
        
           	if (MONO_HANDLE_IS_NULL (errorSource)) { 
        
           		g_warning ("Process terminated.");

cc @kouvel

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=456551
Error message validated: MONO.*src/mono/mono/metadata/icall.c:618. <disabled>
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 10/31/2023 4:57:48 PM UTC

Report

Build	Definition	Test	Pull Request
504118	dotnet/runtime	WasmTestOnBrowser-Invariant.Tests.WorkItemExecution	#95910
501644	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#95292
501422	dotnet/runtime	Workloads-NoWebcil-Wasm.Build.Tests.IcuTests.WorkItemExecution	#95610
488625	dotnet/runtime	Workloads-Wasm.Build.Tests.IcuTests.WorkItemExecution	#95610
487764	dotnet/runtime	normal-Microsoft.Extensions.FileProviders.Physical.Tests.WorkItemExecution	#95324
486016	dotnet/runtime	normal-System.ComponentModel.Tests.WorkItemExecution	#95324

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	3	6

This might be a related failure too - #87672 (comment) .

@pavelsavara

cc @pavelsavara @maraf

I can't reproduce the same error. When we run wasm-library-aot-tests we add /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=true. When reproducing this gives us an error AOT is not supported without IL trimming (PublishTrimmed=true required), so I added it. Am I missing something else? Log of such a run differs a lot:

/workspaces/runtime/src/mono/wasm/build/WasmApp.Native.targets(689,5): error : Precompiling failed for /workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/wasm_build/obj/wasm/for-build/aot-in/Microsoft.CSharp.dll with exit code 1. [/workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/publish/ProxyProjectForAOTOnHelix.proj]
  /workspaces/runtime/src/mono/wasm/build/WasmApp.Native.targets(689,5): error : Failed to load method 0x600005d from '/workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/wasm_build/obj/wasm/for-build/aot-in/Microsoft.CSharp.dll' due to Could not resolve type with token 01000015 from typeref (expected class 'System.Collections.Generic.IEqualityComparer`1' in assembly 'System.Runtime, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a') assembly:System.Runtime, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a type:System.Collections.Generic.IEqualityComparer`1 member:(null). [/workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/publish/ProxyProjectForAOTOnHelix.proj]
  /workspaces/runtime/src/mono/wasm/build/WasmApp.Native.targets(689,5): error : Run with MONO_LOG_LEVEL=debug for more information. [/workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/publish/ProxyProjectForAOTOnHelix.proj]
....
 /workspaces/runtime/src/mono/wasm/build/WasmApp.Native.targets(689,5): error : Could not load signature of Microsoft.DotNet.XHarness.TestRunners.Common.TestRunner:WriteResultsToFile due to: Could not resolve type with token 01000019 from typeref (expected class 'System.IO.TextWriter' in assembly 'System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e') assembly:System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e type:System.IO.TextWriter member:(null) [/workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/publish/ProxyProjectForAOTOnHelix.proj]
  /workspaces/runtime/src/mono/wasm/build/WasmApp.Native.targets(689,5): error : Can't find custom attr constructor image: /workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/wasm_build/obj/wasm/for-build/aot-in/Microsoft.DotNet.XHarness.TestRunners.Common.dll mtoken: 0x0a00000a due to: Could not resolve type with token 01000011 from typeref (expected class 'System.Runtime.CompilerServices.AsyncStateMachineAttribute' in assembly 'System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e') assembly:System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e type:System.Runtime.CompilerServices.AsyncStateMachineAttribute member:(null) [/workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/publish/ProxyProjectForAOTOnHelix.proj]
  /workspaces/runtime/src/mono/wasm/build/WasmApp.Native.targets(689,5): error : Failed to load custom attributes from method System.Threading.Tasks.Task`1<System.Collections.Generic.List`1<string>> Microsoft.DotNet.XHarness.TestRunners.Common.ApplicationEntryPoint:GetIgnoredCategories () due to Could not resolve type with token 01000011 from typeref (expected class 'System.Runtime.CompilerServices.AsyncStateMachineAttribute' in assembly 'System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e') assembly:System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e type:System.Runtime.CompilerServices.AsyncStateMachineAttribute member:(null) [/workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/publish/ProxyProjectForAOTOnHelix.proj]
  /workspaces/runtime/src/mono/wasm/build/WasmApp.Native.targets(689,5): error : FullAOT cannot continue if there are loader errors. [/workspaces/runtime/artifacts/bin/System.Globalization.Calendars.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/publish/ProxyProjectForAOTOnHelix.proj]

Can you share how exactly you are running the tests?

/p:BuildAOTTestsOnHelix=true - this isn't required. Use it if you are not able to reproduce without it. Essentially, it uses the same proxy project that we use on helix, for your local run too.

Running this on macos with v8 11.6.6, I get:

  info: console.info: Initializing dotnet version 9.0.0-dev commit hash e4fbdb907bb187d7b5ba0668a84347c1058e3219
  info: [MONO] Process terminated.
  info: [MONO] Encountered infinite recursion while looking up resource 'Arg_NullReferenceException' in System.Private.CoreLib. Verify the installation of .NET is complete and does not need repairing, and that the state of the process has not become corrupted.
  info: Process v8 exited with 1

.. with a fresh build of e4fbdb9 .

Building with /p:UseSystemResourceKeys=true, I get:

  info: console.info: Initializing dotnet version 9.0.0-dev commit hash e4fbdb907bb187d7b5ba0668a84347c1058e3219
  info: MONO_WASM:    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[<RunAsync>d__4](<RunAsync>d__4& )
  info:    at Microsoft.DotNet.XHarness.TestRunners.Xunit.WasmApplicationEntryPoint.Run()
  info:    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[<RunAsync>d__4](<RunAsync>d__4& )
  info:    at SimpleWasmTestRunner.Main(String[] args)
  info: Error: TypeInitialization_Type, Microsoft.DotNet.XHarness.TestRunners.Common.ApplicationOptions
  info:     at Qn (/Users/ankj/dev/runtime/artifacts/bin/System.Globalization.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/_framework/dotnet.runtime.js:3:32681)
  info:     at Zn (/Users/ankj/dev/runtime/artifacts/bin/System.Globalization.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/_framework/dotnet.runtime.js:3:31798)
  info:     at p.javaScriptExports.call_entry_point (/Users/ankj/dev/runtime/artifacts/bin/System.Globalization.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/_framework/dotnet.runtime.js:3:210195)
  info:     at Object.ol [as runMain] (/Users/ankj/dev/runtime/artifacts/bin/System.Globalization.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/_framework/dotnet.runtime.js:3:141624)
  info:     at run (/Users/ankj/dev/runtime/artifacts/bin/System.Globalization.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/test-main.js:410:50)
  info:     at async /Users/ankj/dev/runtime/artifacts/bin/System.Globalization.Hybrid.WASM.Tests/Release/net9.0-browser/browser-wasm/AppBundle/test-main.js:426:1

The issue seems to be that CultureInfo..cctor indirectly calls CompareInfo..cctor, which then tries to access a static field from CultureInfo (s_InvariantCultureInfo), but that is null because it's cctor hasn't completed yet!

CultureInfo..cctor
    -> CultureData..cctor
    -> interp_runtime_invoke: >> method: System.Globalization.CalendarData:.cctor ()
    -> interp_runtime_invoke: >> method: System.Globalization.GlobalizationMode/Settings:.cctor ()
       -> Environment..cctor
       -> Marshal..cctor
       -> CompareInfo..cctor
            - tries to use CultureInfo.s_invariant .. which hasn't been initialized yet!

This patch works around the issue causing the tests to pass:

diff --git a/src/libraries/System.Private.CoreLib/src/System/Globalization/CultureInfo.cs b/src/libraries/System.Private.CoreLib/src/System/Globalization/CultureInfo.cs
index 31b17f3c4f5..c87c290a3fa 100644
--- a/src/libraries/System.Private.CoreLib/src/System/Globalization/CultureInfo.cs
+++ b/src/libraries/System.Private.CoreLib/src/System/Globalization/CultureInfo.cs
@@ -102,7 +102,7 @@ public partial class CultureInfo : IFormatProvider, ICloneable
         private static volatile CultureInfo? s_userDefaultUICulture;

         // The Invariant culture;
-        private static readonly CultureInfo s_InvariantCultureInfo = new CultureInfo(CultureData.Invariant, isReadOnly: true);
+        private static CultureInfo? s_InvariantCultureInfo;

         // These are defaults that we use if a thread has not opted into having an explicit culture
         private static volatile CultureInfo? s_DefaultThreadCurrentUICulture;
@@ -462,7 +462,7 @@ public static CultureInfo InvariantCulture
         {
             get
             {
-                Debug.Assert(s_InvariantCultureInfo != null);
+                s_InvariantCultureInfo ??= new CultureInfo(CultureData.Invariant, isReadOnly: true);
                 return s_InvariantCultureInfo;
             }
         }

@vargaz

Cc @vargaz

I can reproduce.

@vargaz

@vargaz are you looking into this one?

What probably happens is AOT invokes cctors in a slighly different order.

So what happens is that a call is made to String.Equals(.., StringComparison.Ordinal), but the aot runtime executes class ctors too eagerly, so it executes the CompareInfo cctor because of this line:

                case StringComparison.InvariantCulture:
                case StringComparison.InvariantCultureIgnoreCase:
                    return CompareInfo.Invariant.Compare(this, value, GetCaseCompareOfComparisonCulture(comparisonType)) == 0;

will try to fix.

CompareInfo is a beforefieldinit class, which means the runtime is supposed to be able to execute its cctor at any time before the moment of first access to its static members. So what mono is currently doing is correct. Its possible to change this, but it would mean that more type init checks would be done by the generated code, so it would be somewhat bigger/slower.

@tarekgh

If the mono's behavior is correct then I believe the change in libs, similar to what Ankit proposed in #94225 (comment) will be needed to fix it. @tarekgh, are you fine with moving the initialization to the getter?

Navigation Menu

Repository navigation

[wasm] AOT: `System.Globalization.Calendars.Hybrid.WASM.Tests` and `System.Globalization.Hybrid.WASM.Tests` getting terminated #94225

Build Information

Error Message

Known issue validation

Report

Summary

16 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	void
	ves_icall_System_Environment_FailFast (MonoStringHandle message, MonoExceptionHandle exception, MonoStringHandle errorSource, MonoError *error)
	{
	if (MONO_HANDLE_IS_NULL (errorSource)) {
	g_warning ("Process terminated.");

[wasm] AOT: System.Globalization.Calendars.Hybrid.WASM.Tests and System.Globalization.Hybrid.WASM.Tests getting terminated #94225

Description

Build Information

Error Message

Known issue validation

Report

Summary

Activity

radical commented on Oct 31, 2023

radical commented on Oct 31, 2023

ilonatommy commented on Nov 1, 2023

radical commented on Nov 1, 2023

radical commented on Nov 1, 2023

radical commented on Nov 1, 2023

16 remaining items

radical commented on Nov 4, 2023

radical commented on Nov 4, 2023

vargaz commented on Nov 4, 2023

lewing commented on Nov 7, 2023

vargaz commented on Nov 7, 2023

vargaz commented on Nov 8, 2023

vargaz commented on Nov 8, 2023

ilonatommy commented on Nov 20, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[wasm] AOT: `System.Globalization.Calendars.Hybrid.WASM.Tests` and `System.Globalization.Hybrid.WASM.Tests` getting terminated #94225