Repository navigation
mono minijit runtime tests failing with coreclr_initialize failed - Error: 0x80004005 #72114
Description
Failures 4/5-9/8 (incl. PRs and 7.0 branches):
| Day | Run | OS |
|---|---|---|
| 9/5 | Rolling run 5191 (only in Runfo) | mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open |
| 9/5 | Rolling run 4698 (only in Runfo) | mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open |
| 8/30 | Rolling run | Ubuntu.1804.Arm64.Open - Mono Interpreter |
| 8/30 | PR #74831 | OSX.1200.Amd64.Open - Mono Interpreter |
| 8/24 | PR #74459 | OSX 12 amd64 - Moni minijit |
| 8/8 | PR #73416 | OSX 12 amd64 - Mono Interpreter |
| 8/4 | PR #73230 | OSX 12 amd64 - Mono Interpreter |
| 8/2 | PR #73157 | OSX 12 amd64 - Moni minijit |
| 7/29 | Rolling run | Ubuntu.1804 arm64 - Mono llvm full aot |
| 7/12-7/27 | 8x unique PRs | OSX 12 amd64 and Ubuntu.1804 arm64 |
| 6/1-6/30 | 6x unique PRs | OSX 12 amd64 and Ubuntu.1804 arm64 |
| 5/1-5/31 | 2x unique PRs | OSX 12 amd64 |
| 4/1-4/30 | 6x unique PRs | OSX 12 amd64 and Ubuntu.1804 arm64 |
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
| where TestName contains testNameSubstring
| where includePassedOnRerun or (Outcome == 'Failed')
| extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
| extend Method = substring(TestName, startOfTestName)
| extend Type = substring(TestName, 0, startOfTestName - 1)
| project-away startOfTestName
| where (methodName == '') or (Method == methodName)
| where Message contains messageSubstr
| distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
| where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (Branch startswith 'refs/heads/release/7.0') or (includePR and (Source startswith "pr/")))
| where Type startswith "test/functional/cli/"
and not(Properties contains "runtime-staging")
| summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
| extend PropertiesJson = parse_json(Properties)
| extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
| extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
| extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
| extend Architecture = PropertiesJson.architecture
| extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
//| extend DefinitionName = PropertiesJson.DefinitionName
| project-away PropertiesJson
};
failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true);Hit on #72094 .
Build. From one of the logs:
JIT/Math/Functions/Functions_ro/Functions_ro.sh [FAIL]
BEGIN: coreclr_initialize failed - Error: 0x80004005
END: coreclr_initialize failed - Error: 0x80004005
Return code: 1
Raw output file: /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll ''
Exe path: /private/tmp/helix/working/B5FB0986/p/corerun
Properties:
TRUSTED_PLATFORM_ASSEMBLIES = <see console log for the full list>
APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/
NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/:
System.Reflection.Metadata.MetadataUpdater.IsSupported = false
Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll
Arguments (0):
Expected: 100
Actual: 255
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/tmp/helix/working/B5FB0986/p
> /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh
Expected: True
Actual: False
Stack Trace:
at JIT_Math._Functions_Functions_ro_Functions_ro_._Functions_Functions_ro_Functions_ro_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Output:
BEGIN: coreclr_initialize failed - Error: 0x80004005
END: coreclr_initialize failed - Error: 0x80004005
Return code: 1
Raw output file: /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll ''
Exe path: /private/tmp/helix/working/B5FB0986/p/corerun
Properties:
TRUSTED_PLATFORM_ASSEMBLIES = <see console log for the full list>
APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/
NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/:
System.Reflection.Metadata.MetadataUpdater.IsSupported = false
Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll
Arguments (0):
Expected: 100
Actual: 255
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/tmp/helix/working/B5FB0986/p
> /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh
cc @lewing
Activity
lewing commentedon Jul 13, 2022
marking future until we see it again
karelz commentedon Aug 5, 2022
@lewing it happens more often - see updated top post with Kusto query. Overall about 2x per week - marking it as 'blocking-clean-ci'.
Marking as untriaged again.
Also, it is causing massive failures on the runs, which makes it hard to analyze results -- see for example Rolling build from 7/29 - 1912276 has 200 failures.
[-]mono `minijit` runtime tests failing with `coreclr_initialize failed - Error: 0x80004005` on OSX/x64[/-][+]mono `minijit` runtime tests failing with `coreclr_initialize failed - Error: 0x80004005`[/+]fanyang-mono commentedon Aug 12, 2022
The complete list of test failures with this error:
https://runfo.azurewebsites.net/search/tests/?q=started%3A%7E14+definition%3Aruntime+message%3Acoreclr_initialize+jobName%3AMono
Build from 8/8
https://dev.azure.com/dnceng/public/_build/results?buildId=1929242&view=logs&j=76620f41-b685-5972-e5ab-3722b69632c4&t=e5054d15-9cd7-5687-4154-57982228d697
The failed lane was
mono monointerpreter Pri0 Runtime Tests Run OSX x64 releaseBuild from 8/4 RP #73230
https://dev.azure.com/dnceng/public/_build/results?buildId=1922628&view=logs&j=76620f41-b685-5972-e5ab-3722b69632c4&t=e5054d15-9cd7-5687-4154-57982228d697
The failed lane was
mono monointerpreter Pri0 Runtime Tests Run OSX x64 releaseBuild from 7/29 rolling build https://dev.azure.com/dnceng/public/_build/results?buildId=1912276&view=logs&j=d631b51d-8d68-5da3-5a13-95f6a763c7bb&t=e848a3c9-1589-513e-d5da-7aef4a24d916
The failed lane was
mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release@karelz I couldn't find the build containing this error from 8/2 PR #73157. Could you please comment it here?
fanyang-mono commentedon Aug 12, 2022
I tried to run
Interop/PInvoke/SetLastError/SetLastErrorTest/SetLastErrorTest.shwith interpreter locally multiple times, but wasn't able to reproduce this failure.fanyang-mono commentedon Aug 12, 2022
According to https://support.microsoft.com/en-us/topic/you-may-receive-error-code-0x80004005-or-other-error-codes-when-you-try-to-start-a-windows-xp-based-computer-a15f5b2f-642d-24ac-4912-1570a6bcedec.
Error: 0x80004005seems to be a windows related issue. However, the failures are happening on non-win platforms.@lambdageek Do you have any thoughts on this?
10 remaining items
SamMonoRT commentedon Aug 22, 2022
Seems outside Mono scope at this time. Not certain what the best area label is though. Also moving to 8.0.0 given the failure is not deterministic and seems like a test/infrastructure issue, not a real customer issue.
cc @LakshanF - is this something you can help investigate? Please update area label appropriately.
karelz commentedon Sep 8, 2022
Another massive hit on:
@SamMonoRT who is taking action on this? Can you please find someone on your team or another team? This is blocking CI ...
elinor-fung commentedon Oct 11, 2022
Hit in #75858
Example of one of the failures:
Test run logs
elinor-fung commentedon Oct 11, 2022
I believe it is as @lambdageek said:
When generating CORE_ROOT, both the coreclr and mono versions of
libcoreclr.soare being copied over, so it is a race of which is last. In the case of the runtime pipeline build from my previous comment, the coreclr version is last.Relevant logs from the runtime pipeline build:
Layout.Linux.x64.Release.binlog.txt (actually a binlog, just renamed)
Layout.Linux.x64.Release.log
elinor-fung commentedon Oct 11, 2022
The coreclr version is being included
RunTimeDependencyCopyLocalfrom:runtime/src/tests/Common/Directory.Build.targets
Lines 101 to 111 in 3435b39
Perhaps we just need to condition those under
'$(RuntimeFlavor)' == 'coreclr'.