Activity

1. ghost added

   area-System.Net

   on Jul 15, 2021
2. ghost added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Jul 15, 2021
4. ManickaP commented on Jul 15, 2021

   

   ManickaP

   on Jul 15, 2021

   Member

   More actions

   There's several problems combined here.

   1. Infra issue - some of the console logs without a dump

   Tests:

   === TEST EXECUTION SUMMARY ===
      System.Net.Requests.Tests  Total: 1109, Errors: 0, Failed: 0, Skipped: 6, Time: 48.794s
   /root/helix/work/workitem
   ----- end Thu Jul 15 00:15:40 UTC 2021 ----- exit code 0 ----------------------------------------------------------
   exit code 0 means Exited Successfully
   + export _commandExitCode=0
   

   Infra after the tests passing:

   2021-07-15 00:18:53,404: INFO: 548316835384: packing_test_reporter(34): report_results: Packed 317813 bytes
   Traceback (most recent call last):
     File "/root/helix/work/correlation/reporter/run.py", line 73, in <module>
       main()
     File "/root/helix/work/correlation/reporter/run.py", line 69, in main
       if os.path.isfile(packed_file):
     File "/usr/lib/python3.8/genericpath.py", line 30, in isfile
       st = os.stat(path)
   TypeError: stat: path should be string, bytes, os.PathLike or integer, not method
   + exit 1
   + export '_commandExitCode=1'
   + chmod -R 777 /home/helixbot/dotnetbuild/dumps
   + exit 1
   
   [END EXECUTION]
   Exit Code:1
   

   cc: @premun

   2. Mono crashes - with dump and runtime is mono

   Console.log

   =================================================================
   	Native Crash Reporting
   =================================================================
   Got a SIGABRT while executing native code. This usually indicates
   a fatal error in the mono runtime or one of the native libraries 
   used by your application.
   =================================================================
   ...
   

   cc: @marek-safar

   3. CoreCLR crashes - #54580 draft pr

   Different errors, looks like work in progress:

   ----- end Wed 07/14/2021  1:03:58.23 ----- exit code -1073740791 ----------------------------------------------------------
   

   ----- end Wed Jul 14 01:49:23 UTC 2021 ----- exit code 139 ----------------------------------------------------------
   exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
   

   cc: @jkoritzinsky

   4. Seg fault in FTP tests - majority of CoreCLR with a dump

   Two errors, console.log:

   ./RunTests.sh: line 162:    24 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
   /root/helix/work/workitem
   ----- end Mon Jul 12 19:32:39 UTC 2021 ----- exit code 139 ----------------------------------------------------------
   exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
   

   Assert failure(PID 59 [0x0000003b], Thread: 139 [0x008b]): !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n" "Possible cause: you used new (executable) to allocate the memory, but didn't use DeleteExecutable() to free it."
       File: /__w/1/s/src/coreclr/utilcode/clrhost_nodependencies.cpp Line: 296
       Image: /root/helix/work/correlation/dotnet
   
   ./RunTests.sh: line 162:    59 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
   

   This is the first that looks like it might be System.Net problem. Or it could have been caused by #54954, since this PR has higher occurrence of this error.
   cc: @karelz @janvorli

   5. Only on PR #55467

   Looks like work in progress, console.log:

   Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'xunit.runner.utility.netcoreapp10, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'. Uncaught exception during type initialization. (0x80131534)
   File name: 'xunit.runner.utility.netcoreapp10, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'
    ---> System.TypeInitializationException: The type initializer for 'Xunit.DependencyContextAssemblyCache' threw an exception.
    ---> System.NullReferenceException: Object reference not set to an instance of an object.
      at System.Text.RegularExpressions.RegexParser.AssignNameSlots() in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs:line 1957
      at System.Text.RegularExpressions.RegexParser.CountCaptures() in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs:line 1907
      at System.Text.RegularExpressions.RegexParser.Parse(String pattern, RegexOptions options, CultureInfo culture) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs:line 88
      at System.Text.RegularExpressions.Regex.Init(String pattern, RegexOptions options, TimeSpan matchTimeout, CultureInfo culture) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Regex.cs:line 107
      at System.Text.RegularExpressions.Regex..ctor(String pattern, CultureInfo culture) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Regex.cs:line 68
      at System.Text.RegularExpressions.Regex..ctor(String pattern) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Regex.cs:line 46
      at Xunit.DependencyContextAssemblyCache..cctor() in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/DependencyContextAssemblyCache.cs:line 20
      --- End of inner exception stack trace ---
      at Xunit.DependencyContextAssemblyCache.ResolveManagedAssembly(String assemblyName, Func`2 managedAssemblyLoader) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/DependencyContextAssemblyCache.cs:line 210
      at Xunit.DependencyContextAssemblyCache.LoadManagedDll(String assemblyName, Func`2 managedAssemblyLoader) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/DependencyContextAssemblyCache.cs:line 159
      at Xunit.AssemblyHelper.OnResolving(AssemblyLoadContext context, AssemblyName name) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/AssemblyHelper.cs:line 83
      at System.Runtime.Loader.AssemblyLoadContext.GetFirstResolvedAssemblyFromResolvingEvent(AssemblyName assemblyName) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.cs:line 625
      at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(AssemblyName assemblyName) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.cs:line 693
      at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr gchManagedAssemblyLoadContext, AssemblyName assemblyName) in /_/src/coreclr/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.CoreCLR.cs:line 134
      at Xunit.ConsoleClient.ConsoleRunner..ctor(Object consoleLock)
      at Xunit.ConsoleClient.Program.Main(String[] args) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/Program.cs:line 16
   

   cc: @eiriktsarpalis

   6. Another CoreCLR crash, only #55372 PR

   Happened only on this PR, so seems like fixed before it was merged, console.log:

   Assert failure(PID 23 [0x00000017], Thread: 33 [0x0021]): At least one interface has to implement the method, otherwise there's a bug in JIT/verification.
       File: /__w/1/s/src/coreclr/vm/methodtable.cpp Line: 9578
       Image: /root/helix/work/correlation/dotnet
   
   ./RunTests.sh: line 162:    23 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
   

   7. The last batch without a dump

   CI timeout, console.log:

   If you’re reading this, that means the Helix work item you’re trying to find the logs for has dead-lettered.
   

   Looks like vast majority of problems is not networking related.
5. janvorli commented on Jul 15, 2021

   

   janvorli

   on Jul 15, 2021

   Member

   More actions

   I have fixed the issue 4 from this list yesterday, the fix is already in.
6. janvorli commented on Jul 15, 2021

   

   janvorli

   on Jul 15, 2021

   Member

   More actions

   I mean, the 2nd one in the issue 4.

   👍React with 👍1Reacted by Marie Píchová
7. karelz commented on Jul 15, 2021

   

   karelz

   on Jul 15, 2021

   Member

   More actions

   @jaredpar this seems to be mix of many issues. We will take out the one crash that may be Networking related.
   How do you want to track the rest?

   Also, it seems that Runfo should not mix up the crashes as it did, but perhaps that is too much to ask from a simple tool?
8. karelz

   added

   needs more info

   on Jul 15, 2021
9. jaredpar commented on Jul 15, 2021

   

   jaredpar

   on Jul 15, 2021

   Member

   More actions

   Also, it seems that Runfo should not mix up the crashes as it did, but perhaps that is too much to ask from a simple tool?

   The tool right now only has indexes over certain strings: test names and issues in the timeline. Can design queries for pretty much anything along those lines.

   The detail here to differentiate is in the console logs. That doesn't have any indexes yet. It's possible to do just havent' had time to do so.

   👍React with 👍1Reacted by Karel Zikmund
10. karelz commented on Jul 20, 2021

    

    karelz

    on Jul 20, 2021

    Member

    More actions

    Closing this mega-issue in favor of folks tracking down sub-problems as they are tagged above ...
11. karelz

    closed this as completedon Jul 20, 2021
12. karelz

    added this to the 6.0.0 milestone on Jul 20, 2021
13. ghost locked as resolved and limited conversation to collaborators on Aug 19, 2021
14. karelz

    removed

    untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

    on Oct 20, 2022