Repository navigation
Test failure: JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd #106675
Description
Failed in: runtime-coreclr outerloop 20240819.3
Failed tests:
R2R-CG2 linux x64 Checked no_tiered_compilation @ (Alpine.317.Amd64.Open)Ubuntu.2204.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-amd64
- JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.cmd
Error message:
/root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh: line 327: -r:/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/*.dll: No such file or directory
/root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh: line 266: 7570 Killed $__Command
Return code: 1
Raw output file: /root/helix/work/workitem/uploads/JitBlue/DevDiv_255294/DevDiv_255294/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/DevDiv_255294.dll
19:53:10
Response file: /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.rsp
/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/IL-CG2/DevDiv_255294.dll
-o:/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll
-r:/root/helix/work/correlation/System.*.dll
-r:/root/helix/work/correlation/Microsoft.*.dll
-r:/root/helix/work/correlation/xunit.*.dll
-r:/root/helix/work/correlation/mscorlib.dll
--verify-type-and-field-layout
--method-layout:random
--targetarch:x64
--targetos:linux
Running CrossGen2: dotnet /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.rsp
Running R2RDump: dotnet /root/helix/work/correlation/R2RDump/R2RDump.dll --header --sc --in /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll --out /root/helix/work/workitem/e/JIT/Regression/JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.dll.r2rdump --val
Error: System.BadImageFormatException: The file is not a ReadyToRun image
at ILCompiler.Reflection.ReadyToRun.ReadyToRunReader.Initialize(IAssemblyMetadata metadata) in /_/src/coreclr/tools/aot/ILCompiler.Reflection.ReadyToRun/ReadyToRunReader.cs:line 467
at R2RDump.Program.Run() in /_/src/coreclr/tools/r2rdump/Program.cs:line 449
19:53:22
Crossgen2 failed with exitcode: 137
in ReleaseLock
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 8/19/2024 7:53:10 PM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /root/helix/work/workitem/e/JIT/Regression/Regression_3/../JitBlue/DevDiv_255294/DevDiv_255294/DevDiv_255294.sh
Stack trace:
at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 128
at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
at Program.<<Main>$>g__TestExecutor322|0_323(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)
Known Issue Error Message
Fill the error message using step by step known issues guidance.
{
"ErrorMessage": "Error: System.BadImageFormatException: The file is not a ReadyToRun image",
"ErrorPattern": "",
"BuildRetry": false,
"ExcludeConsoleLog": false
}Known issue validation
Build: 🔎
Result validation:
Validation performed at: 6/1/2025 9:40:33 PM UTC
Report
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 0 | 1 | 13 |
Activity
ivdiazsa commentedon Aug 21, 2024
This
The file is not a ReadyToRun imageerror has been intermittently hitting on and off for a long time, with no apparent pattern. We have to investigate further to try to understand it. Setting it for .NET 10.v-wenyuxu commentedon Nov 27, 2024
Failed in: runtime-coreclr outerloop 20241126.1
Failed tests:
Error message:
Stack trace:
v-wenyuxu commentedon Dec 19, 2024
Failed in: runtime-coreclr outerloop 20241218.4
Failed tests:
Error message:
Stack trace:
v-wenyuxu commentedon Feb 12, 2025
Failed in: runtime-coreclr outerloop 20250211.8
Failed tests:
Error message:
Stack trace:
jkotas commentedon Jun 1, 2025
Hit in #116096
SakeTao commentedon Jun 5, 2025
Failed in: runtime-coreclr outerloop 20250604.1
Failed tests:
Error message:
Stack trace:
SakeTao commentedon Jun 11, 2025
Failed in: runtime-coreclr outerloop 20250610.6
Failed tests:
Error message:
Stack trace:
27 remaining items
EgorBo commentedon Aug 15, 2025
Thanks @kg for the analysis!
EgorBo commentedon Aug 15, 2025
So looks like the problematic invocation is this one:
runtime/src/coreclr/jit/ifconversion.cpp
Lines 670 to 675 in 59ce21a
The most obvious fix is to never invoke this logic if a method is known to never have loops which we care about here.
Although, I guess we should use this repro to just generally optimize this routine, as we can just wrap the entire thing into an unnatural loop.
EgorBo commentedon Aug 15, 2025
So I presume we can use the reachability sets, but we'll have to compute them. We probably can do it only for methods with a lot of blocks.
EgorBo commentedon Aug 15, 2025
@AndyAyersMS so if I calculate the reachability sets in the beginning of the optIfConversion phase (because at that point they're always null/invalid), it drops the time needed to compile the method from 60s to 30s which is still pretty bad. The problem that ReachabilitySets cosume O(n^2) memory (as stated in the comments) and it does show up, ~4GB (main) -> 6.5Gb(reachability set) for working set.
In the ifConversionPhase that
is block reached from A to Bis used to detect whether the blocks we work with inside any kind of a loop (either natural or unnatural) so it doesn't have to be precise or even always invoked.I propose we drop the check for methods with lots of blocks. I guess this particular repro is not realistic, so in the real world we don't expect that to trigger too often.
kg commentedon Aug 15, 2025
The most likely real world scenario I can think of would be large IEnumerator/async state machines since their MoveNexts turn into switches, but they'd need to have a bunch of awaits to hit bblock counts that really inflate the memory usage and CPU time, yeah?
AndyAyersMS commentedon Aug 15, 2025
That would probably be fine.
What we really want to know is if this if conversion candidate is on the critical path of a short loop. A "local neighborhood" search to find the set of blocks reachable in 3 jumps or so might be enough. I think that would just require two bit vectors.
We could also compute the SCCs which will be fairly fast. I could use this info during escape analysis to improve its cycle detection.
The preferred method is still I think Tarjan's SCC though we'd want to have a non-recursive implementation. Kosaraju's is somewhat simpler and easily made non-recursive. #109656 has a reverse graph DFS you could steal.
SakeTao commentedon Aug 19, 2025
Failed in: runtime-coreclr outerloop 20250818.3
Failed tests:
Error message:
Stack trace:
SakeTao commentedon Aug 21, 2025
Failed in: runtime-coreclr outerloop 20250820.1
Failed tests:
Error message:
Stack trace:
kg commentedon Sep 10, 2025
Test is now disabled on crossgen2 CI due to the oom killer
jkotas commentedon Sep 10, 2025
The test is disabled against #85663
runtime/src/tests/issues.targets
Line 529 in dcf8680
kg commentedon Sep 10, 2025
Yeah, I'll close this and update the other one with details.