Repository navigation
Cross built ILCompiler NuGet contains HostOS (Linux) ELFs not TargetOS (FreeBSD) ELFs #104497
Description
Overview:
When cross compiling runtime, the ILCompiler (and NuGet) that is produced contains HostOS (Linux) and not TargetOS (FreeBSD) ELFs.
Reproduction:
Using a docker container on Linux pulled from: https://raw.githubusercontent.com/dotnet/versions/master/build-info/docker/image-info.dotnet-dotnet-buildtools-prereqs-docker-main.json containing a FreeBSD ROOTFS and the most recent net9 preview tag from runtime: v9.0.0-preview.5.24306.7
docker run -e ROOTFS_DIR=/crossrootfs/$ROOTFSARCH -v ${BUILD_SOURCESDIRECTORY}/runtime:/runtime $DOTNETDOCKERCONTAINERUSED /runtime/build.sh -c ${{ parameters.buildType }} -cross -os freebsd -arch $ROOTFSARCH -ci /p:OfficialBuildId=$OFFICIALBUILDID --subset clr+mono+mono.manifests+tools+libs+host+packs
Expected behavior:
ELFs should be like other items generated for TargetOS:
$file crossgen2
crossgen2: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.2, FreeBSD-style, BuildID[sha1]=54a7f1c2a4752435c2cffd15eeb959f609966907, stripped
Actual behavior:
The resulting runtime.freebsd-x64.Microsoft.DotNet.ILCompiler.9.0.0-preview.5.24306.7.nupkg contains Linux ELFs and FreeBSD libs.
$find ./ * | xargs file
./: directory
./ILCompiler.RyuJit.pdb: Microsoft Roslyn C# debugging symbols version 1.0
./libclrjit_unix_x64_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=66177aebc4ab51f16fe1e6a5faa90a7ade09b674, stripped
./libclrjit_win_x86_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=79ecdf1053497bde0393928dee1a727bc6b6b6a1, stripped
./libclrjit_universal_arm_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=b15d888e793cca18b6dd42b3b672f7144fbe45ec, stripped
./libjitinterface_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=02a2d4a17bcbd0ff35f3caca9252853f95529a3c, stripped
./ILCompiler.TypeSystem.pdb: Microsoft Roslyn C# debugging symbols version 1.0
./ILCompiler.DependencyAnalysisFramework.pdb: Microsoft Roslyn C# debugging symbols version 1.0
./ILCompiler.Compiler.pdb: Microsoft Roslyn C# debugging symbols version 1.0
./libclrjit_universal_arm64_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=0b476dc684291af72ab673b77dccbbf7f386cbf8, stripped
./libclrjit_win_x64_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=a58827fd1b7ac68612408fa5e13c7db9091938a2, stripped
./ilc.pdb: Microsoft Roslyn C# debugging symbols version 1.0
./ilc: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=41cb9e347020ad19b6402190528b968b6850c46f, stripped
Regression:
To the best of my knowledge, it has always been this way. This was uncaught until recently when I tried to use a cross built package to bootstrap a native build.
Known Workarounds:
None?
Other info:
TargetOS=linux only appears three places in the .binlog for the build that are after Evaluation. All three are from ILCompiler.cspoj : The first seems to come as a return from ResolveReadyToRunCompilers and the other two (_PrepareForReadyToRunCompilation) and (_CreateR2RImages) use it. NativeAotSupported is reassigned here:
ilc binary and the process does not error from runtime/src/tasks/Crossgen2Tasks/ResolveReadyToRunCompilers.cs
Lines 116 to 120 in a5cc707
There is no "Property reassignment" note in the binlog when ResolveReadyToRunCompilers changes(?) the TargetOS=freebsd to linux
.binlog is not an allowed attachment type so hopefully a screenshot from the MSBuild Structured Log Viewer is enough to help explain what I am seeing.
The Crossgen2 project seems to avoid the ReadyToRun part during packaging:
Activity
dotnet-policy-service commentedon Jul 5, 2024
Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.
jkotas commentedon Jul 5, 2024
This was refactored in #103508 . Does the problem still exist in current main?
Thefrank commentedon Jul 6, 2024
@jkotas
Just checked. It still exists as of e125e93
jkotas commentedon Jul 6, 2024
Ok, here is the problem:
The regular build uses "last known good" toolchain (the one that comes from https://github.com/dotnet/runtime/blob/main/global.json#L3) for aot compilation of most managed tools and components. The one exception is crossgen2 that uses live built aot toolchain that leads to interesting problems every once in a while.
When cross-compiling from Linux to FreeBSD, there is no "last known good" aot toolchain that is able to cross-compile from Linux to FreeBSD. We end up doing Linux->Linux publish instead due to some incorrect build logic and produce binaries for wrong architecture as you have observed.
I can think about two options for addressing this:
Thefrank commentedon Jul 9, 2024
Would disabling AOT for managed items still give FreeBSD (TargetOS) ELF's on components like ILCompiler?
If yes, that might be the simplest solution for the time being.
The multi-stage build sounds like the best solution for this issue. But, unless there is something I am missing, this issue seems limited to a niche case of cross compiling from Linux to FreeBSD/Illumos/Haiku/Other. If it helps other build scenarios for officially supported platforms then there is a better case for this but I am not familiar enough with intricacies here.
jkotas commentedon Jul 9, 2024
I think having the recipe for how to use live-built AOT compilers would be generally useful. We keep having long discussions about whether it is better to use live-built AOT compiler or last-known-good AOT compiler. So having both options and use the one that's more appropriate for given situation would be best.
It would require some extra build logic. Nothing impossible.
jkotas commentedon Jul 9, 2024
cc @jkoritzinsky @lambdageek
12 remaining items
Thefrank commentedon Aug 20, 2024
Which FreeBSD version and which LLVM/Clang version?
sec commentedon Aug 20, 2024
13.3-RELEASE-p3 and llvm18 (had the same with llvm15). Tried to native build runtime with
-p:LinkerFlavor=bfdswitch, but this does not changed anything. Also there's one version of ilc that does work from native build:edit: hm, looks like ilc is build without symbol exported from #105587 when doing native build, will look into that.
Thefrank commentedon Aug 20, 2024
./artifacts/bin/coreclr/freebsd.arm64.Release/ilc/ilcmight be the same from./.packages/runtime.freebsd-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.7.24405.7/tools/ilcwhich might explain why it works but not why there is a difference between the crossbuilt and native build ILCsec commentedon Aug 20, 2024
The size of working ones are different - native one is 72480 bytes and the one from cross nuget is 16606596. the other two, that don't work don't have symbols exported that added them on crossbuild - have build running with symbol export added without
Condition="'$(_targetOS)' == 'freebsd' and '$(IsNativeExecutable)' == 'true'"to verify if this is the issue or is it somewhere else, will know if few minutes.edit: it sill segfault and there's no progname and environ symbol exported
@am11 maybe you could point in right direction, why those symbol exports could be missing when doing native build of ILC?
sec commentedon Aug 22, 2024
Could anyone explain why with the same target WriteIlcRspFileForCompilation, arguments list is different?
this one is missing export of __progname and environ:
while this one have them:
am11 commentedon Aug 22, 2024
@sec, I haven't tested it with freebsd-arm64, but if all things being equal (to x64) it should work the same way. I think the first run of the target is getting targetOS == linux for some reason in your environment.
sec commentedon Aug 22, 2024
I have changed those conditions to
<IlcArg Include="--export-dynamic-symbol:__progname" />so they should always be included, but I'm missing something.am11 commentedon Aug 22, 2024
Have you changed it in all
find ~/.nuget -name '*Unix.targets'?sec commentedon Aug 22, 2024
Hm, based on your PR, you only added those to
src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targetsI tried to look where
--export-dynamic-symbol:DotNetRuntimeDebugHeaderis added and it's only also in this file and it's added in both runs.Microsoft.NETCore.Native.Unix.targetsdon't contains any IlcArgsam11 commentedon Aug 22, 2024
Ah I meant find ~/.nuget -name '*Native.targets'. I'll create a VM on osx arm64 and try to debug it on the weekend.
sec commentedon Aug 22, 2024
Ok, I think I've found the issue, indeed I had some old nuget's cached - will confirm that in few minutes.
edit: yes, I had old
Microsoft.DotNet.ILCompiler.9.0.0-previ ew.7.24405.7insideruntime/.packagesthat didn't had those args added (that was from the crossbuild before ilc patch...) - thanks @am11 :)full build of runtime/aspnet/sdk was completed under freebsd-arm64 :)
Thefrank commentedon Aug 22, 2024
One of those
IlcArgitems in both projects might(should?) contain a--targetos:value. If it does readlinuxor that entry is not there, then you might have to fish around the log and find where it got misassigned.