Repository navigation
CoreCLR native build on Linux / Linux_musl arm64 / arm causes clang crashes #84802
Description
OS: Linux / Linux_musl
Architecture: arm64, arm
Going over the recent runs it seems to be pretty recent - first repro I've found is from yesterday afternoon:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237112&view=results
Diagnostics: clang crashes when compiling src/native/external/libunwind/src/elf32.c or elf64.c based on targeting architecture. I don't see any recent changes to their source code so I tend to suspect either a clang version update or some scripting update. The problem may not be fully deterministic - the above example run is the one where this also happens on Linux arm, in the remaining runs I saw it was only reproing on Linux / Linux_musl arm64.
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script. Stack dump: 0. Program arguments: /usr/local/bin/clang-12 --target=arm-linux-gnueabihf --gcc-toolchain=/crossrootfs/arm/usr --sysroot=/crossrootfs/arm -DBUILDENV_DEBUG=1 -DCLR_ARM_FPU_CAPABILITY=0x7 -DDEBUG -DDISABLE_CONTRACTS -DHAVE_CONFIG_H=1 -DHAVE___THREAD=0 -DHOST_ARM -DHOST_UNIX -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_STRING=\"\" -DTARGET_32BIT -DTARGET_ARM -DTARGET_LINUX -DTARGET_UNIX -DURTBLDENV_FRIENDLY=Debug -DUSE_STL -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Darm_search_unwind_table=UNW_OBJ(arm_search_unwind_table) -Dasm=__asm__ -I/__w/1/s/artifacts/obj/external/libunwind -I/__w/1/s/src/native/external/libunwind_extras -I/__w/1/s/src/native -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/src -I/__w/1/s/src/coreclr/pal/../inc -I/__w/1/s/src/native/external/libunwind/include -I/__w/1/s/src/native/external/libunwind/include/tdep -I/__w/1/s/artifacts/obj/external/libunwind/include -I/__w/1/s/artifacts/obj/external/libunwind/include/tdep -I/__w/1/s/src/native/external/libunwind/include/tdep-arm -I/__w/1/s/src/native/external/libunwind/src -g -fPIC -mthumb -mfpu=vfpv3 -O0 -g -Wall -Wno-null-conversion -fno-omit-frame-pointer -fms-extensions -fwrapv -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wno-unused-but-set-variable -ffp-contract=off -Wno-unknown-warning-option -ferror-limit=4096 -Wno-unused-private-field -Wno-constant-logical-operand -Wno-pragma-pack -Wno-incompatible-ms-struct -Wno-reserved-identifier -Wno-unsafe-buffer-usage -Wno-single-bit-bitfield-constant-conversion -Wno-cast-function-type-strict -fsigned-char -fvisibility=hidden -ffunction-sections -march=armv7-a -fexceptions -Wno-format -Wno-format-security -Wno-implicit-fallthrough -Wno-header-guard -Wno-absolute-value -Wno-inline-asm -Wno-implicit-function-declaration -Wno-implicit-int-conversion -std=gnu11 -MD -MT /__w/1/s/artifacts/obj/external/libunwind/CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -MF CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o.d -o CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -c /__w/1/s/src/native/external/libunwind/src/elf32.c 1. parser at end of file 2. Code generation [ 8%] Building CXX object md/enc/CMakeFiles/mdruntimerw_wks_obj.dir/cmake_pch.hxx.pch #0 0x000056228ba9805e llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/bin/clang-12+0x213f05e) #1 0x000056228ba95e54 llvm::sys::RunSignalHandlers() (/usr/local/bin/clang-12+0x213ce54) #2 0x000056228ba0c438 CrashRecoverySignalHandler(int) (/usr/local/bin/clang-12+0x20b3438) #3 0x00007f29380e9e30 __restore_rt (/lib/libc.so.6+0x42e30) #4 0x000056228c53e310 llvm::DIE::getUnitDie() const (/usr/local/bin/clang-12+0x2be5310) #5 0x000056228c54869c llvm::DwarfDebug::finishEntityDefinitions() (/usr/local/bin/clang-12+0x2bef69c) #6 0x000056228c561a52 llvm::DwarfDebug::finalizeModuleInfo() (/usr/local/bin/clang-12+0x2c08a52) #7 0x000056228c564a68 llvm::DwarfDebug::endModule() (/usr/local/bin/clang-12+0x2c0ba68) #8 0x000056228c535aa9 llvm::AsmPrinter::doFinalization(llvm::Module&) (/usr/local/bin/clang-12+0x2bdcaa9) #9 0x000056228b3ef55d llvm::FPPassManager::doFinalization(llvm::Module&) (.localalias) (/usr/local/bin/clang-12+0x1a9655d) #10 0x000056228b3fb020 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/local/bin/clang-12+0x1aa2020) #11 0x000056228bd2bf2c (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::unique_ptr >) (.constprop.0) (/usr/local/bin/clang-12+0x23d2f2c) #12 0x000056228bd2dc7c clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr >) (/usr/local/bin/clang-12+0x23d4c7c) #13 0x000056228c96670c clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/usr/local/bin/clang-12+0x300d70c) #14 0x000056228d430b39 clang::ParseAST(clang::Sema&, bool, bool) (/usr/local/bin/clang-12+0x3ad7b39) #15 0x000056228c33b269 clang::FrontendAction::Execute() (/usr/local/bin/clang-12+0x29e2269) #16 0x000056228c2deccb clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/local/bin/clang-12+0x2985ccb) #17 0x000056228c3e9c58 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/usr/local/bin/clang-12+0x2a90c58) #18 0x000056228a50788f cc1_main(llvm::ArrayRef, char const*, void*) (/usr/local/bin/clang-12+0xbae88f) #19 0x000056228a5052a0 ExecuteCC1Tool(llvm::SmallVectorImpl&) (/usr/local/bin/clang-12+0xbac2a0) #20 0x000056228c18d5b5 void llvm::function_ref::callback_fn >, std::__cxx11::basic_string, std::allocator >*, bool*) const::'lambda'()>(long) (/usr/local/bin/clang-12+0x28345b5) #21 0x000056228ba0c533 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref) (/usr/local/bin/clang-12+0x20b3533) #22 0x000056228c18e6df clang::driver::CC1Command::Execute(llvm::ArrayRef >, std::__cxx11::basic_string, std::allocator >*, bool*) const (.part.0) (/usr/local/bin/clang-12+0x28356df) #23 0x000056228c164ad7 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/usr/local/bin/clang-12+0x280bad7) #24 0x000056228c165519 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl >&) const (/usr/local/bin/clang-12+0x280c519) #25 0x000056228c172f29 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl >&) (/usr/local/bin/clang-12+0x2819f29) #26 0x000056228a47ecb2 main (/usr/local/bin/clang-12+0xb25cb2) #27 0x00007f29380d557d __libc_start_call_main (/lib/libc.so.6+0x2e57d) #28 0x00007f29380d5630 __libc_start_main_impl (/lib/libc.so.6+0x2e630) #29 0x000056228a504c85 _start (/usr/local/bin/clang-12+0xbabc85) clang-12: error: clang frontend command failed with exit code 139 (use -v to see invocation) clang version 12.0.1 Target: arm-unknown-linux-gnueabihf Thread model: posix InstalledDir: /usr/local/bin [ 8%] Building CXX object md/compiler/CMakeFiles/mdcompiler_wks_obj.dir/assemblymd.cpp.o clang-12: note: diagnostic msg: ******************** PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT: Preprocessed source(s) and associated run script(s) are located at: clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.c clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.sh clang-12: note: diagnostic msg: ********************
I must admit I don't yet fully understand what the "parser at end of file" refers to, whether it's the compiler command line or the source file.
/cc @dotnet/runtime-infrastructure @janvorli
Activity
sbomer commentedon Apr 13, 2023
This looks like #84503. I was seeing it in linux arm64 mono build, but hadn't seen it in other builds. dotnet/dotnet-buildtools-prereqs-docker#842 seemed to fix it for me. But the images in dotnet-buildtools-prereqs-docker have already been updated to llvm 16, which we are not yet ready to use. I think we'll want to roll back the images to llvm 12 to fix this ASAP.
janvorli commentedon Apr 13, 2023
@sbomer I am a bit confused - the failure logged above was caused by clang 12, why do we need to revert the upgrade to clang 16?
sbomer commentedon Apr 13, 2023
Runtime is currently building with a set of mariner images that have clang 12, built with gcc. There is no published set of images that has clang 12, built with clang - because the llvm 16 upgrade went in before the build-with-clang fix. So I'm reverting the upgrade temporarily to get us a new set of images that have clang-built clang 12.
janvorli commentedon Apr 13, 2023
Ah, got it, thank you for the explanation.
trylek commentedon Apr 13, 2023
Frankly speaking I'm not getting why it doesn't fail the same way in the runtime (PR / CI) pipeline, e.g. in this Egor's PR run started about half an hour ago:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=238687&view=results
sbomer commentedon Apr 13, 2023
I'm not sure either - I was able to reproduce it consistently when I saw the failure building mono arm64. But it might be very sensitive to the inputs.
sbomer commentedon Apr 14, 2023
I believe this is fixed with #84851. Please reopen if you hit this again.