Repository navigation
panic code is not deterministically lto'ed #52044
Description
Activity
Oh dear now this is a little terrifying. The following script also reproduces the error here and is a bit smaller in that it only uses rustc. This script generates only the IR and object file for the compilation at hand:
Details
#!/bin/bash set -ex input=" #[no_mangle] pub extern fn foo() { panic!(\"foo\"); } " rustc="rustc /dev/stdin -O -C lto -C panic=abort -C codegen-units=1" rustc="$rustc --emit llvm-ir,obj --crate-type staticlib" for i in `seq 1 100`; do rm -rf a b mkdir a b $rustc --out-dir a <<< $input $rustc --out-dir b <<< $input a=$(md5sum a/stdin.ll | awk '{print $1}') b=$(md5sum b/stdin.ll | awk '{print $1}') if [ "$a" != "$b" ]; then echo IR is different exit 1 fi a=$(md5sum a/stdin.o | awk '{print $1}') b=$(md5sum b/stdin.o | awk '{print $1}') if [ "$a" != "$b" ]; then echo object is different exit 1 fi done
The scary part here is that the IR is the same when the object files are different. This means that somehow LLVM's code generator looks like it's being non-deterministic. I can't reproduce this with
llceither, LLVM's standalone code generator.Somehow this means that the in-process state of LLVM is nondeterministic in just the right way between compilations that it affects the final output file. As to how... that's a bit of a mystery.
This isn't at least an obvious determinism bug in rustc in that the IR is the same so we're feeding the same input to LLVM both times. It's seemingly something else that's going awry!
- addedA-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.A-codegenArea: Code generationArea: Code generationT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.Relevant to the compiler team, which will review and decide on the PR/issue.
on Jul 4, 2018 Trace diff for rustc_codegen_llvm is non-empty but surface level appears to be just pointers changing addresses and hashes being different (still odd, but at least reasonable): https://gist.github.com/Mark-Simulacrum/5d04c7ad104aa8936d45c4ac71f301f3.
Trace diff for the whole compilation (RUST_LOG=trace) is here: https://gist.github.com/Mark-Simulacrum/a1dcc978a6b65cd48307ad9407c60cf9. This shows that there is certainly non-determinism inside rustc, but also does not look like there's anything obvious here.
This seems likely to have something to do with recent changes to
panic_implementationthough IIRC they didn't touch codegen directly. However, perhaps we're seeing some side effect of the specific code being generated -- theBoxMeUpand similar code is relatively low-level for Rust.I cannot reproduce with a minimal
no_stdprogram within the ~100 iterations:#![no_std] #![feature(panic_implementation)] use core::panic::PanicInfo; #[panic_implementation] fn rust_begin_panic(_info: &PanicInfo) -> ! { loop {} } #[no_mangle] pub extern fn foo() { panic!(\"foo\"); }FWIW, this doesn't reproduce with 1.25, which was the first version with LLVM 6.
I'll attempt a bisection tomorrow.
This reproduces with nightly-2018-06-04, which is the first one with #50338 (panic_implementation) merged. But it also does reproduce with nightly-2018-06-03.
Both the merge of #49051 and the parent merge fail for me, so it seems your bisection went wrong.
The merge of #49051, 36b6687, reproduces via the script I wrote above for me locally. The previous merge of #47813, 3926453, did not reproduce via the same script after 100 executions of trying to get a different hash. Additionally using 3926453 I am unable to reproduce via the example you gave in the OP.
How are you testing 3926453? I'm using
rustup-toolchain-install-masterto download the prebuilt binaries and execute locally. Did you build from scratch?To hopefully weed out weird build issues as the problem I've tested the next previous merge as well of #48138, ff2d506, but it also isn't reproducing the bug here.
Local bisection with builds from scratch point me to the upgrade to LLVM 6, quite consistently. Has the compiler used on automation to build C/C++ code changed later?
Mmmm it fails both when llvm is built with GCC 5.4 (which is what I was originally bisecting with) and GCC 4.8 (which seems to be what rust automation used for some old builds I looked at the logs for). I wonder if the fact that this is on Ubuntu, where GCC defaults to have some features enabled that upstream GCC doesn't makes a difference.
8 remaining items
So it turns out the first difference I'm seeing in
-print-before-all -print-after-alloutput is in "IR Dump Before Expand ISel Pseudo-instructions". The last(adjacent) non-different dump is "IR Dump After Module Verifier". That one is still llvm-ir, but the dump before "Expand ISel Pseudo-instructions" is pseudo asm.-print-isel-inputis identical, and-print-machine-instrsshows the difference for_ZN3std9panicking20rust_panic_with_hook17hc482345e1eead25bE"After Instruction Selection". So in fact, no new information since #52044 (comment)I modified llc to not execute a program when doing
-view-isel-dagsor-view-legalize-dags(because that launches 2811 xdot processes), instead just dumping the dag dot files, and compared them. There didn't seem to be notable differences besides things like register numbers, which also differ between runs that are identical. At least there doesn't seem to be structural differences in the DAGs.In case this helps:
Basic block containing the first
-print-machine-instrsdifference, in "After Instruction Selection":Variant 1
%bb.118: derived from LLVM BB %555 Predecessors according to CFG: %bb.117 %390:vr128 = MOVAPSrm %noreg, 1, %noreg, target-flags(x86-tpoff) @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 + 16, %fs; mem:LD16[bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*)(align=32)+16](align=16)(alias.scope=!15687,!15689)(noalias=!15692,!15693,!15694,!15695,!15677,!15679,!15646)(dereferenceable) VR128:%390 dbg:/checkout/src/libcore/ptr.rs:221 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] %391:vr128 = V_SET0; VR128:%391 dbg:/checkout/src/libcore/ptr.rs:222 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] MOVAPSmr %noreg, 1, %noreg, target-flags(x86-tpoff) @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 + 16, %fs, killed %391; mem:ST16[bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*)+16](noalias=!15695,!15677,!15679,!15646) VR128:%391 dbg:/checkout/src/libcore/ptr.rs:222 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] %392:gr32 = MOV32ri64 1; GR32:%392 %393:gr64 = SUBREG_TO_REG 0, killed %392, sub_32bit; GR64:%393 GR32:%392 %394:vr128 = MOV64toPQIrr killed %393; VR128:%394 GR64:%393 dbg:/checkout/src/libcore/ptr.rs:222 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] %395:gr64 = MOV64rm %noreg, 1, %noreg, target-flags(x86-tpoff) @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381, %fs; mem:LD8[bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*)](alias.scope=!15687,!15689)(noalias=!15692,!15693,!15694,!15695,!15677,!15679,!15646)(dereferenceable) GR64:%395 dbg:/checkout/src/libcore/mem.rs:695 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] MOVAPSmr %noreg, 1, %noreg, target-flags(x86-tpoff) @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381, %fs, killed %394; mem:ST16[bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*)](align=32)(noalias=!15695,!15677,!15679,!15646) VR128:%394 dbg:/checkout/src/libcore/ptr.rs:222 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] %60:gr64 = MOVPQIto64rr %390; GR64:%60 VR128:%390 dbg:/checkout/src/libcore/mem.rs:695 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] %396:vr128 = PSHUFDri %390, 78; VR128:%396,%390 dbg:/checkout/src/libcore/mem.rs:695 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] %61:gr64 = MOVPQIto64rr killed %396; GR64:%61 VR128:%396 dbg:/checkout/src/libcore/mem.rs:695 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] TEST64rr %395, %395, implicit-def %eflags; GR64:%395 dbg:/checkout/src/libcore/ptr.rs:59 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] JE_1 %bb.121, implicit %eflags; dbg:/checkout/src/libcore/ptr.rs:59 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] JMP_1 %bb.147; dbg:/checkout/src/libcore/ptr.rs:59 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] Successors according to CFG: %bb.121(0x20000000 / 0x80000000 = 25.00%) %bb.147(0x60000000 / 0x80000000 = 75.00%)Variant 2
%bb.118: derived from LLVM BB %555 Predecessors according to CFG: %bb.117 %390:vr128 = MOVAPSrm %noreg, 1, %noreg, target-flags(x86-tpoff) @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 + 16, %fs; mem:LD16[bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*)(align=32)+16](align=16)(alias.scope=!15687,!15689)(noalias=!15692,!15693,!15694,!15695,!15677,!15679,!15646)(dereferenceable) VR128:%390 dbg:/checkout/src/libcore/ptr.rs:221 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] %391:vr128 = V_SET0; VR128:%391 dbg:/checkout/src/libcore/ptr.rs:222 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] MOVAPSmr %noreg, 1, %noreg, target-flags(x86-tpoff) @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 + 16, %fs, killed %391; mem:ST16[bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*)+16](noalias=!15695,!15677,!15679,!15646) VR128:%391 dbg:/checkout/src/libcore/ptr.rs:222 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] %392:gr32 = MOV32ri64 1; GR32:%392 %393:gr64 = SUBREG_TO_REG 0, killed %392, sub_32bit; GR64:%393 GR32:%392 %394:vr128 = MOV64toPQIrr killed %393; VR128:%394 GR64:%393 dbg:/checkout/src/libcore/ptr.rs:222 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] CMP64mi8 %noreg, 1, %noreg, target-flags(x86-tpoff) @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381, %fs, 0, implicit-def %eflags; mem:LD8[bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*)](alias.scope=!15687,!15689)(noalias=!15692,!15693,!15694,!15695,!15677,!15679,!15646)(dereferenceable) dbg:/checkout/src/libcore/ptr.rs:59 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] MOVAPSmr %noreg, 1, %noreg, target-flags(x86-tpoff) @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381, %fs, killed %394; mem:ST16[bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*)](align=32)(noalias=!15695,!15677,!15679,!15646) VR128:%394 dbg:/checkout/src/libcore/ptr.rs:222 @[ /checkout/src/libcore/ptr.rs:187 @[ /checkout/src/libcore/mem.rs:636 @[ /checkout/src/libcore/mem.rs:694 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] ] ] ] %60:gr64 = MOVPQIto64rr %390; GR64:%60 VR128:%390 dbg:/checkout/src/libcore/mem.rs:695 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] %395:vr128 = PSHUFDri %390, 78; VR128:%395,%390 dbg:/checkout/src/libcore/mem.rs:695 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] %61:gr64 = MOVPQIto64rr killed %395; GR64:%61 VR128:%395 dbg:/checkout/src/libcore/mem.rs:695 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] JE_1 %bb.121, implicit %eflags; dbg:/checkout/src/libcore/ptr.rs:59 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] JMP_1 %bb.147; dbg:/checkout/src/libcore/ptr.rs:59 @[ libstd/thread/local.rs:270 @[ libstd/thread/local.rs:296 @[ libstd/thread/local.rs:248 @[ libstd/panicking.rs:223 @[ libstd/panicking.rs:511 ] ] ] ] ] Successors according to CFG: %bb.121(0x20000000 / 0x80000000 = 25.00%) %bb.147(0x60000000 / 0x80000000 = 75.00%)I think the matching basic block in the
-print-isel-inputIR is:Details
; <label>:555: ; preds = %552 %556 = load <4 x i64>, <4 x i64>* bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*), align 32, !dbg !15680, !alias.scope !15686, !noalias !15691 store <4 x i64> <i64 1, i64 0, i64 0, i64 undef>, <4 x i64>* bitcast (<{ [40 x i8] }>* @_ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h5e656a3d08607c39E.llvm.17348687890923447381 to <4 x i64>*), align 32, !dbg !15697, !noalias !15698 %557 = extractelement <4 x i64> %556, i32 0, !dbg !15699 %558 = extractelement <4 x i64> %556, i32 2, !dbg !15699 %559 = extractelement <4 x i64> %556, i32 3, !dbg !15699 %560 = icmp eq i64 %557, 0, !dbg !15700 %561 = icmp eq i64 %558, 0, !dbg !15702 %562 = or i1 %560, %561, !dbg !15700 br i1 %562, label %573, label %563, !dbg !15700This seems to have been fixed by #51966
Reacted by Hanna KruppeBisection on the llvm side says this was fixed by... the additions for retpoline, which makes no sense, as it's not supposed to change anything if retpoline is not enabled. And as crazy as it seems, the simple fact that there's an extra PreEmitPass2 that doesn't run makes it disappear O_O.
Reacted by kennytmEven better, the retpoline patch has already been backported to llvm 6 used by rust...
So... this might, in fact, not be fixed. That is, if I build beta locally, it's not happening. But if I use the released beta, it does.
The patch that you listed as closing this (the LLVM 7 upgrade) wasn't backported to beta, so I'm unclear why you'd expect it to be fixed by it on beta. Perhaps I'm misinterpreting something?
What I'm saying is that the build environment in which rustc is built makes the problem appear or not, for the same version of rustc. There hasn't been a nightly produced since llvm 7 landed, so I did the build locally, and I marked this issue as fixed because it didn't happen on that build. But since a local build of beta with llvm 6 is also not affected, while the official beta is, that means master with llvm 7 is maybe only not affected on my end because of the compiler I'm using, and the official rustc might still be affected.
I also can't reproduce with the CI build of 64f7de9 (latest master commit as of now) so it's possible that the upgrade did in fact fix this, if not fully.
I can reproduce on a locally built beta if I use clang 6 (which is apparently what rustc automation is using) instead of gcc 5.4 (which is what's installed in the VM I'm using). And it does, indeed, not happen with master built with clang 6.
This is driving me crazy. Here's how things go in my VM:
-
rustc with llvm 6, built with clang 6.0 -> happens
-
rustc with llvm 6, built with gcc 5.4 -> doesn't happen
-
llc from llvm 6, built with clang 6.0 -> doesn't happen !
-
llc from llvm 6, built with gcc 5.4 -> happens !
-
rustc with llvm 7, built with clang 6.0 -> doesn't happen
-
rustc with llvm 7, built with gcc 6.0 -> doesn't happen
-
llc from llvm 7, built with clang 6.0 -> doesn't happen
-
llc from llvm 7, built with gcc 5.4 -> doesn't happen
At least it's more consistent with llvm 7, but considering it disappeared by adding a pass that does nothing (retpoline) while bisecting with gcc 5.4 doesn't make me confident that the actual problem is fixed, even if, in practice, it looks like it is.
-
At least I can confirm that this does indeed not happen with the now latest nightly, with llvm 7.
So...
¯\_(ツ)_/¯, I guess.
This is something I noticed when comparing two builds of Firefox on automation. Both builds were done with the same flags, same toolchains, same paths, same everything. Without any sort of caching. And yet, they differed in exactly one way: the contents of
std::panicking::default_hook.See https://taskcluster-artifacts.net/IWpRV77rTgmt-9RKwTzJiA/0/public/diff.html
I tried multiple times, and there seems to be only two variants of the generated code, and that's the only code, in all of what's generated by rust in Firefox build, that differs.
It turns out this is reproducible at a much smaller scale:
It can take a few attempts repeating the last three commands before differences show up, because there are only two variants, and you may end up getting the same variant multiple times in a row.
Cc: @alexcrichton @nikomatsakis