Repository navigation
ICE: unstable fingerprints for native_libraries #85502
Description
Code
Compiling oasis-core with nightly 2021-05-06 and incremental compilation enabled. Seems to happen during compilation of grpcio v0.8.2.
Meta
rustc --version --verbose:
rustc 1.54.0-nightly (676ee1472 2021-05-06)
binary: rustc
commit-hash: 676ee14729462585b969bbc52f32c307403f4126
commit-date: 2021-05-06
host: x86_64-unknown-linux-gnu
release: 1.54.0-nightly
LLVM version: 12.0.0
Error output
Compiling oasis-core-runtime v0.3.0-alpha (/workdir/runtime)
Compiling oasis-core-keymanager-api-common v0.3.0-alpha (/workdir/keymanager-api-common)
Compiling grpcio v0.8.2
Compiling oasis-core-client v0.3.0-alpha (/workdir/client)
Compiling oasis-core-keymanager-client v0.3.0-alpha (/workdir/keymanager-client)
thread 'rustc' panicked at 'assertion failed: `(left == right)`
left: `Some(Fingerprint(3744085167471343643, 10066438777687008338))`,
right: `Some(Fingerprint(10047237832729537343, 9265653581121216388))`: found unstable fingerprints for native_libraries(grpcio_sys[1373]): [NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_civil_time"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_throw_delegate"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_city"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_time"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_raw_hash_set"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_bad_variant_access"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("upb"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("re2"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_str_format_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_hashtablez_sampler"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_log_severity"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_hash"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_graphcycles_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_synchronization"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_bad_optional_access"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("address_sorting"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_int128"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("z"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_strings"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_statusor"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_symbolize"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("gpr"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_stacktrace"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_debugging_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("grpc"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_exponential_biased"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_raw_logging_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_status"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("cares"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_demangle_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_base"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_malloc_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_time_zone"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_cord"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_strings_internal"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("absl_spinlock_wait"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("ssl"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("crypto"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Static { bundle: None, whole_archive: None }, name: Some("grpc_wrap"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }, NativeLib { kind: Unspecified, name: Some("stdc++"), cfg: None, foreign_module: None, wasm_import_module: None, verbatim: None }]', /rustc/676ee14729462585b969bbc52f32c307403f4126/compiler/rustc_query_system/src/query/plumbing.rs:585:5
stack backtrace:
<snip>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: rustc 1.54.0-nightly (676ee1472 2021-05-06) running on x86_64-unknown-linux-gnu
note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental -C target-feature=+aes,+ssse3 --crate-type lib
note: some of the compiler flags provided by cargo are hidden
query stack during panic:
#0 [native_libraries] looking up the native libraries of a linked crate
end of query stack
Backtrace
0: rust_begin_unwind
at /rustc/676ee14729462585b969bbc52f32c307403f4126/library/std/src/panicking.rs:493:5
1: core::panicking::panic_fmt
at /rustc/676ee14729462585b969bbc52f32c307403f4126/library/core/src/panicking.rs:92:14
2: core::panicking::assert_failed_inner
3: core::panicking::assert_failed
4: rustc_query_system::query::plumbing::incremental_verify_ich
5: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
6: rustc_query_system::query::plumbing::get_query_impl
7: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::native_libraries
8: rustc_codegen_ssa::base::<impl rustc_codegen_ssa::CrateInfo>::new
9: rustc_codegen_ssa::back::write::start_async_codegen
10: rustc_codegen_ssa::base::codegen_crate
11: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
12: rustc_interface::passes::QueryContext::enter
13: rustc_interface::queries::Queries::ongoing_codegen
14: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
15: rustc_span::with_source_map
16: rustc_interface::interface::create_compiler_and_run
17: scoped_tls::ScopedKey<T>::set
Activity
Aaron1011 commentedon May 20, 2021
@kostko: Looking at the log in https://buildkite.com/oasisprotocol/oasis-core-ci/builds/4881#_, I can only see
oasis-core-clientbeing built once. For this crash to occur, there needs to be an existingtarget/debug/incrementalortarget/release/incrementaldirectory. Is your CI setup performing any kind of caching across runs? If so, can you link me to the code that performs that?kostko commentedon May 20, 2021
Yes, it uses caching (used until we disabled it in the PR linked above) by sharing the
incrementalcompile caches across invocations.Aaron1011 commentedon May 20, 2021
@kostko: I'm not familiar with Buildkit - does that share the
incrementaldirectories across different builds (e.g. of different PRs or commits), or just across the different steps (e.g. 'Lint Rust crate versions' and 'Build Rust runtime loader') in a build like https://buildkite.com/oasisprotocol/oasis-core-ci/builds/4881#_ ?kostko commentedon May 20, 2021
Oh so we have custom Buildkite agents and these builds are running in a Docker container where the
incrementaldirectories are bound to a common directory on the host (agent). So the cache is shared across different builds (e.g., different PRs/commits/etc.) that happen on the same agent not only across different steps.Aaron1011 commentedon May 20, 2021
This issue should be fixed by #83901 (which isn't part of the
2021-05-06nightly build). Try updating to the latest nightly - if the issue occurs again, let me know.kostko commentedon May 20, 2021
Cool, will try the most recent nightly.
kostko commentedon May 24, 2021
Tried with
f94942d84 2021-05-19(and clearing the IC caches) but it still seems to hit the same problem (example).Aaron1011 commentedon May 26, 2021
PR #85702 fixes another issue that affects
native_libraries. However, it's possible that there are still more undiscovered issues affectednative_libraries.Aaron1011 commentedon Jun 2, 2021
The latest nightly contains a fix for another bug that affects
native_libraries. @kostko Could you upgrade your Rust nightly again?Enselic commentedon Sep 7, 2023
Triage: Assuming this is fixed since there was no follow-up. Closing.