Repository navigation
[DWARF] Crate Debug Info incomplete? #88521
Description
Activity
@ayermolo I'm not familiar with debuginfo generation - you'll probably get a more prompt response if you ask in T-compiler on Zulip: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler
I'm curious though why this has never come up before? It sounds like a lot of tools depend on this.
Ok, I'll ask there.
No idea. As I said I am not familiar with Rust and just dealing with it's output on tools side of things. So any tips and whatnot are appreciated. Maybe my understanding of what's happening is completely wrong.Note that modules, crates, and codegen units are all different. "Modules" in Rust only affect privacy, they don't affect codegen (it sounds like you were treating LLVM modules the same as Rust modules above, and they're not the same). "Crates" are "codegen units for the rust compiler", they're all compiled at once and there's various things you can do within a crate that you can't do outside it (e.g. implement traits for a type). But the rust compiler will often split up crates into multiple LLVM codegen units before passing them to LLVM, so that it can cache them more often: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/monomorphize/collector/index.html
Can you give info on how to reproduce the problem? It sounds like you are using
-C split-debuginfo=packed -Zunstable-optionswhich is unstable and probably not widely tested.As a wild guess, maybe
-Z share-genericshelps?To me it seems that there is debug information missing on some level that represents what crate this CU belongs.
That doesn't sound like something that DWARF consumers should ever need to know.
Here's a reproduction using the xtask crate within rust-analyzer (6b77e32 with
lto = trueanddebug = 2added toCargo.toml).$ rustc +nightly --version rustc 1.57.0-nightly (ac2d9fc50 2021-09-21) $ RUSTFLAGS="-C split-debuginfo=packed -Zunstable-options -C save-temps" cargo +nightly build --release Compiling xtask v0.1.0 (/home/philipc/code/rust/rust-analyzer/xtask) error: linking dwarf objects with `rust-llvm-dwp` failed: exit status: 1 | = note: "rust-llvm-dwp" "-e" "/home/philipc/code/rust/rust-analyzer/target/release/deps/xtask-be398824d5b61cec" "-o" "/home/philipc/code/rust/rust-analyzer/target/release/deps/xtask-be398824d5b61cec.dwp" = note: = note: error: duplicate DWO ID (C6ED43DCB82FE226) in 'xtask/src/main.rs' and 'xtask/src/main.rs'The executable has 4 skeleton units pointing to the same .dwo:
UNIT<.debug_info+0x00022cfd>: length = 0x24, format = Dwarf32, version = 4, address_size = 8, abbrev_offset = 0x0 < 0><0x0000000b> DW_TAG_compile_unit DW_AT_stmt_list <.debug_line+0x0001095b> DW_AT_comp_dir /home/philipc/code/rust/rust-analyzer DW_AT_GNU_dwo_name /home/philipc/code/rust/rust-analyzer/target/release/deps/xtask-be398824d5b61cec.write_json-94b9108f588c9a54.write_json.1ae45326-cgu.0.rcgu.o.rcgu.o DW_AT_GNU_dwo_id 0xc6ed43dcb82fe226 DW_AT_GNU_ranges_base <.debug_rnglists+0x00000000> DW_AT_GNU_addr_base <.debug_addr+0x00000000> UNIT<.debug_info+0x00023013>: length = 0x34, format = Dwarf32, version = 4, address_size = 8, abbrev_offset = 0x0 < 0><0x0000000b> DW_TAG_compile_unit DW_AT_stmt_list <.debug_line+0x00010f9e> DW_AT_comp_dir /home/philipc/code/rust/rust-analyzer DW_AT_GNU_dwo_name /home/philipc/code/rust/rust-analyzer/target/release/deps/xtask-be398824d5b61cec.write_json-94b9108f588c9a54.write_json.1ae45326-cgu.0.rcgu.o.rcgu.o DW_AT_GNU_dwo_id 0xc6ed43dcb82fe226 DW_AT_GNU_ranges_base <.debug_rnglists+0x00000000> DW_AT_GNU_addr_base <.debug_addr+0x00000000> < 1><0x00000028> DW_TAG_subprogram DW_AT_name new<&str> < 1><0x0000002d> DW_TAG_subprogram DW_AT_name branch<alloc::string::String, xshell::error::Error> < 1><0x00000032> DW_TAG_subprogram DW_AT_name from_residual<xtask::dist::Patch, xshell::error::Error, anyhow::Error> < 1><0x00000037> DW_TAG_null UNIT<.debug_info+0x00029acb>: length = 0x2f, format = Dwarf32, version = 4, address_size = 8, abbrev_offset = 0x0 < 0><0x0000000b> DW_TAG_compile_unit DW_AT_stmt_list <.debug_line+0x00014b26> DW_AT_comp_dir /home/philipc/code/rust/rust-analyzer DW_AT_GNU_dwo_name /home/philipc/code/rust/rust-analyzer/target/release/deps/xtask-be398824d5b61cec.write_json-94b9108f588c9a54.write_json.1ae45326-cgu.0.rcgu.o.rcgu.o DW_AT_GNU_dwo_id 0xc6ed43dcb82fe226 DW_AT_GNU_ranges_base <.debug_rnglists+0x00000000> DW_AT_GNU_addr_base <.debug_addr+0x00000000> < 1><0x00000028> DW_TAG_subprogram DW_AT_name from_raw<(dyn std::error::Error + core::marker::Send + core::marker::Sync)> < 1><0x0000002d> DW_TAG_subprogram DW_AT_name by_ref<anyhow::error::ErrorImpl<()>> < 1><0x00000032> DW_TAG_null UNIT<.debug_info+0x00029b77>: length = 0xb6, format = Dwarf32, version = 4, address_size = 8, abbrev_offset = 0x0 < 0><0x0000000b> DW_TAG_compile_unit DW_AT_stmt_list <.debug_line+0x00014b80> DW_AT_comp_dir /home/philipc/code/rust/rust-analyzer DW_AT_GNU_dwo_name /home/philipc/code/rust/rust-analyzer/target/release/deps/xtask-be398824d5b61cec.write_json-94b9108f588c9a54.write_json.1ae45326-cgu.0.rcgu.o.rcgu.o DW_AT_GNU_dwo_id 0xc6ed43dcb82fe226 DW_AT_GNU_ranges_base <.debug_rnglists+0x00000000> DW_AT_GNU_addr_base <.debug_addr+0x00000000> < 1><0x00000028> DW_TAG_subprogram DW_AT_name from<xshell::error::Error> < 1><0x0000002d> DW_TAG_subprogram DW_AT_name alloc < 1><0x00000032> DW_TAG_subprogram DW_AT_name alloc_impl < 1><0x00000037> DW_TAG_subprogram DW_AT_name allocate < 1><0x0000003c> DW_TAG_subprogram DW_AT_name exchange_malloc < 1><0x00000041> DW_TAG_subprogram DW_AT_name new<anyhow::error::ErrorImpl<xshell::error::Error>> < 1><0x00000046> DW_TAG_subprogram DW_AT_name construct<xshell::error::Error> < 1><0x0000004b> DW_TAG_subprogram DW_AT_name from_std<xshell::error::Error> < 1><0x00000050> DW_TAG_subprogram DW_AT_name new<anyhow::error::ErrorImpl<anyhow::wrapper::MessageError<&str>>> < 1><0x00000055> DW_TAG_subprogram DW_AT_name construct<anyhow::wrapper::MessageError<&str>> < 1><0x0000005a> DW_TAG_subprogram DW_AT_name from_adhoc<&str> < 1><0x0000005f> DW_TAG_subprogram DW_AT_name from<xflags::Error> < 1><0x00000064> DW_TAG_subprogram DW_AT_name new<anyhow::error::ErrorImpl<xflags::Error>> < 1><0x00000069> DW_TAG_subprogram DW_AT_name construct<xflags::Error> < 1><0x0000006e> DW_TAG_subprogram DW_AT_name from_std<xflags::Error> < 1><0x00000073> DW_TAG_subprogram DW_AT_name new<anyhow::error::ErrorImpl<anyhow::error::ContextError<&str, xshell::error::Error>>> < 1><0x00000078> DW_TAG_subprogram DW_AT_name construct<anyhow::error::ContextError<&str, xshell::error::Error>> < 1><0x0000007d> DW_TAG_subprogram DW_AT_name from_context<&str, xshell::error::Error> < 1><0x00000082> DW_TAG_subprogram DW_AT_name new<anyhow::error::ErrorImpl<anyhow::error::ContextError<&str, anyhow::Error>>> < 1><0x00000087> DW_TAG_subprogram DW_AT_name construct<anyhow::error::ContextError<&str, anyhow::Error>> < 1><0x0000008c> DW_TAG_subprogram DW_AT_name context<&str> < 1><0x00000091> DW_TAG_subprogram DW_AT_name ext_context<&str> < 1><0x00000096> DW_TAG_subprogram DW_AT_name new<anyhow::error::ErrorImpl<anyhow::wrapper::MessageError<alloc::string::String>>> < 1><0x0000009b> DW_TAG_subprogram DW_AT_name construct<anyhow::wrapper::MessageError<alloc::string::String>> < 1><0x000000a0> DW_TAG_subprogram DW_AT_name from_adhoc<alloc::string::String> < 1><0x000000a5> DW_TAG_subprogram DW_AT_name from<std::io::error::Error> < 1><0x000000aa> DW_TAG_subprogram DW_AT_name new<anyhow::error::ErrorImpl<std::io::error::Error>> < 1><0x000000af> DW_TAG_subprogram DW_AT_name construct<std::io::error::Error> < 1><0x000000b4> DW_TAG_subprogram DW_AT_name from_std<std::io::error::Error> < 1><0x000000b9> DW_TAG_nullNote that none of these contain any definitions with addresses.
And
xtask-be398824d5b61cec.write_json-94b9108f588c9a54.write_json.1ae45326-cgu.0.rcgu.o.rcgu.ocontains 4 identicalDW_TAG_compile_unitwith thatDW_AT_GNU_dwo_id.Here's one of them:
UNIT<.debug_info+0x0004bec9>: length = 0xbf, format = Dwarf32, version = 4, address_size = 8, abbrev_offset = 0x0 < 0><0x0000000b> DW_TAG_compile_unit DW_AT_producer (indirect string, index 0x39dc): clang LLVM (rustc version 1.57.0-nightly (ac2d9fc50 2021-09-21)) DW_AT_language DW_LANG_Rust DW_AT_name (indirect string, index 0x39df): xtask/src/main.rs DW_AT_GNU_dwo_name (indirect string, index 0x39de): /home/philipc/code/rust/rust-analyzer/target/release/deps/xtask-be398824d5b61cec.write_json-94b9108f588c9a54.write_json.1ae45326-cgu.0.rcgu.o.rcgu.o DW_AT_GNU_dwo_id 0xc6ed43dcb82fe226 < 1><0x0000001c> DW_TAG_namespace DW_AT_name (indirect string, index 0x0): core < 2><0x0000001e> DW_TAG_namespace DW_AT_name (indirect string, index 0x43): ffi < 3><0x00000020> DW_TAG_enumeration_type DW_AT_type 0x00000031<.debug_info+0x0004befa> DW_AT_enum_class yes DW_AT_name (indirect string, index 0x46): c_void DW_AT_byte_size 0x00000001 DW_AT_alignment 0x00000001 < 4><0x00000028> DW_TAG_enumerator DW_AT_name (indirect string, index 0x44): __variant1 DW_AT_const_value 0x00000000 < 4><0x0000002b> DW_TAG_enumerator DW_AT_name (indirect string, index 0x45): __variant2 DW_AT_const_value 0x00000001 < 4><0x0000002e> DW_TAG_null < 3><0x0000002f> DW_TAG_null < 2><0x00000030> DW_TAG_null < 1><0x00000031> DW_TAG_base_type DW_AT_name (indirect string, index 0x4): u8 DW_AT_encoding DW_ATE_unsigned DW_AT_byte_size 0x00000001 < 1><0x00000035> DW_TAG_namespace DW_AT_name (indirect string, index 0x17): std < 2><0x00000037> DW_TAG_namespace DW_AT_name (indirect string, index 0x18): io < 3><0x00000039> DW_TAG_namespace DW_AT_name (indirect string, index 0x19): error < 4><0x0000003b> DW_TAG_enumeration_type DW_AT_type 0x00000031<.debug_info+0x0004befa> DW_AT_enum_class yes DW_AT_name (indirect string, index 0x42): ErrorKind DW_AT_byte_size 0x00000001 DW_AT_alignment 0x00000001 < 5><0x00000043> DW_TAG_enumerator DW_AT_name (indirect string, index 0x1a): NotFound DW_AT_const_value 0x00000000 < 5><0x00000046> DW_TAG_enumerator DW_AT_name (indirect string, index 0x1b): PermissionDenied DW_AT_const_value 0x00000001 < 5><0x00000049> DW_TAG_enumerator DW_AT_name (indirect string, index 0x1c): ConnectionRefused DW_AT_const_value 0x00000002 < 5><0x0000004c> DW_TAG_enumerator DW_AT_name (indirect string, index 0x1d): ConnectionReset DW_AT_const_value 0x00000003 < 5><0x0000004f> DW_TAG_enumerator DW_AT_name (indirect string, index 0x1e): HostUnreachable DW_AT_const_value 0x00000004 < 5><0x00000052> DW_TAG_enumerator DW_AT_name (indirect string, index 0x1f): NetworkUnreachable DW_AT_const_value 0x00000005 < 5><0x00000055> DW_TAG_enumerator DW_AT_name (indirect string, index 0x20): ConnectionAborted DW_AT_const_value 0x00000006 < 5><0x00000058> DW_TAG_enumerator DW_AT_name (indirect string, index 0x21): NotConnected DW_AT_const_value 0x00000007 < 5><0x0000005b> DW_TAG_enumerator DW_AT_name (indirect string, index 0x22): AddrInUse DW_AT_const_value 0x00000008 < 5><0x0000005e> DW_TAG_enumerator DW_AT_name (indirect string, index 0x23): AddrNotAvailable DW_AT_const_value 0x00000009 < 5><0x00000061> DW_TAG_enumerator DW_AT_name (indirect string, index 0x24): NetworkDown DW_AT_const_value 0x0000000a < 5><0x00000064> DW_TAG_enumerator DW_AT_name (indirect string, index 0x25): BrokenPipe DW_AT_const_value 0x0000000b < 5><0x00000067> DW_TAG_enumerator DW_AT_name (indirect string, index 0x26): AlreadyExists DW_AT_const_value 0x0000000c < 5><0x0000006a> DW_TAG_enumerator DW_AT_name (indirect string, index 0x27): WouldBlock DW_AT_const_value 0x0000000d < 5><0x0000006d> DW_TAG_enumerator DW_AT_name (indirect string, index 0x28): NotADirectory DW_AT_const_value 0x0000000e < 5><0x00000070> DW_TAG_enumerator DW_AT_name (indirect string, index 0x29): IsADirectory DW_AT_const_value 0x0000000f < 5><0x00000073> DW_TAG_enumerator DW_AT_name (indirect string, index 0x2a): DirectoryNotEmpty DW_AT_const_value 0x00000010 < 5><0x00000076> DW_TAG_enumerator DW_AT_name (indirect string, index 0x2b): ReadOnlyFilesystem DW_AT_const_value 0x00000011 < 5><0x00000079> DW_TAG_enumerator DW_AT_name (indirect string, index 0x2c): FilesystemLoop DW_AT_const_value 0x00000012 < 5><0x0000007c> DW_TAG_enumerator DW_AT_name (indirect string, index 0x2d): StaleNetworkFileHandle DW_AT_const_value 0x00000013 < 5><0x0000007f> DW_TAG_enumerator DW_AT_name (indirect string, index 0x2e): InvalidInput DW_AT_const_value 0x00000014 < 5><0x00000082> DW_TAG_enumerator DW_AT_name (indirect string, index 0x2f): InvalidData DW_AT_const_value 0x00000015 < 5><0x00000085> DW_TAG_enumerator DW_AT_name (indirect string, index 0x30): TimedOut DW_AT_const_value 0x00000016 < 5><0x00000088> DW_TAG_enumerator DW_AT_name (indirect string, index 0x31): WriteZero DW_AT_const_value 0x00000017 < 5><0x0000008b> DW_TAG_enumerator DW_AT_name (indirect string, index 0x32): StorageFull DW_AT_const_value 0x00000018 < 5><0x0000008e> DW_TAG_enumerator DW_AT_name (indirect string, index 0x33): NotSeekable DW_AT_const_value 0x00000019 < 5><0x00000091> DW_TAG_enumerator DW_AT_name (indirect string, index 0x34): FilesystemQuotaExceeded DW_AT_const_value 0x0000001a < 5><0x00000094> DW_TAG_enumerator DW_AT_name (indirect string, index 0x35): FileTooLarge DW_AT_const_value 0x0000001b < 5><0x00000097> DW_TAG_enumerator DW_AT_name (indirect string, index 0x36): ResourceBusy DW_AT_const_value 0x0000001c < 5><0x0000009a> DW_TAG_enumerator DW_AT_name (indirect string, index 0x37): ExecutableFileBusy DW_AT_const_value 0x0000001d < 5><0x0000009d> DW_TAG_enumerator DW_AT_name (indirect string, index 0x38): Deadlock DW_AT_const_value 0x0000001e < 5><0x000000a0> DW_TAG_enumerator DW_AT_name (indirect string, index 0x39): CrossesDevices DW_AT_const_value 0x0000001f < 5><0x000000a3> DW_TAG_enumerator DW_AT_name (indirect string, index 0x3a): TooManyLinks DW_AT_const_value 0x00000020 < 5><0x000000a6> DW_TAG_enumerator DW_AT_name (indirect string, index 0x3b): FilenameTooLong DW_AT_const_value 0x00000021 < 5><0x000000a9> DW_TAG_enumerator DW_AT_name (indirect string, index 0x3c): ArgumentListTooLong DW_AT_const_value 0x00000022 < 5><0x000000ac> DW_TAG_enumerator DW_AT_name (indirect string, index 0x3d): Interrupted DW_AT_const_value 0x00000023 < 5><0x000000af> DW_TAG_enumerator DW_AT_name (indirect string, index 0x3e): Unsupported DW_AT_const_value 0x00000024 < 5><0x000000b2> DW_TAG_enumerator DW_AT_name (indirect string, index 0x3f): UnexpectedEof DW_AT_const_value 0x00000025 < 5><0x000000b5> DW_TAG_enumerator DW_AT_name (indirect string, index 0x40): OutOfMemory DW_AT_const_value 0x00000026 < 5><0x000000b8> DW_TAG_enumerator DW_AT_name (indirect string, index 0x14): Other DW_AT_const_value 0x00000027 < 5><0x000000bb> DW_TAG_enumerator DW_AT_name (indirect string, index 0x41): Uncategorized DW_AT_const_value 0x00000028 < 5><0x000000be> DW_TAG_null < 4><0x000000bf> DW_TAG_null < 3><0x000000c0> DW_TAG_null < 2><0x000000c1> DW_TAG_null < 1><0x000000c2> DW_TAG_nullI'm guessing those 4 units aren't really meant to be identical, so the duplicate DWO id is causing some to be lost. A reason for that is I think the full units are meant to have entries that correspond to the
DW_AT_namein the skeleton units.Disabling LTO or using thin LTO works okay in this instance.
Reacted by jynSorry didn't get a chance to get to this before PTO. I asked someone to do it internally, if not I'll priorities once I get back.
- added a commit that references this issue
on Dec 17, 2021
Hello.
To preface this. I know nothing about RUST and its compiler. I am coming to this issue on toolchain side BOLT/DWP and dealing with debug information that is produced.
High level summary:
I am seeing multiple Skeleton CUs that have exact same DWO ID. So for example with ThinLTO when Split dwarf is enabled in main binary there are multiple CUs that point to different .dwo files, but have the same DWO ID.
Looking in to those files in to .debug_info.dwo section. They are either exactly the same, or differ by DW_AT_linkage_name.
Although LLVM can be changed to include DW_AT_linkage_name as part of the hashing algorithm, it won't solve the issue fully.
Tools like BOLT (https://github.com/facebookincubator/BOLT), or llvm-dwp rely on DWO ID to be unique.
From DWARF 5 spec (when this was all standardized), but I think also applies to DWARF4
Digging a bit in to it:
Looking in to one of the cases where only difference was DW_AT_linkage_name.
There are multiple modules, crates in rust land?, that are created. They define the same templated functions, and import the same functions.
For example looking at LLVM IR of one of the modules and one of the functions:
We have a function:
alloc_in that is part of templated raw_vec
https://doc.rust-lang.org/src/alloc/raw_vec.rs.html#188
https://doc.rust-lang.org/src/alloc/raw_vec.rs.html#115
From LinkageName this is relavant?
pub mod bar { pub use foo_bar::*; }In a different module it will have:
and on rust side:
pub mod bar2 { pub use foo_bar2::*; }I haven't looked in to cases where debug information is identical, beyond that it didn't have any functions just type information {DW_TAG_namespace , DW_TAG_enumeration_type , DW_TAG_enumerator}.
To me it seems that there is debug information missing on some level that represents what crate this CU belongs.
Any thoughts?