You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support Apple .dSYM bundles in symbol packages/indexing #92911
Apple's supported symbol file format is the .dSYM bundle. A .dSYM bundle is a directory tree in a specific format that contains various files, including the DWARF format symbols for the executable, as well as various meta-data. In the macOS Finder it appears to be a single file (but can be explored using the "Show Package Contents" command). Confusingly, much documentation uses the term ".dSYM file", which is used essentially interchangeably with ".dSYM bundle". Note, however, that the .dSYM is always a bundle (directory tree), never a single file.
The .NET build process currently creates a single file for each executable file's symbols, with the file extension .dwarf. This is not the standard way that symbol files are created and used on macOS. There is some concern that this might not be supported long-term. There may also be developer experience improvements to using .dSYM format symbols, such as automatic Spotlight integration for symbol finding.
Creating a .dwarf file is done by running the dsymutil --flat command on an executable. Simply removing the --flat argument during the build process allows creating .dSYM bundles. However, there are lots of parts of the .NET build and packaging process that need to be updated to deal with the fact that symbols are now directory trees and not just single files.
It is best to consider a .dSYM bundle as opaque: do not make assumptions about the structure and contents of the bundle. There may be reasons to break this rule. For instance, one file inside the .dSYM bundle is the DWARF symbol file. We might want to read that file directly in order to index the symbols. However, even that might be better done by addressing the .dSYM bundle as a whole, e.g., using the following commands to get the build UUID for the contained symbols:
A description of some of the challenges in converting the .NET build to using .dSYM bundles follows.
MSBuild files versus directories
The .NET build uses msbuild for most activities. MSBuild has a strong bias towards files and not directories, at least when it comes to item glob expressions. For example, using:
<Files Include="*.pdb;*.dSYM"/>
Will include all the *.pdb files in the directory, but will ignore any *.dSYM bundles.
Using:
<Files Include="*.dSYM/**"/>
Will include all the files in the .dSYM bundle in the Files item, individually. This can be useful sometimes, but often it is useful to treat the .dSYM bundle as a "single", "whole" thing, and not as a set of pieces.
MSBuild item transform globbing
A lot of MSBuild code does transforms on items to form new items. One example is to transform a list of generated executables to a list of the related symbol files (or bundles):
Note: if using the "expand all files" option in MSBuild, it seems to be very difficult to then associate a per-.dSYM attribute to each of the expanded files, such as the target directory each should be copied to in the NuGet package.
MSBuild copy directories
MSBuild has a built-in Copy task to copy files. However, to copy directories, preserving directory structure, requires a bit of a hack. A December 2023 change to the Copy task did add folder tree copying that might be helpful, but it isn't available in our build tools yet, and it requires distinguishing files and directories to be copied.
If we want to treat .dSYM bundles as an atomic unit, we might need to implement a CopyFileOrFolder custom MSBuild task that can handle either files (e.g., *.pdb) or folders (e.g. *.dSYM) and copy them appropriately.
NuGet packing
.NET packs native builds and symbols into NuGet "legacy" symbols packages. For example, the Microsoft.NETCore.App.Runtime.osx-arm64.9.0.0-dev.symbols.nupkg package contains (this is a small subset):
To support .dSYM, these .dwarf files would need to be replaced with their full .dSYM bundle replacements.
It appears that NuGet itself has assumptions about symbol files being single files with a distinguished file extension. Neither of these things are true for .dSYM bundles: they are directories, and the files they contain both have common file extensions not specific to debug information (e.g., ".plist", ".yml") and have no file extension at all (for the DWARF info file).
The MSBuild Shared Framework SDK file in the arcade repo, in src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets, defines an MSBuild property that determines how NuGet packages symbols. Specifically:
This is passed through many layers before ending up being used in the NuGet client packing code to determine how to pack symbols packages, here.
It needs more investigation to determine if NuGet needs to learn about .dSYM bundles as directories with multiple files, or whether all the packing changs that are needed can be accomplished without touching NuGet.
Symbol server uploading
One use for the generated symbols packages is apparently to give to the symbol server indexer. It cracks the NuGet package, looks for symbol files, and index and uploads the files. It is presumed that if the symbol package contains a .dSYM bundle, including the DWARF debug info file, that the symbol indexer will be able to understand that the debug info file actually is a debug info file (despite having no file extension), and handle it properly. This might need to be verified.
Question: where is the symbol server uploading code?
It is a separate issue about whether the symbol indexer should be augmented such that (1) the entire .dSYM bundle is indexed, (2) Mac native debuggers (lldb) can use the symbol server to automatically find and download the archived .dSYM bundle.
dSYM-in-a-file
It's easy to get the build to produce .dSYM bundles for native code symbols. The biggest problems seem to be in how to package the bundles in NuGet. It has been suggested that perhaps a halfway point would be to (1) produce the .dSYM bundle, then (2) immediately compress the bundle into a single file using zip or tar, e.g., create a .dSYMzip file. This could be packaged like we currently package .dwarf files. Since it's a single file, it might be easier to deal with than teaching all layers of the build system about folders. It would almost certainly require changes to the symbol server indexer.
MSBuild complexity
General comment on working in MSBuild in the .NET build system: the rules that affect the build are spread out widely, between the eng tree, the src/installer tree, the arcade repo (both MSBuild files and custom MSBuild targets), and the NuGet source tree. It is extremely difficult to find documentation on many MSBuild properties and items, determine where they are defined and used, and generally understand the order of operations. There is no IDE support nor any debugger.
It's easy to get the build to produce .dSYM bundles for native code symbols. The biggest problems seem to be in how to package the bundles in NuGet. It has been suggested that perhaps a halfway point would be to (1) produce the .dSYM bundle, then (2) immediately compress the bundle into a single file using zip or tar, e.g., create a .dSYMzip file. This could be packaged like we currently package .dwarf files. Since it's a single file, it might be easier to deal with than teaching all layers of the build system about folders. It would almost certainly require changes to the symbol server indexer.
I'm not sure if dSYM bundles can have symlinks, but if they can, then they can't be stored as-is in nupkgs (for at least two reasons: they're not preserved correctly when expanded and end up being the target file instead of the symlink, and it wouldn't work on Windows anyway).
The solution here is to store the .dSYM as a zip file as you've suggested. This also works around all the other problems where .dSYM bundles are directories.
Activity
BruceForstall commentedon Feb 25, 2024
Some notes on this issue:
Apple's supported symbol file format is the
.dSYMbundle. A.dSYMbundle is a directory tree in a specific format that contains various files, including the DWARF format symbols for the executable, as well as various meta-data. In the macOS Finder it appears to be a single file (but can be explored using the "Show Package Contents" command). Confusingly, much documentation uses the term ".dSYM file", which is used essentially interchangeably with ".dSYM bundle". Note, however, that the.dSYMis always a bundle (directory tree), never a single file.The .NET build process currently creates a single file for each executable file's symbols, with the file extension
.dwarf. This is not the standard way that symbol files are created and used on macOS. There is some concern that this might not be supported long-term. There may also be developer experience improvements to using.dSYMformat symbols, such as automatic Spotlight integration for symbol finding.Creating a .dwarf file is done by running the
dsymutil --flatcommand on an executable. Simply removing the--flatargument during the build process allows creating.dSYMbundles. However, there are lots of parts of the .NET build and packaging process that need to be updated to deal with the fact that symbols are now directory trees and not just single files.It is best to consider a
.dSYMbundle as opaque: do not make assumptions about the structure and contents of the bundle. There may be reasons to break this rule. For instance, one file inside the.dSYMbundle is the DWARF symbol file. We might want to read that file directly in order to index the symbols. However, even that might be better done by addressing the.dSYMbundle as a whole, e.g., using the following commands to get the build UUID for the contained symbols:A description of some of the challenges in converting the .NET build to using
.dSYMbundles follows.MSBuild files versus directories
The .NET build uses msbuild for most activities. MSBuild has a strong bias towards files and not directories, at least when it comes to item glob expressions. For example, using:
Will include all the
*.pdbfiles in the directory, but will ignore any*.dSYMbundles.Using:
Will include all the files in the
.dSYMbundle in theFilesitem, individually. This can be useful sometimes, but often it is useful to treat the.dSYMbundle as a "single", "whole" thing, and not as a set of pieces.MSBuild item transform globbing
A lot of MSBuild code does transforms on items to form new items. One example is to transform a list of generated executables to a list of the related symbol files (or bundles):
This is a 1-to-1 mapping. However, if we need to expand this to the full list of files in the bundle (instead of just the bundle itself), we might do:
(Note the
/**wildcard.)This doesn't work, however, as the "globbing" does not get expanded by MSBuild during the transform. A workaround is to "go through" a property:
Note: if using the "expand all files" option in MSBuild, it seems to be very difficult to then associate a per-.dSYM attribute to each of the expanded files, such as the target directory each should be copied to in the NuGet package.
MSBuild copy directories
MSBuild has a built-in
Copytask to copy files. However, to copy directories, preserving directory structure, requires a bit of a hack. A December 2023 change to theCopytask did add folder tree copying that might be helpful, but it isn't available in our build tools yet, and it requires distinguishing files and directories to be copied.If we want to treat
.dSYMbundles as an atomic unit, we might need to implement aCopyFileOrFoldercustom MSBuild task that can handle either files (e.g.,*.pdb) or folders (e.g.*.dSYM) and copy them appropriately.NuGet packing
.NET packs native builds and symbols into NuGet "legacy" symbols packages. For example, the
Microsoft.NETCore.App.Runtime.osx-arm64.9.0.0-dev.symbols.nupkgpackage contains (this is a small subset):To support
.dSYM, these.dwarffiles would need to be replaced with their full.dSYMbundle replacements.It appears that NuGet itself has assumptions about symbol files being single files with a distinguished file extension. Neither of these things are true for
.dSYMbundles: they are directories, and the files they contain both have common file extensions not specific to debug information (e.g., ".plist", ".yml") and have no file extension at all (for the DWARF info file).The MSBuild Shared Framework SDK file in the arcade repo, in src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets, defines an MSBuild property that determines how NuGet packages symbols. Specifically:
This is passed through many layers before ending up being used in the NuGet client packing code to determine how to pack symbols packages, here.
It needs more investigation to determine if NuGet needs to learn about
.dSYMbundles as directories with multiple files, or whether all the packing changs that are needed can be accomplished without touching NuGet.Symbol server uploading
One use for the generated symbols packages is apparently to give to the symbol server indexer. It cracks the NuGet package, looks for symbol files, and index and uploads the files. It is presumed that if the symbol package contains a
.dSYMbundle, including the DWARF debug info file, that the symbol indexer will be able to understand that the debug info file actually is a debug info file (despite having no file extension), and handle it properly. This might need to be verified.Question: where is the symbol server uploading code?
It is a separate issue about whether the symbol indexer should be augmented such that (1) the entire
.dSYMbundle is indexed, (2) Mac native debuggers (lldb) can use the symbol server to automatically find and download the archived.dSYMbundle.dSYM-in-a-file
It's easy to get the build to produce
.dSYMbundles for native code symbols. The biggest problems seem to be in how to package the bundles in NuGet. It has been suggested that perhaps a halfway point would be to (1) produce the.dSYMbundle, then (2) immediately compress the bundle into a single file using zip or tar, e.g., create a.dSYMzipfile. This could be packaged like we currently package .dwarf files. Since it's a single file, it might be easier to deal with than teaching all layers of the build system about folders. It would almost certainly require changes to the symbol server indexer.MSBuild complexity
General comment on working in MSBuild in the .NET build system: the rules that affect the build are spread out widely, between the
engtree, thesrc/installertree, the arcade repo (both MSBuild files and custom MSBuild targets), and the NuGet source tree. It is extremely difficult to find documentation on many MSBuild properties and items, determine where they are defined and used, and generally understand the order of operations. There is no IDE support nor any debugger.BruceForstall commentedon Feb 26, 2024
A partial implementation, or, a set of attempts: #96456
[-]Support dsym bundles in symbol packages/indexing[/-][+]Support Apple .dSYM bundles in symbol packages/indexing[/+]Add build option to build Mac .dSYM debug symbol bundles
Add build option to build Mac .dSYM debug symbol bundles
Add build option to build Mac .dSYM debug symbol bundles (#100617)
Add build option to build Mac .dSYM debug symbol bundles (dotnet#100617)
Add build option to build Mac .dSYM debug symbol bundles (dotnet#100617)
rolfbjarne commentedon Jun 18, 2025
I'm not sure if dSYM bundles can have symlinks, but if they can, then they can't be stored as-is in nupkgs (for at least two reasons: they're not preserved correctly when expanded and end up being the target file instead of the symlink, and it wouldn't work on Windows anyway).
The solution here is to store the .dSYM as a zip file as you've suggested. This also works around all the other problems where .dSYM bundles are directories.