feat(cc): introduce py_extension rule for C/C++ Python extension modules - #3973
Draft
rickeylev wants to merge 45 commits into
Draft
feat(cc): introduce py_extension rule for C/C++ Python extension modules#3973rickeylev wants to merge 45 commits into
rickeylev wants to merge 45 commits into
Conversation
...and sycophantic toasters are bad collaborators
…into py-extension
This builds on the existing `py-extension` branch. It adds support for platform and abi tags in the resulting library filename. The `:py_extension_test`, `:py_extension_analysis_tests`, and `:py_limited_api_tests` test targets in `//tests/cc/py_extension` now build and pass. Relates to #3283
…tension (#3875) Per feedback from #3851, this PR re-works the platform detection logic to rely on platform constraints instead of the Python runtime/toolchain. ### Description This PR contains updates to the experimental `py_extension` implementation. It re-works the platform tag detection to rely on modern platform constraints, and refactors the compilation and linking mechanism to delegate to Bazel's native `cc_shared_library` and `cc_library` rules. This PR targets the main repository's `py-extension` branch (not `main`). ### Key Changes #### 1. Platform Tag & ABI Tag Derivation * **Introduced `abi_tag` to `PyCcToolchainInfo`**: Added the `abi_tag` field to the `PyCcToolchainInfo` provider, populated by the `py_cc_toolchain` rule. It defaults to deriving the tag from `python_version` (e.g. `cpython-311`) for backward compatibility. * **Constraint-Based Detection**: Replaced legacy parsing of `cc_toolchain` CPU names with direct lookup in rules_python's central `PLATFORMS` registry using platform constraints. * **Limited API Configuration**: Updated how limited API (`.abi3.so`) suffixes are appended based on the toolchain configuration. #### 2. Compilation & Linking Delegation (Refactor to `cc_shared_library`) * **Macro Fusion**: Replaced the custom linking logic in the `py_extension` rule with a macro of the same name. It now accepts C/C++ source/header files directly (`srcs`, `hdrs`, `copts`, `defines`), implicitly wrapping them in a private `cc_library` under the hood. * **Consolidated Attributes**: Consolidated static linkage under standard `deps` (removing the redundant `static_deps` attribute) and aligned linker arguments with `cc_shared_library`'s `user_link_flags`. * **Wrapper Rule (`_py_extension_wrapper`)**: Added a lightweight, private rule that wraps the `cc_shared_library` output to: * Query the Python toolchain for platform/ABI tags to resolve PEP 3149 compliant filenames (e.g. `.cpython-311-x86_64-linux-gnu.so` or `.abi3.so`). * Create a cheap `symlink` from the CSL output to the PEP 3149 name. * Propagate `PyInfo` (for python rules) and `CcSharedLibraryInfo` (for dynamic C++ dependencies). * **ODR Validation**: Leveraging native `cc_shared_library` ensures strict analysis-time validation against One Definition Rule (ODR) violations (e.g., duplicate static linkage in dynamic chains). #### 3. Runfiles and Data Support * **Runfiles Propagation**: Re-implemented runfiles collection using the repository's standard `builders.RunfilesBuilder` to support runtime assets (`data` attribute) and dynamic library dependency propagation. ### Verification & Testing * Added parity test cases in `dependency_graph_tests.bzl` to verify dynamic dependency chains and static sharing behavior. * Added runfiles validation in `py_extension_tests.bzl` to verify data asset propagation. * Updated limited API tests to match CSL output structure. * All 14 tests in `//tests/cc/py_extension/...` are compiling and passing. ────── TAG=agy CONV=23f8a5e8-2d99-401a-9903-1256b7d42b0e --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Why: Merge the latest upstream changes into py-extension to keep the branch up to date and compatible with Starlarkified rules_cc in Bazel 9. How: - Resolve conflict in dev/pyproject.toml and remove stale docs/requirements.txt. - Explicitly load CcSharedLibraryInfo from rules_cc. - Use cc_common.link without main_output to avoid private API restrictions, naming the intermediate library with a leading underscore and .pyextimpl suffix. - Add missing bzl_library targets and fix header includes in tests.
Why: Integrate upstream/py-extension (#3875) platform detection logic and cc_shared_library macro wrapper architecture into our local branch. How: - Resolve conflict in py_extension_rule.bzl in favor of upstream wrapper implementation. - Preserve runfiles dynamic lookup test in py_extension_test.py. - Fix function docstring header formatting in py_cc_toolchain_rule.bzl.
…e type annotations
Why:
Mark py_extension as a public but volatile API in documentation and improve docstring parameter type readability.
How:
- Add volatile API inclusion block to py_extension macro docstring.
- Cross-reference py_library.imports using {attr}.
- Add {type} annotations for all parameters and remove redundant Optional wording.
…3971) Per feedback from #3875 , this PR makes a few changes: * Removed an unnecessary hasattr check on py_cc_toolchain * Renamed the arbitrary "os" and "cpu" identifiers to terms standardized in PEP 508. Also marks `py_extension` as experimental. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Richard Levasseur <richardlev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Why: The py_cc_toolchain_rule.bzl loads flags.bzl, but :flags was missing from the deps attribute of //python/private:py_cc_toolchain_rule, breaking starlark doc extraction in docs build test. How: Add :flags to deps of //python/private:py_cc_toolchain_rule in python/private/BUILD.bazel.
…brary Why: py_cc_toolchain_macro.bzl loads //python/private/pypi:pep508_env.bzl, but the dependency was missing from the py_cc_toolchain_macro target in //python/private:BUILD.bazel. How: Add //python/private/pypi:pep508_env to deps of py_cc_toolchain_macro in python/private/BUILD.bazel.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
rickeylev
marked this pull request as draft
July 29, 2026 03:48
…D file Why: CI failed because tests/cc/py_extension/BUILD.bazel failed Buildifier checks and python/private/cc/BUILD.bazel referenced invalid bzl_library target names from @rules_cc. How: - Run buildifier to format tests/cc/py_extension/BUILD.bazel. - Update py_extension_macro and py_extension_rule bzl_library deps in python/private/cc/BUILD.bazel to use correct target names.
Why: WORKSPACE mode builds failed on Buildkite because bzlmod-only target names were used for rules_cc bzl_library dependencies. How: Use @rules_cc//cc:core_rules and @rules_cc//cc/common in python/private/cc/BUILD.bazel, which are supported in both WORKSPACE mode and bzlmod mode.
Why: Linking C extension shared libraries (.dylib) on macOS failed with undefined Python C symbols because ld requires -undefined dynamic_lookup when symbols are resolved at runtime. How: Add default user_link_flags with -undefined dynamic_lookup for @platforms//os:osx and @platforms//os:macos in py_extension_macro.bzl.
Why: GitHub actions CI failed on the ruff format check due to an extra blank line in py_extension_pkg_test.py. How: Reformat tests/cc/py_extension/py_extension_pkg_test.py using ruff format.
Collaborator
Author
|
cc @rsartor-cmd : something that you might run into integrating on the protobuf side: 65886db tldr, some flags necessary for building on mac are missing (that commit adds them) |
Why: py_extension_bzl is superseded by the py_extension bzl_library target in //python/cc:BUILD.bazel. How: Remove the py_extension_bzl bzl_library target definition from python/cc/BUILD.bazel.
Why: pyelftools is ELF-specific and fails when parsing Mach-O (.dylib on macOS) or PE (.dll on Windows) dynamic libraries. How: Decorate test_inspect_elf with @unittest.skipIf(sys.platform != 'linux', ...) in tests/cc/py_extension/py_extension_test.py.
…ames Why: Analysis tests in py_extension_tests.bzl and py_cc_toolchain_tests.bzl failed on macOS because extension file basenames and toolchain target platform tags vary across operating systems. How: Set explicit platform_machine and sys_platform on fake_py_cc_toolchain_impl in tests/support/cc_toolchains/BUILD.bazel, and use file_path_matches pattern matching for extension filenames in tests/cc/py_extension/py_extension_tests.bzl.
Why: On macOS, Building ext_static under config_settings libc=musl still targets macOS (producing ext_static.cpython-311-darwin.so). How: Update test_musl_platform_impl in tests/cc/py_extension/py_extension_tests.bzl to match ext_static.cpython-311-*.
Why: MSVC link.exe failed to link C extension DLLs on Windows because no symbols were exported, causing LNK1104. How: Add /EXPORT:PyInit_<module_name> to user_link_flags for @platforms//os:windows in py_extension_macro.bzl.
Why: MSVC link.exe failed on Windows with LNK1120 unresolved externals because Python import libraries (.lib) were not included in cc_shared_library deps. How: Append @rules_python//python/cc:current_py_cc_libs to cc_shared_library csl_deps for @platforms//os:windows in py_extension_macro.bzl.
Why: MSVC link.exe failed with LNK1120 unresolved externals because the implicit _impl cc_library lacked Python import libraries on Windows. How: Add current_py_cc_libs to py_cc_headers_and_win_libs for Windows platform in py_extension_macro.bzl.
…encies Why: MSVC link.exe on Windows failed with LNK1120 unresolved externals because default exports_filter excluded external C import libraries (@...). How: Set default exports_filter to ["//...", "@..."] for @platforms//os:windows in py_extension_macro.bzl when not explicitly provided.
Why: Trigger a clean Buildkite build on PR 3973 for the exports_filter Windows fix. How: Empty commit to force Buildkite pipeline run.
Why: In Bazel cc_shared_library, label patterns like //... throw invalid label syntax errors during relative label parsing on Windows. How: Construct package_prefix using native.package_name() so exports_filter entries are valid package prefixes on Windows.
…lter on Windows
Why:
On Windows, exports_filter filtered out @rules_python//python/cc:current_py_cc_libs because hardcoded repository strings did not match canonical Bzlmod repository names, causing MSVC link.exe LNK1104/LNK2001 errors. Also remove unnecessary /EXPORT flag on Windows as PyMODINIT_FUNC already includes __declspec(dllexport).
How:
Pass str(Label("//python/cc:current_py_cc_libs")) to exports_filter and remove /EXPORT from user_link_flags on Windows.
Why:
Hermetic Python import libraries (e.g., python311.lib) reside in the root package of external toolchain repositories. Including //:__subpackages__ in exports_filter matches root package targets across repositories without using invalid wildcard labels.
How:
Pass :__subpackages__ and str(Label("//:__subpackages__")) to default exports_filter for Windows in py_extension_macro.bzl.
…r on Windows
Why:
current_py_cc_libs is located in package python/cc. Including //python/cc:__subpackages__ in exports_filter on Windows ensures cc_shared_library matches and includes current_py_cc_libs without dropping python311.lib.
How:
Pass :__subpackages__ and str(Label("//python/cc:__subpackages__")) to default exports_filter for Windows in py_extension_macro.bzl.
Why: cc_shared_library exports_filter filters all deps against package rules. External python C import libraries (python311.lib, python3.lib) from external toolchain repositories were being excluded by exports_filter. Passing current_py_cc_libs via additional_linker_inputs bypasses exports_filter while ensuring link.exe receives the python import libraries. How: Pass current_py_cc_libs via additional_linker_inputs for @platforms//os:windows in py_extension_macro.bzl.
…lt exports_filter on Windows Why: native.package_relative_label transforms string label entries in exports_filter into Label objects. Including @rules_python//python/cc:current_py_cc_libs in default exports_filter on Windows ensures cc_shared_library exports current_py_cc_libs without dropping python311.lib or python3.lib. How: Pass :__subpackages__ and @rules_python//python/cc:current_py_cc_libs in default exports_filter for Windows in py_extension_macro.bzl.
Why: cc_shared_library additional_linker_inputs expects targets that provide DefaultInfo with files. Returning DefaultInfo containing the .lib import library files from py_cc_toolchain.libs[CcInfo] enables passing current_py_cc_libs via additional_linker_inputs on Windows, ensuring link.exe receives python import libraries without being filtered by exports_filter. How: 1. Update _current_py_cc_libs_impl in current_py_cc_libs.bzl to extract static, interface, and dynamic libraries from CcInfo and return them in DefaultInfo. 2. Pass current_py_cc_libs via additional_linker_inputs on @platforms//os:windows in py_extension_macro.bzl.
…c_libs Why: In WORKSPACE mode, py_cc_toolchain.libs is a struct, so 'CcInfo in py_cc_toolchain.libs' raises a Starlark type error. Additionally, py_cc_toolchain.libs.providers_map.values() may already contain DefaultInfo, causing duplicate provider key errors if a second DefaultInfo is appended. How: Use py_cc_toolchain.libs.providers_map.get(CcInfo) to retrieve CcInfo safely, and filter out existing DefaultInfo instances from providers before appending custom DefaultInfo.
…atibility Why: In Bazel 7, dict keys in providers_map are strings or types that cause 'unhashable type: Provider' errors when passed to providers_map.get(CcInfo). How: Find CcInfo provider in providers_map.values() by checking for the 'linking_context' attribute (hasattr(p, 'linking_context')), which is uniquely present on CcInfo across all Bazel versions.
Why: Re-running individual jobs retained cached failure states from build 16556. How: Trigger fresh buildkite pipeline build.
Why: Re-creating DefaultInfo without specifying default_runfiles and data_runfiles dropped runfiles provided by py_cc_toolchain.libs, causing test_current_toolchain_libs to fail when asserting runfiles content. How: Extract default_runfiles and data_runfiles from the original DefaultInfo in py_cc_toolchain.libs.providers_map and pass them to the new DefaultInfo.
…cc_libs Why: On Windows, MSVC link.exe requires import libraries (.lib) provided in the original DefaultInfo.files of py_cc_toolchain.libs. Omission of these files caused LNK1104 cannot open file python311.lib linking errors on Windows. How: Pass original DefaultInfo.files as transitive depset into the new DefaultInfo returned by current_py_cc_libs.
Why: MSVC link.exe relies on python3XX.lib import library paths on the command line when linking extension DLLs. cc_shared_library dropped import libraries coming from cc_library deps, causing LNK1104 cannot open file python311.lib linking errors on Windows. How: Add to effective_user_link_flags for Windows in py_extension_macro so import library paths are passed explicitly on the link.exe command line.
…urrent_py_cc_libs Why: Including python3XX.dll (dynamic_library) alongside python3XX.lib (interface_library) in DefaultInfo.files caused MSVC link.exe to receive .dll files on the link command line, triggering LNK1107 invalid or corrupt file errors on Windows. How: Prefer interface_library or static_library in DefaultInfo.files and exclude dynamic_library (.dll) when interface_library (.lib) is present.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(cc): introduce py_extension rule for C/C++ Python extension modules
Building C/C++ Python extension modules currently requires manually
configuring complex
cc_binaryorcc_shared_librarytargets andsetting up platform, ABI3, and extension tags. The new
py_extensionrule provides a high-level Starlark API for compilingnative extensions that integrate seamlessly with
rules_pythontoolchains and
py_librarydependencies.Closes #3283.