fix(build): link libc after the IREE archives so integration tests link - #1275
Merged
Merged
Conversation
Any integration test built with `xla-iree` failed to link on aarch64:
/usr/bin/ld: libiree_runtime_unified.a(call.c.o): undefined reference to
symbol '__stack_chk_guard@@GLIBC_2.17'
/usr/bin/ld: /lib/ld-linux-aarch64.so.1: error adding symbols:
DSO missing from command line
`rustc-link-arg` can only append, so the IREE archives land after rustc's own
`-lc`. `call.c.o` is built with the stack protector and references
`__stack_chk_guard`, and that reference therefore appears with no libc left
after it. The symbol is not in libc itself (`libc.so.6` lists it as UND); the
definition is in `ld-linux-aarch64.so.1`, reachable through libc's DT_NEEDED,
which is why ld names the dynamic linker rather than libc. Repeating `-lc`
after the archives puts libc where the pending reference can reach it.
Established by ablation rather than assumed: the trailing `-lc` alone fixes the
link, and `-Wl,--copy-dt-needed-entries` alone does not, because that flag only
governs inputs following it and rustc appends our args after its `-lc`. The
policy flag is therefore not shipped; it would have been a global relaxation
that can hide a genuinely missing `-l` elsewhere in the same link, for no gain.
Verified by linking four targets: `molmo2_xla_vision_parity` and
`cli_help_consistency` under `cuda,xla-iree` (both previously failed),
`molmo2_xla_vision_parity` under `xla-diagnostics` and `chat_template_kwargs`
under `cuda,xla-iree` (both previously passed, still pass), plus
`cargo check --features cuda --all-targets` for the default build.
The `IREE_DIST` and macOS recipes are left alone. `IREE_DIST` emits a nearly
identical group and is likely affected the same way, but neither an
`IREE_DIST` tree nor a macOS host is available here to confirm it by an actual
link, and this file should not carry unverified link changes.
Also drops the `link_args.insert(5, ...)` magic index. Adding one entry ahead
of it would have moved the vendored printf archive out of the group silently,
so the conditional entry is now pushed positionally, and each library in the
group carries the reason it is there.
inureyes
added
type:bug
4 tasks
inureyes
added a commit
that referenced
this pull request
Aug 21, 2026
`TECHNICAL_REPORTS/.keep-reports` opts this repository into tracking reports, and the report is meant to land inside the squash merge rather than trail it. These three merged without one, so they are added here after the fact. The reports carry the parts that do not survive in a diff. #916's recorded blocker was a TF32 artifact in the MLX oracle rather than an emitter defect, and its gate had only ever run on a single-crop solid-color image. #1275's first fix attempt shipped a redundant global linker flag until each half was ablated separately. #1273 is trivial as a change and only interesting because a compile error sat in `main` unseen, which is what #1270 tracks. Both language versions are included, matching the existing convention.
inureyes
added a commit
that referenced
this pull request
Aug 22, 2026
…ot reach main (#1305) ## Summary `xla-compile` runs `cargo check`, which never invokes the linker, so a regression in the IREE link recipe in `build.rs` can reach `main` unobserved; #1274 was exactly that failure, found by hand after the fact rather than by CI. This PR adds a new `xla-link` job that actually links an OpenXLA target on the self-hosted GB10 runner, closing the gap `xla-compile`'s own comment already recorded as open. ## What changed - `changes` job: a new `xla_link` output from a sibling `dorny/paths-filter` filter keyed on `build.rs`, `src/lib/mlxcel-xla/build.rs`, `src/lib/mlxcel-xla/csrc/**`, `scripts/iree/**`, `rust-toolchain.toml`, and `.github/workflows/ci.yml`. That is narrower than the existing `rust` filter, so the link job does not run on every Rust PR, but it covers the whole causal surface of a link failure: the root `build.rs` holds the IREE recipe, `scripts/iree/**` pins the distribution whose archive set the recipe names, `mlxcel-xla`'s build script and its `csrc/**` sources produce the shim object whose undefined symbols those archives resolve, and `rust-toolchain.toml` is on the path because #1274 was entirely about where rustc places its own `-lc` relative to appended `rustc-link-arg` entries. - New `xla-link` job, placed immediately after `xla-compile`, `needs: changes`, gated on `github.repository == 'lablup/mlxcel' && needs.changes.outputs.xla_link == 'true'`, `runs-on: GB10`, `permissions: contents: read`, `timeout-minutes: 120`, a job-scoped `concurrency` group keyed per PR with `cancel-in-progress`, and its own `CARGO_TARGET_DIR` (`$HOME/.cargo-target/mlxcel-xla-link-ci`) separate from `xla-compile`'s and from the release job's. - The link step runs `cargo test --release --features cuda,xla-iree --test xla_prepared_prefill --no-run`. `--release` is mandatory because the debug profile cannot link these targets on this host at all (hundreds of `relocation truncated to fit: R_AARCH64_CALL26` errors against ordinary libstd symbols, the unoptimized binary exceeding the AArch64 direct-branch range). `--no-run` links without executing, so the job never touches the GPU. Because cargo also builds the package's `[[bin]]` targets whenever an integration test is selected, this command links `mlxcel`, `mlxcel-server`, `speculative_bench`, and `mlxcel-bench-decode` in addition to the test binary, so it is a superset of the `--bin mlxcel-server` alternative rather than a cheaper substitute for it. `RUSTFLAGS` is deliberately left unset, unlike `xla-compile`, so a red run here is unambiguously a link failure rather than a lint failure. - The `concurrency` group exists because `xla-link` is the first job on the shared GB10 runner whose warm cost is minutes rather than seconds. Without it, three pushes to a `build.rs` PR queue three link jobs of up to 120 minutes each on the one runner that also serves `cargo-clippy` for every Rust PR and the release build. - Comment blocks recording the trigger rationale, the measured cost, what the gate was demonstrated on, and the residual coverage gaps: the `IREE_DIST` and macOS `IREE_MACOS_HOME` recipes in `build.rs` remain unverified because no runner in this repository holds either distribution, and a link regression arriving through a path outside the filter is still not caught at PR time. - Corrected the note above the `clippy` job, which asserted that the repository guard means fork PRs never queue on the self-hosted runner. It does not. On a `pull_request` event opened from a fork against this repository, `github.repository` is the base repository, so the guard is true and the job runs here. The guard's real effect is to stop the job queueing forever in a fork of the repository. What gates the fork-PR case is the repository's Actions fork-PR approval policy. That reading applies equally to `xla-compile` and to the new job, and the issue specifying this work repeated the incorrect version, so it was propagating. ## Verification All of the following ran on the GB10 host with the IREE runtime provisioned (`bash scripts/iree/setup-cuda.sh`, `eval "$(bash scripts/iree/setup-cuda.sh --env)"`, `MLX_CUDA_ARCHITECTURES=121`). - The job's link command passes from a purged `CARGO_TARGET_DIR`: exit 0 in 15m40s, ending at `Executable tests/xla_prepared_prefill.rs`. Most of that is MLX's CUDA sources compiling from scratch, not the link. - Warm cost, which is what a `build.rs` edit actually pays because it invalidates the `mlxcel` crate but not `mlxcel-core`'s MLX build: 6m01s and 7m24s across two runs, both exit 0. - **The gate catches a link-only regression that `xla-compile` cannot.** With `-l:libflatcc_parsing.a` dropped from the `IREE_CUDA_HOME` branch of `build.rs`, `xla-compile`'s command `cargo check --features cuda,xla-iree --all-targets` still exits 0 in 59 seconds, while this job's command exits 101 with `error: linking with cc failed` and undefined references to `flatcc_verify_string_field`, `flatcc_verify_table_vector_field`, and `flatcc_verify_vector_field`. Restoring the archive returns the link to exit 0. That is the same shape as #1274, on the same tree, and is the demonstration the issue asked for. - The job runs green on this PR itself, since the PR changes `.github/workflows/ci.yml` and so matches the filter. `build.rs` is unchanged by this PR; every control above was run outside the branch and reverted. ## On the negative control the issue specified The issue's acceptance criteria asked for the demonstration to be done by reverting the `-lc` entry added in #1275. That control was tried first and did not fail the link: the command exits 0 with `rustc-link-arg=-lc` confirmed absent from the emitted build-script output and the IREE archives confirmed present on the link line. Why it does not fail is unresolved, and nothing here should be read as evidence that the entry is dead. An earlier revision of this PR claimed the pinned runtime was built without the stack protector; that claim was wrong and has been removed. It came from running `nm` in a shell where `nm` is an alias for an unrelated command, so the search silently matched nothing. Against the pinned distribution, using `/usr/bin/nm`: `__stack_chk_guard` is undefined in 176 objects of `libiree_runtime_unified.a`, including the `call.c.o` named in #1274; it is undefined in `libc.so.6`; and it is defined only in `ld-linux-aarch64.so.1`. Every precondition the `build.rs` comment records still holds today. Two further explanations were tested and ruled out: `-lpthread` and `-ldl` resolve to stub archives on this glibc rather than to scripts that would pull libc in late, and `libm.so` groups only `libm.so.6`. So `build.rs` is left alone, the flatcc control above is what demonstrated the gate, and the workflow comment records all of this so the next reader does not delete the `-lc` entry on the strength of one control that failed to reproduce. Closes #1303
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repeat
-lcafter the IREE archives in theIREE_CUDA_HOMElink recipe, so integration tests built withxla-ireelink on aarch64.Before this, any such test failed with:
Why the obvious reading of that error is wrong
The error names the dynamic linker, which invites the conclusion that the fix is to link it, or that libc is missing entirely. Neither holds:
libc.so.6does not define__stack_chk_guard. It isUNDthere. The definition is inld-linux-aarch64.so.1, reachable only through libc'sDT_NEEDED.rustc-link-argcan only append, so the IREE archives land after rustc's own-lc.call.c.ois compiled with the stack protector, so its reference to__stack_chk_guardappears at a point where no libc follows it.Repeating
-lcafter the archives puts libc where the pending reference can reach it, and ld then resolves through libc'sDT_NEEDEDas it does for any ordinary C program.The fix is minimal because it was ablated, not guessed
The first attempt here added
-Wl,--copy-dt-needed-entries, on the theory that the transitive resolution was what ld was refusing. That flag alone does not work: rustc appends our args after its-lc, so the flag lands at argument 80 while-lcis at argument 29, and it only governs inputs that follow it.Ablating each half on the real link:
-lconly-Wl,--copy-dt-needed-entriesonlySo the policy flag is not shipped. It would have been a global relaxation that can hide a genuinely missing
-lelsewhere in the same link, in exchange for nothing.Verification
Four targets linked, covering both previously-failing and previously-passing cases:
molmo2_xla_vision_paritycuda,xla-ireecli_help_consistencycuda,xla-ireemolmo2_xla_vision_parityxla-diagnosticschat_template_kwargscuda,xla-ireePlus
cargo check --features cuda --all-targetsclean, andcargo fmt --all -- --check.The two previously-passing rows matter as much as the failing ones: the failure was never uniform across targets, so a fix verified only on a failing target could have regressed the rest.
Not changed
The
IREE_DISTand macOS recipes are untouched.IREE_DISTemits a nearly identical group and is likely affected the same way, but this host has neither anIREE_DISTtree nor macOS, so neither can be confirmed by an actual link. Issue #1274 asks for those recipes to be either unchanged or verified, and unverified link changes do not belong in this file.Also in this change
link_args.insert(5, ...)is gone. Adding a single entry ahead of it would have moved the vendored printf archive out of the--start-groupblock with no error, so the conditional entry is now pushed positionally. Each library in the group also carries the reason it is present, which #1274 asks for.Left open
Why the failure was target-dependent and feature-dependent is not fully explained.
chat_template_kwargslinked undercuda,xla-ireewhilemolmo2_xla_vision_paritydid not, and the samemolmo2_xla_vision_paritylinked underxla-diagnostics. Both working binaries carryld-linux-aarch64.so.1as an explicitDT_NEEDEDand the failing links do not, so something in those link lines was already pulling it in. Thesurgerydefault feature was suspected and is neither confirmed nor ruled out. This fix removes the ordering dependency for every target, so the asymmetry stops mattering in practice, but it is recorded here rather than left as folklore.Closes #1274