Rustup proxies not longer working with symlink resolution since 1.28.0

Verification

I searched for recent similar issues at https://github.com/rust-lang/rustup/issues?q=is%3Aissue+is%3Aopen%2Cclosed and found no duplicates.
I am on the latest version of Rustup according to https://github.com/rust-lang/rustup/tags and am still able to reproduce my issue.

Problem

coming from this report: rust-lang/rustwide#94 , currently docs.rs builds are broken because of this issue.

There seems to be something wrong with the symlinks that are now generated instead of hardlinks for the binaries (#4023).

We are using fs::canonicalize before passing it to the binaries:
https://github.com/rust-lang/rustwide/blob/744375a9050e0584124ea11adaeab41781fa8a50/src/utils.rs#L111-L112

Steps

What information would you need to be able to track this down?

we can reproduce it when just trying to run any docs.rs build,

the two errors I saw where both just showing that the call does to rustup install instead of cargo install (or other calls to cargo are done on rustup)

Possible Solution(s)

No response

Notes

No response

Rustup version

1.28.0

Installed toolchains

it happens with just one toolchain installed, tested with `stable-aarch64-apple-darwin` or `stable-x86_64-unknown-linux-gnu`

OS version

Ubuntu 18.04.2 LTS on the server, or newer ubuntu in docker images

@skius

cc @skius @Skgland

Would be nice to get a bit closer to the root cause of this issue, do you think you can do that? Do you have a reproduction scenario that doesn't involve rustwide/docs.rs?

Quick workarounds I can think of:

Maybe std::path::absolute would help?
Or else just canonicalise the parent path than rejoin the file name?

Would be nice to get a bit closer to the root cause of this issue, do you think you can do that? Do you have a reproduction scenario that doesn't involve rustwide/docs.rs?

not yet, below is something that works with rustwide only.

I don't know rustwide internals / the build process very well, so a more low level example will take more time.

I'll probably first try to find a way to downgrade rustup for rustwide.

the rustwide / rustup example

The rustwide docs-builder example lets us see the error directly.

It can be made a little faster by choosing a smaller docker base image:

[...]
    // Create a new workspace in .workspaces/docs-builder
    let workspace =
        WorkspaceBuilder::new(Path::new(".workspaces/docs-builder"), "rustwide-examples")
            // this is new
            .sandbox_image(SandboxImage::remote(
                "ghcr.io/rust-lang/crates-build-env/linux-micro",
            )?)
            .init()?;
[...]

Quick workarounds I can think of:

* Maybe [`std::path::absolute`](https://doc.rust-lang.org/stable/std/path/fn.absolute.html) would help?

* Or else just `canonicalise` the parent path than rejoin the file name?

You mean in our rustwide code? Or rustup?

if rustwide, I first need to figure out why we actually canonicalize the paths before calling, that needs digging into old crater code before rustwide was extracted

On the rustwide side.

I don't think canonicalizing the filename can ever work with symlinked proxies because rustup shims rely on the link name to know where to direct the command to whereas canonicalizing a symlink removes that information by design. Canonicalizing the parent directory should always be fine though.

On the rustup side we could remove symlink support and only hard link (we've had this symlink behaviour for a long time but hard linking used to take precedence). Or else there might be something we can do to fix whatever problem necessitated canonicalizing.

Or else there might be something we can do to fix whatever problem necessitated canonicalizing.

I'm still trying to figure that out. It was added in rust-lang/crater@2c3e40c, but no specific explaination as to why.

Canonicalizing the parent directory should always be fine though.

I will see if I can figure out something here. ( hoping it's the only issue )

I don't think canonicalizing the filename can ever work with symlinked proxies because rustup shims rely on the link name to know where to direct the command to whereas canonicalizing a symlink removes that information by design. Canonicalizing the parent directory should always be fine though.

Explicitly setting arg0 could work but that's only available on Unix and not on Windows.
Not canonicalizing is probably easier.

Oh of course. It can be done on Windows too but we haven't added it to the standard library yet. We really should. And possibly make a cross-platform arg0 function. But I realise that doesn't help in this moment.

so, path::absolute doesn't work. My current hunch is that this is related to how rustwide maps the toolchain into the docker container, and it somehow breaks when we try to use symlinks

Canonicalizing the parent directory should always be fine though.

this also doesn't work. Not sure why yet

My current hunch is that this is related to how rustwide maps the toolchain into the docker container, and it somehow breaks when we try to use symlinks

Ah, could this be #4222? Which is definitely something we should fix.

My current hunch is that this is related to how rustwide maps the toolchain into the docker container, and it somehow breaks when we try to use symlinks

Ah, could this be #4222? Which is definitely something we should fix.

could be. From what I remember, the toolchain installation is done outside of docker on the host, and then the directory is mapped into the docker container.

an additional thought:

I'm not 100% sure how docker maps filesystems, but I could imagine that even when we would switch to relative symlinks, this might not work, if the relative links points outside of the docker-mapped fileystem.

I got docs.rs build again by forcing rustwide to use 1.27.1 and blocking the self update.

I'll probably wait for #4222 / #4226 to be released and try again. If that doesn't work I have to dig a little deeper into how rustup works with the links/ shims, and also how exactly the toolchain is set up & linked into the rustwide docker containers.

Ok, new release should be out now. Sorry for the delay, unfortunately the release process is not anywhere near as simple as pressing a button.

Ok, new release should be out now. Sorry for the delay, unfortunately the release process is not anywhere near as simple as pressing a button.

so #4226 is in that release?

sorry, saw it in the release notes.

Will look if it works now later

There is a similar problem with the change to symlinks instead of hard links in the new version of rustup.
If you are on WSL, Windows can't follow symlinks. Due to this, RustRover users on windows can't select a WSL based toolchain, because both cargo and rustc are now symlinks to rustup, but windows can't follow those links: RUST-17235 Unable to add Rust toolchain on WSL

I don't know their code but judging from the error "can't fine Cargo" I guess it's testing existence of cargo by attempting to follow the symlink? Ideally they would only test if the symlink itself exists (in Rust that can be done with std::fs::symlink_metadata and in Java you'd use the NOFOLLOW_LINKS link option).

@ChrisDenton

@ChrisDenton If this was Unix that's possible, but on Windows unfortunately it's not possible to follow symlinks on a linux filesystem, see my link above. So Java at least is unable to follow the link, or even consider the file as existing.

Right yes, what I was trying to say is that there doesn't appear to be a reason to follow the link. They are presumably only testing for existence so they don't need to follow symlinks.

short update also here, coming from this comment:

changing the symlinks to relative didn't help, so rustwide (= docs.rs & crater & other users like shipyard) are currently frozen to the old rustup version.

I need to find time if it would help to change what exact folders are mapped into the docker container to see if I can make it work with symlinks, but I don't know much about docker mapping.

The nextest also reports an error in nextest-rs/nextest#2246, due to symlink resolution with new rustup.

I have a simliar problem here. As commend in nextest-rs/nextest#2246, Our action got this error:

error: command `/home/runner/.cargo/bin/rustup metadata '--format-version=1' --all-features --filter-platform x86_64-unknown-linux-gnu` failed with exit status: 1

and another error here:

TEST_WORKDIR_PREFIX="/tmp" /home/runner/.cargo/bin/cargo llvm-cov --codecov --output-path codecov.json --workspace  --features=virtiofs  -- --skip integration --nocapture  --test-threads=8
error: process didn't exit successfully: `/home/runner/.cargo/bin/rustup locate-project --message-format plain` (exit status: 1)
--- stdout
error: unexpected argument '--message-format' found

  tip: to pass '--message-format' as a value, use '-- --message-format'

Usage: rustup[EXE] <--verbose|--quiet|+toolchain>

The proxy seems to work abnormally.

And if i run make ut-nextest directly on github actions runner's shell, the command run successfully. However run CARGO=/home/runner/.cargo/bin/cargo make ut-nextest reports error:

runner@fv-az1075-407:~/work/nydus/nydus$ CARGO=/home/runner/.cargo/bin/cargo make ut-nextest
Running unit tests with nextest...
TEST_WORKDIR_PREFIX="/tmp" RUST_BACKTRACE=1 /home/runner/.cargo/bin/cargo nextest run --no-fail-fast --filter-expr 'test(test) - test(integration)' --workspace  --features=virtiofs --release
error: command `/home/runner/.cargo/bin/rustup metadata '--format-version=1' --all-features --filter-platform x86_64-unknown-linux-gnu` failed with exit status: 1
make: *** [Makefile:113: ut-nextest] Error 102

Version info here:

runner@fv-az1075-407:~/work/nydus/nydus$ rustup -V
rustup 1.28.1 (f9edccde0 2025-03-05)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.84.0 (9fc6b4312 2025-01-07)`
runner@fv-az1075-407:~/work/nydus/nydus$ rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /home/runner/.rustup

installed toolchains
--------------------
stable-x86_64-unknown-linux-gnu (default)
1.84.0-x86_64-unknown-linux-gnu (active)

active toolchain
----------------
name: 1.84.0-x86_64-unknown-linux-gnu
active because: overridden by '/home/runner/work/nydus/nydus/rust-toolchain.toml'
installed targets:
  x86_64-unknown-linux-gnu

You can see here: rust-lang/rustwide#97

how it was solved in rustwide, not sure if it helps

@sunshowers

You can see here: rust-lang/rustwide#97

how it was solved in rustwide, not sure if it helps

CC @sunshowers.

We're also affected by this issue. Our Windows builds currently rely on a slightly outdated Cygwin environment, and when cargo is called from Cygwin's sh.exe (through the symlink), we see error messages from rustup.

The following version of sh.exe is affected:

GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin)

The latest sh.exe from Cygwin is not:

GNU bash, version 5.2.21(1)-release (x86_64-pc-cygwin)

I have not tested any versions in between, so I don't know which change in Cygwin caused rustup's symlinks to start working.

Unfortunately, updating the Cygwin environment used by our build is not trivial, so I have to ask: Is there a chance the change to using symlinks could be reverted due to the reported issues?

With the latest rustup you can use the RUSTUP_HARDLINK_PROXIES=1 environment variable to hard link proxies.

Navigation Menu

Repository navigation

Rustup proxies not longer working with symlink resolution since 1.28.0 #4224

Verification

Problem

Steps

Possible Solution(s)

Notes

Rustup version

Installed toolchains

OS version

the rustwide / rustup example

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Rustup proxies not longer working with symlink resolution since 1.28.0 #4224

Description

Verification

Problem

Steps

Possible Solution(s)

Notes

Rustup version

Installed toolchains

OS version

Activity

syphar commented on Mar 4, 2025

djc commented on Mar 4, 2025

ChrisDenton commented on Mar 4, 2025

syphar commented on Mar 4, 2025

the rustwide / rustup example

syphar commented on Mar 4, 2025

ChrisDenton commented on Mar 4, 2025

syphar commented on Mar 4, 2025

Skgland commented on Mar 4, 2025

ChrisDenton commented on Mar 4, 2025

syphar commented on Mar 4, 2025

syphar commented on Mar 4, 2025

ChrisDenton commented on Mar 4, 2025

syphar commented on Mar 4, 2025

syphar commented on Mar 5, 2025

1 remaining item

syphar commented on Mar 5, 2025

ChrisDenton commented on Mar 5, 2025

syphar commented on Mar 5, 2025

syphar commented on Mar 5, 2025

ceronman commented on Mar 7, 2025

ChrisDenton commented on Mar 7, 2025

ceronman commented on Mar 7, 2025

ChrisDenton commented on Mar 7, 2025

syphar commented on Mar 17, 2025

BraveY commented on Mar 25, 2025

syphar commented on Mar 25, 2025

BraveY commented on Mar 25, 2025

mbs-c commented on Jun 26, 2025

ChrisDenton commented on Jun 26, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions