VirtioFS causes intermittent failures with Rust

[markedwards]

Description

This is a duplicate of #6511 which for some reason was just totally ignored.

In my case I intermittently get the following sort of errors when compiling Rust using cargo build or cargo run:

/usr/bin/ld: cannot find /workspaces/rust/target/debug/deps/rust_sandbox-52f451132105a34b.287q8y3urr1rjxzw.rcgu.o: No such file or directory

Switching from VirtioFS to osxfs eliminates the errors, but I don't want to do this.

Reproduce

1. spin up a Rust image (https://hub.docker.com/_/rust)
2. cargo new
3. cargo build

Step 3 fails intermittently.

Expected behavior

Compilation should never produce these kinds of errors.

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.25.0 (126437)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/markedwards/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.0-desktop.1
    Path:     /Users/markedwards/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/markedwards/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/markedwards/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.9
    Path:     /Users/markedwards/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/markedwards/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/markedwards/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.9
    Path:     /Users/markedwards/.docker/cli-plugins/docker-scout

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 8
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.758GiB
 Name: linuxkit-1637e9d1e2c1
 ID: 66439519-58e4-4975-9f07-998f86ae4ebd
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 197
  Goroutines: 258
  System Time: 2023-11-04T12:29:41.338283877Z
  EventsListeners: 11
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

4E3078F8-1424-4C4D-9526-93439747F1F3/20231104123413

Additional Info

No response

[richardpringle]

Seems like this is a concurrency issue of some sort.

When I run until cargo build; do echo retry; done;, eventually my builds will pass. Seems like a dependent compilation step starts but can't find the files that were supposed to be created as an artifact of the previous step. Then, when running cargo build again in the next iteration of the until loop, it can find the file.

If you are unfamiliar, cargo/rustc will save outputs from previous steps so you aren't starting from scratch every time

[richardpringle]

Seems like other had similar issues
#6511

[ShaneMurphy2]

Seeing this as well using cross. Can confirm repeated builds eventually succeed, and that disabling VirtioFS seems to fix it.

[ShaneMurphy2]

I have some time I can devote to this, but I have no idea how to even begin to debug it. If any maintainers have some tips I'm happy to help.

[richardpringle]

For Rust, the workaround is to put your CARGO_TARGET_DIR in a directory that isn't mounted. Not ideal in some cases, but works for me.

[mxfactorial]

cross-rs/cross#1272

[dgageot]

Hi everyone, the issue is fixed on our side, it'll ship with Docker Desktop 4.28.0.