File system performance improvements

Recent Docker versions (17.04 CE Edge onwards) add additional flags to the -v option for docker run that make it possible to specify the consistency requirements for a bind-mounted directory. The flags are

consistent: Full consistency. The container runtime and the host maintain an identical view of the mount at all times. This is the default.
cached: The host's view of the mount is authoritative. There may be delays before updates made on the host are visible within a container.
delegated: The container runtime's view of the mount is authoritative. There may be delays before updates made in a container are visible on the host.

In 17.04 cached is significantly faster than consistent for many workloads. However, delegated currently behaves identically to cached. The Docker for Mac team plans to release an improved implementation of delegated in the future, to speed up write-heavy workloads. We also plan to further improve the performance of cached and consistent.

We will post updates relating to Docker for Mac file sharing performance in comments to this issue. Users interested in news about performance improvements should subscribe here.

To keep the signal-to-noise ratio high we will actively moderate this issue, removing off-topic advertisements, etc. Questions about the implementation, the behaviour, and the design are very welcome, as are realistic benchmarks and real-world use cases.

@yallop

A rudimentary real-worked example (Drupal 8):

Docker for Mac : Version 17.05.0-ce-rc1-mac8 (16582)
Mac-mini :: MacOS Sierra : 10.12.2 (16C67)

A simple command line curl test (taken average of 10 calls to URL) Drupal 8 clean install frontend:

old UNISON (custom synced container approach) Volume mount : 0.470s
standard Volume mount: 1.401s
new :cached Volume mount: 0.490

v.easy implementation to add to my compose.yaml files and happy with any delay between host/output using cached on host codebase.

@yallop Is there a rough/expected release date for 17.04 (stable) ?

@yallop

@yallop Is there a rough/expected release date for 17.04 (stable)

The next stable version will be 17.06, and is likely to be released some time in June. There's been a change to the Docker version numbering scheme recently so that the numbers now indicate the release date, with stable releases every three months (March, June, September, December), and edge releases in the months in between. For example, 17.04 is the Edge release in April 2017 and 17.06 is the stable release in June 2017.

ok, thanks for that. do you expect 17.06 to contain at least the current edge implementation of :cached?

Yes, that's the plan.

How should this work in a docker-compose.yml file? I tried appending :cached to the existing volume setting:

volumes:
  - .:/srv:cached

... but that got me an error (on OSX):

ERROR: Cannot start service my_service: Mounts denied: 9p: Stack overflow

(Docker version 17.05.0-ce-rc1, build 2878a85)

Note: having a global setting or environment variable to switch my local default to "cached" would also be fine (or rather preferable).

@reinout

The syntax looks correct @reinout
here's a stripped but working example docker-compose.yml

version: '2'
services:
  php:
    image: php:7.1-fpm
    ports:
      - 9000
    volumes:
      - .:/var/www/project:cached

tested on Docker version 17.05.0-ce-rc1, build 2878a85

Your example works. My own one still not (even after really making sure there were no left-over old mounted volumes). So I rebooted. Afterwards, it worked.

So: a reboot might be needed if you've run a docker-compose before upgrading docker to the latest version. Possibly related: I switched from docker stable to edge.

Is there a possibility of a global setting? I don't really want to add this option to the docker-compose.yml that all my linux colleagues are using.

not as far as I know. When you linux colleagues are running edge the flag should work.

Another workaround would be (that's what we do right now for separating linux and mac volume mounting )
to just put the mount settings for mac in a separate file, like docker-compose-mac.yml and then run docker-compose -f docker-compose.yml -f docker-compose.mac.yml up -d

See https://docs.docker.com/compose/extends/

@reinout

@reinout You could use an additional compose file to override that of your colleagues? For instance, we have:

docker-compose.yml:

version: '2'

services:

  build_docs:
    image: docs/sphinx
    build: .
    environment:
      - DOCS_NAME='docs'
      - SRC_DIR=src
      - DST_DIR=build
    volumes:
      - "./../../docs/dev:/docs"
    command: /docs/source /docs/build

and volumes-cached.yml:

services:
  build_docs:
    volumes:
      - "./../../docs/dev:/docs:cached"

Which can be run with:

$ docker-compose -f docker-compose.yml -f volumes-cached.yml up

Yes, I could do that. But.... I'd have to do that for each of the 12 docker-compose projects. And I'd have to keep it in sync with changes to the "master" docker-compose.yml.

As an intermediary measure: fine. Long term: no, as it is not very don't-repeat-yourself :-)

If someone wants to enable the "cached" behaviour, that person probably wants to use it for all/most of the dockers. Would it make sense as a config setting in the Docker app itself? In the preferences' "File sharing" tab? (This should probably be its own ticket, I assume?)

@reinout

@reinout: :cached is supported across platforms, so there should be no issue in adding it directly to compose files.

Provided that everybody uses the latest edge version, right? And it seems a bit strange to add an option that only has effect on osx to everybody's docker-compose.yml.

Anyway, it works for now. I won't drag the signal/noise ratio further down :-)

@back-2-95

Thanks @back-2-95 ! I've now edited my comment from Tuesday to reflect that.

I've installed MacOS 12.2 and the prototype Docker with these settings

It's absolutely crawling, it's so slow it's pretty much unusable with our large PHP application with lots of vendors.

Here are the numbers, I don't know how to interpret these runes but it may be of utility to a cleverererr person than me!

docker run --rm -ti alpine sh -c "apk add p7zip ; 7z b"

...
RAM size:    1985 MB,  # CPU hardware threads:   6
RAM usage:   1323 MB,  # Benchmark threads:      6

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       7007   521   1308   6817  |      87193   588   1265   7436
23:       5449   538   1032   5553  |      85882   584   1273   7431
24:       3799   538    759   4085  |      81722   587   1222   7173
25:       4527   538    962   5169  |      78720   582   1205   7006
----------------------------------  | ------------------------------
Avr:             534   1015   5406  |              585   1241   7261
Tot:             559   1128   6334

I can see that it's using virtiofs:

docker run -it --rm -v $(pwd):/vol alpine mount | grep /vol
virtiofs0 on /vol type virtiofs (rw,relatime)

@owsygavin

Thanks, @owsygavin. There are known performance problems with the new virtualization framework on Intel on Monterey (independent of virtiofs). We've reported it to Apple, and I believe they're working on it.

(@christophermclellan It could be worth adding that to your earlier message too, we've had a few people report it).

I am running an integration test suite against dependencies (MySQL and Redis) which live in Docker. With MySQL and Redis running on the host, this suite takes approximately 2-3s, the same amount of time it takes on Linux with Docker dependencies.

With Docker for Mac, it takes roughly 12-13s.

With the experimental build above, and both features enabled, on 12.2 w/ M1 Pro, it takes 75.5s.

In keeping with the trend in this thread, I ran the following benchmark.

w/ experimental features:

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,5 CPUs LE)

LE
CPU Freq: - - - - - - - - -

RAM size:    3918 MB,  # CPU hardware threads:   5
RAM usage:    882 MB,  # Benchmark threads:      5

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:      25566   366   6800  24871  |     213299   398   4567  18198
23:      24763   390   6471  25231  |     209774   399   4547  18151
24:      24452   399   6596  26291  |     206299   399   4544  18110
25:      23321   399   6681  26628  |     200921   398   4498  17881
----------------------------------  | ------------------------------
Avr:             388   6637  25755  |              398   4539  18085
Tot:             393   5588  21920

I then tried to disable the experimental features by unchecking the top checkbox ("Use the new Virtualization framework"), which grayed out the bottom one ("Enable VirtioFS accelerated directory sharing") but left it checked. I hit save and Docker for Mac has hung ever since. Restarting results in it hanging with the message "Docker Desktop starting..."

None of the troubleshooting options - "Clean / Purge data", "Reset to factory defaults", and "Uninstall" - appear to be responsive. I downloaded and installed the stable version of Docker for Mac, and the issue persists. Since it's still starting I am unable to modify any settings to try unchecking both checkboxes.

edit w/o experimental features:

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,5 CPUs LE)

LE
CPU Freq: - - - - - - - - -

RAM size:    3933 MB,  # CPU hardware threads:   5
RAM usage:    882 MB,  # Benchmark threads:      5

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:      23257   374   6046  22625  |     196733   396   4237  16785
23:      22883   389   5993  23316  |     193574   397   4216  16749
24:      22070   402   5903  23730  |     188944   397   4173  16587
25:      21050   400   6014  24034  |     185981   398   4156  16552
----------------------------------  | ------------------------------
Avr:             391   5989  23426  |              397   4195  16668
Tot:             394   5092  20047

I hit save and Docker for Mac has hung ever since. Restarting results in it hanging with the message "Docker Desktop starting..."

None of the troubleshooting options - "Clean / Purge data", "Reset to factory defaults", and "Uninstall" - appear to be responsive. I downloaded and installed the stable version of Docker for Mac, and the issue persists. Since it's still starting I am unable to modify any settings to try unchecking both checkboxes.

Which describes the issue I also had. docker/roadmap#7 (comment)

FWIW I was able to resolve the issue by deleting various Docker directories out of the ~/Library folder and /usr/lib/docker.

@johnmaguire

@johnmaguire sorry for the inconvenience. If anyone gets stuck, they remove the settings file ~/Library/Group Containers/group.com.docker/settings.json

@owsygavin

Thank you to everyone that has provided feedback on the latest experimental build. Apple have just released the macOS 12.3 beta, in which the known speed problems with the new virtualization framework on Intel + Monterey should now be resolved (for example - the issue that @owsygavin reported here). If those that have previously experienced performance issues on Intel would be willing to try out 12.3 we'd be very appreciative of any feedback.

Hi, thank you for this amazing work! Performance of bind mount with virtiofs is far better than gRPC-fuse.

Now I found that chmod to bind-mounted directory fails randomly on macOS 12.3 beta1, like:

$ mkdir test
$ chmod 777 test
$ chmod 777 test
chmod: changing permissions of 'test': Operation not permitted
$ chmod 777 test
$ chmod 777 test
chmod: changing permissions of 'test': Operation not permitted
$ chmod 777 test
$ chmod 777 test
chmod: changing permissions of 'test': Operation not permitted
$ chmod 777 test
chmod: changing permissions of 'test': Operation not permitted

Environment:

macOS 12.3 Beta (21E5196i)
Intel MacBook Pro 2020
Docker Desktop 4.4.2 (73862)

I can provide any further informations if needed!

@sarisia

@sarisia, that's interesting. I can't reproduce here. Could you share your full command line for starting the container, in case the difference is more subtle? Are there other containers running on this host?

@fredericdalleau

@fredericdalleau

I usually use VSCode devcontainer, but I've confirmed it can be reproduced
with the commands below:

$ cat Dockerfile
FROM ubuntu:latest

RUN apt-get update && apt-get -y install git
RUN groupadd user && \
    useradd -s /bin/bash --gid user -m user

$ docker build -t dockertest .
[+] Building 1.2s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                       0.5s
 => => transferring dockerfile: 579B                                       0.0s
 => [internal] load .dockerignore                                          0.7s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest           0.0s
 => [1/3] FROM docker.io/library/ubuntu:latest                             0.0s
 => CACHED [2/3] RUN apt-get update && apt-get -y install git              0.0s
 => CACHED [3/3] RUN groupadd user &&     useradd -s /bin/bash --gid user  0.0s
 => exporting to image                                                     0.2s
 => => exporting layers                                                    0.0s
 => => writing image sha256:befa7326e73b8f7c3c541cb52092f113bd7e5ee4fbc21  0.1s
 => => naming to docker.io/library/dockertest                              0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

$ mkdir test
$ docker run -it --rm --mount type=bind,src=(pwd)/test,target=/test,consistency=delegated -u user dockertest /bin/bash

user@e66a3e99705e:/$ cd /test
user@e66a3e99705e:/test$ git init
Initialized empty Git repository in /test/.git/
user@e66a3e99705e:/test$ git submodule add https://github.com/aws/aws-iot-device-sdk-embedded-C.git
Cloning into '/test/aws-iot-device-sdk-embedded-C'...
remote: Enumerating objects: 25668, done.
remote: Counting objects: 100% (9029/9029), done.
remote: Compressing objects: 100% (1997/1997), done.
remote: Total 25668 (delta 6917), reused 8726 (delta 6765), pack-reused 16639
Receiving objects: 100% (25668/25668), 32.90 MiB | 5.92 MiB/s, done.
Resolving deltas: 100% (16389/16389), done.
user@e66a3e99705e:/test$ cd aws-iot-device-sdk-embedded-C/
user@e66a3e99705e:/test/aws-iot-device-sdk-embedded-C$ git submodule update --init --recursive
Submodule 'demos/jobs/jobs_demo_mosquitto/libmosquitto' (https://github.com/eclipse/mosquitto.git) registered for path 'demos/jobs/jobs_demo_mosquitto/libmosquitto'
Submodule 'libraries/3rdparty/CMock' (https://github.com/ThrowTheSwitch/CMock) registered for path 'libraries/3rdparty/CMock'
error: chmod on /test/.git/modules/aws-iot-device-sdk-embedded-C/config.lock failed: Operation not permitted
Submodule 'libraries/3rdparty/mbedtls' (https://github.com/ARMmbed/mbedtls) registered for path 'libraries/3rdparty/mbedtls'
Submodule 'libraries/3rdparty/tinycbor' (https://github.com/intel/tinycbor) registered for path 'libraries/3rdparty/tinycbor'
Submodule 'libraries/aws/device-defender-for-aws-iot-embedded-sdk' (https://github.com/aws/device-defender-for-aws-iot-embedded-sdk.git) registered for path 'libraries/aws/device-defender-for-aws-iot-embedded-sdk'
error: chmod on /test/.git/modules/aws-iot-device-sdk-embedded-C/config.lock failed: Operation not permitted
fatal: Failed to register url for submodule path 'libraries/aws/device-shadow-for-aws-iot-embedded-sdk'

Also, I tested both root and non-root user inside the container and
it only happens when using non-root user.

any news about new build? 😄

Ah yes, sorry, there is a new one at docker/roadmap#7 (comment). We should have posted it here too, that's our mistake. However, as the main conversation is at docker/roadmap#7, I'm actually going to lock this ticket now to keep everything in one place.

Navigation Menu

Repository navigation

File system performance improvements #1592

428 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

File system performance improvements #1592

Description

Activity

joemewes commented on May 6, 2017

yallop commented on May 8, 2017

joemewes commented on May 8, 2017

yallop commented on May 8, 2017

reinout commented on May 9, 2017

DanielSchwiperich commented on May 9, 2017

reinout commented on May 9, 2017

DanielSchwiperich commented on May 9, 2017

carn1x commented on May 9, 2017

reinout commented on May 9, 2017

yallop commented on May 9, 2017

reinout commented on May 9, 2017

428 remaining items

christophermclellan commented on Jan 27, 2022

owsygavin commented on Jan 27, 2022

stephen-turner commented on Jan 27, 2022

johnmaguire commented on Jan 28, 2022

johnmaguire commented on Jan 28, 2022

PhilETaylor commented on Jan 28, 2022

johnmaguire commented on Jan 28, 2022

fredericdalleau commented on Jan 29, 2022

christophermclellan commented on Feb 3, 2022

sarisia commented on Feb 8, 2022

fredericdalleau commented on Feb 8, 2022

sarisia commented on Feb 8, 2022

Peredery commented on Mar 2, 2022

stephen-turner commented on Mar 2, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions