Since 20.10.6 it's not possible to run docker on a machine with disabled IPv6 interfaces

This is a bug report
This is a feature request
I searched existing issues before opening this one

Expected behavior

When exposing ports with docker run -p hostport:containerport the image is started and the ports are exposed.

Actual behavior

Docker throws an error:

docker: Error response from daemon: driver failed programming external connectivity on endpoint elegant_dirac (e9f85245f5f83cc770c5383a370015c84c63a235c159c5abf3ebad61225d46e0): 
Error starting userland proxy: listen tcp6 [::]:8888: socket: address family not supported by protocol.

Steps to reproduce the behavior

I'm not entirely sure how the machine I run this on was configured, so I cannot explain how to reproduce my configuration exactly.
Using the commands found here I do know that the OS (Ubuntu 20.04.2 LTS) is compiled with IPv6 support, but it's disabled with boot parameters.

user@server: ~$ grep CONFIG_IPV6= /boot/config-$(uname -r)
CONFIG_IPV6=y
user@server: ~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.4.0-71-generic root=/dev/mapper/vg00-root ro ipv6.disable=1 maybe-ubiquity
user@server: ~$ sudo modprobe -c |grep "options ipv6"
options ipv6 disable=1

I'm not the machine admin, so I don't want to configure IPv6, I want to keep it in the state it is now (as this used to work with docker < 20.10.6).

Running a simple docker run -p command triggers the error (I understand that busybox has no ports to publish, but this is a mere example):

user@server:~$ sudo docker run -p 8888:8888 -it --rm busybox  
docker: Error response from daemon: driver failed programming external connectivity on endpoint compassionate_bouman (c0571cb8e44da78ee31fcc6b99327f11aec943070d4fa492fed17a1da636a40b): 
Error starting userland proxy: listen tcp6 [::]:8888: socket: address family not supported by protocol.

Running the same command, but this time specifying --net=host without any published ports works:

user@server:~$ sudo docker run -it --rm --net=host busybox 
/ #

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:47:17 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:45:28 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.7.0)

Server:
 Containers: 11
  Running: 1
  Paused: 0
  Stopped: 10
 Images: 25
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-71-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 15.64GiB
 Name: REDACTED
 ID: REDACTED
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http://REDACTED:8080/
 Username: REDACTED
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

@arkodg

ping @arkodg

the host ipv6 capability needs to be checked in https://github.com/moby/libnetwork/blob/b3507428be5b458cb0e2b4086b13531fb0706e46/drivers/bridge/port_mapping.go#L95
to fix this the right way

we already have a similar check for ipv6 when setting up the IP address
https://github.com/moby/libnetwork/blob/b3507428be5b458cb0e2b4086b13531fb0706e46/drivers/bridge/setup_ipv6.go#L32

caching that value and using it in the bridge port-mapper code is one way to solve this issue

the issue is we are trying to setup a userland-proxy based port mapping even when the ipv6 address is nil to satisfy moby/libnetwork#2608

@cpuguy83

@ba-tno as a workaround, can you
try disabling userland-proxy in dockerd
https://docs.docker.com/engine/reference/commandline/dockerd/#daemon thereby relying to iptables for port mapping
or
specifically map a host ipv4 address to a container port -p 127.0.0.1:8080:8080

@cpuguy83 I'm no longer active in libnetwork
the code in libnetwork such as setting up a ipv6 bridge and enabling ipv6 on the host seems to suggest that we use dockerd's --ipv6 opt-in configuration value as the way to build intent, and also overrides host intent if we need to.
We again had to go against that principle to support backward compatibility for userland-proxy moby/libnetwork#2608

there is unfortunately no clear principle in libnetwork today regarding intent and capability

I'm having the same issue, same versions.

Workaround: docker run -p 0.0.0.0:80:80 instead of docker run -p 80:80

Workaround: docker run -p 0.0.0.0:80:80 instead of docker run -p 80:80

that would mean a rewrite of all composefiles on the hosts! 👎

Downgrade to previous version until fix

…

On 15 Apr 2021, at 9:20 am, ba-tno ***@***.***> wrote: that would mean a rewrite of all composefiles on the hosts! 👎 My problem exactly 😞 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

There is a daemon option to tell it the default ip to bind to for port forwarding. "--ip=0.0.0.0" ought to do it.

@cpuguy83

I did the downgrades too. @cpuguy83 can you give example how to implement that to service file or something?

I'm not sure I understand.
That's the flag you add to the daemon, or the corresponding config option in /etc/docker/daemon.json ({"ip": "0.0.0.0"})

https://docs.docker.com/engine/reference/commandline/dockerd/

@cpuguy83

@cpuguy83 0.0.0.0 wont help, its the default value set in --ip, and used to bind to all addresses
@bor8
you can append this line in /etc/docker/daemon.json and bind to localhost

{
        "ip": "127.0.0.1"
}

if you are using the default docker bridge

or set com.docker.network.bridge.host_binding_ipv4=127.0.0.1 if you are using custom networks
https://docs.docker.com/engine/reference/commandline/network_create/#bridge-driver-options

Is it that simple? Can someone test and confirm this? BTW: It was not a good idea to disable userland-proxy -> no connections from container inside to other VMs/ports possible here! (Maybe not related: We have HTTP connect proxy Squid. I don't understand what userland-proxy is.)

127.0.0.1 in daemon.json means you won't be hitting your containers from another host from my testing. Can confirm the 0.0.0.0 doesn't work either, you get the userland-proxy still tilting at the v6 windmill that way. Could probably add a DNAT that would make 127.0.0.1 work but that seems dreadful from a security perspective

You can add whatever IP you want that is on the host system.

That is true, so for folks with an address they don't expect to be changing frequently, that's a good fit. Doesn't help me tho :'(

Well, localhost would make it easy (same config on all machines). Host IP would it make a bit more complicated (but my IPs don't change often). I want to make sure that this "advice" works! This time!

@bor8

@bor8 I tried @arkodg solution and is not working. I also tried to disable ipv6 entirely in /etc/docker/daemon.json but not working either.

Personally the only fix that worked for me was to downgrade to 20.10.5

apt install docker-ce=5:20.10.5~3-0~ubuntu-bionic

You might also want to “hold” the package for now

apt-mark hold docker-ce

Cross posting from moby issue:

If you have a machine affected by this and would like to submit a fix, I believe we just need to check if ipv6 is enabled here: https://github.com/moby/libnetwork/blob/b3507428be5b458cb0e2b4086b13531fb0706e46/drivers/bridge/port_mapping.go#L111

Being fixed in moby/moby#42322

In case someone with little experience (as myself) stumbles on this I've written down all the steps needed to rollback docker.

These are the steps I followed, perhaps there are more efficient or elegant methods.

In short you will first "blacklist" the 20.10.6 version, then remove docker-ce and related packages and finally re-install docker. Due to the blacklist, the last step will install the 20.10.5 version (at the time of writing).
When later 20.10.7 comes out you should be able to run apt upgrade without an issue (not tested myself).

Find the exact version name to exclude
This command lists all versions of docker-ce that the apt command can install.
 sudo apt-cache madison docker-ce
You're looking for the string in the second column, looking something like 5:20.10.6~3-0~ubuntu-focal.

Configure apt to ignore version 20.10.6
There are different methods, I chose apt pin using preference files.
Below commands are specifically for Ubuntu Focal, in the previous command you can find
the version string that is applicable in your case. Replace the string 5:20.10.6~3-0~ubuntu-focal accordingly.
You probably need root access to write to /etc/apt/preferences.d/
echo -e "Package: docker-ce\nPin: version 5:20.10.6~3-0~ubuntu-focal\nPin-Priority: -10" > /etc/apt/preferences.d/docker-ce
echo -e "Package: docker-ce-rootless-extras\nPin: version 5:20.10.6~3-0~ubuntu-focal\nPin-Priority: -10" > /etc/apt/preferences.d/docker-ce-rootless-extras
echo -e "Package: docker-ce-cli\nPin: version 5:20.10.6~3-0~ubuntu-focal\nPin-Priority: -10" > /etc/apt/preferences.d/docker-ce-cli
Stop docker
sudo systemctl stop docker
Remove docker-ce and related from your system
sudo apt remove "^docker-ce.*"
This allows you confirm if the pins set earlier are in effect
sudo apt-cache policy "^docker-ce.*"
This should show for docker-ce, docker-ce-rootless-extras and docker-ce-cli that version 20.10.6 has a priority of -10, while other version have a positive number, e.g. 500.

Install docker again.
This should also install the required components. Note that it should install 20.10.5.
If not, I don't know what went wrong.
sudo apt install docker-ce
Reboot
I tried simply restarting the service, but that gave me errors.
After a reboot (sudo reboot) the docker service was started and I could expose ports again without having to do any other workarounds.

Thanks @ba-tno, commenting so that I can find this later for my team.

Has there been any word from the docker team about how an issue like this was able to make it into production?

They are headless and have no tests. - Why does this sound familiar? E.g. https://github.com/weaveworks/weave ran into the same problem. No offence.

Hey, I was also thinking about my personal situation. Docker and weave - you are not alone ;)

Navigation Menu

Repository navigation

Since 20.10.6 it's not possible to run docker on a machine with disabled IPv6 interfaces #1233

Expected behavior

Actual behavior

Steps to reproduce the behavior

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Since 20.10.6 it's not possible to run docker on a machine with disabled IPv6 interfaces #1233

Description

Expected behavior

Actual behavior

Steps to reproduce the behavior

Activity

cpuguy83 commented on Apr 13, 2021

arkodg commented on Apr 13, 2021

arkodg commented on Apr 14, 2021

TravisRoy commented on Apr 14, 2021

AkihiroSuda commented on Apr 15, 2021

logopk commented on Apr 15, 2021

avisual commented on Apr 15, 2021

cpuguy83 commented on Apr 15, 2021

bor8 commented on Apr 15, 2021

cpuguy83 commented on Apr 15, 2021

arkodg commented on Apr 15, 2021

bor8 commented on Apr 15, 2021

DeathRabbit679 commented on Apr 15, 2021

cpuguy83 commented on Apr 15, 2021

DeathRabbit679 commented on Apr 15, 2021

bor8 commented on Apr 15, 2021

victorigualada commented on Apr 20, 2021

jflambert commented on Apr 21, 2021

cpuguy83 commented on Apr 22, 2021

AkihiroSuda commented on Apr 26, 2021

gjolund commented on May 8, 2021

bor8 commented on May 8, 2021

bor8 commented on May 8, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions