Repository navigation
Supports multiple IP address assignment for nerdctl run and nerdctl compose up #3663
Description
Per (#3556 (comment)) I am opening an issue on what seems to be CNI concurrency issue.
I am having CNI issues when using multiple networks. One is macvlan network and the other is a bridge network. I've also attempted dual macvlan with similar results. What happens looks like an ordering issue. Sometimes the bridge interface would try and come up as a macvlan for example or as below shows the macvlan tries to come up as a bridge . I made a basic test compose file with hello-world and it happens there are well. Nothing is currently assigned the ip. I've tried other unused ip's as well.
I found this when trying to use 2.0.0-rc3 and I retried when 2.0.0 was released and the problem persists. The same compose file works without issue in docker compose.
This is the result from a "nerdctl compose up". The home.local is macvlan, proxy.home.local is a bridge. Single network services start with no issues.
# nerdctl compose up
INFO[0000] Ensuring image hello-world
INFO[0000] Creating container test
FATA[0003] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2024-11-09T11:06:52-06:00" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): failed to allocate all requested IPs: 10.0.0.171": unknown
FATA[0003] error while creating container test: exit status 1
The hello-world compose file:
services:
hello:
image: hello-world
container_name: test
hostname: test
networks:
home.local:
ipv4_address: 10.0.0.171
mac_address: 02:42:0a:00:01:47
proxy.home.local:
ipv4_address: 10.100.100.127
networks:
home.local:
name: home.local
external: true
proxy.home.local:
name: proxy.home.local
external: true
Network creation commands and cni versions:
# nerdctl network create -d macvlan --subnet=10.0.0.0/24 --gateway=10.0.0.1 -o parent=bond0 -o macvlan_mode=bridge home.local
# nerdctl network create --subnet=10.100.100.0/24 proxy.home.local
# /opt/cni/bin/macvlan -v
CNI macvlan plugin v1.6.0
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0
# /opt/cni/bin/bridge -v
CNI bridge plugin v1.6.0
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0
Activity
Zheaoli commentedon Nov 10, 2024
Bug confirmed, Let us figure it out
Zheaoli commentedon Nov 26, 2024
OK, here's the detail
I'm not sure if this a bug or not. For now, we do not support multiple IP address assignment yet.
https://github.com/containerd/nerdctl/blob/main/pkg/cmd/container/create.go#L634-L662
Your compose file will be converted to the command
nerdctl run --pull=never --net=home.local --ip=10.0.0.171 --mac-address=02:42:0a:00:01:47 --net=proxy.home.local --ip=10.100.100.127 --hostname=test --restart=no hello-worldThe last IP address will override the first IP address. So the issue comes here.
I'm not sure we should support multiple IP address assignment. WDYT @AkihiroSuda
AkihiroSuda commentedon Nov 26, 2024
It should correspond to whether Docker supports multiple IP address assignment
Zheaoli commentedon Nov 26, 2024
Same idea, Let us figure it out
[-]CNI Concurrency issue with two networks in a single container[/-][+]Supports multiple IP address assignment for `nerdctl run` and `nerdctl compose up`[/+]gflohr commentedon Feb 28, 2025
I have figured it out.
I have the same issue using the nerdctl that comes with Rancher Desktop on macOS. Switching to Docker fixes it. So, Docker supports multiple IP address assignment.
apostasie commentedon May 23, 2025
Docker does not support passing multiple networks and multiple ips on the run command: the last specified ip will be used for the first specified network (<- the logic in that escapes me, but whatever).
I personally think this ^ is just buggy and should support multiple ips.
eg:
The same is likely true for mac and ipv6 as well. Same for port bindings.
I suspect what docker does with that compose file is
create, thenconnect(for each network), thenrun.If I am not mistaken, fixing this in nerdctl requires a good many different things to happen and likely significant effort:
network connectthat we apparently do not haveconnectwhen presented with multiple networksI need some of this.
Not personally interested in the compose part, but I need multi-ip with multi-networks, so, I am likely to go for 2. and 3, and either 1. or allow multi-ips to be specified on
run.If anyone else is interested in this and wants to chime in, give me a ping so that we either collaborate on this or at least don't duplicate effort.
apostasie commentedon May 24, 2025
Actually, docker DOES support specifying details for multiple networks on
run- just with this syntax:docker run -itd --network=name=my-net1,ip=192.0.2.42 --network=name=my-net2,ip=192.0.3.42 busybox