I’ve previously written about using macvlan networks with docker, this has proved to be a great way to make containers more like lightweight VMs as you can assign a unique IP on your network to them. Unfortunately when I did this I only allocated 4 IPs to the network, and 1 of those is used to provide a communication path from the host to the macvlan network.
Here is how I’ve used up those 4 IPs:
- wireguard – allows clients on wireguard to see other docker services on the host
- mqtt broker – used to bridge between my IoT network and the lan network without exposing all of my lan to the IoT network
- nginx – a local only webserver, useful for fronting Home Assistant and other web based apps I use
- shim – IP allocated to supporting routing from the host to the macvlan network.
If I had known how useful giving a container a unique IP on the network was, I would have allocated more up front. Unfortunately you can’t easily grow a docker network, you need to delete and recreate it.
As an overview here is what we need to do.
- Stop any docker container that is attached to the macvlan network
- Undo the shim routing
- Delete the docker network
- Recreate the docker network (expanded)
- Redo the shim routing
- Recreate the existing containers
This ends up not being too hard, and the only slightly non-obvious step is undoing the shim routing, which is the reverse of the setup.
|
1 2 3 4 |
$ sudo ip route del 192.168.1.64/30 dev myNewNet-shim $ sudo ip link set myNewNet-shim down $ sudo ip addr del 192.168.1.67/32 dev myNewNet-shim $ sudo ip link del myNewNet-shim link enp3s0 type macvlan mode bridge |
The remainder of this post is a walk through of setting up a 4 IP network, then tearing it down and setting up a larger 8 IP network.
The first thing we’ll do is inspect the existing docker macvlan network. That will give us information on what we provisioned originally
|
1 |
$ docker network inspect myNewNet |
There is quite a bit of output, but the chunk we care about is the “Config” section
|
1 2 3 4 5 6 7 8 9 10 |
"Config": [ { "Subnet": "192.168.1.0/24", "IPRange": "192.168.1.64/30", "Gateway": "192.168.1.1", "AuxiliaryAddresses": { "host": "192.168.1.67" } } ] |
This aligns with what we setup previously. However, if we pretend for a second we are starting on a brand new machine here are the docker macvlan setup steps we would do.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# Create the network $ docker network create -d macvlan -o parent=enp3s0 \ --subnet 192.168.1.0/24 \ --gateway 192.168.1.1 \ --ip-range 192.168.1.64/30 \ --aux-address 'host=192.168.1.67' \ myNewNet # Setup the shim routing so the host can see the new containers via IP $ sudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge $ sudo ip addr add 192.168.1.67/32 dev myNewNet-shim $ sudo ip link set myNewNet-shim up $ sudo ip route add 192.168.1.64/30 dev myNewNet-shim |
Cool, so that’s a 4 IP macvlan network setup. We can test it out with a new docker container
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# Best to do this in a second shell window # Create new container on the new network and specify the IP we want to assign $ docker run -it --network myNewNet --ip 192.168.1.50 ubuntu bash # Now inside of that container, we install some tools $ apt-get update $ apt-get install net-tools $ apt-get install iputils-ping # And we can ping the host $ ping 192.168.1.88 PING 192.168.1.88 (192.168.1.88) 56(84) bytes of data. 64 bytes from 192.168.1.88: icmp_seq=1 ttl=64 time=0.236 ms 64 bytes from 192.168.1.88: icmp_seq=2 ttl=64 time=0.206 ms 64 bytes from 192.168.1.88: icmp_seq=3 ttl=64 time=0.134 ms |
Now from the host – we can via the shim also see the new container
|
1 2 3 4 5 |
$ sudo ping 192.168.1.50 PING 192.168.1.50 (192.168.1.50) 56(84) bytes of data. 64 bytes from 192.168.1.50: icmp_seq=1 ttl=64 time=0.086 ms 64 bytes from 192.168.1.50: icmp_seq=2 ttl=64 time=0.064 ms 64 bytes from 192.168.1.50: icmp_seq=3 ttl=64 time=0.064 ms |
At this point we have a functioning 4 IP macvlan network. Time to tear it down and recreate it with more IPs. If we were to repeat the docker network inspect we would see the same information we had above.
Stop the docker container(s) running on it. If you don’t, you will not be able to remove the docker network.
Now we undo the shim routing
|
1 2 3 4 |
$ sudo ip route del 192.168.1.64/30 dev myNewNet-shim $ sudo ip link set myNewNet-shim down $ sudo ip addr del 192.168.1.67/32 dev myNewNet-shim $ sudo ip link del myNewNet-shim link enp3s0 type macvlan mode bridge |
If we don’t do this, we won’t be able to re-do the shim setup because the old configuration will be present
Now we can remove the network, and re-create it. We will also redo the shim work.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
$ docker network rm myNewNet $ docker network create -d macvlan -o parent=enp3s0 \ --subnet 192.168.1.0/24 \ --gateway 192.168.1.1 \ --ip-range 192.168.1.64/29 \ --aux-address 'host=192.168.1.71' \ myNewNet $ sudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge $ sudo ip addr add 192.168.1.71/32 dev myNewNet-shim $ sudo ip link set myNewNet-shim up $ sudo ip route add 192.168.1.64/29 dev myNewNet-shim |
This is almost identical to what we did for the 4 IP network, but we’ve expanded the IP range, and changed the aux-address to be the last IP in the range. For the shim, similarly we’ve specified a different IP and range.
While it might seem like we can just restart the containers, we actually need to recreate them. Why? Well, the docker containers aren’t bound to a network name, but the instance ID of the network. With this delete / re-create we’ve used the same names – but have new unique IDs. If we try to restart the old containers we will get errors that the network is missing.
Note: The shim-routing instructions are not persistent, so you’ll want to capture those into a shell script. Mine lives in /usr/local/bin/macvlansetup and is triggered with a cron @reboot entry.
Messing with my macvlan network setup seemed pretty scary. While I was doing this I even made a few mistakes, forgetting initially to undo the shim so I had to go back an fix that. I also discovered that restarting my containers didn’t work because of the network ID changing. In all cases I had reasonable error messages to guide me forward. This blog post is the perfect version of what I did, without my errors but notes on what to look out for. That’s it – now I’ve got 4 additional IPs to play with.
Great post! Thank you. I’ve been wondering why so many guides suggest setting up a separate bridge network so that a container can interact with the host. Doesn’t this basically make everything work? Are there security issues to consider when creating a much larger macvlan network, perhaps with every IP address from 192.168.1.192 to 192.168.1.254, and having the shim post back to it?
docker network create -d macvlan -o parent=enp3s0 \
--subnet 192.168.1.0/24 \
--gateway 192.168.1.1 \
--ip-range 192.168.1.192/26 \
--aux-address 'host=192.168.1.192' \
myNewNet
sudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge
sudo ip addr add 192.168.1.192/32 dev myNewNet-shim
sudo ip link set myNewNet-shim up
sudo ip route add 192.168.1.192/26 dev myNewNet-shim
Glad the post helped.
You comment “doesn’t this make everything work?” – for me, it is magic. I’m able to use a container bound to one of the macvlan network IPs like it is a stand alone machine on my network.
As for security concerns. This is all within my local network. I have the same security concerns I would have with any machine on my network.