{"id":2287,"date":"2023-12-09T13:47:12","date_gmt":"2023-12-09T17:47:12","guid":{"rendered":"https:\/\/lowtek.ca\/roo\/?p=2287"},"modified":"2023-12-09T13:47:12","modified_gmt":"2023-12-09T17:47:12","slug":"expanding-a-docker-macvlan-network","status":"publish","type":"post","link":"https:\/\/lowtek.ca\/roo\/2023\/expanding-a-docker-macvlan-network\/","title":{"rendered":"Expanding a docker macvlan network"},"content":{"rendered":"<p>I&#8217;ve <a href=\"https:\/\/lowtek.ca\/roo\/2020\/docker-and-macvlan-networking-ipv4\/\">previously written about using macvlan networks with docker<\/a>, this has proved to be a great way to make containers more like lightweight VMs as you can assign a unique IP on your network to them. Unfortunately when I did this I only allocated 4 IPs to the network, and 1 of those is used to provide a communication path from the host to the macvlan network.<\/p>\n<p>Here is how I&#8217;ve used up those 4 IPs:<\/p>\n<ol>\n<li><a href=\"https:\/\/docs.linuxserver.io\/images\/docker-wireguard\/\">wireguard<\/a> &#8211; allows clients on wireguard to see other docker services on the host<\/li>\n<li><a href=\"https:\/\/mosquitto.org\/\">mqtt broker<\/a> &#8211; used to bridge between my IoT network and the lan network without exposing all of my lan to the IoT network<\/li>\n<li><a href=\"https:\/\/docs.linuxserver.io\/images\/docker-nginx\/\">nginx<\/a> &#8211; a local only webserver, useful for fronting <a href=\"https:\/\/docs.linuxserver.io\/images\/docker-homeassistant\/\">Home Assistant<\/a> and other web based apps I use<\/li>\n<li>shim &#8211; IP allocated to supporting routing from the host to the macvlan network.<\/li>\n<\/ol>\n<p>If I had known how useful giving a container a unique IP on the network was, I would have allocated more up front. Unfortunately <a href=\"https:\/\/stackoverflow.com\/questions\/64596780\/how-to-change-configuration-of-existing-docker-network\">you can&#8217;t easily grow a docker network<\/a>, you need to delete and recreate it.<\/p>\n<p>As an overview here is what we need to do.<\/p>\n<ul>\n<li>Stop any docker container that is attached to the macvlan network<\/li>\n<li>Undo the shim routing<\/li>\n<li>Delete the docker network<\/li>\n<li>Recreate the docker network (expanded)<\/li>\n<li>Redo the shim routing<\/li>\n<li>Recreate the existing containers<\/li>\n<\/ul>\n<p>This ends up not being too hard, and the only slightly non-obvious step is undoing the shim routing, which is the reverse of the setup.<\/p>\n<pre class=\"lang:default decode:true\">$ sudo ip route del 192.168.1.64\/30 dev myNewNet-shim\r\n$ sudo ip link set myNewNet-shim down\r\n$ sudo ip addr del 192.168.1.67\/32 dev myNewNet-shim\r\n$ sudo ip link del myNewNet-shim link enp3s0 type macvlan mode bridge\r\n<\/pre>\n<p>The remainder of this post is a walk through of setting up a 4 IP network, then tearing it down and setting up a larger 8 IP network.<\/p>\n<p><!--more--><\/p>\n<p>The first thing we&#8217;ll do is inspect the existing docker macvlan network. That will give us information on what we provisioned originally<\/p>\n<pre class=\"lang:default decode:true\">$ docker network inspect myNewNet<\/pre>\n<p>There is quite a bit of output, but the chunk we care about is the &#8220;Config&#8221; section<\/p>\n<pre class=\"lang:default decode:true \">            \"Config\": [\r\n                {\r\n                    \"Subnet\": \"192.168.1.0\/24\",\r\n                    \"IPRange\": \"192.168.1.64\/30\",\r\n                    \"Gateway\": \"192.168.1.1\",\r\n                    \"AuxiliaryAddresses\": {\r\n                        \"host\": \"192.168.1.67\"\r\n                    }\r\n                }\r\n            ]<\/pre>\n<p>This aligns with what we setup previously. However, if we pretend for a second we are starting on a brand new machine here are the docker macvlan setup steps we would do.<\/p>\n<pre class=\"lang:default decode:true\"># Create the network\r\n$ docker network create -d macvlan -o parent=enp3s0 \\\r\n  --subnet 192.168.1.0\/24 \\\r\n  --gateway 192.168.1.1 \\\r\n  --ip-range 192.168.1.64\/30 \\\r\n  --aux-address 'host=192.168.1.67' \\\r\n  myNewNet\r\n\r\n# Setup the shim routing so the host can see the new containers via IP\r\n$ sudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge\r\n$ sudo ip addr add 192.168.1.67\/32 dev myNewNet-shim\r\n$ sudo ip link set myNewNet-shim up\r\n$ sudo ip route add 192.168.1.64\/30 dev myNewNet-shim<\/pre>\n<p>Cool, so that&#8217;s a 4 IP macvlan network setup. We can test it out with a new docker container<\/p>\n<pre class=\"lang:default decode:true\"># Best to do this in a second shell window\r\n\r\n# Create new container on the new network and specify the IP we want to assign\r\n$ docker run -it --network myNewNet --ip 192.168.1.50 ubuntu bash\r\n\r\n# Now inside of that container, we install some tools\r\n$ apt-get update\r\n$ apt-get install net-tools\r\n$ apt-get install iputils-ping\r\n\r\n# And we can ping the host\r\n$ ping 192.168.1.88\r\nPING 192.168.1.88 (192.168.1.88) 56(84) bytes of data.\r\n64 bytes from 192.168.1.88: icmp_seq=1 ttl=64 time=0.236 ms\r\n64 bytes from 192.168.1.88: icmp_seq=2 ttl=64 time=0.206 ms\r\n64 bytes from 192.168.1.88: icmp_seq=3 ttl=64 time=0.134 ms\r\n<\/pre>\n<p>Now from the host &#8211; we can via the shim also see the new container<\/p>\n<pre class=\"lang:default decode:true \">$ sudo ping 192.168.1.50\r\nPING 192.168.1.50 (192.168.1.50) 56(84) bytes of data.\r\n64 bytes from 192.168.1.50: icmp_seq=1 ttl=64 time=0.086 ms\r\n64 bytes from 192.168.1.50: icmp_seq=2 ttl=64 time=0.064 ms\r\n64 bytes from 192.168.1.50: icmp_seq=3 ttl=64 time=0.064 ms<\/pre>\n<p>At this point we have a functioning 4 IP macvlan network. Time to tear it down and recreate it with more IPs. If we were to repeat the docker network inspect we would see the same information we had above.<\/p>\n<p>Stop the docker container(s) running on it. If you don&#8217;t, you will not be able to remove the docker network.<\/p>\n<p>Now we undo the shim routing<\/p>\n<pre class=\"lang:default decode:true \">$ sudo ip route del 192.168.1.64\/30 dev myNewNet-shim \r\n$ sudo ip link set myNewNet-shim down \r\n$ sudo ip addr del 192.168.1.67\/32 dev myNewNet-shim \r\n$ sudo ip link del myNewNet-shim link enp3s0 type macvlan mode bridge<\/pre>\n<p>If we don&#8217;t do this, we won&#8217;t be able to re-do the shim setup because the old configuration will be present<\/p>\n<p>Now we can remove the network, and re-create it. We will also redo the shim work.<\/p>\n<pre class=\"lang:default decode:true\">$ docker network rm myNewNet\r\n\r\n$ docker network create -d macvlan -o parent=enp3s0 \\\r\n  --subnet 192.168.1.0\/24 \\\r\n  --gateway 192.168.1.1 \\\r\n  --ip-range 192.168.1.64\/29 \\\r\n  --aux-address 'host=192.168.1.71' \\\r\n  myNewNet\r\n\r\n$ sudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge\r\n$ sudo ip addr add 192.168.1.71\/32 dev myNewNet-shim\r\n$ sudo ip link set myNewNet-shim up\r\n$ sudo ip route add 192.168.1.64\/29 dev myNewNet-shim<\/pre>\n<p>This is almost identical to what we did for the 4 IP network, but we&#8217;ve expanded the IP range, and changed the aux-address to be the last IP in the range. For the shim, similarly we&#8217;ve specified a different IP and range.<\/p>\n<p>While it might seem like we can just restart the containers, we actually need to recreate them. Why? Well, the docker containers aren&#8217;t bound to a network name, but the instance ID of the network. With this delete \/ re-create we&#8217;ve used the same names &#8211; but have new unique IDs. If we try to restart the old containers we will get errors that the network is missing.<\/p>\n<p><strong>Note:<\/strong> The shim-routing instructions are not persistent, so you&#8217;ll want to capture those into a shell script. Mine lives in <code>\/usr\/local\/bin\/macvlansetup<\/code> and is triggered with a <a href=\"https:\/\/www.linode.com\/docs\/guides\/run-jobs-or-scripts-using-crontab-on-boot\/\">cron @reboot entry<\/a>.<\/p>\n<p>Messing with my macvlan network setup seemed pretty scary. While I was doing this I even made a few mistakes, forgetting initially to undo the shim so I had to go back an fix that. I also discovered that restarting my containers didn&#8217;t work because of the network ID changing. In all cases I had reasonable error messages to guide me forward. This blog post is the perfect version of what I did, without my errors but notes on what to look out for. That&#8217;s it &#8211; now I&#8217;ve got 4 additional IPs to play with.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve previously written about using macvlan networks with docker, this has proved to be a great way to make containers more like lightweight VMs as you can assign a unique IP on your network to them. Unfortunately when I did this I only allocated 4 IPs to the network, and 1 of those is used &hellip; <a href=\"https:\/\/lowtek.ca\/roo\/2023\/expanding-a-docker-macvlan-network\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Expanding a docker macvlan network&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,12,21],"tags":[],"class_list":["post-2287","post","type-post","status-publish","format-standard","hentry","category-computing","category-how-to","category-network"],"_links":{"self":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/2287","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/comments?post=2287"}],"version-history":[{"count":3,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/2287\/revisions"}],"predecessor-version":[{"id":2290,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/2287\/revisions\/2290"}],"wp:attachment":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/media?parent=2287"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/categories?post=2287"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/tags?post=2287"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}