{"id":1750,"date":"2020-10-04T14:35:17","date_gmt":"2020-10-04T18:35:17","guid":{"rendered":"https:\/\/lowtek.ca\/roo\/?p=1750"},"modified":"2023-12-08T11:22:54","modified_gmt":"2023-12-08T15:22:54","slug":"docker-and-macvlan-networking-ipv4","status":"publish","type":"post","link":"https:\/\/lowtek.ca\/roo\/2020\/docker-and-macvlan-networking-ipv4\/","title":{"rendered":"Docker and macvlan networking (IPv4)"},"content":{"rendered":"

Docker<\/a> is the well known spin on Linux containers (LXC<\/a>), if you’re not already playing with containers it’s probably time to jump in and get familiar. I’ve been (very slowly) migrating my personal infrastructure over to a container centric setup.<\/p>\n

For me, containers are really nice for managing the set of software dependencies needed to run any particular application. It allows me to keep my RSS feed reader<\/a> up to date, and avoids me breaking something my WordPress install needs or vice versa. Containers are a light weight virtualization.<\/p>\n

The default networking model (default bridge<\/a>) allows you to easily expose (map) a set of ports from the container, onto the host. This makes it easy to host an nginx container as your webserver on port 80.<\/p>\n

Docker does some interesting network tricks to keep things more secure, but this gets problematic too. Containers can’t easily see the host they are on<\/a>, making it difficult for container A to see container B’s port on the host – however, you can put both containers on the same docker network to allow them to see each other. This is a subject for another blog post entirely.<\/p>\n

The macvlan support in docker<\/a> is very cool. It allows you to provision a second IP address on the same network card, giving your docker container a full IP on the local network. In the world of virtual machines, similar macvlan support is available, and when you want to treat a docker container like a mini-VM, this is very useful.<\/p>\n

<\/p>\n

Credit to this great blog post<\/a> that pretty much spells out what you need to do, but I’ll explain it in my own way below.<\/p>\n

Here are the steps:<\/p>\n

    \n
  1. Allocate an IP range for the network<\/li>\n
  2. Select an IP in that range to allow host access<\/li>\n
  3. Create a docker macvlan network<\/li>\n
  4. Create a docker container connected to that network<\/li>\n
  5. Modify the host network to route to the macvlan network<\/li>\n<\/ol>\n

    These 5 steps, really break down to this series of command lines<\/p>\n

    (3) Create docker macvlan network<\/p>\n

    $ docker network create -d macvlan -o parent=enp3s0 \\\r\n  --subnet 192.168.1.0\/24 \\\r\n  --gateway 192.168.1.1 \\\r\n  --ip-range 192.168.1.64\/30 \\\r\n  --aux-address 'host=192.168.1.67' \\\r\n  myNewNet<\/pre>\n
    (4) Create docker container connected to that network<\/p>\n
    $ docker run -it --network myNewNet ubuntu bash<\/pre>\n
    (5) Modify the host network to route to the macvlan network<\/p>\n
    $ sudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge\r\n$ sudo ip addr add 192.168.1.67\/32 dev myNewNet-shim\r\n$ sudo ip link set myNewNet-shim up\r\n$ sudo ip route add 192.168.1.64\/30 dev myNewNet-shim\r\n<\/pre>\n
    Now let’s walk through the breakdown of how those 3 sets of commands are composed, so you can understand how you would map these onto your specific situation.<\/p>\n
    Step 1<\/strong><\/p>\n
    We will assume here that there is a router on your local network which is providing DHCP services and handing out IP addresses. With the docker macvlan setup, docker will be assigning IP addresses and we need to make sure that we don’t have any conflicts.<\/p>\n
    Often the DHCP server is using a subset of IP addresses, and leaving many available for static allocation. We’ll just use some of those. My setup starts at 100 for DHCP, so any number below that is fine.<\/p>\n
    I’ll pick a CIDR range starting at 64 – but limit it to a few bits (2) –\u00a0 thus my range will be x.x.x.64\/30<\/b>\u00a0– or specifically\u00a0192.168.1.64\/30<\/code>\u00a0 I found this CIDR calculator<\/a> useful for helping me sort out the correct notation. This setup allocates only 4 addresses to docker to manage, but that’s fine for what I want.<\/p>\n
    Step 2<\/strong><\/p>\n
    By default, the docker container that is created will not have visibility from the host. Thus if the host is on 192.168.1.100 the new container we will create will predictably get 192.168.1.64 – the first out of the 4 reserved addresses – but the networking rules will prevent the host from seeing the new container. The inverse is also true, the new container won’t be able to see the host.<\/p>\n
    If we use another host on the same network, say 192.168.1.120 – we can see the new docker container (192.168.1.64) just like it was a regular machine on the network. We need to add a new route on the host so it can see this new network (and vice-versa), and to do that we need to reserve an address on that same network.<\/p>\n
    Let’s pick the last address of the 4 and reserve that (192.168.1.67).<\/p>\n
    Step 3<\/strong><\/p>\n
    Before we create the macvlan network, we need three more bits of information, the name of the network device we are going to attach to and the gateway on our network.<\/p>\n
    The first we can get this by running the following on the host, and looking for the name of the network card<\/p>\n
    $ ifconfig -a\r\n<\/pre>\n
    In my case, this is enp3s0<\/code>.<\/p>\n
    The same ifconfig command should show us the subnet that we are running on our local network. This can be figured out\u00a0 from ifconfig<\/code> output as the netmask is 255.255.255.0<\/code>\u00a0which should be typical for your local network – then the CIDR block is going to end with\u00a0\/24<\/code><\/p>\n
    The last bit we need is the gateway, this is probably your main router that acts as your connection to the internet.<\/p>\n
    $ ip route show<\/pre>\n
    The above will get us both the gateway and the subnet. The gateway to pick is from the first line that starts with default via <gateway><\/code><\/p>\n
    Now we have enough information to create the docker network<\/p>\n
    $ docker network create -d macvlan -o parent=enp3s0 \\\r\n --subnet 192.168.1.0\/24 \\\r\n --gateway 192.168.1.1 \\\r\n --ip-range 192.168.1.64\/30 \\\r\n --aux-address 'host=192.168.1.67' \\\r\n myNewNet<\/pre>\n
    Step 4<\/strong><\/p>\n
    Creating the docker container is very easy<\/p>\n
    docker run -it --network myNewNet ubuntu bash\r\n<\/pre>\n
    Now, from inside that container we can install some of the basic networking tools we will want<\/p>\n
    # apt-get update\r\n# apt-get install net-tools\r\n# apt-get install iputils-ping<\/pre>\n
    This will let us poke around from inside the container and see that we’re live, and see what IP address we think we have<\/p>\n
    root@b98e9b55d9fe:\/# ping otherhost\r\nPING otherhost.lan (192.168.1.120) 56(84) bytes of data.\r\n64 bytes from otherhost.lan (192.168.1.120): icmp_seq=1 ttl=64 time=0.196 ms\r\n64 bytes from otherhost.lan (192.168.1.120): icmp_seq=2 ttl=64 time=0.108 ms\r\n64 bytes from otherhost.lan (192.168.1.120): icmp_seq=3 ttl=64 time=0.166 ms<\/pre>\n
    We can also use another machine on the network to see that the new IP address (192.168.1.64) is alive and well.<\/p>\n
    mylaptop ~ % ping 192.168.1.64\r\nPING 192.168.1.64 (192.168.1.64): 56 data bytes\r\n64 bytes from 192.168.1.64: icmp_seq=0 ttl=64 time=2.056 ms\r\n64 bytes from 192.168.1.64: icmp_seq=1 ttl=64 time=2.000 ms\r\n64 bytes from 192.168.1.64: icmp_seq=2 ttl=64 time=1.963 ms\r\n<\/pre>\n
    But, we can’t see it from the docker host machine<\/p>\n
    dockerhost $ ping 192.168.1.64\r\nPING 192.168.1.64 (192.168.1.64) 56(84) bytes of data.\r\nFrom 192.168.1.100 icmp_seq=1 Destination Host Unreachable\r\nFrom 192.168.1.100 icmp_seq=2 Destination Host Unreachable\r\nFrom 192.168.1.100 icmp_seq=3 Destination Host Unreachable<\/pre>\n
    Step 5<\/strong><\/p>\n
    As we saw in step 4, the docker host can’t see the new container on the network. This is a problem if you want to run other containers on the same host, but have those interact with the container that is on its own IP address.<\/p>\n
    If you remember back in step 2 we reserved an address, this is what we’re going to use it for – to build a bridge on the host to the new network.<\/p>\n
    We are going to create a new interface linked to the same ethernet device, assign the reserved IP address to it, bring the interface up, and add a routing rule.<\/p>\n
    $ sudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge\r\n$ sudo ip addr add 192.168.1.67\/32 dev myNewNet-shim\r\n$ sudo ip link set myNewNet-shim up\r\n$ sudo ip route add 192.168.1.64\/30 dev myNewNet-shim<\/pre>\n
    At this point, we now have a way to route traffic from the host – to the new container.<\/p>\n
    dockerhost $ ping 192.168.1.64\r\nPING 192.168.1.64 (192.168.1.64) 56(84) bytes of data.\r\n64 bytes from 192.168.1.64: icmp_seq=1 ttl=64 time=0.077 ms\r\n64 bytes from 192.168.1.64: icmp_seq=2 ttl=64 time=0.035 ms<\/pre>\n
    The ip<\/code> commands above are not persistent, so a reboot will return us to the same state where the host can’t see the new container on it’s own IP address.<\/p>\n
    We can fix this easily by creating (or editing) the \/etc\/rc.local<\/code> file and adding these commands there (obviously without the sudo<\/code> as this file will run as root). Make sure that \/etc\/rc.local<\/code> has the right permissions and is set to executable.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Docker is the well known spin on Linux containers (LXC), if you’re not already playing with containers it’s probably time to jump in and get familiar. I’ve been (very slowly) migrating my personal infrastructure over to a container centric setup. For me, containers are really nice for managing the set of software dependencies needed to … Continue reading “Docker and macvlan networking (IPv4)”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,12],"tags":[],"class_list":["post-1750","post","type-post","status-publish","format-standard","hentry","category-computing","category-how-to"],"_links":{"self":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/1750","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/comments?post=1750"}],"version-history":[{"count":8,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/1750\/revisions"}],"predecessor-version":[{"id":2286,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/1750\/revisions\/2286"}],"wp:attachment":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/media?parent=1750"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/categories?post=1750"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/tags?post=1750"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}