Docker and macvlan networking (IPv4)

Docker is the well known spin on Linux containers (LXC), if you’re not already playing with containers it’s probably time to jump in and get familiar. I’ve been (very slowly) migrating my personal infrastructure over to a container centric setup.

For me, containers are really nice for managing the set of software dependencies needed to run any particular application. It allows me to keep my RSS feed reader up to date, and avoids me breaking something my WordPress install needs or vice versa. Containers are a light weight virtualization.

The default networking model (default bridge) allows you to easily expose (map) a set of ports from the container, onto the host. This makes it easy to host an nginx container as your webserver on port 80.

Docker does some interesting network tricks to keep things more secure, but this gets problematic too. Containers can’t easily see the host they are on, making it difficult for container A to see container B’s port on the host – however, you can put both containers on the same docker network to allow them to see each other. This is a subject for another blog post entirely.

The macvlan support in docker is very cool. It allows you to provision a second IP address on the same network card, giving your docker container a full IP on the local network. In the world of virtual machines, similar macvlan support is available, and when you want to treat a docker container like a mini-VM, this is very useful.

Credit to this great blog post that pretty much spells out what you need to do, but I’ll explain it in my own way below.

Here are the steps:

1. Allocate an IP range for the network
2. Select an IP in that range to allow host access
3. Create a docker macvlan network
4. Create a docker container connected to that network
5. Modify the host network to route to the macvlan network

These 5 steps, really break down to this series of command lines

(3) Create docker macvlan network

(4) Create docker container connected to that network

(5) Modify the host network to route to the macvlan network

Now let’s walk through the breakdown of how those 3 sets of commands are composed, so you can understand how you would map these onto your specific situation.

Step 1

We will assume here that there is a router on your local network which is providing DHCP services and handing out IP addresses. With the docker macvlan setup, docker will be assigning IP addresses and we need to make sure that we don’t have any conflicts.

Often the DHCP server is using a subset of IP addresses, and leaving many available for static allocation. We’ll just use some of those. My setup starts at 100 for DHCP, so any number below that is fine.

I’ll pick a CIDR range starting at 64 – but limit it to a few bits (2) –  thus my range will be x.x.x.64/30 – or specifically 192.168.1.64/30  I found this CIDR calculator useful for helping me sort out the correct notation. This setup allocates only 4 addresses to docker to manage, but that’s fine for what I want.

Step 2

By default, the docker container that is created will not have visibility from the host. Thus if the host is on 192.168.1.100 the new container we will create will predictably get 192.168.1.64 – the first out of the 4 reserved addresses – but the networking rules will prevent the host from seeing the new container. The inverse is also true, the new container won’t be able to see the host.

If we use another host on the same network, say 192.168.1.120 – we can see the new docker container (192.168.1.64) just like it was a regular machine on the network. We need to add a new route on the host so it can see this new network (and vice-versa), and to do that we need to reserve an address on that same network.

Let’s pick the last address of the 4 and reserve that (192.168.1.67).

Step 3

Before we create the macvlan network, we need three more bits of information, the name of the network device we are going to attach to and the gateway on our network.

The first we can get this by running the following on the host, and looking for the name of the network card

In my case, this is enp3s0.

The same ifconfig command should show us the subnet that we are running on our local network. This can be figured out  from ifconfig output as the netmask is 255.255.255.0 which should be typical for your local network – then the CIDR block is going to end with /24

The last bit we need is the gateway, this is probably your main router that acts as your connection to the internet.

The above will get us both the gateway and the subnet. The gateway to pick is from the first line that starts with default via <gateway>

Now we have enough information to create the docker network

Step 4

Creating the docker container is very easy

Now, from inside that container we can install some of the basic networking tools we will want

This will let us poke around from inside the container and see that we’re live, and see what IP address we think we have

We can also use another machine on the network to see that the new IP address (192.168.1.64) is alive and well.

But, we can’t see it from the docker host machine

Step 5

As we saw in step 4, the docker host can’t see the new container on the network. This is a problem if you want to run other containers on the same host, but have those interact with the container that is on its own IP address.

If you remember back in step 2 we reserved an address, this is what we’re going to use it for – to build a bridge on the host to the new network.

We are going to create a new interface linked to the same ethernet device, assign the reserved IP address to it, bring the interface up, and add a routing rule.

At this point, we now have a way to route traffic from the host – to the new container.

The ip commands above are not persistent, so a reboot will return us to the same state where the host can’t see the new container on it’s own IP address.

We can fix this easily by creating (or editing) the /etc/rc.local file and adding these commands there (obviously without the sudo as this file will run as root). Make sure that /etc/rc.local has the right permissions and is set to executable.