{"id":2643,"date":"2026-04-06T17:51:46","date_gmt":"2026-04-06T21:51:46","guid":{"rendered":"https:\/\/lowtek.ca\/roo\/?p=2643"},"modified":"2026-04-06T17:51:46","modified_gmt":"2026-04-06T21:51:46","slug":"update-nixos-docker-macvlan","status":"publish","type":"post","link":"https:\/\/lowtek.ca\/roo\/2026\/update-nixos-docker-macvlan\/","title":{"rendered":"Update: NixOS + Docker + MacVLAN"},"content":{"rendered":"<p><a href=\"https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2026\/04\/arp-flux.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-2644\" src=\"https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2026\/04\/arp-flux-500x155.jpg\" alt=\"\" width=\"500\" height=\"155\" srcset=\"https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2026\/04\/arp-flux-500x155.jpg 500w, https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2026\/04\/arp-flux.jpg 600w\" sizes=\"auto, (max-width: 500px) 85vw, 500px\" \/><\/a><\/p>\n<p>This is an updated version of my previous article <a href=\"https:\/\/lowtek.ca\/roo\/2025\/nixos-docker-with-macvlan-ipv4\/\">NixOS + Docker with MacVLAN (IPv4)<\/a> &#8211; which addresses ARP Flux which causing networking problems. Skip to the bottom <strong>summary<\/strong> if you want to just read the conclusion.<\/p>\n<p>Ever since, but very intermittently, my <a href=\"https:\/\/lowtek.ca\/roo\/2025\/nixos-with-mirrored-zfs-boot-volume\/\">new server build out<\/a> &#8211; I had been having issues with my macbook (<a href=\"https:\/\/en.wikipedia.org\/wiki\/MacOS\">OSX<\/a>) randomly losing connection to the server, but only for a brief time. I noticed mostly when using <a href=\"https:\/\/mosh.org\/\">mosh<\/a> and it would indicate that I&#8217;d lost connection, but self hosted web UIs like <a href=\"https:\/\/github.com\/grafana\/grafana\">Grafana<\/a> also gave me problems.<\/p>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Ping_(networking_utility)\">Ping<\/a> would also fail, but then it would start to work after a minute or so. What was weird is that while I could not ping my new server (192.168.1.79) I could have a stable mosh connection to another box (192.168.1.140), which could see the new server (192.168.1.79) just fine. This made it seem very much like it was something wrong with OSX. I was guilty of mostly ignoring this because it did impact me, but not frequently &#8212; it was still very annoying.<\/p>\n<p>This is also an exploration of using <a href=\"https:\/\/gemini.google.com\/\">Google Gemini<\/a>, which you can use from an incognito tab with no login required. It didn&#8217;t immediately get me to the right answer, but we eventually got there.<\/p>\n<p>The first iteration didn&#8217;t really get me very far, only a suggestion to check the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Routing_table\">route<\/a> on my OSX machine when the problem was happening. This meant I had to wait around for it to break &#8211; and catch it in the act.<\/p>\n<pre class=\"lang:default decode:true \">% route -n get myserver.lan\r\n   route to: 192.168.1.79\r\ndestination: 192.168.1.79\r\n  interface: en0\r\n      flags: &lt;UP,HOST,DONE,LLINFO,WASCLONED,IFSCOPE,IFREF&gt;\r\n recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire\r\n       0         0         0         0         0         0      1500      1167<\/pre>\n<p>Unfortunately the output from this didn&#8217;t change with broken networking vs. working. At this point I added the detail that this networking problem seemed to happen most often after waking the OSX machine from sleep. Gemini was then able to reason:<\/p>\n<blockquote><p>In your route output, the flag LLINFO stands for Link Layer Information. This means macOS has a cached MAC address for 192.168.1.79.<\/p>\n<p>When a Mac sleeps, it often hands off its networking responsibilities to a &#8220;Sleep Proxy&#8221; (like an Apple TV or HomePod) or simply drops its ARP table. When it wakes up, it might be trying to use a stale hardware address, or the Linux machine (myserver.lan) might not be responding to the Mac&#8217;s &#8220;Who has 192.168.1.79?&#8221; request fast enough.<\/p><\/blockquote>\n<p>The next suggestion was to check the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Address_Resolution_Protocol\">arp<\/a> information when networking was broken.<\/p>\n<pre class=\"lang:default decode:true\">arp -a | grep 192.168.1.79\r\n<\/pre>\n<blockquote>\n<ul>\n<li>If it says (incomplete), the Mac is sending requests but getting no answer.<\/li>\n<li>If it shows a MAC address, compare it to the actual MAC address of the Linux box. If they match, the physical path is clear but the packets are being dropped higher up.<\/li>\n<\/ul>\n<\/blockquote>\n<p>This got me looking at my <a href=\"https:\/\/github.com\/andrewlow\/WatchYourLAN-Patch\">WatchYourLan<\/a> deployment, this is a networking monitoring tool that I use to alert me to new devices (based on <a href=\"https:\/\/en.wikipedia.org\/wiki\/MAC_address\">MAC addresses<\/a>) on my network.<\/p>\n<p>To my surprise &#8211; I found another MAC <code>82:d4:9a:8f:06:40<\/code> for myserver.lan, one that I didn&#8217;t expect to see. Gemini called this a smoking gun, as it is a Locally Administered Address (<a href=\"https:\/\/en.wikipedia.org\/wiki\/MAC_address#Locally_administered_address\">LAA<\/a>) &#8211; also known as a generated MAC. Gemini also concludes that this may be a Docker issue, which somewhat confirms what I was thinking. The advice from Gemini is getting more specific now, but I also find that in general this is where AI generated output can go off the rails so caution is required.<\/p>\n<blockquote><p><strong>The Pattern: You have a &#8220;Bridge&#8221; problem<\/strong><\/p>\n<p>Seeing four different randomized MACs for a single IP (192.168.1.79) on a wired Linux machine is a classic symptom of Docker Userland Proxy or Bridge-utils misconfiguration.<\/p>\n<p>When you run containers or virtual machines in &#8220;Bridge Mode,&#8221; the Linux kernel creates a virtual switch (br0, docker0, etc.).<\/p>\n<p>The physical NIC (eth0) has its own hardware MAC.<\/p>\n<p>The Bridge (br0) often generates its own MAC (like one of these A6 or 52 ones).<\/p>\n<p>Each container gets a virtual interface (veth) with a randomized MAC.<\/p><\/blockquote>\n<p>I then remember to add the detail that this Linux machine is a <a href=\"https:\/\/nixos.org\/\">NixOS<\/a> installation, and Gemini comes back with<\/p>\n<blockquote><p>NixOS with a complex Docker setup is the &#8220;perfect storm&#8221; for this. NixOS tends to be very explicit about interface creation, and Docker&#8217;s default bridge behavior often clashes with how NixOS manages its own networking stack.<\/p><\/blockquote>\n<p>It suggests a &#8220;ghost hunter&#8221; command to identify bad MACs<\/p>\n<pre class=\"lang:default decode:true \">ip link show | grep -iE '..:[0-9a-f][26ae]:' -B 1\r\n<\/pre>\n<blockquote><p>If these &#8220;ghost&#8221; MACs are responding to ARP requests on your physical LAN (a phenomenon called ARP Flux), it\u2019s usually because the Linux kernel is being too &#8220;helpful&#8221; by answering ARP requests for an IP on any interface, even if the request came in on a different one.<\/p><\/blockquote>\n<p>It also suggests a change to my <code>configuration.nix<\/code> file, which in retrospect was pretty close to the final solution &#8211; but not quite. At this point I&#8217;m pretty impressed with Gemini being able to help with NixOS specifics. It also asks the question<\/p>\n<blockquote><p>Does your NixOS config happen to have any bridge interfaces (networking.bridges) or macvlan setups defined manually?<\/p><\/blockquote>\n<p>Why, yes &#8211; yes I do I reply. Gemini then fires back:<\/p>\n<blockquote><p>This macvlan-shim is the &#8220;smoking gun&#8221; for your macOS networking issues.<\/p>\n<p><strong>Why this breaks your Mac&#8217;s connection<\/strong><br \/>\nBy default, Linux is very &#8220;promiscuous&#8221; with ARP. When your Mac sends an ARP request asking &#8220;Who is 192.168.1.79?&#8221;, the Linux kernel sees that request on enp3s0. Because the macvlan-shim is bridged to that same physical card, the kernel might answer the Mac using the <strong>macvlan&#8217;s generated MAC address<\/strong> instead of the physical NIC&#8217;s hardware address.<\/p>\n<p>This is known as <strong>ARP Flux<\/strong>. Your Mac gets confused because it sees multiple MAC addresses (the &#8220;ghosts&#8221; we found earlier) all claiming to be the same IP or living on the same segment. When the Mac wakes from sleep, it&#8217;s a race: if the &#8220;shim&#8221; MAC wins the ARP race, the Mac sends traffic to a virtual interface that isn&#8217;t configured to handle the host&#8217;s primary traffic.<\/p><\/blockquote>\n<p>It sure likes the term &#8220;smoking gun&#8221;, but it feels like we&#8217;re close to identifying the real root problem in my setup. I had previously done some changes to the way I deployed the various docker containers on my macvlan network to set static MAC addresses, but had not yet figured out how to make the shim IP have a static MAC address. The evidence I have and the advice from Gemini are starting to align. Odd that I got to this configuration from a <a href=\"https:\/\/lowtek.ca\/roo\/2020\/docker-and-macvlan-networking-ipv4\/\">previous setup using Ubuntu<\/a> which didn&#8217;t have an issue, or maybe it was just luck due to timing.<\/p>\n<p>I asked a lot of clarifying questions about the solutions it was proposing, which helped iterate towards both my more complete understanding of the solution &#8211; but it also helped generate some options. A minimalist solution to this was suggested, but felt like a lot of &#8216;trust in the force&#8217;:<\/p>\n<pre class=\"lang:default decode:true\">boot.kernel.sysctl = {\r\n  # This is the \"Nuclear Option\" that covers everything at once\r\n  \"net.ipv4.conf.all.arp_ignore\" = 1;\r\n  \"net.ipv4.conf.all.arp_announce\" = 2;\r\n};<\/pre>\n<p>If you added this to my previous article on <a href=\"https:\/\/lowtek.ca\/roo\/2025\/nixos-docker-with-macvlan-ipv4\/\">NixOS + Docker + MacVLAN<\/a> you&#8217;d probably be fine. However, here is the more complete solution I ended up using:<\/p>\n<pre class=\"lang:default decode:true\">{\r\n  networking.macvlans.\"myNewNet-shim\" = {\r\n    mode = \"bridge\";\r\n    interface = \"enp3s0\";\r\n  };\r\n\r\n  networking.interfaces.\"myNewNet-shim\" = {\r\n    macAddress = \"06:00:00:00:00:67\";\r\n    ipv4.addresses = [{ address = \"192.168.1.67\"; prefixLength = 32; }];\r\n    ipv4.routes = [{ address = \"192.168.1.64\"; prefixLength = 30; }];\r\n  };\r\n\r\n  # We need to prevent Arp Flux and silence any ghost replies\r\n  # Ensure the kernel only answers for 192.168.1.79 on the physical \r\n  # card and for containers on the virtual IPs\r\n  boot.kernel.sysctl = {\r\n    # Only reply if the target IP is on the specific \r\n    # interface receiving the request\r\n    \"net.ipv4.conf.all.arp_ignore\" = 1;          # Required, only reply\r\n                                                 # to ARP for your interface\r\n    \"net.ipv4.conf.enp5s0.arp_ignore\" = 1;       # Redundant, covered by\r\n                                                 # above all setting\r\n    \"net.ipv4.conf.macvlan-shim.arp_ignore\" = 1; # Recommended, keeps \r\n                                                 # shim silent\r\n\r\n    # Ensure ARP announcements use the primary IP of the interface\r\n    \"net.ipv4.conf.all.arp_announce\" = 2;        # Required, always use\r\n                                                 # the right interface\r\n    \"net.ipv4.conf.enp5s0.arp_announce\" = 2;     # Redundant, covered by\r\n                                                 # above all setting\r\n    \"net.ipv4.conf.macvlan-shim.arp_announce\" = 2;  # Recommended, do not\r\n                                                    # leak MACs\r\n  };\r\n}<\/pre>\n<p>Again, if you compare with the <a href=\"https:\/\/lowtek.ca\/roo\/2025\/nixos-docker-with-macvlan-ipv4\/\">original article<\/a> we can see that I&#8217;ve added a specific MAC address for the shim, and used a naming convention to make the last digit match the IP address. The big change is the <code>boot.kernel.sysctl<\/code> which is very similar to the minimal setup above.<\/p>\n<p>I also use the new MAC address numbering scheme for each of my macvlan containers &#8211; assigning them <code>06:00:00:00:00:XX<\/code> where <code>XX<\/code> is the IP. Very handy to see they are the right assignment.<\/p>\n<p>Now this had some interesting side-effects. The shim IP continued to offer up a stale MAC generated MAC address. I was able to fix this by forcing a recreate (but I suspect a reboot may have solved the problem).<\/p>\n<pre class=\"lang:default decode:true\">sudo ip link set myNewNet-shim down\r\nsudo ip link delete myNewNet-shim\r\nsudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge\r\nsudo ip link set myNewNet-shim address 06:00:00:00:00:67\r\nsudo ip link set myNewNet-shim up<\/pre>\n<p>A few other things also broke due to this more restrictive ARP control. Previously a container living on the macvlan network (like my nginx for local web services &#8211; on IP 192.168.1.65) could see my server (192.168.1.79). The <code>myNewNet-shim<\/code> provided visibility from the host to the nginx (192.168.1.65).<\/p>\n<p>In this new no ARP Flux world, I have to use the shim address (192.168.1.67) in order to connect to my server (192.168.1.79) &#8211; so everywhere I reference <code>x.x.x.79<\/code> I now needed to use <code>x.x.x.67<\/code>.<\/p>\n<p>I&#8217;ve realized that this makes my wireguard setup a bit more annoying because if I use wireguard to connect home from remote, and I want to connect to my server I need to use the shim address instead of the local DNS name that maps to the real address. This was too big a trade off so I went back to Gemini and had a long discussion (with a lot of dead ends) to arrive at a solution that adds back the visibility of the host (192.168.1.79) from the macvlan containers (say 192.168.1.64 for example).<\/p>\n<pre class=\"lang:default decode:true\">  # Create a \"stealth\" host IP on the macvlan bridge.\r\n  # This allows containers\/WireGuard to talk to the host at 192.168.1.79.\r\n  # 'scope host' is critical: it prevents ARP Flux by ensuring the host \r\n  # never advertises this IP to the physical network (enp3s0).\r\n  systemd.services.\"network-addresses-myNewNet-shim\" = {\r\n    # We use the absolute path to 'ip' to avoid needing to set the 'path' variable\r\n    postStart = \"${pkgs.iproute2}\/bin\/ip addr add 192.168.1.79\/32 dev myNewNet-shim scope host || true\";\r\n  };<\/pre>\n<p>Now using wireguard, I can see the full local network &#8211; host included.<\/p>\n<p>Less ARP chaos is a good thing. Gemini did suggest that instead of declaring the route for the shim with a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Classless_Inter-Domain_Routing\">CIDR<\/a> block that includes the shim address, I could be even more specific and simply have a route per IP address:<\/p>\n<pre class=\"lang:default decode:true\">  ipv4.routes = [\r\n    # 1. Route the WireGuard container specifically\r\n    { address = \"192.168.1.64\"; prefixLength = 32; } \r\n    \r\n    # 2. Route your other containers specifically\r\n    { address = \"192.168.1.65\"; prefixLength = 32; }\r\n    { address = \"192.168.1.66\"; prefixLength = 32; } \r\n\r\n    # 3. Keep your WireGuard tunnel route\r\n    { address = \"10.13.13.0\"; prefixLength = 24; via = \"192.168.1.64\"; }\r\n  ];<\/pre>\n<p>The key difference here is using <code>\/32<\/code> instead of <code>\/30<\/code>. Being specific avoids the Linux Kernel from having to figure out what to do with the shim address (192.168.1.67) but it seems to do the right thing so I&#8217;ve gone with the simpler declaration. I&#8217;ve also got a special route in there for wireguard addresses so my host can see the wireguard clients directly.<\/p>\n<p>Verifying the fix can be done on the OSX machine<\/p>\n<pre class=\"lang:default decode:true \">arp -a | grep 192.168.1<\/pre>\n<p>We want to review all of the MAC addresses to make sure we have the expected ones that follow the pattern <code>06:00:00:00:00:xx<\/code>. If all goes well, this is the end of the ghosting problem (ARP Flux) that will cause the annoying interruption in networking from my OSX machine to the server.<\/p>\n<p><strong>In summary &#8211; the full docker macvlan setup on NixOS<\/strong> &#8211; this is basically a brief re-telling of the <a href=\"https:\/\/lowtek.ca\/roo\/2025\/nixos-docker-with-macvlan-ipv4\/\">original post<\/a>, with all of the updates above merged in.<\/p>\n<p>First enable docker support on NixOS. You need a single line added to your <code>\/etc\/nixos\/configuration.nix<\/code><\/p>\n<pre class=\"lang:default decode:true\">virtualisation.docker.enable = true;<\/pre>\n<p>Create your docker macvlan network.<\/p>\n<pre class=\"lang:default decode:true \">$ docker network create -d macvlan -o parent=enp3s0 \\\r\n  --subnet 192.168.1.0\/24 \\\r\n  --gateway 192.168.1.1 \\\r\n  --ip-range 192.168.1.64\/30 \\\r\n  --aux-address 'host=192.168.1.67' \\\r\n  myNewNet<\/pre>\n<p>Docker will persist this network configuration across reboots.<\/p>\n<p>Now we need to modify <code>\/etc\/nixos\/configuration.nix<\/code> to fix routing to\/from our macvlan network IPs from the host &#8211; and avoid causing APR Flux.<\/p>\n<pre class=\"lang:default decode:true\">  networking.macvlans.\"myNewNet-shim\" = {\r\n    mode = \"bridge\";\r\n    interface = \"enp3s0\";\r\n  };\r\n\r\n  networking.interfaces.\"myNewNet-shim\" = {\r\n    macAddress = \"06:00:00:00:00:67\";\r\n    ipv4.addresses = [{ address = \"192.168.1.67\"; prefixLength = 32; }];\r\n    ipv4.routes = [{ address = \"192.168.1.64\"; prefixLength = 30; }];\r\n  };\r\n\r\n  # Create a \"stealth\" host IP on the macvlan bridge. \r\n  # This allows containers\/WireGuard to talk to the host at 192.168.1.79. \r\n  # 'scope host' is critical: it prevents ARP Flux by ensuring the host \r\n  # never advertises this IP to the physical network (enp3s0). \r\n  systemd.services.\"network-addresses-myNewNet-shim\" = { \r\n  # We use the absolute path to 'ip' to avoid needing to set \r\n  # the 'path' variable \r\n  postStart = \"${pkgs.iproute2}\/bin\/ip addr add 192.168.1.79\/32 dev myNewNet-shim scope host || true\"; \r\n  };\r\n\r\n  # ARP Flux Protection (Defense in Depth)\r\n  # While 'scope host' on the macvlan-shim prevents the host from \r\n  # advertising 192.168.1.79 to the physical network, these sysctl \r\n  # settings ensure the kernel's ARP behavior is strictly tied \r\n  # to the specific interface receiving the request.\r\n  boot.kernel.sysctl = {\r\n    # Only reply if the target IP is on the specific \r\n    # interface receiving the request\r\n    \"net.ipv4.conf.all.arp_ignore\" = 1;          # Required, only reply\r\n                                                 # to ARP for your interface\r\n    \"net.ipv4.conf.enp5s0.arp_ignore\" = 1;       # Redundant, covered by\r\n                                                 # above all setting\r\n    \"net.ipv4.conf.macvlan-shim.arp_ignore\" = 1; # Recommended, keeps \r\n                                                 # shim silent\r\n\r\n    # Ensure ARP announcements use the primary IP of the interface\r\n    \"net.ipv4.conf.all.arp_announce\" = 2;        # Required, always use\r\n                                                 # the right interface\r\n    \"net.ipv4.conf.enp5s0.arp_announce\" = 2;     # Redundant, covered by\r\n                                                 # above all setting\r\n    \"net.ipv4.conf.macvlan-shim.arp_announce\" = 2;  # Recommended, do not\r\n                                                    # leak MACs\r\n  };\r\n<\/pre>\n<p>This was a bit of a journey, but if you were just looking for a clean way to get macvlan networks working with NixOS and Docker hopefully it is presented in a way that is straight forward and you can follow. Also, we touched on using AI to help us explore solutions &#8211; and I do encourage you to use it as a tool vs. a magic eight ball. Along the way many wrong answers were presented, but by asking for more details and ways to test the assumptions, and changes &#8211; I was able to learn more about the solution and come to what I think was a good solution.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is an updated version of my previous article NixOS + Docker with MacVLAN (IPv4) &#8211; which addresses ARP Flux which causing networking problems. Skip to the bottom summary if you want to just read the conclusion. Ever since, but very intermittently, my new server build out &#8211; I had been having issues with my &hellip; <a href=\"https:\/\/lowtek.ca\/roo\/2026\/update-nixos-docker-macvlan\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Update: NixOS + Docker + MacVLAN&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23,6,12],"tags":[],"class_list":["post-2643","post","type-post","status-publish","format-standard","hentry","category-ai","category-computing","category-how-to"],"_links":{"self":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/2643","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/comments?post=2643"}],"version-history":[{"count":8,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/2643\/revisions"}],"predecessor-version":[{"id":2652,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/2643\/revisions\/2652"}],"wp:attachment":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/media?parent=2643"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/categories?post=2643"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/tags?post=2643"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}