How To – Roo's View

Update: NixOS + Docker + MacVLAN

This is an updated version of my previous article NixOS + Docker with MacVLAN (IPv4) – which addresses ARP Flux which causing networking problems. Skip to the bottom summary if you want to just read the conclusion.

Ever since, but very intermittently, my new server build out – I had been having issues with my macbook (OSX) randomly losing connection to the server, but only for a brief time. I noticed mostly when using mosh and it would indicate that I’d lost connection, but self hosted web UIs like Grafana also gave me problems.

Ping would also fail, but then it would start to work after a minute or so. What was weird is that while I could not ping my new server (192.168.1.79) I could have a stable mosh connection to another box (192.168.1.140), which could see the new server (192.168.1.79) just fine. This made it seem very much like it was something wrong with OSX. I was guilty of mostly ignoring this because it did impact me, but not frequently — it was still very annoying.

This is also an exploration of using Google Gemini, which you can use from an incognito tab with no login required. It didn’t immediately get me to the right answer, but we eventually got there.

The first iteration didn’t really get me very far, only a suggestion to check the route on my OSX machine when the problem was happening. This meant I had to wait around for it to break – and catch it in the act.

% route -n get myserver.lan
   route to: 192.168.1.79
destination: 192.168.1.79
  interface: en0
      flags: <UP,HOST,DONE,LLINFO,WASCLONED,IFSCOPE,IFREF>
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         0         0         0      1500      1167

% route -n get myserver.lan

route to: 192.168.1.79

destination: 192.168.1.79

interface: en0

flags: <UP,HOST,DONE,LLINFO,WASCLONED,IFSCOPE,IFREF>

recvpipe sendpipe ssthresh rtt,msec rttvar hopcount mtu expire

0 0 0 0 0 0 1500 1167

Unfortunately the output from this didn’t change with broken networking vs. working. At this point I added the detail that this networking problem seemed to happen most often after waking the OSX machine from sleep. Gemini was then able to reason:

In your route output, the flag LLINFO stands for Link Layer Information. This means macOS has a cached MAC address for 192.168.1.79.

When a Mac sleeps, it often hands off its networking responsibilities to a “Sleep Proxy” (like an Apple TV or HomePod) or simply drops its ARP table. When it wakes up, it might be trying to use a stale hardware address, or the Linux machine (myserver.lan) might not be responding to the Mac’s “Who has 192.168.1.79?” request fast enough.

The next suggestion was to check the arp information when networking was broken.

arp -a | grep 192.168.1.79

1	arp -a \| grep 192.168.1.79

If it says (incomplete), the Mac is sending requests but getting no answer.

If it shows a MAC address, compare it to the actual MAC address of the Linux box. If they match, the physical path is clear but the packets are being dropped higher up.

This got me looking at my WatchYourLan deployment, this is a networking monitoring tool that I use to alert me to new devices (based on MAC addresses) on my network.

To my surprise – I found another MAC 82:d4:9a:8f:06:40 for myserver.lan, one that I didn’t expect to see. Gemini called this a smoking gun, as it is a Locally Administered Address (LAA) – also known as a generated MAC. Gemini also concludes that this may be a Docker issue, which somewhat confirms what I was thinking. The advice from Gemini is getting more specific now, but I also find that in general this is where AI generated output can go off the rails so caution is required.

The Pattern: You have a “Bridge” problem

Seeing four different randomized MACs for a single IP (192.168.1.79) on a wired Linux machine is a classic symptom of Docker Userland Proxy or Bridge-utils misconfiguration.

When you run containers or virtual machines in “Bridge Mode,” the Linux kernel creates a virtual switch (br0, docker0, etc.).

The physical NIC (eth0) has its own hardware MAC.

The Bridge (br0) often generates its own MAC (like one of these A6 or 52 ones).

Each container gets a virtual interface (veth) with a randomized MAC.

I then remember to add the detail that this Linux machine is a NixOS installation, and Gemini comes back with

NixOS with a complex Docker setup is the “perfect storm” for this. NixOS tends to be very explicit about interface creation, and Docker’s default bridge behavior often clashes with how NixOS manages its own networking stack.

It suggests a “ghost hunter” command to identify bad MACs

ip link show | grep -iE '..:[0-9a-f][26ae]:' -B 1

1	ip link show \| grep -iE '..:[0-9a-f][26ae]:' -B 1

If these “ghost” MACs are responding to ARP requests on your physical LAN (a phenomenon called ARP Flux), it’s usually because the Linux kernel is being too “helpful” by answering ARP requests for an IP on any interface, even if the request came in on a different one.

It also suggests a change to my configuration.nix file, which in retrospect was pretty close to the final solution – but not quite. At this point I’m pretty impressed with Gemini being able to help with NixOS specifics. It also asks the question

Does your NixOS config happen to have any bridge interfaces (networking.bridges) or macvlan setups defined manually?

Why, yes – yes I do I reply. Gemini then fires back:

This macvlan-shim is the “smoking gun” for your macOS networking issues.

Why this breaks your Mac’s connection
By default, Linux is very “promiscuous” with ARP. When your Mac sends an ARP request asking “Who is 192.168.1.79?”, the Linux kernel sees that request on enp3s0. Because the macvlan-shim is bridged to that same physical card, the kernel might answer the Mac using the macvlan’s generated MAC address instead of the physical NIC’s hardware address.

This is known as ARP Flux. Your Mac gets confused because it sees multiple MAC addresses (the “ghosts” we found earlier) all claiming to be the same IP or living on the same segment. When the Mac wakes from sleep, it’s a race: if the “shim” MAC wins the ARP race, the Mac sends traffic to a virtual interface that isn’t configured to handle the host’s primary traffic.

It sure likes the term “smoking gun”, but it feels like we’re close to identifying the real root problem in my setup. I had previously done some changes to the way I deployed the various docker containers on my macvlan network to set static MAC addresses, but had not yet figured out how to make the shim IP have a static MAC address. The evidence I have and the advice from Gemini are starting to align. Odd that I got to this configuration from a previous setup using Ubuntu which didn’t have an issue, or maybe it was just luck due to timing.

I asked a lot of clarifying questions about the solutions it was proposing, which helped iterate towards both my more complete understanding of the solution – but it also helped generate some options. A minimalist solution to this was suggested, but felt like a lot of ‘trust in the force’:

boot.kernel.sysctl = {
  # This is the "Nuclear Option" that covers everything at once
  "net.ipv4.conf.all.arp_ignore" = 1;
  "net.ipv4.conf.all.arp_announce" = 2;
};

boot.kernel.sysctl = {

# This is the "Nuclear Option" that covers everything at once

"net.ipv4.conf.all.arp_ignore" = 1;

"net.ipv4.conf.all.arp_announce" = 2;

};

If you added this to my previous article on NixOS + Docker + MacVLAN you’d probably be fine. However, here is the more complete solution I ended up using:

{
  networking.macvlans."myNewNet-shim" = {
    mode = "bridge";
    interface = "enp3s0";
  };

  networking.interfaces."myNewNet-shim" = {
    macAddress = "06:00:00:00:00:67";
    ipv4.addresses = [{ address = "192.168.1.67"; prefixLength = 32; }];
    ipv4.routes = [{ address = "192.168.1.64"; prefixLength = 30; }];
  };

  # We need to prevent Arp Flux and silence any ghost replies
  # Ensure the kernel only answers for 192.168.1.79 on the physical 
  # card and for containers on the virtual IPs
  boot.kernel.sysctl = {
    # Only reply if the target IP is on the specific 
    # interface receiving the request
    "net.ipv4.conf.all.arp_ignore" = 1;          # Required, only reply
                                                 # to ARP for your interface
    "net.ipv4.conf.enp5s0.arp_ignore" = 1;       # Redundant, covered by
                                                 # above all setting
    "net.ipv4.conf.macvlan-shim.arp_ignore" = 1; # Recommended, keeps 
                                                 # shim silent

    # Ensure ARP announcements use the primary IP of the interface
    "net.ipv4.conf.all.arp_announce" = 2;        # Required, always use
                                                 # the right interface
    "net.ipv4.conf.enp5s0.arp_announce" = 2;     # Redundant, covered by
                                                 # above all setting
    "net.ipv4.conf.macvlan-shim.arp_announce" = 2;  # Recommended, do not
                                                    # leak MACs
  };
}

{

networking.macvlans."myNewNet-shim" = {

mode = "bridge";

interface = "enp3s0";

};

networking.interfaces."myNewNet-shim" = {

macAddress = "06:00:00:00:00:67";

ipv4.addresses = [{ address = "192.168.1.67"; prefixLength = 32; }];

ipv4.routes = [{ address = "192.168.1.64"; prefixLength = 30; }];

};

# We need to prevent Arp Flux and silence any ghost replies

# Ensure the kernel only answers for 192.168.1.79 on the physical

# card and for containers on the virtual IPs

boot.kernel.sysctl = {

# Only reply if the target IP is on the specific

# interface receiving the request

"net.ipv4.conf.all.arp_ignore" = 1; # Required, only reply

# to ARP for your interface

"net.ipv4.conf.enp5s0.arp_ignore" = 1; # Redundant, covered by

# above all setting

"net.ipv4.conf.macvlan-shim.arp_ignore" = 1; # Recommended, keeps

# shim silent

# Ensure ARP announcements use the primary IP of the interface

"net.ipv4.conf.all.arp_announce" = 2; # Required, always use

# the right interface

"net.ipv4.conf.enp5s0.arp_announce" = 2; # Redundant, covered by

# above all setting

"net.ipv4.conf.macvlan-shim.arp_announce" = 2; # Recommended, do not

# leak MACs

};

}

Again, if you compare with the original article we can see that I’ve added a specific MAC address for the shim, and used a naming convention to make the last digit match the IP address. The big change is the boot.kernel.sysctl which is very similar to the minimal setup above.

I also use the new MAC address numbering scheme for each of my macvlan containers – assigning them 06:00:00:00:00:XX where XX is the IP. Very handy to see they are the right assignment.

Now this had some interesting side-effects. The shim IP continued to offer up a stale MAC generated MAC address. I was able to fix this by forcing a recreate (but I suspect a reboot may have solved the problem).

sudo ip link set myNewNet-shim down
sudo ip link delete myNewNet-shim
sudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge
sudo ip link set myNewNet-shim address 06:00:00:00:00:67
sudo ip link set myNewNet-shim up

sudo ip link set myNewNet-shim down

sudo ip link delete myNewNet-shim

sudo ip link add myNewNet-shim link enp3s0 type macvlan mode bridge

sudo ip link set myNewNet-shim address 06:00:00:00:00:67

sudo ip link set myNewNet-shim up

A few other things also broke due to this more restrictive ARP control. Previously a container living on the macvlan network (like my nginx for local web services – on IP 192.168.1.65) could see my server (192.168.1.79). The myNewNet-shim provided visibility from the host to the nginx (192.168.1.65).

In this new no ARP Flux world, I have to use the shim address (192.168.1.67) in order to connect to my server (192.168.1.79) – so everywhere I reference x.x.x.79 I now needed to use x.x.x.67.

I’ve realized that this makes my wireguard setup a bit more annoying because if I use wireguard to connect home from remote, and I want to connect to my server I need to use the shim address instead of the local DNS name that maps to the real address. This was too big a trade off so I went back to Gemini and had a long discussion (with a lot of dead ends) to arrive at a solution that adds back the visibility of the host (192.168.1.79) from the macvlan containers (say 192.168.1.64 for example).

  # Create a "stealth" host IP on the macvlan bridge.
  # This allows containers/WireGuard to talk to the host at 192.168.1.79.
  # 'scope host' is critical: it prevents ARP Flux by ensuring the host 
  # never advertises this IP to the physical network (enp3s0).
  systemd.services."network-addresses-myNewNet-shim" = {
    # We use the absolute path to 'ip' to avoid needing to set the 'path' variable
    postStart = "${pkgs.iproute2}/bin/ip addr add 192.168.1.79/32 dev myNewNet-shim scope host || true";
  };

# Create a "stealth" host IP on the macvlan bridge.

# This allows containers/WireGuard to talk to the host at 192.168.1.79.

# 'scope host' is critical: it prevents ARP Flux by ensuring the host

# never advertises this IP to the physical network (enp3s0).

systemd.services."network-addresses-myNewNet-shim" = {

# We use the absolute path to 'ip' to avoid needing to set the 'path' variable

postStart = "${pkgs.iproute2}/bin/ip addr add 192.168.1.79/32 dev myNewNet-shim scope host || true";

};

Now using wireguard, I can see the full local network – host included.

Less ARP chaos is a good thing. Gemini did suggest that instead of declaring the route for the shim with a CIDR block that includes the shim address, I could be even more specific and simply have a route per IP address:

  ipv4.routes = [
    # 1. Route the WireGuard container specifically
    { address = "192.168.1.64"; prefixLength = 32; } 
    
    # 2. Route your other containers specifically
    { address = "192.168.1.65"; prefixLength = 32; }
    { address = "192.168.1.66"; prefixLength = 32; } 

    # 3. Keep your WireGuard tunnel route
    { address = "10.13.13.0"; prefixLength = 24; via = "192.168.1.64"; }
  ];

ipv4.routes = [

# 1. Route the WireGuard container specifically

{ address = "192.168.1.64"; prefixLength = 32; }

# 2. Route your other containers specifically

{ address = "192.168.1.65"; prefixLength = 32; }

{ address = "192.168.1.66"; prefixLength = 32; }

# 3. Keep your WireGuard tunnel route

{ address = "10.13.13.0"; prefixLength = 24; via = "192.168.1.64"; }

];

The key difference here is using /32 instead of /30. Being specific avoids the Linux Kernel from having to figure out what to do with the shim address (192.168.1.67) but it seems to do the right thing so I’ve gone with the simpler declaration. I’ve also got a special route in there for wireguard addresses so my host can see the wireguard clients directly.

Verifying the fix can be done on the OSX machine

arp -a | grep 192.168.1

1	arp -a \| grep 192.168.1

We want to review all of the MAC addresses to make sure we have the expected ones that follow the pattern 06:00:00:00:00:xx. If all goes well, this is the end of the ghosting problem (ARP Flux) that will cause the annoying interruption in networking from my OSX machine to the server.

In summary – the full docker macvlan setup on NixOS – this is basically a brief re-telling of the original post, with all of the updates above merged in.

First enable docker support on NixOS. You need a single line added to your /etc/nixos/configuration.nix

virtualisation.docker.enable = true;

1	virtualisation.docker.enable = true;

Create your docker macvlan network.

$ docker network create -d macvlan -o parent=enp3s0 \
  --subnet 192.168.1.0/24 \
  --gateway 192.168.1.1 \
  --ip-range 192.168.1.64/30 \
  --aux-address 'host=192.168.1.67' \
  myNewNet

$ docker network create -d macvlan -o parent=enp3s0 \

--subnet 192.168.1.0/24 \

--gateway 192.168.1.1 \

--ip-range 192.168.1.64/30 \

--aux-address 'host=192.168.1.67' \

myNewNet

Docker will persist this network configuration across reboots.

Now we need to modify /etc/nixos/configuration.nix to fix routing to/from our macvlan network IPs from the host – and avoid causing APR Flux.

  networking.macvlans."myNewNet-shim" = {
    mode = "bridge";
    interface = "enp3s0";
  };

  networking.interfaces."myNewNet-shim" = {
    macAddress = "06:00:00:00:00:67";
    ipv4.addresses = [{ address = "192.168.1.67"; prefixLength = 32; }];
    ipv4.routes = [{ address = "192.168.1.64"; prefixLength = 30; }];
  };

  # Create a "stealth" host IP on the macvlan bridge. 
  # This allows containers/WireGuard to talk to the host at 192.168.1.79. 
  # 'scope host' is critical: it prevents ARP Flux by ensuring the host 
  # never advertises this IP to the physical network (enp3s0). 
  systemd.services."network-addresses-myNewNet-shim" = { 
  # We use the absolute path to 'ip' to avoid needing to set 
  # the 'path' variable 
  postStart = "${pkgs.iproute2}/bin/ip addr add 192.168.1.79/32 dev myNewNet-shim scope host || true"; 
  };

  # ARP Flux Protection (Defense in Depth)
  # While 'scope host' on the macvlan-shim prevents the host from 
  # advertising 192.168.1.79 to the physical network, these sysctl 
  # settings ensure the kernel's ARP behavior is strictly tied 
  # to the specific interface receiving the request.
  boot.kernel.sysctl = {
    # Only reply if the target IP is on the specific 
    # interface receiving the request
    "net.ipv4.conf.all.arp_ignore" = 1;          # Required, only reply
                                                 # to ARP for your interface
    "net.ipv4.conf.enp5s0.arp_ignore" = 1;       # Redundant, covered by
                                                 # above all setting
    "net.ipv4.conf.macvlan-shim.arp_ignore" = 1; # Recommended, keeps 
                                                 # shim silent

    # Ensure ARP announcements use the primary IP of the interface
    "net.ipv4.conf.all.arp_announce" = 2;        # Required, always use
                                                 # the right interface
    "net.ipv4.conf.enp5s0.arp_announce" = 2;     # Redundant, covered by
                                                 # above all setting
    "net.ipv4.conf.macvlan-shim.arp_announce" = 2;  # Recommended, do not
                                                    # leak MACs
  };

networking.macvlans."myNewNet-shim" = {

mode = "bridge";

interface = "enp3s0";

};

networking.interfaces."myNewNet-shim" = {

macAddress = "06:00:00:00:00:67";

ipv4.addresses = [{ address = "192.168.1.67"; prefixLength = 32; }];

ipv4.routes = [{ address = "192.168.1.64"; prefixLength = 30; }];

};

# Create a "stealth" host IP on the macvlan bridge.

# This allows containers/WireGuard to talk to the host at 192.168.1.79.

# 'scope host' is critical: it prevents ARP Flux by ensuring the host

# never advertises this IP to the physical network (enp3s0).

systemd.services."network-addresses-myNewNet-shim" = {

# We use the absolute path to 'ip' to avoid needing to set

# the 'path' variable

postStart = "${pkgs.iproute2}/bin/ip addr add 192.168.1.79/32 dev myNewNet-shim scope host || true";

};

# ARP Flux Protection (Defense in Depth)

# While 'scope host' on the macvlan-shim prevents the host from

# advertising 192.168.1.79 to the physical network, these sysctl

# settings ensure the kernel's ARP behavior is strictly tied

# to the specific interface receiving the request.

boot.kernel.sysctl = {

# Only reply if the target IP is on the specific

# interface receiving the request

"net.ipv4.conf.all.arp_ignore" = 1; # Required, only reply

# to ARP for your interface

"net.ipv4.conf.enp5s0.arp_ignore" = 1; # Redundant, covered by

# above all setting

"net.ipv4.conf.macvlan-shim.arp_ignore" = 1; # Recommended, keeps

# shim silent

# Ensure ARP announcements use the primary IP of the interface

"net.ipv4.conf.all.arp_announce" = 2; # Required, always use

# the right interface

"net.ipv4.conf.enp5s0.arp_announce" = 2; # Redundant, covered by

# above all setting

"net.ipv4.conf.macvlan-shim.arp_announce" = 2; # Recommended, do not

# leak MACs

};

This was a bit of a journey, but if you were just looking for a clean way to get macvlan networks working with NixOS and Docker hopefully it is presented in a way that is straight forward and you can follow. Also, we touched on using AI to help us explore solutions – and I do encourage you to use it as a tool vs. a magic eight ball. Along the way many wrong answers were presented, but by asking for more details and ways to test the assumptions, and changes – I was able to learn more about the solution and come to what I think was a good solution.

Re-encoding/transcoding video for space savings

There are lots of good reasons to run a media server at home to host your own content. There are also plenty of legitimate ways to get content, but also many that are less so. I think it is important to pay creators for their efforts, I also really like “owning” the media I enjoy and not paying repeatedly to re-watch things.

Sometimes you’ll end up with a version of the content (say captured OTA) that is both high quality, but huge. An hour long 1080p HDTV capture from my HD Homerun lands in around 7.7GB. A more efficient encoding can significantly drop that, of course these are lossy encodings meaning that the quality can be less. The trick is to figure out how to pick an encoding of the video that retains most of the quality, but is significantly smaller.

Different codecs have different approaches, all with trade-offs, but sometimes it’s really hard to see the differences. Many of the content creators are also faced with the same dilemma, they have high bit-rate master copies of the content and need to squeeze it down to fit on a DVD or Blu-ray.

Let’s dive into how we do this. For this example we’ll take a 45min TV show that starts at 1.9G. It is encoded in AVC1/H.264 at 6000 kb/s. The image quality is very good, but it’s also sort of big and maybe you don’t care that much about this particular TV series but you do want to retain a copy.

FFmpeg can fix this for us. We’ll be moving to H.265 as an encoding, burning a bunch of CPU cycles to get there, and ideally get a video that is substantially the same but uses less storage.

$ ffmpeg -i original.mp4 -c:v libx265 \
  -vtag hvc1 -crf 20 -preset fast \
  -c:a copy smaller.mp4

$ ffmpeg -i original.mp4 -c:v libx265 \

-vtag hvc1 -crf 20 -preset fast \

-c:a copy smaller.mp4

You can see we’ve told it to re-encode to the H.265 codec, but to simply copy the audio. The crf flag is important for maintaining quality, lower values will be more quality, higher less. If you don’t supply the flag, a default of 28 is used resulting in more space savings, but to my eyes a softness in the output.

Original  2033893144 (1.9G)
Smaller    560561334 (535M)

1 2	Original 2033893144 (1.9G) Smaller 560561334 (535M)

That is a significant space savings, the new file is only 27% of the original – 3 times smaller! Sure, it’s small, but is it any good?

Let’s look at some stats. While we have maintained the 1920×1080 resolution, we’ve dropped the bitrate to 1510 kb/s. This explains a lot of the saving, we’ve reduce the number of bits used to create each frame of the image.

We will use ffmpeg to extract a few frames of the two videos for comparison. The use of the .png format will give us “lossless” images (but recall the video itself has been encoded in a lossy format).

$ ffmpeg -ss 00:04:10 -i original.mp4 -update true -frames:v 1 original-1.png
$ ffmpeg -ss 00:22:08 -i original.mp4 -update true -frames:v 1 original-2.png

$ ffmpeg -ss 00:04:10 -i smaller.mp4 -update true -frames:v 1 smaller-1.png
$ ffmpeg -ss 00:22:08 -i smaller.mp4 -update true -frames:v 1 smaller-2.png

$ ffmpeg -ss 00:04:10 -i original.mp4 -update true -frames:v 1 original-1.png

$ ffmpeg -ss 00:22:08 -i original.mp4 -update true -frames:v 1 original-2.png

$ ffmpeg -ss 00:04:10 -i smaller.mp4 -update true -frames:v 1 smaller-1.png

$ ffmpeg -ss 00:22:08 -i smaller.mp4 -update true -frames:v 1 smaller-2.png

Above are the the frames from 4:10, the first/top is from the original, and the second/bottom is from the smaller version. You can click through to view the original 1920×1080 image. Visually they seem identical.

Using the idiff tool we can create a difference image and get data about how different these two images are from each other.

$ idiff -o diff-1.png original-1.png smaller-1.png 
Comparing "original-1.png" and "smaller-1.png"
  Mean error = 0.00534948
  RMS error = 0.00772274
  Peak SNR = 42.2446
  Max error  = 0.145098 @ (834, 553, R)  values are 0.19215688, 0.1764706, 0.15294118 vs 0.3372549, 0.29411766, 0.26666668
  1614504 pixels (77.9%) over 1e-06
  1614504 pixels (77.9%) over 1e-06
FAILURE

$ idiff -o diff-1.png original-1.png smaller-1.png

Comparing "original-1.png" and "smaller-1.png"

Mean error = 0.00534948

RMS error = 0.00772274

Peak SNR = 42.2446

Max error = 0.145098 @ (834, 553, R) values are 0.19215688, 0.1764706, 0.15294118 vs 0.3372549, 0.29411766, 0.26666668

1614504 pixels (77.9%) over 1e-06

FAILURE

Ok, so they are different but we knew that. What does the difference image look like?

Yup, just a big black box not a lot of obvious differences, this agrees with what we we can see ourselves. Let’s tweak the idiff command to highlight the differences by a scale of 20

$ idiff -scale 20 -o diff-1X.png original-1.png smaller-1.png 
Comparing "original-1.png" and "smaller-1.png"
  Mean error = 0.00534948
  RMS error = 0.00772274
  Peak SNR = 42.2446
  Max error  = 0.145098 @ (834, 553, R)  values are 0.19215688, 0.1764706, 0.15294118 vs 0.3372549, 0.29411766, 0.26666668
  1614504 pixels (77.9%) over 1e-06
  1614504 pixels (77.9%) over 1e-06
FAILURE

$ idiff -scale 20 -o diff-1X.png original-1.png smaller-1.png

Comparing "original-1.png" and "smaller-1.png"

Mean error = 0.00534948

RMS error = 0.00772274

Peak SNR = 42.2446

Max error = 0.145098 @ (834, 553, R) values are 0.19215688, 0.1764706, 0.15294118 vs 0.3372549, 0.29411766, 0.26666668

1614504 pixels (77.9%) over 1e-06

FAILURE

Now we can see the small differences magnified, and it’s in areas you’d expect, around the edges of objects and in the small details.

Let’s look at another frame using the same approach.

Again, the original is the first, and the smaller is the second. Let’s do the diff, but only grab the scaled version.

$ idiff -scale 20 -o diff-2X.png original-2.png smaller-2.png 
Comparing "original-2.png" and "smaller-2.png"
  Mean error = 0.00534642
  RMS error = 0.00788875
  Peak SNR = 42.0598
  Max error  = 0.105882 @ (704, 188, B)  values are 0.34117648, 0.3254902, 0.24313727 vs 0.43921572, 0.42352945, 0.34901962
  1617659 pixels (78%) over 1e-06
  1617659 pixels (78%) over 1e-06
FAILURE

$ idiff -scale 20 -o diff-2X.png original-2.png smaller-2.png

Comparing "original-2.png" and "smaller-2.png"

Mean error = 0.00534642

RMS error = 0.00788875

Peak SNR = 42.0598

Max error = 0.105882 @ (704, 188, B) values are 0.34117648, 0.3254902, 0.24313727 vs 0.43921572, 0.42352945, 0.34901962

1617659 pixels (78%) over 1e-06

FAILURE

Again, in the scaled version of the diff image we see that the fine details are were the very small differences are. Try opening the full images, one in each tab and toggling between them – can you see any changes? Yes, we know there are differences – the smaller file is 1/3 of the size – but is this enough to matter? Or even notice?

I’m going to continue to use MakeMKV to rip blu-rays of movies I consider ‘reference material’ – like Dune, because I want all the bits. However, if it’s just some TV show I captured OTA and I’m going to watch once, or at most a handful of times – I’ll take the space savings.

Forgejo – self hosted git server

Source control is really great, and very accessible today. Many developers starts with just a pile of files and no source control, but you soon have a disaster and wish you had a better story than a bunch of copies of your files. In the early days of computing, source control was a lot more gnarly to get setup – now we have things like GitHub that make it easy.

Forgejo brings a GitHub like experience to your homelab. I can’t talk about Forgejo without mentioning Gitea. I’ve been running Gitea for about the last 7 years, it’s fairly lightweight and gives you a nice web UI that feels a lot like GitHub, but it’s self-hosted. Unfortunately there was a break in the community back in 2022, when Gitea Ltd took control – this did not go well and Forgejo was born.

The funny thing is that Gitea orginally came out of Gogs. The wikipedia page indicates Gogs was controlled by a single maintainer and Gitea as a fork opened up more community contribution. It’s unfortunate that open source projects often struggle either due to commercial influences, or the various parties involved in the project. Writing code can be hard, but working with other people is harder.

For a while Forgejo was fully Gitea compatible, this changed in early 2024 when they stopped maintaining compatibility. I only became aware of Forgejo in late 2024, but decided Gitea was still an acceptable solution for my needs. It was only recently that I was reminded about Forgejo and re-evaluated if it was finally time to move (yes, it was).

Forgejo has a few installation options, I unsurprisingly selected the docker path. I opted to use the rootless docker image which may limit some future expansion such as supporting actions, but I have basic source control needs and I can always change things later if I need.

My docker-compose.yml uses the built in sqlite3 DB, but as mentioned above is using the rootless version which is a bit more secure.

networks:
  forgejo:
    external: false

services:
  server:
    image: codeberg.org/forgejo/forgejo:14-rootless
    container_name: forgejo
    user: 1000:100
    environment:
      - USER_UID=1000
      - USER_GID=100
    restart: always
    networks:
      - forgejo
    volumes:
      - ./forgejo:/var/lib/gitea
      - ./conf:/etc/gitea
      - /etc/localtime:/etc/localtime:ro
    ports:
      - '3001:3000'
      - '222:2222'

networks:

forgejo:

external: false

services:

server:

image: codeberg.org/forgejo/forgejo:14-rootless

container_name: forgejo

user: 1000:100

environment:

- USER_UID=1000

- USER_GID=100

restart: always

networks:

- forgejo

volumes:

- ./forgejo:/var/lib/gitea

- ./conf:/etc/gitea

- /etc/localtime:/etc/localtime:ro

ports:

- '3001:3000'

- '222:2222'

As I’m based on NixOS, my gid is 100, not the typical 1000. I had to modify my port mapping to avoid a conflict with Gitea which is already using 3000.

Now, the next thing I need (ok, want) to do is configure my nginx as a reverse proxy so I can give my new source control system a hostname instead of having to deal with port numbers. I actually run two nginx containers – one based on swag for internet visible web, and another nginx for internal systems. With a more complex configuration I could use just one, but having a hard separation gives me peace of mind that I haven’t accidentally exposed an internal system to the internet.

I configured a hostname (forge.lan) in my openwrt router which I use for local DNS. My local nginx is running with a unique IP address thanks to macvlan magic. If I map forge.lan to the IP of my nginx (192.168.1.66) then the nginx configuration is fairly simple, I treat it like a site creating a file config/nginx/site-confs/forge.conf that looks like:

# Git server (Forgejo) - https://forgejo.org/docs/latest/admin/setup/reverse-proxy/#basic-http
server {
        listen 80;
# Disable IPV6
#       listen [::]:80;
        server_name forge.lan;

        location / {
                proxy_pass http://192.168.1.79:3001;

                proxy_set_header Connection $http_connection;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;

                client_max_body_size 512M;
        }
}

# Git server (Forgejo) - https://forgejo.org/docs/latest/admin/setup/reverse-proxy/#basic-http

server {

listen 80;

# Disable IPV6

# listen [::]:80;

server_name forge.lan;

location / {

proxy_pass http://192.168.1.79:3001;

proxy_set_header Connection $http_connection;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header X-Forwarded-Proto $scheme;

client_max_body_size 512M;

}

Most of this is directly from the forgejo doc on reverse proxies. When my nginx gets traffic on port 80 for a server named forge.lan, it will proxy the connection to my main server (192.168.1.79) running the forgejo container.

With this setup, we can now start the docker container

docker compose pull; docker compose up -d

1	docker compose pull; docker compose up -d

And visit http://forge.lan to be greeted by the bootstrap setup screen. At this point we can mostly just accept all the defaults because it should self detect everything correctly.

When we interact with this new self hosted git server, at least some of the time we’ll be on the command line. This means we’ll be wanting to use a ssh connection so we can do things like this

# Check out our NixOS configurations
git clone ssh://git@forge.lan:222/myuser/nixos-configs.git

1 2	# Check out our NixOS configurations git clone ssh://git@forge.lan:222/myuser/nixos-configs.git

There is a problem here. If you recall the webserver (192.168.1.66) is not on the same IP as the host (192.168.1.79) of my forgejo container. Since I want the hostname forge.lan to map to the webserver IP, I’ve introduced a challenge for myself.

When I hit this problem with Gitea, my solution was simply to switch to using my swag based public facing webserver (which runs on my main host IP) and use a deny list to prevent anyone from getting to gitea unless they were on my local network. This works, but means I had some worry that one day I’d mess that up and expose my self hosted git server to the internet. It turns out there is a better way, nginx knows how to proxy ssh connections.

This stackexchange post pointed me in the right direction, but it’s simply a matter of adding a stream configuration to your main nginx.conf file.

stream {
        upstream ssh {
                server 192.168.1.79:222;
                }
        server {
                listen 222;
                proxy_pass ssh;
        }
}

stream {

upstream ssh {

server 192.168.1.79:222;

}

server {

listen 222;

proxy_pass ssh;

}

After restarting nginx, I can now perform ssh connections to forgejo. This feels pretty slick.

I then proceeded to clone my git repos from my gitea server to my new forgejo server. This is a bit repetitive, and to avoid too many mistakes I cooked up a script based on this pattern. Oh yeah, and I did need to enable push-to-create on forgejo to make this work.

#!/usr/bin/env bash

echo "clone repo: " $1
dir=`echo "${1##*/}"`
echo " directory: " $dir

read -p "Press enter to continue OR CTRL-C to abort"

git clone --mirror $1
cd $dir
git remote set-url origin ssh://git@forge.lan:222/myuser/$dir
git push --mirror origin
cd ..
rm -rf $dir

#!/usr/bin/env bash

echo "clone repo: " $1

dir=`echo "${1##*/}"`

echo " directory: " $dir

read -p "Press enter to continue OR CTRL-C to abort"

git clone --mirror $1

cd $dir

git remote set-url origin ssh://git@forge.lan:222/myuser/$dir

git push --mirror origin

cd ..

rm -rf $dir

This script takes a single parameter which is the URL for the source (Gitea) server. It then strips off the project name which is the directory of the project. We are using the --mirror option to tell git we want everything, not just the main branch.

Using this I was able to quickly move all 29 repositories I had. My full commit history came along just fine. I did lose the fun commit graph, but apparently if you aren’t afraid to do some light DB hacking you can move it over from Gitea as the formats are basically the same. The action table is the one you want to move / migrate. I’m ok with my commit graph being reset.

You also don’t get anything migrated outside of the git repos, this means issues for example will be lost. For me this isn’t a big deal, I had 3 issues all created 4 years ago. If you want a more complete migration you might investigate this set of scripts.

The last thing for me is to work my way through any place I’ve got any of those repositories checked out, and change the origin URL. For example consider my nix-configs project

$ cd nix-configs

# check the current repo
$ git remote -v
origin  ssh://git@mygitea.lan:3022/myuser/nixos-configs.git (fetch)
origin  ssh://git@mygitea.lan:3022/myuser/nixos-configs.git (push)

# change to the new server
$ git remote set-url origin ssh://git@forge.lan:222/myuser/nixos-configs.git

# verify we have it all updated
$ git remote -v
origin  ssh://git@forge.lan:222/myuser/nixos-configs.git (fetch)
origin  ssh://git@forge.lan:222/myuser/nixos-configs.git (push)

$ cd nix-configs

# check the current repo

$ git remote -v

origin ssh://git@mygitea.lan:3022/myuser/nixos-configs.git (fetch)

origin ssh://git@mygitea.lan:3022/myuser/nixos-configs.git (push)

# change to the new server

$ git remote set-url origin ssh://git@forge.lan:222/myuser/nixos-configs.git

# verify we have it all updated

$ git remote -v

origin ssh://git@forge.lan:222/myuser/nixos-configs.git (fetch)

origin ssh://git@forge.lan:222/myuser/nixos-configs.git (push)

At this point I’m fully migrated, and can shut down the old gitea server.

If you were interested in a Forgejo setup that has actions configured and is setup for a bit more scale, check out this write up.