OpenWRT 19.07 to 21.02.0 upgrade

I’ve long been a fan of open firmware for my home routers. Way back I started with DD-WRT, but more recently I’ve moved to OpenWRT paired with TP-Link Archer C7 hardware. I actually have two, one as my main gateway and a second configured as a dumb AP (access point). Running two WiFi access points means I get great coverage from the basement, to the second floor.

While I have two, they are not quite identical. One is a v5, and the other is a v2. Still, for hardware you can pick up for under $100 it fits into my sweet spot for hardware. I picked up both of mine used, for less than half the price of new hardware. The other benefit to having two which run the same software stack (OpenWRT 19.07), means I can swap hardware for the main gateway if I have a compatibility problem.

Recently the latest version (21.02) has been declared stable. It’s well past time to upgrade. With the hardware I have, OpenWRT has recently done a transition from ar71xx to ath79. Thankfully In my setup with 19.07 I’d already moved to ath79.

You can check the TARGET that you are running by ssh’ing into the router and looking at the /etc/openwrt-release file

It is a good idea to start with the release notes. The upgrade process is 3 parts.

  1. Prepare
  2. Upgrade
  3. Post Install Configuration, Setup or Restore

We’ve already started the Prepare step since we have looked at the release notes and made sure we have the ath79 version running. Since we’re already on ath79 we can use the normal sysupgrade process. OpenWRT is also moving to DSA Networking, but the hardware I’m using hasn’t switched over yet – the upgrade process will detect this and refuse to upgrade if you have a problem.

I’ve got a full rsync backup of my router configuration, so if something goes really wrong I at least have the files.

Next I need to figure out which packages I have installed on the router(s). This script seems to work well, I’ll duplicate the script here.

This will dump out the list of packages that you have installed beyond the stock configuration. I’ve got rsync installed (of course) along with the prometheus packages. The upgrade will not automatically install these packages, so having the list of them helps us get back to the configuration we are used to.

We should now be ready to Upgrade. I found the easiest way to get the right sysupgrade package was using the Firmware Selector web tool. Since I have slightly different versions, I needed to make two downloads.

It is always a good idea to check the hash (sha256sum) of the files you download to make sure you have a good download. I’ve been burned only a handful of times by this, but once should be enough to teach you the lesson.

Using the Web UI to upgrade can be found in the menu system: LuCI → System → Backup / Flash Firmware → Actions: Flash new firmware image.

Time for a deep breath, and double check we have a backup. Time to flash, also double checking we push the right version to the right device.

After flashing the 1st device, I noted two things. The “flashing” screen seemed to get stuck – well past when the device has refreshed. Using a second browser window, I think I figured out why things got stuck. The LuCI web UI now redirects you to https:// with a self signed certificate. I think the browser on the flash screen got stuck because of the change in protocol (and the bad cert).

It’s good that the web UI is now hosted using https because now when you log in, you’re not sending your passwords in the clear. Sure it’s your own network, but I think I’d rather have to deal with clicking through the advanced screens to tell my browser to accept the self signed certificate than not have a secure connection.

For the Post Install Configuration, Setup or Restore step it’s a matter of going through the packages I’d identified above and re-installing them. As a second check, I also re-installed an re-ran the listuserpackages.awk script and got the same list back once I’d installed all the packages.

I can also verify that all of the configuration files made it by testing the functions these additional packages had. All was good, at least with my dumb AP.

My main gateway router was a bit scarier – it’s got a few more packages installed than the dumb AP and when I update it, I take an internet outage. Also, I ran into trouble trying to update some of the modules:

  • kmod-usb-storage
  • kmod-usb3

Both of these are related to my use of a USB drive as storage for the vnStat package. This appears to be a bug in the web UI – either way, using the cli worked fine to install things. It also looks like a fix is coming in the next patch.

The cli was happy to find the package

Where the web UI simply failed to located it.

In the end all was well, and a reboot to make sure things all came back just fine got me back to a fully working state but on the latest version. This was much easier than I had expected and I shouldn’t have stalled doing this so long.

A few housekeeping details to work through post install / basic check out. The upgrade procedure will create *-opkg files in /etc/config when you install the new packages. Be safe and do a quick diff and review of what has or has not changed (you may need to install the diffutils package, or move the files off the device to a machine that can diff things).

In my case – the sqm package seems to have changed at least some of the options. My setup still works, but I should probably rebase my config file.

Other stuff I have to fiddle with. WPA3 – now  mainline for OpenWRT I need to figure out how to run with this new protocol (while not breaking the world). A quick look around makes it seem like WPA3 is still a bit on the bleeding edge.

There is also New network configuration syntax and board.json change which means some changes in /etc/config/network which I should probably make sure I migrate to (the current build has some backwards compatibility – so my old file is fine for now).

Doing a planned migration / upgrade was way better than my usual emergency restore / rebuild. The last time I was messing with the router firmware I’d accidentally run an rsync backup and gotten the source / target the wrong way around – successfully sync’ing a blank directly onto my operating access point. Doh.

Anyways, hopefully this article helps me (or someone else) upgrade smoothly in the future.

DNS for lowtek.ca

It’s been 20 years I’ve had the domain lowtek.ca. Over that time I’ve had a few different ways to host the domain and the DNS records which point that domain the the IP address.

For a long time I had a static IP, this allowed me to set my DNS record and forget it. It was even, depending on my ISP, possible for me to have the reverse lookup work too. Paying a bit extra for a static IP was well worth it with a DSL connection, as you frequently got a new address. When I moved to cable, a static IP wasn’t an option, but the IP also rarely changes (once every year or two).

Back in the early days, domain registrars would charge for their DNS services or simply didn’t have one – this pushed me to look for free alternatives. I’ve used a few solutions, the most recent being some systems my brother maintains and I’ve got root access on, allowing me to modify the bind configuration files.

Recently I realized that rebel.ca would allow me to manage my DNS records, and it’s free.

Once logged into rebel.ca you can go to manage your domains. For a given domain you can then get to the DNS tab. On that tab under “Nameserver Information” you can pick one of three options:

  • Park with Rebel.ca
  • Forward Domain
  • Use Third Party Hosting

Until very recently I was using the Third Party Hosting option and pointing at the DNS servers managed by my brother. This worked well, but was another place to log into and I would have to re-learn the bind rules for editing my records because I rarely did any updates. With rebel.ca managing it, I get an easy web UI to change things.

I did find it odd that I had to opt to “Park with Rebel.ca” in order to start managing my DNS. Once you’ve selected this option, you can then use the “Advanced DNS Manager” to see the records that they have for your domain.

Here is what I got once I had parked my domain.

The IP address 69.16.231.58 reverse maps to lb04.parklogic.com, and parklogic.com appears to be a domain parking company. Visiting a parked domain appears to redirect to https://simcast.com/ eventually.

Since this particular domain only has a web server on it, I only need to make 2 changes to the default entry that was provided to me.

I could probably safely trim down some of the entries. I don’t need an MX record, nor the localhost or loopback entries. The wildcard could also be reduced to only the www subdomain. This works as is, and the extra entries shouldn’t cause any problems.

One thing that I’ve given up by moving to the rebel.ca DNS is control over the SOA record.  Let’s look at the differences in the SOA records.

My old nameserver

Rebel.ca nameserver

This boils down to these changes (old vs. rebel.ca)

  • Refresh: 10min vs. 3hrs
  • Retry: 2hrs vs. 1hr
  • Expire: 1000hr vs. 168hr
  • Minimum: 10min vs. 1hr

I’m a little concerned with the refresh value being so much different. I was worried this would impact how long it will take to roll out changes to the DNS records in general. Previously with a 10min refresh – I’d see fairly quick DNS changes. This is somewhat mitigated by the fact that my site(s) are not high traffic, so many times the values are not cached at all. This might burn me when my IP address changes – but hopefully for only ~3hrs. I can probably live with that if it turns out to be that long.

The retry and expire values are basically in the same range, so it’ll be fine.

The minimum is actually the SOA default TTL and is used for negative caching. This means that all the queries that don’t have a valid response are cached for this amount of seconds. Thus the 1hr time is aligned with the general recommendations. The previous setting of 10mins I think was wrong and would cause extra network traffic in some cases.

While I do not have control over the SOA record, I can control the TTL for specific records as you will have noticed in the screen shots above where I’ve set some of them to 600.

So for the sparrowsplay.com A record, I have 600 seconds (10min) set.

There is a lot of confusion about DNS records in general as is apparent in this stack overflow post. I know from experience with bind that if I forgot to update the SOA serial number, my changes wouldn’t flow. This I believe may be simply because the bind software was not deciding to change anything based on the serial number. As I read through things, the low TTL for the A records should cause my changes to flow quickly (10min vs. 3hrs).

Doing a quick test changing the record back to 69.16.231.58 and using one of the DNS propagation checking sites, I can see that some of the servers picked up the change immediately.  The ‘big’ nameservers: opendns, google, quad9 seemed to pick up the changes very quickly. I did see google and opendns give back the old address randomly in the middle of my refreshing for updates, so clearly there is a load balancer with many nameservers and not all of them are picking up the change (yet).

At almost precisely the 10min mark, all of the nameservers appear to have gotten the new IP address. This is very good news, meaning that the TTL for the A record determines how long it will take to make an IP change for my website.  This means that my earlier concern about the SOA record was mistaken, with TTL control over the A record I can quickly update my IP address.