Category: Computing

Ubuntu adding a 2nd data drive as a mirror (RAID1)

Over the years I’ve had the expected number of hard drive failures. Some have been more catastrophic to me as I didn’t have a good backup strategy in place, others felt avoidable if I’d paid attention the warning signs.

My current setup for data duplication is based on Snapraid, a non-traditional RAID solution. It allows mixed sizes of drives, and the replication is done via regularly running the sync operation. Mine is done daily, files are sync’d across the drives and a data validation is done from time to time as well. This means while I might lose up to 24hrs of data if the primary drive fails, I have lower usage of the main parity drive and I get the assurance that file corruption hasn’t happened.

Snapraid is very bad when you have either: many small files, frequently changing files. It is ideal for backing up media like photos or movies. To deal with the more rapidly changing data I’ve got a SSD drive for storage. I haven’t yet had a SSD fail on me, but that is assured to happen at one point. Backblaze is already seeing some failure rate information that is concerning. Couple this with the fact that my storage SSD started throwing errors the other day and only a full power cycle of the machine brought it back  – it’s fine now, but for how long? Time to setup a mirror.

For this storage I’m going back to traditional RAID. The SSD is a 480GB drive, and thankfully the price of them has dropped to easily under $70. This additional drive now fills all 6 of the SATA ports on my motherboard, the next upgrade will need to be an SATA port expansion card. I’ve written about RAID a few times here.

I’ve moved away from specifying drives as /dev/sdbX because these values can change. Even this new SSD caused the drive that was at /dev/sdf to move to /dev/sdg allowing the new drive to use /dev/sdf. My /etc/fstab is now setup using /dev/disk/by-id/xxx because these are persistent. Most of the disk utilities understand this format just fine as you an see with this example with fdisk.

Granted, working with /dev/disk/by-id is a lot more verbose – but that id will not change if you re-organize the SATA cables.

Let’s get going on setting up the new drive as a mirror for the existing one. Here’s the basic set of steps

  1. Partition the new drive so it is identical to the existing one
  2. Create a RAID1 array in degraded state
  3. Format and mount the array
  4. Copy the data from the existing drive to the new array
  5. Un-mount both the array and the original drive
  6. Mount the array where the original drive was mounted
  7. Make sure things are good – the next step is destructive
  8. Add the original drive to the degraded RAID1 array making it whole

It may seems like a lot of steps, and some of them are scary – but on the other side we’ll have a software RAID protecting the data. The remainder of this post will be the details of those steps above.

Continue reading “Ubuntu adding a 2nd data drive as a mirror (RAID1)”

OpenWRT Guest Network (and beyond)

As my kids get to the age where both they and their friends have devices, this means granting access to our internet to a growing circle of people. OpenWRT has the ability to support guest networks and I’ve been meaning to set this up for some time.

Beyond simply having a guest network, I also want to setup an IoT network where I can isolate some of the network enabled things but not give them wide access to the rest of my internal infrastructure.

Let’s start with a simple guest network setup. This is well documented on the OpenWRT site. I’ll be using the CLI instructions to make these changes. FWIW this is based on the 21.02.0 version of OpenWRT.

The first part of this will be pretty much copy and paste from the OpenWRT instructions:

I of course modified the ipaddr for my setup, but pretty much used the command as is.

On the Web UI (LuCI) you should now see under Network->Interfaces a new Guest network. All of the changes landed in the /etc/config/network file.

Now we setup a wireless network configuration with the first radio

Again, I’ve pretty much followed the directions but have customized the SSID and not shared the password. I’ve rolled in the extras to isolate guest users and use encryption.

The LuCI web UI now looks a little more concerning (Network->Wireless)

But.. the previous Network->Interfaces now seems to have come to life.

Still the expected changes are reflected in /etc/config/wireless.

It took a bit of head-scratching, but I figured out what was wrong. I had not specified a wireless.guest.key that met the minimum length (8 to 63 characters) – this apparently caused everything to go sideways. Once I fixed this my new wireless interface came to life.

Let’s continue on with the DHCP configuration

And the Firewall

Now not only should you be able to see the new WiFi SSID available to connect to, but when you do you will be isolated from all other devices on the network and only able to see the internet. A network scan will turn up the existence of the router, but attempts to connect the web UI fail – that’s pretty cool isolation.

Devices connected to the guest network still show up in the OpenWRT status page. They are assigned a DHCP address from the network.guest.ipaddr subnet, which is distinct from my normal network.

Apparently I do give up a little performance having two (or more) networks hung off of the same radio, but the utility of having a restrictive guest network is pretty cool.

The Archer C7 has two radios, and we’ve not configured the guest network on the second radio. Let’s do that now.

Cool – now I have a guest network that is on both radio bands. You’ll note that I run both access points with the same SSID, this mostly just works and devices figure it out. I even run my dumb AP with the same SSID. This is one approach that works and let’s people move around the house with seamless connections.

I do know that some people try to force a device to a particular radio type, and will run their legacy network on a different SSID. This is of course also valid, it really depends what you’re looking to achieve. I’m taking the simple to configure devices approach, and giving the devices the responsibility to work out which radio band and which access point to connect to.

Now let’s create a second ‘guest’ like network for IoT devices. This time I’ll just combine all of the steps together

Nice – now I have a third subnet which will hand out DHCP addresses valid for 24hrs. The devices are all isolated from each other.

For devices on the IoT network, while I don’t want the device to be able to see anything other than the internet – for my own monitoring and use, I’d like to be able to see the devices on the IoT network from my normal lan network. This turns out to be very easy.

Connecting to the IoT network and doing as scan, shows me that I can only see myself and the router (because the device has to send traffic somewhere). Again, with the isolation I can’t connect to the web interface of the router. However, with this new “zone->forwardings” I can from my lan network see devices on the IoT network. Super cool, and actually very easy.

There plenty more tweaking we can do here, but to avoid going too far down the rabbit hole we’ll stop here.

OpenWRT 19.07 to 21.02.0 upgrade

I’ve long been a fan of open firmware for my home routers. Way back I started with DD-WRT, but more recently I’ve moved to OpenWRT paired with TP-Link Archer C7 hardware. I actually have two, one as my main gateway and a second configured as a dumb AP (access point). Running two WiFi access points means I get great coverage from the basement, to the second floor.

While I have two, they are not quite identical. One is a v5, and the other is a v2. Still, for hardware you can pick up for under $100 it fits into my sweet spot for hardware. I picked up both of mine used, for less than half the price of new hardware. The other benefit to having two which run the same software stack (OpenWRT 19.07), means I can swap hardware for the main gateway if I have a compatibility problem.

Recently the latest version (21.02) has been declared stable. It’s well past time to upgrade. With the hardware I have, OpenWRT has recently done a transition from ar71xx to ath79. Thankfully In my setup with 19.07 I’d already moved to ath79.

You can check the TARGET that you are running by ssh’ing into the router and looking at the /etc/openwrt-release file

It is a good idea to start with the release notes. The upgrade process is 3 parts.

  1. Prepare
  2. Upgrade
  3. Post Install Configuration, Setup or Restore

We’ve already started the Prepare step since we have looked at the release notes and made sure we have the ath79 version running. Since we’re already on ath79 we can use the normal sysupgrade process. OpenWRT is also moving to DSA Networking, but the hardware I’m using hasn’t switched over yet – the upgrade process will detect this and refuse to upgrade if you have a problem.

I’ve got a full rsync backup of my router configuration, so if something goes really wrong I at least have the files.

Next I need to figure out which packages I have installed on the router(s). This script seems to work well, I’ll duplicate the script here.

This will dump out the list of packages that you have installed beyond the stock configuration. I’ve got rsync installed (of course) along with the prometheus packages. The upgrade will not automatically install these packages, so having the list of them helps us get back to the configuration we are used to.

We should now be ready to Upgrade. I found the easiest way to get the right sysupgrade package was using the Firmware Selector web tool. Since I have slightly different versions, I needed to make two downloads.

It is always a good idea to check the hash (sha256sum) of the files you download to make sure you have a good download. I’ve been burned only a handful of times by this, but once should be enough to teach you the lesson.

Using the Web UI to upgrade can be found in the menu system: LuCI → System → Backup / Flash Firmware → Actions: Flash new firmware image.

Time for a deep breath, and double check we have a backup. Time to flash, also double checking we push the right version to the right device.

After flashing the 1st device, I noted two things. The “flashing” screen seemed to get stuck – well past when the device has refreshed. Using a second browser window, I think I figured out why things got stuck. The LuCI web UI now redirects you to https:// with a self signed certificate. I think the browser on the flash screen got stuck because of the change in protocol (and the bad cert).

It’s good that the web UI is now hosted using https because now when you log in, you’re not sending your passwords in the clear. Sure it’s your own network, but I think I’d rather have to deal with clicking through the advanced screens to tell my browser to accept the self signed certificate than not have a secure connection.

For the Post Install Configuration, Setup or Restore step it’s a matter of going through the packages I’d identified above and re-installing them. As a second check, I also re-installed an re-ran the listuserpackages.awk script and got the same list back once I’d installed all the packages.

I can also verify that all of the configuration files made it by testing the functions these additional packages had. All was good, at least with my dumb AP.

My main gateway router was a bit scarier – it’s got a few more packages installed than the dumb AP and when I update it, I take an internet outage. Also, I ran into trouble trying to update some of the modules:

  • kmod-usb-storage
  • kmod-usb3

Both of these are related to my use of a USB drive as storage for the vnStat package. This appears to be a bug in the web UI – either way, using the cli worked fine to install things. It also looks like a fix is coming in the next patch.

The cli was happy to find the package

Where the web UI simply failed to located it.

In the end all was well, and a reboot to make sure things all came back just fine got me back to a fully working state but on the latest version. This was much easier than I had expected and I shouldn’t have stalled doing this so long.

A few housekeeping details to work through post install / basic check out. The upgrade procedure will create *-opkg files in /etc/config when you install the new packages. Be safe and do a quick diff and review of what has or has not changed (you may need to install the diffutils package, or move the files off the device to a machine that can diff things).

In my case – the sqm package seems to have changed at least some of the options. My setup still works, but I should probably rebase my config file.

Other stuff I have to fiddle with. WPA3 – now  mainline for OpenWRT I need to figure out how to run with this new protocol (while not breaking the world). A quick look around makes it seem like WPA3 is still a bit on the bleeding edge.

There is also New network configuration syntax and board.json change which means some changes in /etc/config/network which I should probably make sure I migrate to (the current build has some backwards compatibility – so my old file is fine for now).

Doing a planned migration / upgrade was way better than my usual emergency restore / rebuild. The last time I was messing with the router firmware I’d accidentally run an rsync backup and gotten the source / target the wrong way around – successfully sync’ing a blank directly onto my operating access point. Doh.

Anyways, hopefully this article helps me (or someone else) upgrade smoothly in the future.