Ubuntu adding a 2nd data drive as a mirror (RAID1)

Over the years I’ve had the expected number of hard drive failures. Some have been more catastrophic to me as I didn’t have a good backup strategy in place, others felt avoidable if I’d paid attention the warning signs.

My current setup for data duplication is based on Snapraid, a non-traditional RAID solution. It allows mixed sizes of drives, and the replication is done via regularly running the sync operation. Mine is done daily, files are sync’d across the drives and a data validation is done from time to time as well. This means while I might lose up to 24hrs of data if the primary drive fails, I have lower usage of the main parity drive and I get the assurance that file corruption hasn’t happened.

Snapraid is very bad when you have either: many small files, frequently changing files. It is ideal for backing up media like photos or movies. To deal with the more rapidly changing data I’ve got a SSD drive for storage. I haven’t yet had a SSD fail on me, but that is assured to happen at one point. Backblaze is already seeing some failure rate information that is concerning. Couple this with the fact that my storage SSD started throwing errors the other day and only a full power cycle of the machine brought it back  – it’s fine now, but for how long? Time to setup a mirror.

For this storage I’m going back to traditional RAID. The SSD is a 480GB drive, and thankfully the price of them has dropped to easily under $70. This additional drive now fills all 6 of the SATA ports on my motherboard, the next upgrade will need to be an SATA port expansion card. I’ve written about RAID a few times here.

I’ve moved away from specifying drives as /dev/sdbX because these values can change. Even this new SSD caused the drive that was at /dev/sdf to move to /dev/sdg allowing the new drive to use /dev/sdf. My /etc/fstab is now setup using /dev/disk/by-id/xxx because these are persistent. Most of the disk utilities understand this format just fine as you an see with this example with fdisk.

Granted, working with /dev/disk/by-id is a lot more verbose – but that id will not change if you re-organize the SATA cables.

Let’s get going on setting up the new drive as a mirror for the existing one. Here’s the basic set of steps

  1. Partition the new drive so it is identical to the existing one
  2. Create a RAID1 array in degraded state
  3. Format and mount the array
  4. Copy the data from the existing drive to the new array
  5. Un-mount both the array and the original drive
  6. Mount the array where the original drive was mounted
  7. Make sure things are good – the next step is destructive
  8. Add the original drive to the degraded RAID1 array making it whole

It may seems like a lot of steps, and some of them are scary – but on the other side we’ll have a software RAID protecting the data. The remainder of this post will be the details of those steps above.

Continue reading “Ubuntu adding a 2nd data drive as a mirror (RAID1)”

OpenWRT Guest Network (and beyond)

As my kids get to the age where both they and their friends have devices, this means granting access to our internet to a growing circle of people. OpenWRT has the ability to support guest networks and I’ve been meaning to set this up for some time.

Beyond simply having a guest network, I also want to setup an IoT network where I can isolate some of the network enabled things but not give them wide access to the rest of my internal infrastructure.

Let’s start with a simple guest network setup. This is well documented on the OpenWRT site. I’ll be using the CLI instructions to make these changes. FWIW this is based on the 21.02.0 version of OpenWRT.

The first part of this will be pretty much copy and paste from the OpenWRT instructions:

I of course modified the ipaddr for my setup, but pretty much used the command as is.

On the Web UI (LuCI) you should now see under Network->Interfaces a new Guest network. All of the changes landed in the /etc/config/network file.

Now we setup a wireless network configuration with the first radio

Again, I’ve pretty much followed the directions but have customized the SSID and not shared the password. I’ve rolled in the extras to isolate guest users and use encryption.

The LuCI web UI now looks a little more concerning (Network->Wireless)

But.. the previous Network->Interfaces now seems to have come to life.

Still the expected changes are reflected in /etc/config/wireless.

It took a bit of head-scratching, but I figured out what was wrong. I had not specified a wireless.guest.key that met the minimum length (8 to 63 characters) – this apparently caused everything to go sideways. Once I fixed this my new wireless interface came to life.

Let’s continue on with the DHCP configuration

And the Firewall

Now not only should you be able to see the new WiFi SSID available to connect to, but when you do you will be isolated from all other devices on the network and only able to see the internet. A network scan will turn up the existence of the router, but attempts to connect the web UI fail – that’s pretty cool isolation.

Devices connected to the guest network still show up in the OpenWRT status page. They are assigned a DHCP address from the network.guest.ipaddr subnet, which is distinct from my normal network.

Apparently I do give up a little performance having two (or more) networks hung off of the same radio, but the utility of having a restrictive guest network is pretty cool.

The Archer C7 has two radios, and we’ve not configured the guest network on the second radio. Let’s do that now.

Cool – now I have a guest network that is on both radio bands. You’ll note that I run both access points with the same SSID, this mostly just works and devices figure it out. I even run my dumb AP with the same SSID. This is one approach that works and let’s people move around the house with seamless connections.

I do know that some people try to force a device to a particular radio type, and will run their legacy network on a different SSID. This is of course also valid, it really depends what you’re looking to achieve. I’m taking the simple to configure devices approach, and giving the devices the responsibility to work out which radio band and which access point to connect to.

Now let’s create a second ‘guest’ like network for IoT devices. This time I’ll just combine all of the steps together

Nice – now I have a third subnet which will hand out DHCP addresses valid for 24hrs. The devices are all isolated from each other.

For devices on the IoT network, while I don’t want the device to be able to see anything other than the internet – for my own monitoring and use, I’d like to be able to see the devices on the IoT network from my normal lan network. This turns out to be very easy.

Connecting to the IoT network and doing as scan, shows me that I can only see myself and the router (because the device has to send traffic somewhere). Again, with the isolation I can’t connect to the web interface of the router. However, with this new “zone->forwardings” I can from my lan network see devices on the IoT network. Super cool, and actually very easy.

There plenty more tweaking we can do here, but to avoid going too far down the rabbit hole we’ll stop here.

Installing docker-mailserver

Everyone should have their own domain name (or several). Having a website on your domain is easy and a sensible use of that domain name. Almost no one should run their own email server, it’s complicated and makes you responsible for all of the problems.

There are lots of providers out there that run email services and allow you to bring your own domain. ProtonMail would be a good choice, you can even bring your own custom domain and still use ProtonMail. Alternatives are offered by Google, and Microsoft if you want to go that route, both provide support for custom domains.

If you are still thinking of running your own mail server, then grab your tinfoil hat and let’s look at the best way to operate a mail server in the age of containers. I’ve run my own email for a long time, mostly following the Ubuntu setup. Using the docker-mailserver solves all of the email problems with a single container.

I will mention there are many alternatives: mailu, iredmail, etc. The docker-mailserver project stands out for me as they have avoided database based configuration and have stuck with ‘files on disk’ as the model.

This is a long overdue mail upgrade. I started doing this way back in 2017, but never really finished the work. The SSD rebuild disrupted how I was doing things, and changing email is a little scary. The hard drive that stores my email is very old. It is a Seagate Barracuda 40GB (ST340014A). The SMART information says that the Power Cycle Count is only 502, but the Power On Hours is an astounding 130442 (that is 14.89 years). Every stat is in pre-fail or old age, it is definitely time to move my email off that drive.

Before starting, take the time to read through the documentation. Once you think you’re ready to start installing thing the ReadMe is the right path to follow. I’m going a slightly different path than the recommended one which uses docker-compose, and will build out a basic docker deployment.

First I grabbed the following two files

And the setup script for v10.0.0 as I intend to use the ‘stable’ version vs. ‘edge’. It is important to get the matching setup.sh script for the version you are deploying.

I used the docker-compose.yml file to inform me how to configure my Makefile based docker approach. Most of the create options are a direct mimic of the compose file.

I walked through the mailserver.env file and made a few changes

  • ONE_DIR=1
    I’m not totally sure about this, but the documentation reads: “consolidate all states into a single directory (/var/mail-state) to allow persistence using docker volumes.” which seems like a good idea.
    My existing email server uses ClamAV 
    I’m a fan of fail2ban for protecting my server from abuse
    My existing email sever uses SpamAssassin

The volume pointing to letsencrypt is sort of a placeholder for now. Once we get things basically setup I will be changing the SSL_TYPE to enable encryption using my existing letsencrypt certificate that my webserver container has setup.

I later added the following additional configuration to enable logwatch.

    Having an email centric logwatch email vs. combining it with my server logwatch, seemed like a good idea.

  • LOGWATCH_RECIPIENT=postmaster@lowtek.ca
    Where to send the email.

With my Makefile based docker approach I have build, start and update targets. I can manually roll-back if needed as the previous container is has a -old name. The first step is to build the container.

At this point we are at the Get up and running section. We need to start the container and configure some email addresses.

Assuming all goes well, the mailserver container will be running. If we poke around the filesystem we’ll see a few files have been created

  • config/dovecot-quotas.cf
  • maillogs/clamav.log
  • maillogs/freshclam.log

We should be able to run the setup script and add a user, and configure some key aliases.

The creation of the account will cause some additional files to be created

  • config/postfix-accounts.cf
  • config/postfix-virtual.cf

At this point we have a running email server – but we need to start getting data to flow there. You may have to open ports in your firewall on the docker host to allow external traffic to connect to this new container.

This is very encouraging, but there is still a list of things I need to do

  1. Create accounts and aliases
  2. Configure smart host sending via AWS SES
  3. Enable SSL_TYPE
  4. Set up DKIM
  5. Change the port forwarding to point to my container host
  6. Migrate email using imapsync

The rest of this post is details on those steps

Continue reading “Installing docker-mailserver”