Month: November 2021

Ubuntu adding a 2nd data drive as a mirror (RAID1)

Over the years I’ve had the expected number of hard drive failures. Some have been more catastrophic to me as I didn’t have a good backup strategy in place, others felt avoidable if I’d paid attention the warning signs.

My current setup for data duplication is based on Snapraid, a non-traditional RAID solution. It allows mixed sizes of drives, and the replication is done via regularly running the sync operation. Mine is done daily, files are sync’d across the drives and a data validation is done from time to time as well. This means while I might lose up to 24hrs of data if the primary drive fails, I have lower usage of the main parity drive and I get the assurance that file corruption hasn’t happened.

Snapraid is very bad when you have either: many small files, frequently changing files. It is ideal for backing up media like photos or movies. To deal with the more rapidly changing data I’ve got a SSD drive for storage. I haven’t yet had a SSD fail on me, but that is assured to happen at one point. Backblaze is already seeing some failure rate information that is concerning. Couple this with the fact that my storage SSD started throwing errors the other day and only a full power cycle of the machine brought it back  – it’s fine now, but for how long? Time to setup a mirror.

For this storage I’m going back to traditional RAID. The SSD is a 480GB drive, and thankfully the price of them has dropped to easily under $70. This additional drive now fills all 6 of the SATA ports on my motherboard, the next upgrade will need to be an SATA port expansion card. I’ve written about RAID a few times here.

I’ve moved away from specifying drives as /dev/sdbX because these values can change. Even this new SSD caused the drive that was at /dev/sdf to move to /dev/sdg allowing the new drive to use /dev/sdf. My /etc/fstab is now setup using /dev/disk/by-id/xxx because these are persistent. Most of the disk utilities understand this format just fine as you an see with this example with fdisk.

Granted, working with /dev/disk/by-id is a lot more verbose – but that id will not change if you re-organize the SATA cables.

Let’s get going on setting up the new drive as a mirror for the existing one. Here’s the basic set of steps

  1. Partition the new drive so it is identical to the existing one
  2. Create a RAID1 array in degraded state
  3. Format and mount the array
  4. Copy the data from the existing drive to the new array
  5. Un-mount both the array and the original drive
  6. Mount the array where the original drive was mounted
  7. Make sure things are good – the next step is destructive
  8. Add the original drive to the degraded RAID1 array making it whole

It may seems like a lot of steps, and some of them are scary – but on the other side we’ll have a software RAID protecting the data. The remainder of this post will be the details of those steps above.

Continue reading “Ubuntu adding a 2nd data drive as a mirror (RAID1)”

Tasmota firmware (pwn your IoT)

Long gone are the days where X10 rules the Smart Home devices space and with ubiquitous WiFi and cheap ESP hardware we’re seeing IoT devices that connect to WiFi. The problem is that almost all of them want to call home and talk to some service in the cloud. Sure you bought the device, but do you really own it?

When I needed a WiFi controlled outlet, I headed off to the Tasmota Supported Devices Repository to determine which one I should buy. Tasmota is one of the options for alternative firmware for ESP devices. This gives you control over the software running on the IoT device, and most importantly the ability to use it without any cloud server that you don’t control. This is still annoyingly difficult, we really need the tech industry to adopt a better way to give people easy to use devices and software without insisting they give up all control.

Buying from Amazon, I didn’t have to wait long to get a cheap WiFi outlet. It is thanks to Michael Steigerwald and his talk “Smart home – Smart hack”  that we have a way to over the air update some of the devices running the Tyua firmware. Unfortunately, to my dismay, I discovered that many of the Tuya based devices ship with a newer and more secure firmware preventing this hack from always working.

The tuya-convert project is pretty comprehensive, but still requires a fairly deep technical understanding to pull off. I tried a couple of ways to run the software before giving up and using a RaspberryPi. Once I decided to go with the Pi things were much easier.

I got lucky as the Moko YX-WS01A appears to ship with old firmware, my next purchase may be more carefully researched. I was very careful to not connect it to the recommended software (smartapp.tyua.com) as that was likely to cause a firmware update. I really didn’t want to have to crack this thing open and hook up to the ESP physically. Maybe the Moko outlets will continue to ship the older, exploitable, firmware – but buyer beware.

Once I had the very basic Tasmota firmware installed, a tasmota_XXXXXX-#### network access point was available (where XXXXXX is a string derived from the device’s MAC address and #### is a number). I can now connect to this access point and configure the device to one of my WiFi networks by opening a browser on 192.168.4.1. Take care, if you mess up the WiFi password you may have trouble recovering the device.

This screen is different than the Tasmota instructions, I suspect this is because the binary provided as part of tuya-convert is stripped down and does not have any specific hardware configured.

Once you configure a connection to a WiFi network, you’ll lose the access point connection, but you will be able to locate the device on the network you connected it to. It will appear with the device name tasmota_XXXXXX-####.

Before we go further, we’ll perform a reset 5 as advised on this page. It may not be needed, but it sounds like a good idea. This is easy to do with the Console provided on the web UI.

We can see that we’re back on version 9.2.0 – so next we’re going to update the firmware. Which firmware should we pick? This page provides a good overview of the various options. There are many ways to perform the upgrade – I’ve elected to download the .gz binary an provide that file to the web UI. I’ve also picked the default and recommended tasmota.bin.gz file. This will update me to version 10.0.0.

The performance of the web UI seemed quite slow, I have to keep reminding myself this is a very basic microcontroller that costs a few dollars. It’s pretty amazing it works. Post firmware upgrade the web performance does seem quite a bit better.

At this point I can hit the Toggle button and see the LED on the outlet turn off an on, but I don’t seem to be triggering the outlet itself. More configuration is needed.

From the web UI, choosing Configure then Configure Module I can see that this is setup as a generic device with only 4 GPIO pins. Using this template as a guide, I select Generic (18) and set the GPIO pins as indicated. This works great, and I can now toggle the outlet on an off via the Web UI.

A word of warning. Back when the device was acting as an access point – you can only attach one device to it, attempts to connect a second client will fail. I also had some weirdness configuring the module, but I think this was because I had multiple browsers / apps pointed at the one device. Go slow, and do one thing at a time.

As for app based control, there are several Android apps which will bypass the need for a MQTT setup and work directly against the HTTP endpoint. I tried several, but decided for my simple needs Tasmota HomeSwitch was a good match.

Using the app seems to mostly work, but has some latency at times depending where the device is at in terms of responding to the HTTP requests. I notice the same type of latency using the web app, but this represents itself more as slowness to load the page vs. waiting for a button press on the app to take effect.

Bonus – the device appears to persist it’s state (on/off) even if you unplug it from power. This is pretty useful as it means that if there a power failure, it will return to the previous state.

Sure it only supports up to 10A, but wifi control over power and I can keep it entirely on my own network is pretty slick.