Category: IoT

Tasmota, MQTT and Prometheus

In my recent IoT adventures I’ve been using Tasmota firmware to give me local only control over my devices. I’ve started to build out some sensors (temperature, flood) that I want to gather data from, this requires that I have a way to alert (and graph) the data coming out of these sensors. I already have Grafana + Prometheus running, thus it is just a matter of adding a prometheus exporter to get the data out.

Tasmota has built in MQTT support. While I could just craft my own prometheus exporter that scraped the data from the various devices, I decided that adding MQTT to the software stack and using an off the shelf MQTT prometheus exporter meant I was only managing configuration instead of writing something custom.

MQTT is a great protocol for IoT devices. It’s light weight and reliable, it’s dubbed “The Standard for IoT Messaging“. It didn’t take long for me to come across the Ecilpse Mosquitto project which based on the pull stats from dockerhub is very popular. The terms MQTT broker and MQTT server seem to be used interchangeably – the thing to remember is it’s a message queue system that supports publish / subscribe.

The Tasmota device is a client, Mosquitto is the broker, and the Prometheus exporter is a client. Once the data is in Prometheus I can make pretty graphs and create alerts in Grafana.

Running Mosquitto in a container was very easy, but quickly ran into a problem I had created myself with the restricted IoT network. Devices on my IoT network can’t see each other, or really much of anything including not being able to see the MQTT broker.

While I could poke a hole in the IoT network configuration to allow it to see the host:port that my MQTT broker is running on, there are a lot of containers running on that host/IP. Then I remember that I could use the docker macvlan support to create a unique IP address. [Security footnote: while this let’s me have a unique IP address, the code is still running on the same host as the other containers so the additional security is somewhat limited. This is still sort of cool and makes it less likely that I’ll goof up some firewall rules and expose too many things, it also sets me up for running an actual second host if I wanted better security.]

I quickly discovered that you can only have 1 macvlan setup on a host. It may be possible to work around this limitation using the 802.1q trunk bridge mode, this quickly started to seem complicated so I bailed. I did discover that with my existing macvlan network I can specify static IP addresses, and since I have unused IPs in my macvlan network this will work fine.

Here is the makefile that manages the docker deployment of my Mosquitto MQTT broker.

And the mosquitto.conf file is simply

Keeping in mine that my wireguard container is running on 192.168.1.64 and I’ve gone back and modified the wireguard Makefile/container to specify that IP address to ensure that things work reliably after reboots.

The last thing I need to do is modify my OpenWRT configuration to allow the IoT network devices to be able to see this new container. Adding the following to my /etc/config/firewall enables that.

Not specified here, but a good idea – is to assign a hostname to the static IP of the container so we can later reference it by name.

Configuring my Tasmota devices to talk with the MQTT broker is straightforward now that there is network visibility, the documentation is pretty easy to follow. Viewing the console of the Tasmota device helps see if the connection is successful.

Another good debugging technique is to shell into the mosquitto container and subscribe to all events.

A quick recap: We have statically assigned an IP to a container from the macvlan network. That container is running a MQTT broker. The IoT devices are sending events to that broker, and moving data from the IoT network onto the network where Prometheus lives.

Now all that is left is to add an exporter to pull data from MQTT and feed Prometheus. Looking at dockerhub it seems https://hub.docker.com/r/kpetrem/mqtt-exporter is very popular, but after some experiments it seemed it didn’t meet my needs. One thing I wanted to support was detecting when a device went missing and there wasn’t an easy way to do this using that exporter.

Using the list of prometheus ports it was easy to find many MQTT exporters. This one (https://github.com/hikhvar/mqtt2prometheus) stood out as a good option. While it doesn’t have a container on dockerhub, it does have one on the github hosted repo (ghcr.io/hikhvar/mqtt2prometheus:latest).

My Makefile for driving the container is

and more importantly the config file

Most interesting for others here is probably the MQTT payloads and the mapping I’ve made to the config file. It took a few iterations to figure out the correct config file based on the documentation.

Now that I’ve got data flowing, I can create pretty graphs and set alerts as needed.

I’ve also achieved a no-code solution to getting data from the Tasmota IoT sensors into graphs and alerts. The pathway is a little longer than I would like for reliability: IoT device -> MQTT Broker -> exporter -> Prometheus -> Grafana. This means that I need 4 containers to work + the IoT device itself. It’s still not a bad solution, and the Prometheus + Grafana solution is used for monitoring other components of my infrastructure so I pay regular attention to it.

Learning Rust by (re)writing a prometheus exporter

As part of my self hosted infrastructure I’ve got prometheus setup gather metrics from various sources, and Grafana to visualize them. My TED5000 gives me power usage information for the whole house, and my thermostat provides temperature data.

When using prometheus, you need exporters to provide metric data. There is an existing TED5000 exporter that I’m using, but there wasn’t one that I found for the thermostat – so I created one. The initial implementation was in python, and this worked fine. However I’d see in my pi-hole dashboard that lookups of the thermostat name were high in the stats (22000+ lookups per 24hr period). Looking at the logs, it seems every 15 second four lookups would happen, two pairs (A and AAAA) separated by 2 seconds. I suspect this was a side effect of the radiotherm library I was using in my code.

The exporter is very simple, it’s just a webserver that responds by making a web request to the thermostat, and responding with the data formatted for prometheus. The response payload looks like:

I figured that this was a good opportunity for me to learn Rust with a practical project.

The first thing I did was get myself setup to compile Rust code. I did this using a docker container as inspired by my previous post.

At this point I was able to create my first hello world rust code. I went from printing “hello” to following a tutorial to create a very simple webserver – which will be sufficient for a prometheus exporter.

I then started to learn about cargo and the crates.io site. It turns out there are a few prometheus crates out there, I found one that looked like a good match but after looking at it in more details I decided it was a lot more code and capability than I was looking for. Consider again the very simple response payload above, I really need very little help from a library.

I located a tutorial on reading data from a URL in Rust. This tutorial was less complete as it made assumptions you had more knowledge of managing crates than I did at the time. In order to take the code snippet and get working code you needed to add two lines to your Cargo.toml file. Furthermore, the code makes use of the blocking feature in the reqwest library which is not on by default. Using crates.io you can find the libraries (error-chain, reqwest) and details on how to configure them. This ends up being what you need:

At this point I now have two samples of code. One which is a single threaded web server, and a second which can read from a URL (the thermostat) and parse out the JSON response. A little bit of hacking to figure out return syntax in Rust and I’ve managed to smash these together and have a basic exporter working.

The existing exporter runs as a container on my server, so all that remains is to wrap this Rust code up in a container and I’ve got a complete replacement.

Looking at the binary I’d been running with cargo run I was a little surprised to see it was 60MB, I was quick to rationalize that this was in the debug tree. Compiling a release version (cargo build --release) resulted in a much smaller binary (8.5MB). This size reduction sent me off to see how small a container I could easily create.

Two things I want: a) a multi-stage build Dockerfile b) a static binary that will run in a scratch image. Luckily this is well travelled ground and is concisely explained in this blog post for creating small rust containers.

The result was a 12MB docker image that only contains the one binary that is my code.

The previous python based implementation built on an alpine base image was 126MB in size, so a 10x reduction in size – plus this new image runs without privilege (ie: as a user). This means this container has a very small attack surface. It appears to use %0.01 CPU vs. the previous %0.02. You can check the complete code out on github.

My pi-hole is showing me that it is still one of the higher names resolved with 11520 lookups in a 24hr period. This maps out to 24 hours * 60 minutes * 4 (every 15 seconds). I’ve improved on the DNS lookups, but it still feels like I can further improve this with a some clever coding.

Flashing Tasmota over serial

In my previous post on Tasmota I was able to make use of tuya-convert which supports over the air (OTA) updating a ‘smart wifi’ device to the Tasmota firmware. Tuya-convert relies on exploiting a defect in the firmware, Tuya has patched this bug and some (many) device manufacturers have started to ship updated versions of the software – and in some cases new, non-ESP based hardware. Thankfully many devices are still based on ESP hardware and many of the circuit boards even have test points exposed that make hooking up a serial adapter possible for someone comfortable with a soldering iron.

The Tasmota doc is a great place to start. The device I’m working on is a Gosund Smart Light Switch SW5. I did try tuya-convert on this and got the id2 response which indicates that there is a newer firmware. Opening up the lightswitch was easy, but you will need T6 torx screwdriver. Once the circuit board was out I could take a close look.

We can see that the chip is an ESP-8285, that will let me find the data sheet and figure out the pin outs. We can also use the Tasmota doc on pin outs as a reference. Right at the top of the image you can see test / solder pads on the edge, these are going to be useful.

I’ve annotated the chip diagram to show the pins I’ll need to flash the device. It’s hard to tell from the photos I’ve shown so far the scale of the chip, but I can tell you that I don’t have the skills to solder directly to the pins of the chip – I’ll need to use the break out pads on the side of the circuit board. Even those solder pads are very small.

I confirmed based on the data sheet that the break out pads mapped to the pins I had identified on the chip using a multi-meter. The pins on the chip are very small and I was working by feel mostly, but it gave me enough confidence that I could use the break out pads to do the connection.

It is very important to only power the ESP-8285 with 3.3V. If you use 5V you are very likely to break things permanently. Off to eBay I went to pick up what I thought was one of the recommended CH340G serial adapter boards. I later learned that the one I bought did not have a voltage regulator making it unable to supply enough power for the ESP-8285.

You can see that while this board supports 3.3V or 5V – you must modify the board to break the solder bridge to the 5V and add one to the 3.3V. I was able to verify the voltage was 3.3V after my modification with a multi-meter.

At first I had decided because the break out pads were evenly spaced and about the same spacing as a pin header. I did try this – but got stumped by the fact that if I did hook it up correctly the LED on the CH340G adapter would go out and things didn’t work. This turned out to be due to the fact that this adapter would not supply enough power to the ESP-8285. This approach might work, but soldering to the pads was easy enough and that’s the path I ended up taking.

I decided to use Linux and the esptool to do the flashing. It turns out that I could just use pip to install the esptool.

On Linux, if you haven’t modified your user to be in the right group you may not be able to use the /dev/ttyUSB0 device. There may also be additional things you need to do in order for the adapter to be recognized by the OS. I’ll leave these challenges up to the reader to overcome with some creative use of a search engine.

Here are the results of my soldering job – along with a micro-SD card for a sense of scale. This is very small but it’s not too difficult if you use a few tricks. Solder paste on the pads applied with a toothpick are a big help to get the solder to flow. Pre-tinning the wires helps. Working under magnification helped my aging eyes. Make sure the soldering iron is completely warmed up. Also add a little solder to the tip of the iron (pre-tin the tip). With these preparation steps, it should be easy to just touch the iron and the wire to the pad and the solder will flow and you’re good.

Initially I had wanted to avoid soldering, but it turned out to be very quick and easy to do. The reliability of the connection is also superior to trying to press fit the wires – and creating a jig would take a lot more time. In future when I need to flash over serial I’ll just go warm up the soldering iron.

I then decided the correct wiring to perform. Note that TX/RX are swapped. IO0 is pulled to ground to force the ESP into programming mode.

It was here where I ran completely into the wall trying to use my cheap eBay CH340G adapter. I did find someone else who’d succeeded flashing the SW5, which gave me hope that I could succeed. It turns out there is a friendly community you can connect with on Discord to talk about Tasmota things. I started a conversation in the #flashing-issues channel and after a short while got some helpful advice.

Now it turns out, I’d been hasty. If I’d done more research I would have found that having a stable 3.3V power supply wasn’t a given for all of the adapters. While the golden CH340G can be had for a few dollars – many of the CH340G adapters do not have a voltage regulator on them and will not work with the ESP devices. Once the Discord folks had kindly educated me on this – I created a pull request to add some additional warnings to the doc (which has been merged).

Fellow Canadians might be able to grab one of these CH340G adapters from Amazon, it has the required voltage regulator. In my case I dug through my pile of electronic projects and  came up with a Circuit Playground Express, it has an onboard voltage regulator and will easily deliver the ~150mA that the ESP needs (as long as I’m not trying to drive the onboard LEDs).

This changes my wiring diagram. I’ve abbreviated Circuit Playground Express (CPE).

By connecting all the GND lines together I ensure both devices are using the same reference. I’m pulling the 3.3V from the CPE, but the CH340G adapter is being used for the programming. This worked like a charm, but was a mess of wires.

Now that I have stable power, I can follow the steps.

1. Backup firmware

Unplug the connections and do them again – (power cycles the setup)

2. Erase flash

Again, reset the world

3. Upload tasmota.bin – make sure you get the right firmware.

Now you can test your work by providing only power (GND and 3.3v) to the SW5 device to see if the Tasmota firmware starts up ok and offers a WiFi access point (tasmota_XXXXXX-####). You can even go through the initial setup and get it connected to your network.

After I’d gotten my SW5 running Tasmota, I then re-assembled the actual light switch. I found that you had to carefully tighten the screws to avoid binding the switching action. I then temporarily wired this to house wiring to confirm that all of the functions worked as expected. Once things were good, I could permanently install this in my desired location.

I’d referenced flashing Tasmota over serial as the ‘hard way’ – but as long as you can disassemble the device to get to the circuit board, and there are test points you can solder to – it isn’t all that hard. You do need to be comfortable with a soldering iron, and have the right gear to program 3.3V serial.