Category: How To

Learning Rust by (re)writing a prometheus exporter

As part of my self hosted infrastructure I’ve got prometheus setup gather metrics from various sources, and Grafana to visualize them. My TED5000 gives me power usage information for the whole house, and my thermostat provides temperature data.

When using prometheus, you need exporters to provide metric data. There is an existing TED5000 exporter that I’m using, but there wasn’t one that I found for the thermostat – so I created one. The initial implementation was in python, and this worked fine. However I’d see in my pi-hole dashboard that lookups of the thermostat name were high in the stats (22000+ lookups per 24hr period). Looking at the logs, it seems every 15 second four lookups would happen, two pairs (A and AAAA) separated by 2 seconds. I suspect this was a side effect of the radiotherm library I was using in my code.

The exporter is very simple, it’s just a webserver that responds by making a web request to the thermostat, and responding with the data formatted for prometheus. The response payload looks like:

I figured that this was a good opportunity for me to learn Rust with a practical project.

The first thing I did was get myself setup to compile Rust code. I did this using a docker container as inspired by my previous post.

At this point I was able to create my first hello world rust code. I went from printing “hello” to following a tutorial to create a very simple webserver – which will be sufficient for a prometheus exporter.

I then started to learn about cargo and the crates.io site. It turns out there are a few prometheus crates out there, I found one that looked like a good match but after looking at it in more details I decided it was a lot more code and capability than I was looking for. Consider again the very simple response payload above, I really need very little help from a library.

I located a tutorial on reading data from a URL in Rust. This tutorial was less complete as it made assumptions you had more knowledge of managing crates than I did at the time. In order to take the code snippet and get working code you needed to add two lines to your Cargo.toml file. Furthermore, the code makes use of the blocking feature in the reqwest library which is not on by default. Using crates.io you can find the libraries (error-chain, reqwest) and details on how to configure them. This ends up being what you need:

At this point I now have two samples of code. One which is a single threaded web server, and a second which can read from a URL (the thermostat) and parse out the JSON response. A little bit of hacking to figure out return syntax in Rust and I’ve managed to smash these together and have a basic exporter working.

The existing exporter runs as a container on my server, so all that remains is to wrap this Rust code up in a container and I’ve got a complete replacement.

Looking at the binary I’d been running with cargo run I was a little surprised to see it was 60MB, I was quick to rationalize that this was in the debug tree. Compiling a release version (cargo build --release) resulted in a much smaller binary (8.5MB). This size reduction sent me off to see how small a container I could easily create.

Two things I want: a) a multi-stage build Dockerfile b) a static binary that will run in a scratch image. Luckily this is well travelled ground and is concisely explained in this blog post for creating small rust containers.

The result was a 12MB docker image that only contains the one binary that is my code.

The previous python based implementation built on an alpine base image was 126MB in size, so a 10x reduction in size – plus this new image runs without privilege (ie: as a user). This means this container has a very small attack surface. It appears to use %0.01 CPU vs. the previous %0.02. You can check the complete code out on github.

My pi-hole is showing me that it is still one of the higher names resolved with 11520 lookups in a 24hr period. This maps out to 24 hours * 60 minutes * 4 (every 15 seconds). I’ve improved on the DNS lookups, but it still feels like I can further improve this with a some clever coding.

Using Docker to isolate development environments

In terms of modern programming languages, I would pick golang as a great language to use. To me, it feels a little bit like a modernized C and has a nice approach to building single file binaries. It has a module system, type safety and garbage collection.

I really dislike the way that you install golang onto your system and the way you end up managing that install with environment variables and the like. It’s not a terrible install story, but I don’t like polluting my development laptop with cruft. Multiple versions make this more annoying.

Docker on the other hand, despite it’s flaws, is worth having installed. It allows you to run lots of different stuff without needing to make a big commitment about the install. So instead of re-imaging my machine to start fresh, I just purge the containers and start again. Another benefit is that it’s relatively easy for me to point someone else at my configuration and for them to re-use it nearly directly.

Getting a docker image that will persist state and make it trivial to compile golang programs turns out to be very easy

A couple of things to note. I’m using a host mounted volume – specifically the current directory that I issue the docker create in. From inside the container it is mapped to /data. I’ve also named the container, making it easy for me to re-run/attach to it for future compiles.

Edit – running with -u is a good idea to make docker run as the right user (you). This will mean that files created by that container on the mounted host volume are owned by you.

As an example, here is how I’d go about compiling a github project that is in golang.

How slick is that? My development machine needs docker and git installed. The rest of this environment is entirely inside the docker container.

Let me now demonstrate persistence of the container from run to run.

Thus if I happen to need a utility which isn’t installed in the base golang bullseye image, it’s easy for me to install. Also, from run to run – I have persistence of the changes I’ve made to my named image.

Ripping CDs to FLAC on Linux

This weekend I was messing around with ripping CDs again. Almost my entire collection has been digital for years, but all in MP3 format. Way back when ripping a CD to MP3 was a trade-off of time to encode, quality of playback, size of file I’d made some choices about what I was going to use as my standard for digitizing my collection.

I arrived at 192 kbps encoding (fixed bitrate), MP3 format encoded with the LAME encoder for the most part. To arrive at this bitrate I’d done a lot of A/B sampling between songs – with my best attempts at blind listening comparison to see which ones I could tell the difference between. After about 160 kbps I couldn’t hear any significant differences, and certainly after 192 kbps it was all the same. If you want to learn more about bitrates and encoding formats, this seems like a good article.

Since then – computers have gotten stupidly faster, so encoding time doesn’t matter . Storage is also cheap and plentiful, so I don’t care about file sizes. My main media server is Plex, which will happily transcode from FLAC for me to MP3 when I need it. There is also the mp3fs userspace filesystem that I can use to map FLAC to MP3 when I need it. I’d arrived at the conclusion that my archive format should be FLAC a couple of years ago, but I’d failed to get Linux setup to rip successfully.

With Windows machines there has been EAC which is basically the default way people who care will rip their music. It ensures ‘perfect’ copies. I found whipper which seemed to provide a similar solution on Linux, but a couple of years ago I failed to get this to work with either of the optical drives I have installed in my box. I could get all the tracks but the last one on the disc, very frustrating.

In my recent revisit to this, I started with abcde. This resulted in a simple docker container that I could run that would rip a disc to FLAC without much fuss. It worked well, but then I re-discovered my notes on using whipper and figured I’d see if the project had progressed.

It had – and whipper works great with one of my optical drives, but not the other. That’s fine as one is enough. My working drive is an ASUS DRW-1814BL and so far no problems except for one disc. The one problematic CD was one that I’d failed to rip in the past as well, there is a little bit of physical damage and whipper would bail dealing with the index.

It turns out my abcde setup worked great on this bad CD and was able to rip it to FLAC. I’ve less confidence that the abcde process is as robust and exact as whipper – but I’d rather have the music ripped than not at all.

For abcde I have a Makefile

And a Dockerfile

You’ll want to customize it to point at the right device, but there isn’t much here. Make build, make run and you’re good to go.

For whipper, it’s ever easier. I just used a shell script to call the pre-built container.

There is a little bit of setup for whipper, best to follow the documentation. Briefly you need to analyze the drive and figure out the offset. I also tweaked the generated config file to change the directory and file naming scheme.

That’s it. Ripping your CDs with high quality is trivial now, and with nicely featured media servers the format doesn’t matter. Now I just have to slowly re-rip CDs that I care about having high quality archives of.