Roo – Page 13 – Roo's View

Lint – the unseen foe

I’ve seen this a few times and it’s always surprised me until I’ve figured it out. Hopefully this brief post helps someone else one day.

Years ago a friend of mine had me over to help take his phone apart. The headphone jack had stopped being reliable (yeah, way back in the day when it was normal for you to plug in headphones). We had fun taking the phone apart, but in the end it turned out that the headphone jack was jammed full of pocket lint. Yup. Some careful digging with a pin and tweezers and we cleared out an alarming amount of lint that had jammed up the port. This fully restored the headphone jack function.

One of my kids had the same thing happen to them. Janky headphone jack, and yup – the bottom was stuffed full with pocket lint. Just be very careful poking around in the port. It’s not very big and you can mess stuff up. Lint is soft and will come out with some gently coaxing.

Lately my ~1.5yr old Pixel 4a had stopped reliably charging. The USB-C cable would fit in fine, but not stay put. It would also pop out very easily. This morning after another failed to charge overnight incident I again inspected the USB-C port. It looked fine. Probing very gently with a pin, it soon became obvious there was some lint in there. Then I pulled out more and more.. an alarming amount. There was a lot of lint. Now I can look into the port and see the shiny plastic bottom, not a dark matted blackness. The USB-C cable seats nice and deeply and doesn’t pop out easily.

Given phones probably live a good percentage of their lives in your pocket, this isn’t a surprising outcome. Still – cleaning out lint wasn’t even close to the first thing I thought of doing in any of these cases. I’d even checked what my warranty and repair options were. The fix was 2 minutes of careful work.

Samsung Galaxy S7 Battery Swap

Sure, the Samsung S7 is a 6 year old phone at this point – but it’s perfect for my son who’s in grade 7 and doesn’t really need a phone. The other day it stopped turning on – when you plugged it in, it would indicate the battery was at 100%. I could even get it to power on while plugged in, but removing the USB power would result in an immediate black screen as it powered off hard.

I sort of dreaded opening this phone up because it’s one of the ones that is glued shut. I was pleasantly surprised, as a little heating with the heat gun and the all metal back came up pretty easily using a suction cup. After that there were some phillips screws to remove and I was able to see the battery.

It was clear there was a problem here – the connector should be squared up with the rest of the circuit board. If you look closely – you can see that the battery has also shifted down significantly within the phone, nearly 2mm.

Taking a close look at the cable – you can see the connector is a little busted up despite my photo being a bit out of focus.

I attempted to reconnect the cable, but soon found out that the connector was badly damaged and it snapped off the cable completely.

Oh well. Off to search up buying a new battery for this phone. A quick look around and it seems there are lots of choices – some as low as $16 (eBay), and the normal crazy mark-up ones at $60-$90. I opted for one of the Chinese made knock off brands off of Amazon that came with tools (junk) and the adhesive to re-attach the back. It also claimed to be 3300 mAh vs the stock 3000. It was at a slight premium vs. eBay, but only a couple of bucks and the reviews were good. Worth the $25 and it shipped to me the next day.

My pricing logic for stuff like this is to avoid the cheapest prices – these are often very cheap for a reason. There is a step up from the cheapest where you’re going to get basically the same part up to the next price plateau – if you can discern the price notches you can basically buy at certain quality levels. The danger with all of these is that lots of unethical sellers will slap OEM labels on parts that are not, so often paying a high premium is not buying quality at all. It’s always a gamble which is frustrating.

The battery I’m replacing was already previously replaced. I think this is why the battery didn’t fit very well in the phone. The poor fit is likely what resulted in it breaking off (when the phone was dropped, probably multiple times if I know my son). If you fit the broken battery into the compartment properly there is a significant gap at the bottom.

Again, this is nearly 2mm gap. The OEM battery is tape/glued in – but I suspect it also fit much more snugly in the space. If you are replacing a battery – consider if it will slide around and either tape – or pad it – to avoid the battery moving. I know that I’ve done battery swaps and left a gap in the past – I probably won’t in the future.

The new battery fits like a glove. Top to bottom, almost no space to move around. So I didn’t bother taping it in place, I’m pretty confident it’ll stay put.

While I’m not a fan of glued shut phones – I did use adhesive to re-seal the phone. Hopefully I won’t have to go back in at all. In a couple of years this phone will basically be too old to use. While it’s still running stock firmware, it does appear that there is an unofficial but current LineageOS build for it.

The S7 got a 3/10 score for repairability – but it wasn’t really that bad to get at the battery. The places where it got hit on the score was replacing some of the other components – I’ve certainly had more than 1 USB charge port go bad, and gluing that to the screen seems like a really bad idea. There really needs to be a better trade of for waterproofing and repairability.

Tasmota, MQTT and Prometheus

In my recent IoT adventures I’ve been using Tasmota firmware to give me local only control over my devices. I’ve started to build out some sensors (temperature, flood) that I want to gather data from, this requires that I have a way to alert (and graph) the data coming out of these sensors. I already have Grafana + Prometheus running, thus it is just a matter of adding a prometheus exporter to get the data out.

Tasmota has built in MQTT support. While I could just craft my own prometheus exporter that scraped the data from the various devices, I decided that adding MQTT to the software stack and using an off the shelf MQTT prometheus exporter meant I was only managing configuration instead of writing something custom.

MQTT is a great protocol for IoT devices. It’s light weight and reliable, it’s dubbed “The Standard for IoT Messaging“. It didn’t take long for me to come across the Ecilpse Mosquitto project which based on the pull stats from dockerhub is very popular. The terms MQTT broker and MQTT server seem to be used interchangeably – the thing to remember is it’s a message queue system that supports publish / subscribe.

The Tasmota device is a client, Mosquitto is the broker, and the Prometheus exporter is a client. Once the data is in Prometheus I can make pretty graphs and create alerts in Grafana.

Running Mosquitto in a container was very easy, but quickly ran into a problem I had created myself with the restricted IoT network. Devices on my IoT network can’t see each other, or really much of anything including not being able to see the MQTT broker.

While I could poke a hole in the IoT network configuration to allow it to see the host:port that my MQTT broker is running on, there are a lot of containers running on that host/IP. Then I remember that I could use the docker macvlan support to create a unique IP address. [Security footnote: while this let’s me have a unique IP address, the code is still running on the same host as the other containers so the additional security is somewhat limited. This is still sort of cool and makes it less likely that I’ll goof up some firewall rules and expose too many things, it also sets me up for running an actual second host if I wanted better security.]

I quickly discovered that you can only have 1 macvlan setup on a host. It may be possible to work around this limitation using the 802.1q trunk bridge mode, this quickly started to seem complicated so I bailed. I did discover that with my existing macvlan network I can specify static IP addresses, and since I have unused IPs in my macvlan network this will work fine.

Here is the makefile that manages the docker deployment of my Mosquitto MQTT broker.

# Mosquitto - a MQTT broker / server 
# https://hub.docker.com/_/eclipse-mosquitto
#
NAME = mosquitto
REPO = eclipse-mosquitto
#
ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))

# Create the container
build:
        docker create \
                --name=$(NAME) \
                -v $(ROOT_DIR)/mosquitto.conf:/mosquitto/config/mosquitto.conf \
                -p 1883:1883 \
                --network myMacvlanNet \
                --ip 192.168.1.65 \
                --restart=unless-stopped \
                $(REPO)

# 
# Start the container
start:
        docker start $(NAME)

# Update the container
update:
        docker pull $(REPO)
        - docker rm $(NAME)-old
        docker rename $(NAME) $(NAME)-old
        make build
        docker stop $(NAME)-old
        make start

# Mosquitto - a MQTT broker / server

# https://hub.docker.com/_/eclipse-mosquitto

NAME = mosquitto

REPO = eclipse-mosquitto

ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))

# Create the container

build:

docker create \

--name=$(NAME) \

-v $(ROOT_DIR)/mosquitto.conf:/mosquitto/config/mosquitto.conf \

-p 1883:1883 \

--network myMacvlanNet \

--ip 192.168.1.65 \

--restart=unless-stopped \

$(REPO)

# Start the container

start:

docker start $(NAME)

# Update the container

update:

docker pull $(REPO)

- docker rm $(NAME)-old

docker rename $(NAME) $(NAME)-old

make build

docker stop $(NAME)-old

make start

And the mosquitto.conf file is simply

listener 1883
allow_anonymous true

1 2	listener 1883 allow_anonymous true

Keeping in mine that my wireguard container is running on 192.168.1.64 and I’ve gone back and modified the wireguard Makefile/container to specify that IP address to ensure that things work reliably after reboots.

The last thing I need to do is modify my OpenWRT configuration to allow the IoT network devices to be able to see this new container. Adding the following to my /etc/config/firewall enables that.

config rule
	option name 'Allow-MQTT-IoT'
	list proto 'tcp'
	option src 'iot'
	option dest 'lan'
	list dest_ip '192.168.1.65'
	option target 'ACCEPT'
	option dest_port '1883'

config rule

option name 'Allow-MQTT-IoT'

list proto 'tcp'

option src 'iot'

option dest 'lan'

list dest_ip '192.168.1.65'

option target 'ACCEPT'

option dest_port '1883'

Not specified here, but a good idea – is to assign a hostname to the static IP of the container so we can later reference it by name.

Configuring my Tasmota devices to talk with the MQTT broker is straightforward now that there is network visibility, the documentation is pretty easy to follow. Viewing the console of the Tasmota device helps see if the connection is successful.

19:53:03.050 MQT: Attempting connection...
19:53:03.072 MQT: Connected
19:53:03.076 MQT: device/sensor1/LWT = Online (retained)
19:53:03.078 MQT: device/sensor1/cmnd/POWER =

19:53:03.050 MQT: Attempting connection...

19:53:03.072 MQT: Connected

19:53:03.076 MQT: device/sensor1/LWT = Online (retained)

19:53:03.078 MQT: device/sensor1/cmnd/POWER =

Another good debugging technique is to shell into the mosquitto container and subscribe to all events.

$ docker exec -it mosquitto sh
/ # mosquitto_sub -t '#' -v

1 2	$ docker exec -it mosquitto sh / # mosquitto_sub -t '#' -v

A quick recap: We have statically assigned an IP to a container from the macvlan network. That container is running a MQTT broker. The IoT devices are sending events to that broker, and moving data from the IoT network onto the network where Prometheus lives.

Now all that is left is to add an exporter to pull data from MQTT and feed Prometheus. Looking at dockerhub it seems https://hub.docker.com/r/kpetrem/mqtt-exporter is very popular, but after some experiments it seemed it didn’t meet my needs. One thing I wanted to support was detecting when a device went missing and there wasn’t an easy way to do this using that exporter.

Using the list of prometheus ports it was easy to find many MQTT exporters. This one (https://github.com/hikhvar/mqtt2prometheus) stood out as a good option. While it doesn’t have a container on dockerhub, it does have one on the github hosted repo (ghcr.io/hikhvar/mqtt2prometheus:latest).

My Makefile for driving the container is

#
# MQTT exporter
# https://github.com/hikhvar/mqtt2prometheus
#
NAME = exporter-mqtt
REPO = ghcr.io/hikhvar/mqtt2prometheus:latest
#
ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))

# Create the container
build:
        docker create \
                --name=$(NAME) \
                -v $(ROOT_DIR)/config.yaml:/config.yaml \
                --network metrics-net \
                --restart=unless-stopped \
                $(REPO)

# 
# Start the container
start:
        docker start $(NAME)

# Update the container
update:
        docker pull $(REPO)
        - docker rm $(NAME)-old
        docker rename $(NAME) $(NAME)-old
        make build
        docker stop $(NAME)-old
        make start

# MQTT exporter

# https://github.com/hikhvar/mqtt2prometheus

NAME = exporter-mqtt

REPO = ghcr.io/hikhvar/mqtt2prometheus:latest

ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))

# Create the container

build:

docker create \

--name=$(NAME) \

-v $(ROOT_DIR)/config.yaml:/config.yaml \

--network metrics-net \

--restart=unless-stopped \

$(REPO)

# Start the container

start:

docker start $(NAME)

# Update the container

update:

docker pull $(REPO)

- docker rm $(NAME)-old

docker rename $(NAME) $(NAME)-old

make build

docker stop $(NAME)-old

make start

and more importantly the config file

mqtt:
 server: tcp://mqttbroker.local:1883
 # Subscribe to all topics
 topic_path: "#"
 # Parse out the device id
 device_id_regex: ".*/(?P<deviceid>.*)/.*"
 # The MQTT QoS level
 qos: 0
#
cache:
 # Each received metric will be presented for this time if no update is send via MQTT.
 timeout: 10m
#
json_parsing:
 separator: .
#
# Payloads
# device/sensor1/SENSOR {"Time":"2022-03-02T17:11:43","Switch1":"ON","AM2301":{"Temperature":11.7,"Humidity":41.2,"DewPoint":-1.1},"TempUnit":"C"}
# device/sensor1/RESULT {"POWER":"OFF"}
#
# Only metrics listed here will be exported
metrics:
 - prom_name: mqtt_temperature
   mqtt_name: AM2301.Temperature
   help: DHT22 temperature reading
   type: gauge

 - prom_name: mqtt_humidity
   mqtt_name: AM2301.Humidity
   help: DHT22 humidity reading
   type: gauge

 - prom_name: mqtt_power
   mqtt_name: POWER
   help: Water sensor
   type: gauge
   # Map "ON" and "OFF" to integer values
   string_value_mapping:
     map:
       ON: 1
       OFF: 0

mqtt:

server: tcp://mqttbroker.local:1883

# Subscribe to all topics

topic_path: "#"

# Parse out the device id

device_id_regex: ".*/(?P<deviceid>.*)/.*"

# The MQTT QoS level

qos: 0

cache:

# Each received metric will be presented for this time if no update is send via MQTT.

timeout: 10m

json_parsing:

separator: .

# Payloads

# device/sensor1/SENSOR {"Time":"2022-03-02T17:11:43","Switch1":"ON","AM2301":{"Temperature":11.7,"Humidity":41.2,"DewPoint":-1.1},"TempUnit":"C"}

# device/sensor1/RESULT {"POWER":"OFF"}

# Only metrics listed here will be exported

metrics:

- prom_name: mqtt_temperature

mqtt_name: AM2301.Temperature

help: DHT22 temperature reading

type: gauge

- prom_name: mqtt_humidity

mqtt_name: AM2301.Humidity

help: DHT22 humidity reading

type: gauge

- prom_name: mqtt_power

mqtt_name: POWER

help: Water sensor

type: gauge

# Map "ON" and "OFF" to integer values

string_value_mapping:

map:

ON: 1

OFF: 0

Most interesting for others here is probably the MQTT payloads and the mapping I’ve made to the config file. It took a few iterations to figure out the correct config file based on the documentation.

Now that I’ve got data flowing, I can create pretty graphs and set alerts as needed.

I’ve also achieved a no-code solution to getting data from the Tasmota IoT sensors into graphs and alerts. The pathway is a little longer than I would like for reliability: IoT device -> MQTT Broker -> exporter -> Prometheus -> Grafana. This means that I need 4 containers to work + the IoT device itself. It’s still not a bad solution, and the Prometheus + Grafana solution is used for monitoring other components of my infrastructure so I pay regular attention to it.