Pi-hole Ubuntu Server (take 2)

My past two posts have been the hardware setup and pi-hole deployment. You rarely get things right the first time and this is no exception.

I’ve already added an update to the first post about my nullmailer configuration. I had forgotten to modify /etc/mailname and thus a lot of the automated email was failing. I happened to notice this because my newly deployed pi-hole was getting a lot of reverse name lookups for my main router, which were triggered by the email system writing the error.

You may also want to go look at / clean out the /var/spool/nullmailer/failed directory, it likely has a bunch of emails that were not sent.

Once corrected, I started to get a bunch of emails. They were two different errors both triggered by visiting the pi-hole web interface

The first issue is a bit curious. Let’s start by looking at the /etc/resolv.conf file which is a generated file based on the DHCP configuration.

Then if we look at the static configuration that was created by the pi-hole install in /etc/dhcpcd.conf

Decoding the interaction here. The pi-hole setup script believes that we should have a static IP and it also has assigned the pi-hole to use DNS servers that are the same DNS servers that I specified as my custom entries for upstream resolution.

This feels like the pi-hole install script intentionally makes the pi-hole host machine not able to do local DNS resolutions. I wonder if all pi-hole installations have this challenge. I asked on the forum, – and after a long discussion I came to agree that I fell into a trap.

A properly configured linux machine will have the name from /etc/hostname also reflected in /etc/hosts (as I have done). My setup was working fine because the DNS service on my OpenWRT router was covering up this gap. While it’s nice that it just worked, the machine should really be able to resolve its own name without relying on DNS.

As my network has IPv6 as well, the pi-hole also gets an IPv6 DNS entry. Since the static configuration doesn’t specify an upstream IPv6 DNS – the ULA for the pi-hole leaks into the configuration. This is a  potentially scary DNS recursion problem, but it doesn’t seem to be getting used to forward the lookup of ‘myhost’ to the pi-hole so I’m going to ignore this for now.

And easy fix is to go modify /etc/hosts

Where myhost is whatever appears in /etc/hostname.

This will fix the first problem, and it seems the second problem was also caused by the same unable to resolve myhost. I think this is exactly what this post was describing when it was seeing the error messages. This is another thread and different solution to a similar problem.

Satisfied I’d solved that mystery, along the way I’d also created a list of things to ‘fix’

  1. Ditch rsyslog remote
  2. Run logwatch locally
  3. Run fluentbit locally
  4. Password-less sudo
  5. SSH key only access
  6. Add mosh

The rest of this post is just details on these 6 changes.

1. Ditch rsyslog remote

I setup rsyslog because I wanted to get logwatch to run over my logs and avoid yet another daily automated email arriving. Unfortunately logwatch combined with rsyslog wasn’t giving me any separation of what was happening where. On one hand, seeing all of the sudo actions in one place is useful, but not knowing where they happened was a problem.

Rsyslog appears to only be forwarding logs sent through the system (auth, kern, mail, syslog) which is useful – but all of the other log files such as /var/log/pihole.log were not being transported.

Turning this off is as easy as removing the configuration file I had added to /etc/rsyslog.d

2. Run logwatch locally

Just install it.

Now we’ll get nightly reports in email via nullmailer.

3. Run fluent-bit locally

While fluent-bit does document installing the agent on the Raspberry Pi, these instructions are for the Raspbian distribution only.

I’m using Ubuntu, so need to follow the instructions for Ubuntu. Thankfully there is an arm architecture build of the agent.

Now that the agent is installed, we need to modify the /etc/td-agent-bit/td-agent-bit.conf configuration file.

The default file just adds some CPU metrics. We’re going to remove those and replace with the following

The input section uses the tail plugin. This watches files and forwards as log data any new data appended to those file patterns. The Tag helps match up INPUT and OUTPUT sections, as you can have multiple of each and they don’t all have to connect. The Path_Key is used to pass along a useful label to the Loki backend, labels are a Loki concept.

This of course assumes we have an instances of Grafana Loki running (as I do). There are plenty of other log destinations you could choose.  Fluent-bit supports many output plugins, heck we could have even targeted rsyslog.

Once the config file is ready:

 4. Password-less sudo

Simply add the following to the end of the /etc/sudoers file

The ‘safe’ way to do this is visudo because it provides some checking to ensure you don’t lock yourself out.

5. SSH key only access

This is a two part process. We need to create a ssh key and install it to allow key based ssh. The GitHub doc on this is actually pretty good.

Once you’ve got a key pair (private/public) copy the contents of the public (.pub) file into the .ssh/authorized_keys file on your pi-hole.

The private key belongs in the .ssh/ directory of the client machine (your laptop). Details on this will vary.

It’s a good idea to confirm that the key is working – plenty of ways to do this, but ssh -v is a good way to check.

Now we go and change  /etc/ssh/sshd_config to disable passwords.

Also don’t forget to restart sshd so the changes take affect

To test this, temporarily move your key such that ssh can’t find it – you should get rejected. Attempts to connect without the key will show

6. Add mosh

Mosh is a great enhancement to your ssh experience. It provides a persistent shell connection even with intermittent networks. What this means is that when you close your laptop and it goes to sleep, you don’t have to reconnect that ssh session.


Leave a Reply

Your email address will not be published. Required fields are marked *