My past two posts have been the hardware setup and pi-hole deployment. You rarely get things right the first time and this is no exception.
I’ve already added an update to the first post about my nullmailer configuration. I had forgotten to modify /etc/mailname and thus a lot of the automated email was failing. I happened to notice this because my newly deployed pi-hole was getting a lot of reverse name lookups for my main router, which were triggered by the email system writing the error.
You may also want to go look at / clean out the /var/spool/nullmailer/failed directory, it likely has a bunch of emails that were not sent.
Once corrected, I started to get a bunch of emails. They were two different errors both triggered by visiting the pi-hole web interface
|
1 2 |
myhost : Jul 4 08:14:10 : www-data : unable to resolve host myhost myhost : Jul 4 08:14:10 : www-data : problem with defaults entries ; TTY=unknown ; PWD=/var/www/html/admin ; USER=root ; |
The first issue is a bit curious. Let’s start by looking at the /etc/resolv.conf file which is a generated file based on the DHCP configuration.
|
1 2 3 4 5 6 |
# Generated by dhcpcd from eth0.dhcp, eth0.ra # /etc/resolv.conf.head can replace this line nameserver 149.112.121.30 nameserver 149.112.122.30 nameserver fea4:decf:1667::8 # /etc/resolv.conf.tail can replace this line |
Then if we look at the static configuration that was created by the pi-hole install in /etc/dhcpcd.conf
|
1 2 3 4 |
interface eth0 static ip_address=192.168.1.1/24 static routers=192.168.1.10 static domain_name_servers=149.112.121.30 149.112.122.30 |
Decoding the interaction here. The pi-hole setup script believes that we should have a static IP and it also has assigned the pi-hole to use DNS servers that are the same DNS servers that I specified as my custom entries for upstream resolution.
This feels like the pi-hole install script intentionally makes the pi-hole host machine not able to do local DNS resolutions. I wonder if all pi-hole installations have this challenge. I asked on the forum, – and after a long discussion I came to agree that I fell into a trap.
A properly configured linux machine will have the name from /etc/hostname also reflected in /etc/hosts (as I have done). My setup was working fine because the DNS service on my OpenWRT router was covering up this gap. While it’s nice that it just worked, the machine should really be able to resolve its own name without relying on DNS.
As my network has IPv6 as well, the pi-hole also gets an IPv6 DNS entry. Since the static configuration doesn’t specify an upstream IPv6 DNS – the ULA for the pi-hole leaks into the configuration. This is a potentially scary DNS recursion problem, but it doesn’t seem to be getting used to forward the lookup of ‘myhost’ to the pi-hole so I’m going to ignore this for now.
And easy fix is to go modify /etc/hosts
|
1 2 3 4 5 6 7 8 9 |
127.0.0.1 localhost myhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts |
Where myhost is whatever appears in /etc/hostname.
This will fix the first problem, and it seems the second problem was also caused by the same unable to resolve myhost. I think this is exactly what this post was describing when it was seeing the error messages. This is another thread and different solution to a similar problem.
Satisfied I’d solved that mystery, along the way I’d also created a list of things to ‘fix’
- Ditch rsyslog remote
- Run logwatch locally
- Run fluentbit locally
- Password-less sudo
- SSH key only access
- Add mosh
The rest of this post is just details on these 6 changes.
1. Ditch rsyslog remote
I setup rsyslog because I wanted to get logwatch to run over my logs and avoid yet another daily automated email arriving. Unfortunately logwatch combined with rsyslog wasn’t giving me any separation of what was happening where. On one hand, seeing all of the sudo actions in one place is useful, but not knowing where they happened was a problem.
Rsyslog appears to only be forwarding logs sent through the system (auth, kern, mail, syslog) which is useful – but all of the other log files such as /var/log/pihole.log were not being transported.
Turning this off is as easy as removing the configuration file I had added to /etc/rsyslog.d
2. Run logwatch locally
Just install it.
|
1 |
sudo apt install logwatch |
Now we’ll get nightly reports in email via nullmailer.
3. Run fluent-bit locally
While fluent-bit does document installing the agent on the Raspberry Pi, these instructions are for the Raspbian distribution only.
I’m using Ubuntu, so need to follow the instructions for Ubuntu. Thankfully there is an arm architecture build of the agent.
|
1 2 3 4 5 6 7 8 9 10 11 |
# Add the key curl https://packages.fluentbit.io/fluentbit.key | sudo apt-key add - # Add the Ubuntu source list - create a file in /etc/apt/sources.list.d echo "deb https://packages.fluentbit.io/ubuntu/focal focal main" > /etc/apt/sources.list.d/fluent-bit.list # Update apt sudo apt update # Install td-agent-bit sudo apt install td-agent-bit |
Now that the agent is installed, we need to modify the /etc/td-agent-bit/td-agent-bit.conf configuration file.
The default file just adds some CPU metrics. We’re going to remove those and replace with the following
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# We leave the top part of the agent config file alone # The bit we will replace are the INPUT and OUTPUT sections [INPUT] # tail all host logs name tail Path /var/log/*.log,/var/log/*/*.log,/var/log/syslog,/var/log/lastlog,/var/log/faillog Tag host.* Path_Key filename [OUTPUT] Name loki Host 192.168.1.35 Match host* Labels host=myhost, $filename |
The input section uses the tail plugin. This watches files and forwards as log data any new data appended to those file patterns. The Tag helps match up INPUT and OUTPUT sections, as you can have multiple of each and they don’t all have to connect. The Path_Key is used to pass along a useful label to the Loki backend, labels are a Loki concept.
This of course assumes we have an instances of Grafana Loki running (as I do). There are plenty of other log destinations you could choose. Fluent-bit supports many output plugins, heck we could have even targeted rsyslog.
Once the config file is ready:
|
1 2 3 4 5 6 7 8 |
# enable the service sudo systemctl enable td-agent-bit.service # start the agent sudo systemctl start td-agent-bit.service # check that things are all ok sudo systemctl status td-agent-bit.service |
4. Password-less sudo
Simply add the following to the end of the /etc/sudoers file
|
1 2 |
# no password sudo for user myid myid ALL=(ALL) NOPASSWD:ALL |
The ‘safe’ way to do this is visudo because it provides some checking to ensure you don’t lock yourself out.
5. SSH key only access
This is a two part process. We need to create a ssh key and install it to allow key based ssh. The GitHub doc on this is actually pretty good.
Once you’ve got a key pair (private/public) copy the contents of the public (.pub) file into the .ssh/authorized_keys file on your pi-hole.
The private key belongs in the .ssh/ directory of the client machine (your laptop). Details on this will vary.
It’s a good idea to confirm that the key is working – plenty of ways to do this, but ssh -v is a good way to check.
Now we go and change /etc/ssh/sshd_config to disable passwords.
|
1 2 |
# Change to no to disable tunnelled clear text passwords PasswordAuthentication no |
|
1 |
sudo systemctl restart sshd.service |
To test this, temporarily move your key such that ssh can’t find it – you should get rejected. Attempts to connect without the key will show
|
1 2 |
ssh user@myhost user@myhost: Permission denied (publickey). |
6. Add mosh
Mosh is a great enhancement to your ssh experience. It provides a persistent shell connection even with intermittent networks. What this means is that when you close your laptop and it goes to sleep, you don’t have to reconnect that ssh session.
|
1 2 |
# Install mosh - you will not regret this sudo apt install mosh |