{"id":1839,"date":"2021-07-05T21:38:18","date_gmt":"2021-07-06T01:38:18","guid":{"rendered":"https:\/\/lowtek.ca\/roo\/?p=1839"},"modified":"2021-07-05T21:38:18","modified_gmt":"2021-07-06T01:38:18","slug":"pi-hole-ubuntu-server-take-2","status":"publish","type":"post","link":"https:\/\/lowtek.ca\/roo\/2021\/pi-hole-ubuntu-server-take-2\/","title":{"rendered":"Pi-hole Ubuntu Server (take 2)"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1844\" src=\"https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2021\/07\/clapper-take2.png\" alt=\"\" width=\"1200\" height=\"400\" srcset=\"https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2021\/07\/clapper-take2.png 1200w, https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2021\/07\/clapper-take2-500x167.png 500w, https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2021\/07\/clapper-take2-1024x341.png 1024w, https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2021\/07\/clapper-take2-768x256.png 768w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p>My past two posts have been the <a href=\"https:\/\/lowtek.ca\/roo\/2021\/raspberry-pi-ubuntu-server\/\">hardware setup<\/a> and <a href=\"https:\/\/lowtek.ca\/roo\/2021\/pi-hole-a-black-hole-for-advertisements\/\">pi-hole deployment<\/a>. You rarely get things right the first time and this is no exception.<\/p>\n<p>I&#8217;ve already added an update to the first post about my nullmailer configuration. I had forgotten to modify <code>\/etc\/mailname<\/code> and thus a lot of the automated email was failing. I happened to notice this because my newly deployed pi-hole was getting a lot of reverse name lookups for my main router, which were triggered by the email system writing the error.<\/p>\n<p>You may also want to go look at \/ clean out the <code>\/var\/spool\/nullmailer\/failed<\/code> directory, it likely has a bunch of emails that were not sent.<\/p>\n<p>Once corrected, I started to get a bunch of emails. They were two different errors both triggered by visiting the pi-hole web interface<\/p>\n<pre class=\"lang:default decode:true \">myhost : Jul \u00a04 08:14:10 : www-data : unable to resolve host myhost\r\nmyhost : Jul \u00a04 08:14:10 : www-data : problem with defaults entries ; TTY=unknown ; PWD=\/var\/www\/html\/admin ; USER=root ;<\/pre>\n<p>The first issue is a bit curious. Let&#8217;s start by looking at the <code>\/etc\/resolv.conf<\/code> file which is a generated file based on the DHCP configuration.<\/p>\n<pre class=\"lang:default decode:true \"># Generated by dhcpcd from eth0.dhcp, eth0.ra\r\n# \/etc\/resolv.conf.head can replace this line\r\nnameserver 149.112.121.30\r\nnameserver 149.112.122.30\r\nnameserver fea4:decf:1667::8\r\n# \/etc\/resolv.conf.tail can replace this line<\/pre>\n<p>Then if we look at the static configuration that was created by the pi-hole install in <code>\/etc\/<span class=\"s1\">dhcpcd.conf<\/span><\/code><\/p>\n<pre class=\"lang:default decode:true \">interface eth0\r\n        static ip_address=192.168.1.1\/24\r\n        static routers=192.168.1.10\r\n        static domain_name_servers=149.112.121.30 149.112.122.30<\/pre>\n<p>Decoding the interaction here. The pi-hole setup script believes that we should have a static IP and it also has assigned the pi-hole to use DNS servers that are the same DNS servers that I specified as my custom entries for upstream resolution.<\/p>\n<p>This feels like the pi-hole install script intentionally makes the pi-hole host machine not able to do local DNS resolutions. I wonder if all pi-hole installations have this challenge. <a href=\"https:\/\/discourse.pi-hole.net\/t\/pi-hole-install-script-and-dhcpcd-conf\/48047\/11\">I asked on the forum,<\/a> &#8211; and after a long discussion I came to agree that I fell into a trap.<\/p>\n<p>A properly configured linux machine will have the name from <code>\/etc\/hostname<\/code> also reflected in <code>\/etc\/hosts<\/code> (as I have done). My setup was working fine because the DNS service on my <a href=\"https:\/\/openwrt.org\/\">OpenWRT<\/a> router was covering up this gap. While it&#8217;s nice that it just worked, the machine should really be able to resolve its own name without relying on DNS.<\/p>\n<p>As my network has IPv6 as well, the pi-hole also gets an IPv6 DNS entry. Since the static configuration doesn&#8217;t specify an upstream IPv6 DNS &#8211; the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Unique_local_address\">ULA<\/a> for the pi-hole leaks into the configuration. This is a\u00a0 potentially scary DNS recursion problem, but it doesn&#8217;t seem to be getting used to forward the lookup of &#8216;myhost&#8217; to the pi-hole so I&#8217;m going to ignore this for now.<\/p>\n<p>And easy fix is to go modify <code>\/etc\/hosts<\/code><\/p>\n<pre class=\"lang:default decode:true \">127.0.0.1 localhost myhost\r\n\r\n# The following lines are desirable for IPv6 capable hosts\r\n::1 ip6-localhost ip6-loopback\r\nfe00::0 ip6-localnet\r\nff00::0 ip6-mcastprefix\r\nff02::1 ip6-allnodes\r\nff02::2 ip6-allrouters\r\nff02::3 ip6-allhosts<\/pre>\n<p>Where <code>myhost<\/code> is whatever appears in <code>\/etc\/hostname<\/code>.<\/p>\n<p>This will fix the first problem, and it seems the second problem was also caused by the same unable to resolve myhost. I think this is exactly what <a href=\"https:\/\/discourse.pi-hole.net\/t\/security-information-for-my-server-problem-with-defaults-entries\/41005\">this post<\/a> was describing when it was seeing the error messages. This is <a href=\"https:\/\/discourse.pi-hole.net\/t\/pi-holes-host-machine-cant-resolve-local-domains\/48008\">another thread<\/a> and different solution to a similar problem.<\/p>\n<p>Satisfied I&#8217;d solved that mystery, along the way I&#8217;d also created a list of things to &#8216;fix&#8217;<\/p>\n<ol>\n<li>Ditch rsyslog remote<\/li>\n<li>Run logwatch locally<\/li>\n<li>Run fluentbit locally<\/li>\n<li>Password-less sudo<\/li>\n<li>SSH key only access<\/li>\n<li>Add mosh<\/li>\n<\/ol>\n<p>The rest of this post is just details on these 6 changes.<\/p>\n<p><!--more--><\/p>\n<p><strong>1. Ditch rsyslog remote<\/strong><\/p>\n<p>I setup rsyslog because I wanted to get logwatch to run over my logs and avoid yet another daily automated email arriving. Unfortunately logwatch combined with rsyslog wasn&#8217;t giving me any separation of what was happening where. On one hand, seeing all of the sudo actions in one place is useful, but not knowing where they happened was a problem.<\/p>\n<p>Rsyslog appears to only be forwarding logs sent through the system (auth, kern, mail, syslog) which is useful &#8211; but all of the other log files such as <code>\/var\/log\/pihole.log<\/code> were not being transported.<\/p>\n<p>Turning this off is as easy as removing the configuration file I had added to <code>\/etc\/rsyslog.d<\/code><\/p>\n<p class=\"p1\"><strong>2. Run logwatch locally<\/strong><\/p>\n<p>Just install it.<\/p>\n<pre class=\"lang:default decode:true \">sudo apt install logwatch<\/pre>\n<p>Now we&#8217;ll get nightly reports in email via nullmailer.<\/p>\n<p><strong>3. Run fluent-bit locally<\/strong><\/p>\n<p>While fluent-bit does document <a href=\"https:\/\/docs.fluentbit.io\/manual\/installation\/linux\/raspbian-raspberry-pi\">installing the agent on the Raspberry Pi<\/a>, these instructions are for the Raspbian distribution only.<\/p>\n<p>I&#8217;m using Ubuntu, so need to follow the <a href=\"https:\/\/docs.fluentbit.io\/manual\/installation\/linux\/ubuntu\">instructions for Ubuntu<\/a>. Thankfully there is an arm architecture build of the agent.<\/p>\n<pre class=\"lang:default decode:true \"># Add the key\r\ncurl https:\/\/packages.fluentbit.io\/fluentbit.key | sudo apt-key add -\r\n\r\n# Add the Ubuntu source list - create a file in \/etc\/apt\/sources.list.d\r\necho \"deb https:\/\/packages.fluentbit.io\/ubuntu\/focal focal main\" &gt;  \/etc\/apt\/sources.list.d\/fluent-bit.list \r\n\r\n# Update apt\r\nsudo apt update\r\n\r\n# Install td-agent-bit\r\nsudo apt install td-agent-bit<\/pre>\n<p class=\"p1\">Now that the agent is installed, we need to modify the <code>\/etc\/td-agent-bit\/td-agent-bit.conf<\/code> configuration file.<\/p>\n<p>The default file just adds some CPU metrics. We&#8217;re going to remove those and replace with the following<\/p>\n<pre class=\"lang:default decode:true \"># We leave the top part of the agent config file alone\r\n# The bit we will replace are the INPUT and OUTPUT sections\r\n\r\n[INPUT]\r\n    # tail all host logs\r\n    name tail\r\n    Path \/var\/log\/*.log,\/var\/log\/*\/*.log,\/var\/log\/syslog,\/var\/log\/lastlog,\/var\/log\/faillog\r\n    Tag host.*\r\n    Path_Key filename\r\n\r\n[OUTPUT]\r\n    Name loki\r\n    Host 192.168.1.35\r\n    Match host*\r\n    Labels host=myhost, $filename<\/pre>\n<p>The input section uses the tail plugin. This watches files and forwards as log data any new data appended to those file patterns. The Tag helps match up INPUT and OUTPUT sections, as you can have multiple of each and they don&#8217;t all have to connect. The Path_Key is used to pass along a useful label to the Loki backend, labels are a Loki concept.<\/p>\n<p>This of course assumes we have an instances of Grafana Loki running (as I do). There are plenty of other log destinations you could choose.\u00a0 Fluent-bit supports <a href=\"https:\/\/docs.fluentbit.io\/manual\/pipeline\/outputs\">many output plugins<\/a>, heck we could have even targeted <a href=\"https:\/\/docs.fluentbit.io\/manual\/pipeline\/outputs\/syslog\">rsyslog<\/a>.<\/p>\n<p>Once the config file is ready:<\/p>\n<pre class=\"lang:default decode:true \"># enable the service\r\nsudo systemctl enable td-agent-bit.service\r\n\r\n# start the agent\r\nsudo systemctl start td-agent-bit.service\r\n\r\n# check that things are all ok\r\nsudo systemctl status td-agent-bit.service<\/pre>\n<p><strong>\u00a04. Password-less sudo<\/strong><\/p>\n<p>Simply add the following to the end of the \/etc\/sudoers file<\/p>\n<pre class=\"lang:default decode:true \"># no password sudo for user myid\r\nmyid ALL=(ALL) NOPASSWD:ALL<\/pre>\n<p>The &#8216;safe&#8217; way to do this is <a href=\"https:\/\/www.sudo.ws\/man\/1.8.13\/visudo.man.html\">visudo<\/a> because it provides some checking to ensure you don&#8217;t lock yourself out.<\/p>\n<p><strong>5. SSH key only access<\/strong><\/p>\n<p>This is a two part process. We need to create a ssh key and install it to allow key based ssh. The <a href=\"https:\/\/docs.github.com\/en\/github\/authenticating-to-github\/connecting-to-github-with-ssh\/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent\">GitHub doc<\/a> on this is actually pretty good.<\/p>\n<p>Once you&#8217;ve got a key pair (private\/public) copy the contents of the public (.pub) file into the <code>.ssh\/authorized_keys<\/code> file on your pi-hole.<\/p>\n<p>The private key belongs in the .ssh\/ directory of the client machine (your laptop). Details on this will vary.<\/p>\n<p>It&#8217;s a good idea to confirm that the key is working &#8211; plenty of ways to do this, but <code>ssh -v<\/code> is a good way to check.<\/p>\n<p>Now we go and change\u00a0 \/<span class=\"crayon-i\">etc<\/span>\/<span class=\"crayon-i\">ssh<\/span>\/<span class=\"crayon-i\">sshd_config to disable passwords.<\/span><\/p>\n<div id=\"crayon-60e1dff64d202585840213-4\" class=\"crayon-line crayon-striped-line\">\n<pre class=\"lang:default decode:true \"># Change to no to disable tunnelled clear text passwords\r\nPasswordAuthentication no<\/pre>\n<\/div>\n<div>Also don&#8217;t forget to restart sshd so the changes take affect<\/div>\n<div><\/div>\n<div>\n<pre class=\"lang:default decode:true \">sudo systemctl restart sshd.service<\/pre>\n<p>To test this, temporarily move your key such that ssh can&#8217;t find it &#8211; you should get rejected. Attempts to connect without the key will show<\/p>\n<pre class=\"lang:default decode:true \">ssh user@myhost\r\nuser@myhost: Permission denied (publickey).<\/pre>\n<\/div>\n<p><strong>6. Add mosh<\/strong><\/p>\n<p><a href=\"https:\/\/mosh.org\/\">Mosh<\/a> is a great enhancement to your ssh experience. It provides a persistent shell connection even with intermittent networks. What this means is that when you close your laptop and it goes to sleep, you don&#8217;t have to reconnect that ssh session.<\/p>\n<pre class=\"lang:default decode:true \"># Install mosh - you will not regret this\r\nsudo apt install mosh<\/pre>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>My past two posts have been the hardware setup and pi-hole deployment. You rarely get things right the first time and this is no exception. I&#8217;ve already added an update to the first post about my nullmailer configuration. I had forgotten to modify \/etc\/mailname and thus a lot of the automated email was failing. I &hellip; <a href=\"https:\/\/lowtek.ca\/roo\/2021\/pi-hole-ubuntu-server-take-2\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Pi-hole Ubuntu Server (take 2)&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,12,20],"tags":[],"class_list":["post-1839","post","type-post","status-publish","format-standard","hentry","category-computing","category-how-to","category-pi-hole"],"_links":{"self":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/1839","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/comments?post=1839"}],"version-history":[{"count":6,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/1839\/revisions"}],"predecessor-version":[{"id":1847,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/1839\/revisions\/1847"}],"wp:attachment":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/media?parent=1839"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/categories?post=1839"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/tags?post=1839"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}