{"id":1620,"date":"2019-03-04T22:53:45","date_gmt":"2019-03-05T02:53:45","guid":{"rendered":"https:\/\/lowtek.ca\/roo\/?p=1620"},"modified":"2019-03-05T11:17:36","modified_gmt":"2019-03-05T15:17:36","slug":"ubuntu-16-04-to-18-04-rebuild-with-new-ssd","status":"publish","type":"post","link":"https:\/\/lowtek.ca\/roo\/2019\/ubuntu-16-04-to-18-04-rebuild-with-new-ssd\/","title":{"rendered":"Ubuntu 16.04 to 18.04 rebuild with new SSD"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-1622\" src=\"https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2019\/02\/ssd-500x160.png\" alt=\"\" width=\"500\" height=\"160\" srcset=\"https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2019\/02\/ssd-500x160.png 500w, https:\/\/lowtek.ca\/roo\/wp-content\/uploads\/2019\/02\/ssd.png 700w\" sizes=\"auto, (max-width: 500px) 85vw, 500px\" \/><\/p>\n<p>Way back in late 2016 I <a href=\"https:\/\/lowtek.ca\/roo\/2016\/server-upgrade-part-1-the-build\/\">started building a new server<\/a> to replace my aging server that runs this website, email, etc. I&#8217;m still slowly working towards actually swapping all of the services I run to the new hardware, a process slowed by the fact that I&#8217;m trying to refactor everything to be container based &#8211; and that I simply don&#8217;t seem to have a lot of free time to hack on these things. This has been complicated by the original motherboard failing to work, and my desire to upgrade the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Solid-state_drive\">SSD<\/a> to a larger one.<\/p>\n<p>Originally I had a 60GB SSD, which I got at a great price (about $60 if I recall). Today SSD prices have fallen through the floor, making it seem sensible to start building 1TB raid arrays out of (highly reliable) SSDs. A little before Christmas I picked up a <a href=\"https:\/\/www.canadacomputers.com\/product_info.php?cPath=179_1229_1088&amp;item_id=105069\">120GB Kingston SSD<\/a> for around $30. Clearly it was time to do an upgrade.<\/p>\n<p>Back in 2016, the right choice was <a href=\"https:\/\/lowtek.ca\/roo\/2016\/server-upgrade-part-2-basic-os-setup\/\">Ubuntu 16.04<\/a>, in 2019 the right version is <a href=\"http:\/\/releases.ubuntu.com\/18.04\/\">18.04 LTS<\/a>. Now while I haven&#8217;t moved all the services I need to the new server (in fact, I&#8217;ve moved very few) &#8212; there are some services which do run on the new server and my infrastructure does rely on them. This means that I want to minimize the downtime of the new server, while still achieving a full clean install.<\/p>\n<p>Taking the server offline for &#8216;hours&#8217; seemed like a bad idea, however since my desktop is a similar enough machine to the new server hardware it seemed like a good idea to use it to build the new OS on the new SSD. In fact, my bright idea was to do the install with the new SSD inside of a USB enclosure.<\/p>\n<p>Things went smoothly, download 18.04.02, create a bootable USB stick, put SSD into a drive enclosure, boot with USB stick and do an install to the external SSD. Initially I picked a single snap, the docker stuff &#8211; I later changed my mind about this and re-did the install. The shell\/curses based installer was new to me &#8211; but easy to use. The only option I picked was to pre-install OpenSSH server because of course I want one of those.<\/p>\n<p>Attempting to boot from the newly installed SSD just didn&#8217;t work. It was hanging my hardware just after POST. This was very weird. Removing the SSD from the enclosure and doing a direct SATA connection worked fine &#8211; and it proved that the install was good to go.<\/p>\n<p>For the next part, I&#8217;m using my old <a href=\"https:\/\/lowtek.ca\/roo\/2016\/server-upgrade-part-2-basic-os-setup\/\">OS setup post<\/a> as a reference.<\/p>\n<p>Either 18.04.02 is recent enough, or the install has changed &#8211; but I didn&#8217;t need to do<\/p>\n<pre class=\"lang:default decode:true\">$ sudo apt-get update\r\n$ sudo apt-get upgrade<\/pre>\n<p>since doing the above made no changes to the system as installed. Still, it doesn&#8217;t hurt to make sure you are current.<\/p>\n<p>Of course, we still want fail2ban<\/p>\n<pre class=\"lang:default decode:true\">$ sudo apt-get install fail2ban<\/pre>\n<p>I didn&#8217;t create a new SSH key, but copied in the keys from the old system. I did need to create the .ssh directory.<\/p>\n<pre class=\"lang:default decode:true\">$ mkdir .ssh\r\n$ chmod 700 .ssh<\/pre>\n<p>Once you&#8217;ve copied in the id_rsa files, don&#8217;t forget to create the authorized_keys file<\/p>\n<pre class=\"lang:default decode:true \">cp id_rsa.pub authorized_keys\r\nchmod 400 authorized_keys<\/pre>\n<p>It&#8217;s a good idea now to verify that we can access the machine via ssh. Once we know we can, let&#8217;s go make ssh more secure by blocking password based authentication. The password is still useful to have for physical (console) logins.<\/p>\n<pre class=\"lang:default decode:true \">$ vi\u00a0\/etc\/ssh\/sshd_config\r\n\r\n# Change to no to disable tunnelled clear text passwords\r\nPasswordAuthentication no<\/pre>\n<p>Let&#8217;s also setup a firewall (<a href=\"https:\/\/help.ubuntu.com\/community\/UFW\">ufw<\/a>)<\/p>\n<pre class=\"lang:default decode:true\">$ sudo ufw allow 22\r\n$ sudo ufw allow 80\r\n$ sudo ufw allow 443\r\n$ sudo ufw enable<\/pre>\n<p>Now as we will almost exclusively be using docker to host things, docker will be managing the iptables for us in parallel to ufw. So strictly speaking we don&#8217;t need 80 or 443 enabled because docker will do that when we need to. I did also discover that <a href=\"https:\/\/serverfault.com\/questions\/925281\/ufw-comment-existing-rule\">ufw has comments<\/a> you can add to each rule &#8211; making a nice way to know why you opened (or closed) a port.<\/p>\n<p>Staying on the security topic, let&#8217;s install logwatch. This requires a mail system &#8211; and as we don&#8217;t want postfix because that will interfere with the plan to put email in a docker container.. we are going to use nullmailer setup to send email to our mail server (hosted on the old server until we migrate).<\/p>\n<pre class=\"lang:default decode:true \">$ sudo apt-get install nullmailer\r\n$ sudo apt-get install logwatch<\/pre>\n<p>Don&#8217;t forget to configure logwatch to email us daily<\/p>\n<pre class=\"lang:default decode:true \">$ vi\u00a0\/etc\/cron.daily\/00logwatch\r\n\r\n# execute\r\n\/usr\/sbin\/logwatch --output mail --mailto myid@lowtek.ca --detail high<\/pre>\n<p>For <a href=\"https:\/\/en.wikipedia.org\/wiki\/Docker_(software)\">docker<\/a> we&#8217;re going to hook up to the docker.com repositories and install the CE version. To do this we need some additional support added to our base install.<\/p>\n<pre class=\"lang:default decode:true \"># things we need to make the install go smoothly\r\nsudo apt-get install \\\r\n    apt-transport-https \\\r\n    ca-certificates \\\r\n    curl \\\r\n    gnupg-agent \\\r\n    software-properties-common\r\n\r\n# add the key from docker.com\r\ncurl -fsSL https:\/\/download.docker.com\/linux\/ubuntu\/gpg | sudo apt-key add -\r\n\r\n# add the repository\r\nsudo add-apt-repository \\\r\n   \"deb [arch=amd64] https:\/\/download.docker.com\/linux\/ubuntu \\\r\n   $(lsb_release -cs) \\\r\n   stable\"\r\n\r\n# update apt\r\nsudo apt-get update<\/pre>\n<p>We can now install docker.<\/p>\n<pre class=\"lang:default decode:true\">$ sudo apt-get install docker-ce docker-ce-cli containerd.io<\/pre>\n<p>I also like to be able to run docker as &#8216;me&#8217; vs needing to sudo for every command<\/p>\n<pre class=\"lang:default decode:true \">$ sudo gpasswd -a ${USER} docker\r\n$ sudo service docker restart<\/pre>\n<p>Don&#8217;t forget that you need to logout\/login to have your shell environment pick up the permissions to do this.<\/p>\n<p>A few more annoyances we&#8217;re going to address. Needing my password to sudo is inconvenient. I&#8217;m willing to trade off some security for making this more convenient &#8211; however, be aware we are making a trade-off here.<\/p>\n<blockquote class=\"twitter-tweet\" data-width=\"550\" data-dnt=\"true\">\n<p lang=\"en\" dir=\"ltr\">TIL visudo is the safe way to edit sudoers <a href=\"https:\/\/t.co\/qop34V5zxf\">https:\/\/t.co\/qop34V5zxf<\/a> .. and yes, I&#39;m currently locked out of my linux box<\/p>\n<p>&mdash; Roo (@andrew_low) <a href=\"https:\/\/twitter.com\/andrew_low\/status\/1096887113033891840?ref_src=twsrc%5Etfw\">February 16, 2019<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>Don&#8217;t be me, either use visudo to verify that you&#8217;re modifying \/etc\/sudoers correctly. Making your user is easy, just add to the end of the file.<\/p>\n<pre class=\"lang:default decode:true\"># no password sudo for user myid\r\nmyid ALL=(ALL) NOPASSWD:ALL<\/pre>\n<p>Another minor nit is while I was asked in the install to give a machine name, the \/etc\/hosts file didn&#8217;t get the name added as expected. Just adding it (machineName) to the end of the 1st line will fix things.<\/p>\n<pre class=\"lang:default decode:true \">127.0.0.1\tlocalhost.localdomain\tlocalhost machineName<\/pre>\n<p>Things are almost perfect, but we haven&#8217;t <a href=\"https:\/\/libre-software.net\/ubuntu-automatic-updates\/\">setup automatic updates<\/a> yet.<\/p>\n<pre class=\"lang:default decode:true \">$ sudo apt install unattended-upgrades<\/pre>\n<p>Then configure the unattended upgrades<\/p>\n<pre class=\"lang:default decode:true\">$ sudo vi \/etc\/apt\/apt.conf.d\/50unattended-upgrades\r\n\r\n# Remove comments to enable updates\r\n        \"${distro_id}:${distro_codename}-updates\";\r\n\r\n# Setup email for any failures\r\nUnattended-Upgrade::Mail \"root\";\r\nUnattended-Upgrade::MailOnlyOnError \"true\";\r\n\r\n# A few more settings\r\nUnattended-Upgrade::Remove-Unused-Kernel-Packages \"true\";\r\nUnattended-Upgrade::Remove-Unused-Dependencies \"true\";\r\nUnattended-Upgrade::Automatic-Reboot \"true\";\r\nUnattended-Upgrade::Automatic-Reboot-Time \"02:00\";\r\n<\/pre>\n<p>Next is ensuring that the unattended upgrades get run, modify the file: \/etc\/apt\/apt.conf.d\/20auto-upgrades to look like below<\/p>\n<pre class=\"lang:default decode:true\">APT::Periodic::Update-Package-Lists \"1\";\r\nAPT::Periodic::Download-Upgradeable-Packages \"1\";\r\nAPT::Periodic::AutocleanInterval \"7\";\r\nAPT::Periodic::Unattended-Upgrade \"1\";\r\n<\/pre>\n<p>We can test this using a dry-run<\/p>\n<pre class=\"lang:default decode:true\">$ sudo unattended-upgrades --dry-run --debug<\/pre>\n<p>I did discover an interesting problem with docker and shutdowns. The automatic reboot will use shutdown to initiate a reboot in the middle of the night. When this happens you can cancel the shutdown, but this seems to cause docker with a &#8211;restart policy to not restart things.<\/p>\n<pre class=\"lang:default decode:true \"># issue a scheduled shutdown\r\n$ sudo shutdown +10 --reboot\r\n# containers are running fine\r\n$ docker ps\r\n# cancel the shutdown\r\n$ sudo shutdown -c\r\n# containers still happy\r\n$ docker ps\r\n# reboot\r\n$ sudo reboot\r\n# containers not automatically restarted<\/pre>\n<p>It does appear that if you allow the shutdown to happen (ie: do not cancel it), then the docker containers do restart as expected.<\/p>\n<p>I can&#8217;t really explain the shutdown and docker interaction issue. Nor the install of Ubuntu onto an external drive, and then that drive causing the machine to fail to POST properly.<\/p>\n<p><strong>Edit<\/strong><\/p>\n<p>Adding 3rd party repository to unattended-upgrade. Since I elected to pull docker from docker.com vs. the package provided via Ubuntu, I want to also have that repository be part of the automatic upgrades.<\/p>\n<p>I first found <a href=\"http:\/\/blog.ezyang.com\/2010\/03\/third-party-unattended-upgrade\/\">this post<\/a> which provides a path to figuring it out, but I found a <a href=\"https:\/\/linux-audit.com\/upgrading-external-packages-with-unattended-upgrade\/\">better description<\/a> which I used to come up with my solution.<\/p>\n<pre class=\"lang:default decode:true \">$ grep Origin \/var\/lib\/apt\/lists\/download.docker.com_linux_ubuntu_dists_bionic_InRelease \r\nOrigin: Docker\r\n$ grep Suite \/var\/lib\/apt\/lists\/download.docker.com_linux_ubuntu_dists_bionic_InRelease \r\nSuite: bionic\r\n<\/pre>\n<p>This gives us the two bits of magic we need to add to <span class=\"s1\">\/etc\/apt\/apt.conf.d\/50unattended-upgrades &#8211; so we modify the Allowed-Origins section to look like:<\/span><\/p>\n<pre class=\"lang:default decode:true\">Unattended-Upgrade::Allowed-Origins {\r\n        \"${distro_id}:${distro_codename}\";\r\n        \"${distro_id}:${distro_codename}-security\";\r\n        \/\/ Extended Security Maintenance; doesn't necessarily exist for\r\n        \/\/ every release and this system may not have it installed, but if\r\n        \/\/ available, the policy for updates is such that unattended-upgrades\r\n        \/\/ should also install from here by default.\r\n        \"${distro_id}ESM:${distro_codename}\";\r\n        \"${distro_id}:${distro_codename}-updates\";\r\n        \"Docker:bionic\";\r\n\/\/      \"${distro_id}:${distro_codename}-proposed\";\r\n\/\/      \"${distro_id}:${distro_codename}-backports\";\r\n};<\/pre>\n<p>Again, we can test that things are cool with<\/p>\n<pre class=\"lang:default decode:true \">$ sudo unattended-upgrades --dry-run --debug<\/pre>\n<p>This will ensure that docker updates get applied automatically.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Way back in late 2016 I started building a new server to replace my aging server that runs this website, email, etc. I&#8217;m still slowly working towards actually swapping all of the services I run to the new hardware, a process slowed by the fact that I&#8217;m trying to refactor everything to be container based &hellip; <a href=\"https:\/\/lowtek.ca\/roo\/2019\/ubuntu-16-04-to-18-04-rebuild-with-new-ssd\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Ubuntu 16.04 to 18.04 rebuild with new SSD&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-1620","post","type-post","status-publish","format-standard","hentry","category-computing"],"_links":{"self":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/1620","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/comments?post=1620"}],"version-history":[{"count":10,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/1620\/revisions"}],"predecessor-version":[{"id":1631,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/posts\/1620\/revisions\/1631"}],"wp:attachment":[{"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/media?parent=1620"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/categories?post=1620"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lowtek.ca\/roo\/wp-json\/wp\/v2\/tags?post=1620"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}