Computing – Page 17

Add drive to RAID5 on Ubuntu

Some time ago I migrated from a RAID1 setup to RAID5, this was on the minimum 3 drives. At some point this summer I spotted a good deal on a matching 1TB drive to what I had in my array and bought it. My purchase sat in my desk drawer for a month (or two) then I finally got around to installing it into the server. At least another couple of months went by until I got to adding it to my array – it turns out to be really simple and I’m kicking myself for dragging my feet.

With any hardware upgrade (specifically drives) it’s a good idea to capture what the system thinks things look like before you make any changes. For the most part Ubuntu talks about UUIDs for drives, but a couple of places (at least in my install) use the /dev/sd*# names and can trip you up when you shuffle hardware around. Capturing the drive assignments is simply a matter of:

$ sudo fdisk -l | grep ^/dev

Post hardware installation I was surprised at how much of a shuffle the /dev/sd*#‘s changed around. I was glad I had a before and after capture of the data, it also let me identify the new drive pretty easily.

Early in my notes I have “could it be this simple?” and a link to the kernel.org wiki on RAID. It turns out that yes, it really is that simple — but you do need to follow the steps carefully. I did also find an Ubuntu Forum post that was a good read for background too.

The new drive I had temporarily used on an OSX system to do some recovery work, so fdisk wasn’t very happy about working with the drive that had a GUID partition table (GPT). It turns out parted was happy to work with the volume and let me even change it back into something fdisk could work with.

I puzzled over the fact that this new drive wanted to start at 2048 instead of 63, I was initially under the incorrect assumption this had something to do with the GPT setup that I hadn’t been able to fix. Consider two basically identical volumes (old followed by new)

$ sudo fdisk -l /dev/sdb

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000

Device Boot Start End Blocks Id System /dev/sdb1 63 1953520064 976760001 83 Linux

$ sudo fdisk -l /dev/sdc

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk identifier: 0x00000000
Device Boot Start End Blocks Id System /dev/sdc1 2048 1953525134 976761543+ 83 Linux

I’ve highlighted the key differences in bold, you can see the physical sector size is 4096 vs. 512 and that is the reason for the different start position. Ok, diversion over – let’s actually follow the wiki and get this drive added to the RAID array.

Start by looking at what we have:

$ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] md_d3 : active raid5 sdf1[1] sdd1[0] sdb1[2] 1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

So, my RAID5 array is /dev/md_d3, and I know my new drive is /dev/sdc1 after my parted/fdisk adventure above.

$ sudo mdadm --add /dev/md_d3 /dev/sdc1

Now we look at mdstat again and it shows we have a spare. This is honestly what I should have at least done with the drive immediately after installing it – having a spare lets the RAID array fail over to the spare drive with no administrator intervention.

$ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] md_d3 : active raid5 sdc1[3](S) sdf1[1] sdd1[0] sdb1[2] 1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

Next we grow the array across the new device

$ sudo mdadm --grow --raid-devices=4 /dev/md_d3

You can peek at /proc/mdstat from time to time (or use the watch command) to monitor progress. This may take a while.

Once this is done, don’t forget to modify /etc/mdadm/mdadm.conf as per the wiki: “To make mdadm find your array edit /etc/mdadm.conf and correct the num-devices information of your Array”

At this point we now have our data spread across more drives, but don’t have a larger volume. We need to resize the volume to take advantage of the new space. It’s recommended you do the resize with the RAID5 volume unmounted (offline). I set about to do this and hit problems unmounting the volume: this turned out to be samba holding on to the volume, turning that service off fixed things.

Then I hit a show stopper, the resize2fs command failed:

$ sudo resize2fs -p /dev/md_d3 resize2fs 1.42 (29-Nov-2011) resize2fs: Device or resource busy while trying to open /dev/md_d3 Couldn't find valid filesystem superblock.

Huh? This is something I’ll one day sort out I suppose, but it really beats me what is going on here. You can resize RAID5 while it’s online too, it’s slower and a bit scarier, but it works.

$ sudo resize2fs /dev/md_d3 resize2fs 1.42 (29-Nov-2011) Filesystem at /dev/md_d3 is mounted on /stuff; on-line resizing required old_desc_blocks = 117, new_desc_blocks = 175 Performing an on-line resize of /dev/md_d3 to 732569952 (4k) blocks.

This was followed by a few moments of terror as I realized that I was doing this over a SSH connection – what if the connection is lost? Next time I’ll use screen, or nohup the process.

It was neat to watch the free space on the drive creep upwards. It was running at about 1Gb every 2 seconds. Once this finishes, you’re done. My RAID volume went from 1.9T to 2.8T with the new drive.

How To: Jenkins with Apache controlled authentication

For a change of pace, I was working with RHEL6 instead of Ubuntu and setting up a Jenkins CI server. I’ve used Jenkins aka Hudson previously, but this was my first time setting it up.

A lot of this is straight from the Jenkins wiki, which is detailed and helpful but at times cryptic. The default access mode of Jenkins is pretty much wide open, this is very handy for getting things done – but probably not what you want if there are a mix of people on the network, many whom you really don’t want to let do stuff like launch/configure your builds. As I had Apache already running and setting up authentication with Apache is relatively straight-forward, I figured the easy solution would be to hide Jenkins behind Apache.

Since Jenkins is a big wad of Java code that is offering up a web interface, we’ve effectively got two web servers running: Apache, and Jenkins (different ports). The solution we’ll use is a proxy on the Apache side and some firewall rules to prevent direct access to Jenkins, forcing people through the proxy and thus the authentication controlled by Apache.

Let’s start by checking to see if mod proxy is enabled. This is simply a matter of verifying if /etc/httpd/conf/httpd.conf has these two lines:

LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so

In my case it was enabled so no work needed. Now we modify /etc/sysconfig/jenkins, at the end of the file we need to add some args to point at the path we want our Jenkins instance to be hosted at:

JENKINS_ARGS="--prefix=/jenkins"

Restart Jenkins (service jenkins restart) to have the changes picked up. You can test to see if it’s working on localhost:8080/jenkins. I tried, and failed to have this work for a nested path (ie: /path/to/jenkins), I suspect this is a Jenkins limitation but didn’t chase down the actual reason.

Next let’s create an Apache configuration file in /etc/httpd/conf.d/jenkins_proxy.conf with the following contents:

ProxyPass /jenkins http://localhost:8080/jenkins ProxyPassReverse /jenkins http://localhost:8080/jenkins ProxyRequests Off

# Local reverse proxy authorization override # Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu) Order deny,allow Allow from all

We need to restart the web server (service httpd restart) to have these changes picked up. As this is RHEL6 and it is running with SELinux enabled, we also need to allow httpd (Apache) to do proxy connections:

# setsebool -P httpd_can_network_connect true

Now at this point you should be able to visit http://yoursite.com/jenkins, and see that our proxy configuration is working. This is cool, but people can still talk to it via http://yoursite.com:8080/jenkins which will bypass Apache.

We’ll be using iptables to accomplish this. As you can see my system was running iptables, but everything was permitted.

# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere
Chain FORWARD (policy ACCEPT) target prot opt source destination

Chain OUTPUT (policy ACCEPT) target prot opt source destination

Use ifconfig to figure out what your ethernet card is (the one where external packets will come from), in my case it was eth6. So we can simply instruct iptables to drop packets destined to 8080 from that source, allowing only internal traffic (ie: the proxy) to pass.

# iptables -A INPUT -p tcp -i eth6 --dport 8080 -j DROP

That’s it, now users are forced to come in the correct front door (Apache). To make the iptables change permanent:

#service iptables save

So now the Apache web server is seeing traffic to Jenkins and can perform authentication, this is simple enough to add to the Apache configuration file we created that defined the proxy (/etc/httpd/conf.d/jenkins_proxy.conf) – I’ll leave that one up to the reader to sort out.

Makejail – limited SSH account on Ubuntu

Jail Cell in the Rock by hadsie, on Flickr

photo by hadsie

Previous I had covered how to setup scponly as a restricted fileserver environment. While this works well, it is very limited and didn’t allow for rsync to run (without heroics beyond what I was willing to do). Using makejail seems to be a better solution for my needs, and it turns out to be quite easy to setup on Ubuntu 12.04. On the journey here I had also tried out rssh which I also decided wasn’t a good fit.

You’ll of course need sshd installed which I’ll assume you have, and makejail which we can install easily:

$ sudo apt-get install makejail

Now we need to modify our openssh configuration by editing /etc/ssh/sshd_config, there are two changes we need to make. Modify the yes setting for UsePrivilegeSeparation:

# Disable Privilege Separation to allow chroot UsePrivilegeSeparation no
and at the bottom of the configuration file we’ll add:

Match User frank ChrootDirectory /home/frank AllowTCPForwarding no X11Forwarding no PasswordAuthentication no

Of course, for each restricted user you need to specify the username and home directory. You may have noticed that for the restricted users I’ve disabled password authentication, this is because changing the password is broken in the ‘jailed’ environment so we just avoid the issue by insisting on the use of keys (yes, you’ll need the restricted user to send you their public key to install in the .ssh/authorized_keys file of the restricted user).

Next we need to create a simple python script file that we can pass to makejail as a configuration file. I called mine jailconf.py and the contents look like:

chroot = "/home/frank" testCommandsInsideJail = ["bash", "ls", "touch", "rm", "rmdir", "less", "cat", "rsync" ]

Then execute makejail with this configuration file.

$ sudo makejail jailconf.py

For some reason, I needed to run makejail twice initially before it ran without errors – but it is something you can run multiple times with no serious side effects, this is handy if you want to add more commands later.

That’s it, now if you take a peek at the filesystem structure that’s been created – it’s a chroot environment. You’ll probably want to go in and create a /home/frank/stuff directory and assign ownership to the user so they can stick files there.

$ sudo ls -l /home/frank total 36 drwxr-xr-x 2 root root 4096 Sep 19 22:59 bin drwxr-xr-x 2 root root 4096 Sep 19 22:55 dev drwxr-xr-x 3 root root 4096 Sep 19 22:56 etc drwxrwxrwx 4 frank frank 4096 Sep 19 23:28 stuff drwxr-xr-x 4 root root 4096 Sep 19 22:55 lib drwxr-xr-x 2 root root 4096 Sep 19 22:55 root drwxr-xr-x 2 root root 4096 Sep 19 22:59 sbin drwxr-xr-x 2 root root 4096 Dec 5 2009 selinux drwxr-xr-x 5 root root 4096 Sep 19 22:55 usr

Now once you sort out the public key login (and remember to make sure the permissions on the .ssh directory and authorized keys are correct), the user frank will be able to log in and see the directory tree /home/frank as if it were the root of the filesystem. Only commands listed in the configuration file (jailconf.py) will be available to that user. Of course, if the filesystem is writeable (and executable) then they could always upload copies of the commands they want to run – but hopefully these are people you trust to some level.

References: I came to this solution initially through this article. There was a serverfault post that helped with the ssh configuration changes related to disabling password authentication.

In my case this is one component in allowing a friend to use my system as a remote (encrypted) backup site using rsync. I’ll post more details on that in the future.