Category: Computing

Some time ago I migrated from a RAID1 setup to RAID5, this was on the minimum 3 drives. At some point this summer I spotted a good deal on a matching 1TB drive to what I had in my array and bought it. My purchase sat in my desk drawer for a month (or two) then I finally got around to installing it into the server. At least another couple of months went by until I got to adding it to my array – it turns out to be really simple and I’m kicking myself for dragging my feet.

With any hardware upgrade (specifically drives) it’s a good idea to capture what the system thinks things look like before you make any changes. For the most part Ubuntu talks about UUIDs for drives, but a couple of places (at least in my install) use the /dev/sd*# names and can trip you up when you shuffle hardware around. Capturing the drive assignments is simply a matter of:

$ sudo fdisk -l | grep ^/dev

Post hardware installation I was surprised at how much of a shuffle the /dev/sd*#‘s changed around. I was glad I had a before and after capture of the data, it also let me identify the new drive pretty easily.

Early in my notes I have “could it be this simple?” and a link to the kernel.org wiki on RAID. It turns out that yes, it really is that simple — but you do need to follow the steps carefully. I did also find an Ubuntu Forum post that was a good read for background too.

The new drive I had temporarily used on an OSX system to do some recovery work, so fdisk wasn’t very happy about working with the drive that had a GUID partition table (GPT). It turns out parted was happy to work with the volume and let me even change it back into something fdisk could work with.

I puzzled over the fact that this new drive wanted to start at 2048 instead of 63, I was initially under the incorrect assumption this had something to do with the GPT setup that I hadn’t been able to fix. Consider two basically identical volumes (old followed by new)

$ sudo fdisk -l /dev/sdb

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdb1 63 1953520064 976760001 83 Linux

$ sudo fdisk -l /dev/sdc

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdc1 2048 1953525134 976761543+ 83 Linux

I’ve highlighted the key differences in bold, you can see the physical sector size is 4096 vs. 512 and that is the reason for the different start position. Ok, diversion over – let’s actually follow the wiki and get this drive added to the RAID array.

Start by looking at what we have:

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md_d3 : active raid5 sdf1[1] sdd1[0] sdb1[2]
1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

So, my RAID5 array is /dev/md_d3, and I know my new drive is /dev/sdc1 after my parted/fdisk adventure above.

$ sudo mdadm --add /dev/md_d3 /dev/sdc1

Now we look at mdstat again and it shows we have a spare. This is honestly what I should have at least done with the drive immediately after installing it – having a spare lets the RAID array fail over to the spare drive with no administrator intervention.

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md_d3 : active raid5 sdc1[3](S) sdf1[1] sdd1[0] sdb1[2]
1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

Next we grow the array across the new device

$ sudo mdadm --grow --raid-devices=4 /dev/md_d3

You can peek at /proc/mdstat from time to time (or use the watch command) to monitor progress. This may take a while.

Once this is done, don’t forget to modify /etc/mdadm/mdadm.conf as per the wiki: “To make mdadm find your array edit /etc/mdadm.conf and correct the num-devices information of your Array”

At this point we now have our data spread across more drives, but don’t have a larger volume. We need to resize the volume to take advantage of the new space. It’s recommended you do the resize with the RAID5 volume unmounted (offline). I set about to do this and hit problems unmounting the volume: this turned out to be samba holding on to the volume, turning that service off fixed things.

Then I hit a show stopper, the resize2fs command failed:

$ sudo resize2fs -p /dev/md_d3
resize2fs 1.42 (29-Nov-2011)
resize2fs: Device or resource busy while trying to open /dev/md_d3
Couldn't find valid filesystem superblock.

Huh? This is something I’ll one day sort out I suppose, but it really beats me what is going on here. You can resize RAID5 while it’s online too, it’s slower and a bit scarier, but it works.

$ sudo resize2fs /dev/md_d3
resize2fs 1.42 (29-Nov-2011)
Filesystem at /dev/md_d3 is mounted on /stuff; on-line resizing required
old_desc_blocks = 117, new_desc_blocks = 175
Performing an on-line resize of /dev/md_d3 to 732569952 (4k) blocks.

This was followed by a few moments of terror as I realized that I was doing this over a SSH connection – what if the connection is lost? Next time I’ll use screen, or nohup the process.

It was neat to watch the free space on the drive creep upwards. It was running at about 1Gb every 2 seconds. Once this finishes, you’re done. My RAID volume went from 1.9T to 2.8T with the new drive.

For a change of pace, I was working with RHEL6 instead of Ubuntu and setting up a Jenkins CI server. I’ve used Jenkins aka Hudson previously, but this was my first time setting it up.

A lot of this is straight from the Jenkins wiki, which is detailed and helpful but at times cryptic. The default access mode of Jenkins is pretty much wide open, this is very handy for getting things done – but probably not what you want if there are a mix of people on the network, many whom you really don’t want to let do stuff like launch/configure your builds. As I had Apache already running and setting up authentication with Apache is relatively straight-forward, I figured the easy solution would be to hide Jenkins behind Apache.

Since Jenkins is a big wad of Java code that is offering up a web interface, we’ve effectively got two web servers running: Apache, and Jenkins (different ports). The solution we’ll use is a proxy on the Apache side and some firewall rules to prevent direct access to Jenkins, forcing people through the proxy and thus the authentication controlled by Apache.

Let’s start by checking to see if mod proxy is enabled. This is simply a matter of verifying if /etc/httpd/conf/httpd.conf has these two lines:

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so

In my case it was enabled so no work needed. Now we modify /etc/sysconfig/jenkins, at the end of the file we need to add some args to point at the path we want our Jenkins instance to be hosted at:

JENKINS_ARGS="--prefix=/jenkins"

Restart Jenkins (service jenkins restart) to have the changes picked up. You can test to see if it’s working on localhost:8080/jenkins. I tried, and failed to have this work for a nested path (ie: /path/to/jenkins), I suspect this is a Jenkins limitation but didn’t chase down the actual reason.

Next let’s create an Apache configuration file in /etc/httpd/conf.d/jenkins_proxy.conf with the following contents:

ProxyPass /jenkins http://localhost:8080/jenkins
ProxyPassReverse /jenkins http://localhost:8080/jenkins
ProxyRequests Off

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
Order deny,allow
Allow from all

We need to restart the web server (service httpd restart) to have these changes picked up. As this is RHEL6 and it is running with SELinux enabled, we also need to allow httpd (Apache) to do proxy connections:

# setsebool -P httpd_can_network_connect true

Now at this point you should be able to visit http://yoursite.com/jenkins, and see that our proxy configuration is working. This is cool, but people can still talk to it via http://yoursite.com:8080/jenkins which will bypass Apache.

We’ll be using iptables to accomplish this. As you can see my system was running iptables, but everything was permitted.

# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Use ifconfig to figure out what your ethernet card is (the one where external packets will come from), in my case it was eth6. So we can simply instruct iptables to drop packets destined to 8080 from that source, allowing only internal traffic (ie: the proxy) to pass.

# iptables -A INPUT -p tcp -i eth6 --dport 8080 -j DROP

That’s it, now users are forced to come in the correct front door (Apache). To make the iptables change permanent:

#service iptables save

So now the Apache web server is seeing traffic to Jenkins and can perform authentication, this is simple enough to add to the Apache configuration file we created that defined the proxy (/etc/httpd/conf.d/jenkins_proxy.conf) – I’ll leave that one up to the reader to sort out.