Having recently setup mirrored volumes with a pair of 1TB drives, I could now migrate data off the pair of 250Gb data drives to allow me to combine those two drives into a single volume. Way back when I purchased these drives I had intended to run a mirrored setup, but at the time decided that having more storage was more important. I had “cleverly” purchased two 250Gb drives from different manufacturers, in theory to avoid concurrent failures. It turns out that not all 250Gb drives are made the same.
Following the instructions from my previous posting, all went well up to where I tried to add the 2nd volume to the mirrored set. If you run into a similar problem you’ll likely see one of the two following errors:
mdadm: add new device failed for /dev/hda1 as 2: No space left on device
mdadm: add new device failed for /dev/hda1 as 2: Invalid argument
I found some good hints on how to diagnose the problem, it turns out you can check the partition sizes manually
$ cat /proc/partitions
major minor #blocks name
8 17 244196001 sdb1
8 65 244198552 sde1
Close, but not quite the same. As it was, I had unluckily chosen /dev/sdb as the 1st drive in the mirrored set. It turns out that fdisk tells an even more interesting story.
$ sudo fdisk -l /dev/sdb
Disk /dev/sdb: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000c0f4f
Device Boot Start End Blocks Id System
/dev/sdb1 1 30401 244196001 83 Linux
$ sudo fdisk -l /dev/sde
Disk /dev/sde: 250.0 GB, 250059350016 bytes
16 heads, 63 sectors/track, 484521 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sde1 1 484521 244198552+ 83 Linux
Yuck, looks messy. At this point I’ve got some of my live data sitting on one half of the mirrored set, and no suitable 2nd drive to act as the mirror. Somewhat predictably there is a solution that minimizes downtime and avoids copying all of the data to a new location.
First you unmount the volume and run resize2fs on it. We don’t need to know the correct size, just any size smaller than the 2nd volume – so I used 200Gb.
$ sudo umount /media/data/
$ sudo e2fsck -f /dev/md2
e2fsck 1.40.8 (13-Mar-2008)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md2: 1890279/15269888 files (0.2% non-contiguous), 32742192/61049616 blocks
$ sudo resize2fs /dev/md2 200G
resize2fs 1.40.8 (13-Mar-2008)
Resizing the filesystem on /dev/md2 to 52428800 (4k) blocks.
The filesystem on /dev/md2 is now 52428800 blocks long.
Now we need to calculate what the correct size of the mirrored partition should be. I looked at two bits of data: the size the mdadm -D reported for the partition I wanted to resize, and the size that was in /proc/partitions for the same. These differed by 88 blocks, so I used the value 88 as a fudge factor – it may not be required but it worked for me. I then also ensured that I supplied a value that was an even multiple of 64 (blocks).
So starting with 244196001 from /proc/partitions:
(244196001 - 88) / 64 = 3815561.14
Drop the decimal places and multiply by 64 to get the number of blocks.
3815561 * 64 = 244195904
Now we feed this new size into mdadm and specify the –grow flag (which can also be used to shrink if you specify a block size smaller than the current which is what we are doing in this case). We then re-run resize2fs without a specified size, which will cause it to expand the filesystem to fill the partition.
$ sudo mdadm --grow /dev/md2 --size=244195904
$ sudo resize2fs /dev/md2
resize2fs 1.40.8 (13-Mar-2008)
Resizing the filesystem on /dev/md2 to 61048976 (4k) blocks.
The filesystem on /dev/md2 is now 61048976 blocks long.
Now all that is left is to run a filesystem check, and remount it.
$sudo e2fsck -f /dev/md2
e2fsck 1.40.8 (13-Mar-2008)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md2: 1890279/15269888 files (0.2% non-contiguous), 32742192/61048976 blocks
$ sudo mount -a
Now when you attempt to add the 2nd volume, it will be a matching size and the mirror will work. In the future, I intend to be a little more careful when I plan to setup mirrored drives and pick the smaller volume as the starting point.
