When ZFS first came out, it was a proprietary filesystem but it had some very interesting characteristics – at the time it’s ability to scale massively and protect your data seemed very cool. My interest in filesystems goes back to my C64 days editing floppy disks to create infinite directory listings and the like. Talking about filesystems reminds me of when I was a COOP student at QNX, they had ‘QFS’ and meeting the developer helped de-mystify filesystems for me.
For some reason ZFS is also linked in my memory with the ‘shouting in the datacenter’ video. As best I can tell this is likely because both DTrace and ZFS both came out of Sun around the same time.
I finally decided to fully decommission my old server and the RAID5 array of 1TB drives. I’ve also recently been experimenting with NixOS, and I’ve really enjoyed that so far. I figured why not setup a dedicated backup server? This also presented a good chance to setup and play with ZFS which now has reliable open source versions available.
First I spent some time learning what I would consider ZFS basics. This video was useful for me. Also, these two blog posts were good starting points.
Since I’m using NixOS as my base operating system, I’ll be following the doc on setting up ZFS on NixOS. Now, while I’m not setting up my boot volume to be ZFS – it turns out you still need to do the same basic setup if you want ZFS capabilities in your NixOS.
You need to generate a unique ‘hostid’ – the doc suggests using
|
1 |
head -c4 /dev/urandom <span class="p">|</span> od -A none -t x4 |
Now we need to modify the /etc/nixos/configuration.nix to include
|
1 2 3 |
boot.supportedFilesystems = [ "zfs" ]; boot.zfs.forceImportRoot = false; networking.hostId = "yourHostId"; |
Rebuild and reboot, then you can query available zpools
|
1 2 |
$ zpool status no pools available |
Now we create a pool, I think in this step we are actually adding a bunch of devices to a vdev, which is then wrapped in a pool. Using fdisk I’m able to identify the four 1TB drives which are all partitioned and ready to roll: sdd, sde, sdf, and sdg.
|
1 |
$ sudo zpool create backup raidz sdd sde sdf sdg |
This process took a short while to complete, but after it was done running sudo fdisk -l /dev/sdd gave me this:
|
1 2 3 4 5 6 7 8 9 10 11 |
Disk /dev/sdd: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors Disk model: WDC WD10EARX-00N Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 2F1E0C4F-A95F-E948-99AC-18E8829496CD Device Start End Sectors Size Type /dev/sdd1 2048 1953507327 1953505280 931.5G Solaris /usr & Apple ZFS /dev/sdd9 1953507328 1953523711 16384 8M Solaris reserved 1 |
It seems new partitions were created and I now have a zpool
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
New partitions were created and I assume initialized to be zfs $ zpool status pool: backup state: ONLINE config: NAME STATE READ WRITE CKSUM backup ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 sdg ONLINE 0 0 0 |
I don’t believe you can reasonably expand or shrink a RAIDZ vdev, this means you need to plan ahead for your storage needs. Also important to remember that the guidance is to not have ZFS volumes at more than 80% usage, beyond this level performance starts to suffer. Storage is cheap, and with pools I think you can have multiple vdev’s in a single pool, so while a single RAIDZ vdev has limitations I think ZFS offers some interesting flexibility.
Unexpectedly, it seems that the newly created ZFS is also mounted and ready to roll
|
1 2 3 |
$ zfs list NAME USED AVAIL REFER MOUNTPOINT backup 523K 2.55T 140K /backup |
That’s not where I want to mount the volume, so let’s go figure out how to move it.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# First let us view the mountpoint $ zfs get mountpoint backup NAME PROPERTY VALUE SOURCE backup mountpoint /backup default # Now we can modify that value $ sudo zfs set mountpoint=/data/raidz backup # And check to see it changed $ zfs get mountpoint backup NAME PROPERTY VALUE SOURCE backup mountpoint /data/raidz local |
Cool. I’ve got a ZFS filesytem. One snag, it isn’t mounted automatically after a reboot. I can manually mount it:
|
1 |
$ sudo zpool import -a |
And digging into the NixOS doc, we find the configuration we need to add
|
1 |
boot.zfs.extraPools = [ “backup” ]; |
This fixed me up, and ZFS is auto mounted on reboots.
One last configuration tweak, let’s enable scrubbing of the ZFS pool in our NixOS configuration
|
1 |
services.zfs.autoScrub.enable = true; |
Setting up ZFS on NixOS is very easy. Why would you want ZFS over another filesystem or storage management system? I’ve been using snapraid.it for a while on my main server, and I like the data integrity that it brings beyond just a RAID5 setup. The snapraid site has an interesting comparison matrix. I will say that setting up ZFS RAIDZ was a lot less scary than any of my adventures using mdadm to setup a software RAID5.
What do I see as the key strengths of ZFS?
- Data integrity verification and automatic repair – all files are check-summed, and with RAIDZ redundancy we can recovery from underlying data corruption.
- Pooled Storage – something I need to explore more, but I think this will give me flexibility over adding more storage over time if needed.
- Copy-on-write – this is about consistency of the filesystem, especially over power failure events.
Remember I started out with some old hardware I was repurposing? Those 1TB drives were all surprisingly in ‘good’ shape, but between 10 and 13 years of power on time (some of them have manufacture data of 2009). In my next blog post we’ll cover how ZFS handles failures as we see these ancient drives start to fail.