ZFS has been pasrt of FreeBSD since version 7.0 so as we’re now on the verge of version 9.0 being released I thought i’d better play catchup and make an effort to find out about ZFS. Firstly ZFS stands for “Zettabyte File System”, apparently “A ZFS file system can store up to 256 quadrillion zettabytes (ZB), where a zettabyte is 270 bytes” whatever that is. I got this information from Wikipedia so you can digest that with as much salt as you need. So what features do we get using ZFS
Features of ZFS (very much a summary!)
- High Capacity – massive filesystems can be built.
- Snapshots
- Ability to grow a zpool
- Redundancy
- On-the-fly compression
Starting Point
In my office I have a HP microserver running PC-BSD, with 8GB RAM. The OS is on a single 160GB HDD which left 3 slots free. I’ve added 3 identical 2TB Wester Digital Green (Cheap) drive into it and thats the starting point. Before we kick off, I’ll be using the instructions from the FreeBSD handbook, it seems like a good place to start. First off lets get ZFS enabled on the box.
# echo 'zfs_enable="YES"' >> /etc/rc.conf # /etc/rc.d/zfs start
Now we’ll create a zpool with a single disk in it.
# zpool create ZTEST /dev/ada3 # df -h Filesystem Size Used Avail Capacity Mounted on /dev/label/rootfs0 1.9G 477M 1.3G 26% / devfs 1.0K 1.0K 0B 100% /dev /dev/label/var0 1.9G 50M 1.7G 3% /var /dev/label/usr0 656G 7.7G 596G 1% /usr procfs 4.0K 4.0K 0B 100% /proc linprocfs 4.0K 4.0K 0B 100% /compat/linux/proc /tmp 1.9G 477M 1.3G 26% /usr/jails/portjail/tmp /media 1.9G 477M 1.3G 26% /usr/jails/portjail/media /usr/home 656G 7.7G 596G 1% /usr/jails/portjail/usr/home devfs 1.0K 1.0K 0B 100% /usr/jails/portjail/dev procfs 4.0K 4.0K 0B 100% /usr/jails/portjail/proc /usr/src 656G 7.7G 596G 1% /usr/jails/portjail/usr/src linprocfs 4.0K 4.0K 0B 100% /usr/jails/portjail/compat/linux/proc ZTEST 1.8T 21K 1.8T 0% /ZTEST
See the ZTEST volume with 1.8T of space in it?
Compression
I’d like to enable compression on the whole volume, after all with a measly 1.8T available it may soon fill up!
zfs set compression=gzip ZTEST
Now i’ll download a large-ish file to my home directory (/usr/home/dan/) and check its size and then copy it into our new ZTEST.
… time passes …
Wow that took ages! it will teach me to do this on the end of a DSL line in the middle of nowhere. I downloaded the FreeBSD memstick image which is about 1GB.
# /usr/home/dan/Downloads# ls -la total 1062838 drwxr-xr-x 2 dan dan 512 Oct 16 19:37 . drwxr-xr-x 23 dan dan 1024 Oct 16 21:29 .. -rw-r--r-- 1 dan dan 66 Oct 13 19:45 .directory -rw-r--r-- 1 root dan 1087774720 Feb 17 2011 FreeBSD-8.2-RELEASE-amd64-memstick.img # /usr/home/dan/Downloads# cp FreeBSD-8.2-RELEASE-amd64-memstick.img /ZTEST/ # /usr/home/dan/Downloads# cd /ZTEST/ # /ZTEST# ls -la total 780155 drwxr-xr-x 2 root wheel 3 Oct 16 21:36 . drwxr-xr-x 20 root wheel 512 Oct 16 19:28 .. -rw-r--r-- 1 root wheel 1087774720 Oct 16 21:37 FreeBSD-8.2-RELEASE-amd64-memstick.img
Hmmmm, that looks like the file size is exactly the same! Hang on, engage brain, thats what I’d expect when I think about it without the ‘new-toy-labrador-like-enthusiasm’. The key is the total figure, which is a chunk less on the compressed drive. Whilst steadying my nerves I found the following command which gives all the settings of the zpool:
# zfs get all ZTEST NAME PROPERTY VALUE SOURCE ZTEST type filesystem - ZTEST creation Sun Oct 16 19:28 2011 - ZTEST used 762M - ZTEST available 1.78T - ZTEST referenced 762M - ZTEST compressratio 1.27x - ZTEST mounted yes - ZTEST quota none default ZTEST reservation none default ZTEST recordsize 128K default ZTEST mountpoint /ZTEST default ZTEST sharenfs off default ZTEST checksum on default ZTEST compression gzip local ZTEST atime on default ZTEST devices on default ZTEST exec on default ZTEST setuid on default ZTEST readonly off default ZTEST jailed off default ZTEST snapdir hidden default ZTEST aclmode groupmask default ZTEST aclinherit restricted default ZTEST canmount on default ZTEST shareiscsi off default ZTEST xattr off temporary ZTEST copies 1 default ZTEST version 4 - ZTEST utf8only off - ZTEST normalization none - ZTEST casesensitivity sensitive - ZTEST vscan off default ZTEST nbmand off default ZTEST sharesmb off default ZTEST refquota none default ZTEST refreservation none default ZTEST primarycache all default ZTEST secondarycache all default ZTEST usedbysnapshots 0 - ZTEST usedbydataset 762M - ZTEST usedbychildren 51K - ZTEST usedbyrefreservation 0 -
So that confirms the type of compression used and the compression rate achieved. For the recored, we can select from the following compression options: on, off, lzjb, gzip, gzip-N. Do a ‘man zfs‘ for more details.
Mounting, Unmounting, etc
Still vaguely following the handbook, they now describe how to mount and unmount the zpools, so i’ll start with un-mounting ZTEST:
# zfs umount ZTEST # df | grep ZTEST #
You can see the volume is now un-mounted, now to put it back again.
# zfs mount ZTEST # df | grep ZTEST ZTEST 1915748301 780172 1914968128 0% /ZTEST
Happy days! Now finally lets destroy the volume completely, from which there is no return.
# zpool destroy ZTEST # df | grep ZTEST
Redundancy
That all worked well, but i think we need to get a RAID set going for the rest of the post. We all have servers that have rubbish RAID cards in them that will only do RAID 1 and 0? Well ZFS provides software RAID as part of the deal. Our options are:
- Mirror (like RAID1)
- RAIDZ or RAIDZ1 (like RAID5)
- RAIDZ2 (like RAID6)
In addition, spares are also supported and for the real enthusiast a cache devices (like a solid state drive) can be used to enhance performance. To business, let make a RAID5 type array:
# zpool create DATA raidz1 ada1 ada2 ada3
We can see whats been created with:
# zpool get all DATA NAME PROPERTY VALUE SOURCE DATA size 5.44T - DATA used 147K - DATA available 5.44T - DATA capacity 0% - DATA altroot - default DATA health ONLINE - DATA guid 8060566178365973247 default DATA version 15 default DATA bootfs - default DATA delegation on default DATA autoreplace off default DATA cachefile - default DATA failmode wait default DATA listsnapshots off default
Lets put the compression back on and this time we’ll leave the default compression type:
zfs compression=on DATA
Check whats happened again with the command:
# zfs get all DATA | grep compression DATA compression on local
There is some other interesting data here as well:
NAME PROPERTY VALUE SOURCE DATA type filesystem - DATA creation Sun Oct 16 22:27 2011 - DATA used 93.9K - DATA available 3.56T - DATA referenced 28.0K - DATA compressratio 1.00x - DATA mounted yes -
Just to sure, we can also keep 2 copies of all the files with the following command:
zfs set copies=2 DATA
That all seemed to go well, so I’ll copy in the test file again and see what happens.
# zfs get all DATA NAME PROPERTY VALUE SOURCE DATA type filesystem - DATA creation Sun Oct 16 22:27 2011 - DATA used 1.64G - DATA available 3.56T - DATA referenced 1.64G - DATA compressratio 1.15x - DATA mounted yes - DATA quota none default DATA reservation none default DATA recordsize 128K default DATA mountpoint /DATA default DATA sharenfs off default DATA checksum on default DATA compression on local DATA atime on default DATA devices on default DATA exec on default DATA setuid on default DATA readonly off default DATA jailed off default DATA snapdir hidden default DATA aclmode groupmask default DATA aclinherit restricted default DATA canmount on default DATA shareiscsi off default DATA xattr off temporary DATA copies 2 local DATA version 4 - DATA utf8only off - DATA normalization none - DATA casesensitivity sensitive - DATA vscan off default DATA nbmand off default DATA sharesmb off default DATA refquota none default DATA refreservation none default DATA primarycache all default DATA secondarycache all default DATA usedbysnapshots 0 - DATA usedbydataset 1.64G - DATA usedbychildren 69.9K - DATA usedbyrefreservation 0 -
Now its taking 1.64GB to store 1GB of data? That will be the two copies kicking in, even with the compression.
