ZFS on FreeBSD – Starting out

ZFS has been pasrt of FreeBSD since version 7.0 so as we’re now on the verge of version 9.0 being released I thought i’d better play catchup and make an effort to find out about ZFS. Firstly ZFS stands for “Zettabyte File System”, apparently “A ZFS file system can store up to 256 quadrillion zettabytes (ZB), where a zettabyte is 270 bytes” whatever that is. I got this information from Wikipedia so you can digest that with as much salt as you need. So what features do we get using ZFS

Features of ZFS (very much a summary!)

  • High Capacity – massive filesystems can be built.
  • Snapshots
  • Ability to grow a zpool
  • Redundancy
  • On-the-fly compression

Starting Point

In my office I have a HP microserver running PC-BSD, with 8GB RAM. The OS is on a single 160GB HDD which left 3 slots free. I’ve added 3 identical 2TB Wester Digital Green (Cheap) drive into it and thats the starting point. Before we kick off,  I’ll be using the instructions from the FreeBSD handbook, it seems like a good place to start. First off lets get ZFS enabled on the box.

# echo 'zfs_enable="YES"' >> /etc/rc.conf
# /etc/rc.d/zfs start

Now we’ll create a zpool with a single disk in it.

# zpool create ZTEST /dev/ada3

# df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
/dev/label/rootfs0    1.9G    477M    1.3G    26%    /
devfs                 1.0K    1.0K      0B   100%    /dev
/dev/label/var0       1.9G     50M    1.7G     3%    /var
/dev/label/usr0       656G    7.7G    596G     1%    /usr
procfs                4.0K    4.0K      0B   100%    /proc
linprocfs             4.0K    4.0K      0B   100%    /compat/linux/proc
/tmp                  1.9G    477M    1.3G    26%    /usr/jails/portjail/tmp
/media                1.9G    477M    1.3G    26%    /usr/jails/portjail/media
/usr/home             656G    7.7G    596G     1%    /usr/jails/portjail/usr/home
devfs                 1.0K    1.0K      0B   100%    /usr/jails/portjail/dev
procfs                4.0K    4.0K      0B   100%    /usr/jails/portjail/proc
/usr/src              656G    7.7G    596G     1%    /usr/jails/portjail/usr/src
linprocfs             4.0K    4.0K      0B   100%    /usr/jails/portjail/compat/linux/proc
ZTEST                 1.8T     21K    1.8T     0%    /ZTEST

See the ZTEST volume with 1.8T of space in it?


I’d like to enable compression on the whole volume, after all with a measly 1.8T available it may soon fill up!

zfs set compression=gzip ZTEST

Now i’ll download a large-ish file to my home directory (/usr/home/dan/) and check its size and then copy it into our new ZTEST.
… time passes …
Wow that took ages! it will teach me to do this on the end of a DSL line in the middle of nowhere. I downloaded the FreeBSD memstick image which is about 1GB.

# /usr/home/dan/Downloads# ls -la
total 1062838
drwxr-xr-x   2 dan   dan         512 Oct 16 19:37 .
drwxr-xr-x  23 dan   dan        1024 Oct 16 21:29 ..
-rw-r--r--   1 dan   dan          66 Oct 13 19:45 .directory
-rw-r--r--   1 root  dan  1087774720 Feb 17  2011 FreeBSD-8.2-RELEASE-amd64-memstick.img

# /usr/home/dan/Downloads# cp FreeBSD-8.2-RELEASE-amd64-memstick.img /ZTEST/
# /usr/home/dan/Downloads# cd /ZTEST/
# /ZTEST# ls -la
total 780155
drwxr-xr-x   2 root  wheel           3 Oct 16 21:36 .
drwxr-xr-x  20 root  wheel         512 Oct 16 19:28 ..
-rw-r--r--   1 root  wheel  1087774720 Oct 16 21:37 FreeBSD-8.2-RELEASE-amd64-memstick.img

Hmmmm, that looks like the file size is exactly the same! Hang on, engage brain, thats what I’d expect when I think about it without the ‘new-toy-labrador-like-enthusiasm’. The key is the total figure, which is a chunk less on the compressed drive. Whilst steadying my nerves I found the following command which gives all the settings of the zpool:

# zfs get all ZTEST
NAME   PROPERTY              VALUE                  SOURCE
ZTEST  type                  filesystem             -
ZTEST  creation              Sun Oct 16 19:28 2011  -
ZTEST  used                  762M                   -
ZTEST  available             1.78T                  -
ZTEST  referenced            762M                   -
ZTEST  compressratio         1.27x                  -
ZTEST  mounted               yes                    -
ZTEST  quota                 none                   default
ZTEST  reservation           none                   default
ZTEST  recordsize            128K                   default
ZTEST  mountpoint            /ZTEST                 default
ZTEST  sharenfs              off                    default
ZTEST  checksum              on                     default
ZTEST  compression           gzip                   local
ZTEST  atime                 on                     default
ZTEST  devices               on                     default
ZTEST  exec                  on                     default
ZTEST  setuid                on                     default
ZTEST  readonly              off                    default
ZTEST  jailed                off                    default
ZTEST  snapdir               hidden                 default
ZTEST  aclmode               groupmask              default
ZTEST  aclinherit            restricted             default
ZTEST  canmount              on                     default
ZTEST  shareiscsi            off                    default
ZTEST  xattr                 off                    temporary
ZTEST  copies                1                      default
ZTEST  version               4                      -
ZTEST  utf8only              off                    -
ZTEST  normalization         none                   -
ZTEST  casesensitivity       sensitive              -
ZTEST  vscan                 off                    default
ZTEST  nbmand                off                    default
ZTEST  sharesmb              off                    default
ZTEST  refquota              none                   default
ZTEST  refreservation        none                   default
ZTEST  primarycache          all                    default
ZTEST  secondarycache        all                    default
ZTEST  usedbysnapshots       0                      -
ZTEST  usedbydataset         762M                   -
ZTEST  usedbychildren        51K                    -
ZTEST  usedbyrefreservation  0                      -

So that confirms the type of compression used and the compression rate achieved. For the recored, we can select from the following compression options: on, off, lzjb, gzip, gzip-N. Do a ‘man zfs‘ for more details.

Mounting, Unmounting, etc

Still vaguely following the handbook, they now describe how to mount and unmount the zpools, so i’ll start with un-mounting ZTEST:

# zfs umount ZTEST
# df | grep ZTEST

You can see the volume is now un-mounted, now to put it back again.

# zfs mount ZTEST
# df | grep ZTEST
ZTEST              1915748301  780172 1914968128     0%    /ZTEST

Happy days! Now finally lets destroy the volume completely, from which there is no return.

# zpool destroy ZTEST
# df | grep ZTEST


That all worked well, but i think we need to get a RAID set going for the rest of the post. We all have servers that have rubbish RAID cards in them that will only do RAID 1 and 0? Well ZFS provides software RAID as part of the deal. Our options are:

  • Mirror (like RAID1)
  • RAIDZ or RAIDZ1 (like RAID5)
  • RAIDZ2 (like RAID6)

In addition, spares are also supported and for the real enthusiast a cache devices (like a solid state drive) can be used to enhance performance. To business, let make a RAID5 type array:

# zpool create DATA raidz1 ada1 ada2 ada3

We can see whats been created with:

# zpool get all DATA
DATA  size           5.44T       -
DATA  used           147K        -
DATA  available      5.44T       -
DATA  capacity       0%          -
DATA  altroot        -           default
DATA  health         ONLINE      -
DATA  guid           8060566178365973247  default
DATA  version        15          default
DATA  bootfs         -           default
DATA  delegation     on          default
DATA  autoreplace    off         default
DATA  cachefile      -           default
DATA  failmode       wait        default
DATA  listsnapshots  off         default

Lets put the compression back on and this time we’ll leave the default compression type:

zfs compression=on DATA

Check whats happened again with the command:

# zfs get all DATA | grep compression
DATA  compression           on                     local

There is some other interesting data here as well:

NAME  PROPERTY              VALUE                  SOURCE
DATA  type                  filesystem             -
DATA  creation              Sun Oct 16 22:27 2011  -
DATA  used                  93.9K                  -
DATA  available             3.56T                  -
DATA  referenced            28.0K                  -
DATA  compressratio         1.00x                  -
DATA  mounted               yes                    -

Just to sure, we can also keep 2 copies of all the files with the following command:

zfs set copies=2 DATA

That all seemed to go well, so I’ll copy in the test file again and see what happens.

# zfs get all DATA
NAME  PROPERTY              VALUE                  SOURCE
DATA  type                  filesystem             -
DATA  creation              Sun Oct 16 22:27 2011  -
DATA  used                  1.64G                  -
DATA  available             3.56T                  -
DATA  referenced            1.64G                  -
DATA  compressratio         1.15x                  -
DATA  mounted               yes                    -
DATA  quota                 none                   default
DATA  reservation           none                   default
DATA  recordsize            128K                   default
DATA  mountpoint            /DATA                  default
DATA  sharenfs              off                    default
DATA  checksum              on                     default
DATA  compression           on                     local
DATA  atime                 on                     default
DATA  devices               on                     default
DATA  exec                  on                     default
DATA  setuid                on                     default
DATA  readonly              off                    default
DATA  jailed                off                    default
DATA  snapdir               hidden                 default
DATA  aclmode               groupmask              default
DATA  aclinherit            restricted             default
DATA  canmount              on                     default
DATA  shareiscsi            off                    default
DATA  xattr                 off                    temporary
DATA  copies                2                      local
DATA  version               4                      -
DATA  utf8only              off                    -
DATA  normalization         none                   -
DATA  casesensitivity       sensitive              -
DATA  vscan                 off                    default
DATA  nbmand                off                    default
DATA  sharesmb              off                    default
DATA  refquota              none                   default
DATA  refreservation        none                   default
DATA  primarycache          all                    default
DATA  secondarycache        all                    default
DATA  usedbysnapshots       0                      -
DATA  usedbydataset         1.64G                  -
DATA  usedbychildren        69.9K                  -
DATA  usedbyrefreservation  0                      -

Now its taking 1.64GB to store 1GB of data? That will be the two copies kicking in, even with the compression.

This entry was posted in FreeBSD Administration and tagged , , , , . Bookmark the permalink.

Leave a Reply