Zfs configurations: Storage Pool (FreeBSD)

Documentation

create/grow mirrored pool

https://www.freebsd.org/doc/handbook/zfs-zpool.html

Tutorials

growing zpool	https://tomasz.korwel.net/2014/01/03/growing-zfs-pool/
replacing HDD	http://ryan.himmelwright.net/post/replace-zfs-mirror-drive/

Creating & Naming Partitions

Creating Partitions

You should create partitions that are slightly smaller than the total disk-size. (by a few megabytes). This way you can account for small differences in avail bytes on future replacement drives.

gpart show

# create GPT 
gpart destroy -F ada3
gpart create -s gpt ada3

# [6T drives]
    # ada1 WD-WXN1H842YH9K
    # ada2 WD-WXB1HB4L2K9C
    gpart add -s 5570G -a 4k -t freebsd-zfs ada1

# [8T drives]
    # ada1p1
    # ada2p1
    gpart add -s 7300G -a 4096 -t freebsd-zfs ada1

# [10T drives]
    # ada3p1 VCG7A2NN
    # ada4p1 VCG7J54N
    gpart add -s 9200G -a 4k -t freebsd-zfs ada3

Disk Labels

As your storage array grows in size, it is handy to have something more informative than randomly assigned da1, da2, da3, ... for your disks. Create a naming/numbering scheme based on the harddrive location in the box.

Manually change the SATA connections, and reboot machine until the Disk-Order is ascending (/dev/da0 at the top). You can then label your disks with:

NOTE:

This type of disk labels does not persist. Look up permanent disk labels instead.

camcontrol identify /dev/ada2 | grep -i serial          ## displays same serial num as printed on physical disk

glabel create   s0d0-2-4HAx0H    /dev/ada0p2            ## create label (shelf0, disk0, partition2, last 6x digits of hdd serial (printed on disk))
glabel destroy /dev/label/s0d0-2-4HAx0H                 ## destroy label

Mirrors and VDEVs

A mirror is a set of 2x, or 3x disks that all store the same contents, and each disk can replicate the other if it fails. a VDEV is a raw device (file,disk,partition, etc).

# create a mirror from regular zfs partition
# (take not of your disk-names, you'll need them)
zpool status

#            <pool-name>     <existing vdev>               <drive or partition to add>
zpool attach   zroot     diskid/DISK-WD-WX61DA4HAX0Hp3    /dev/label/s0d1-1-4BLJ4L

Creating a new mirror, and adding it to an existing pool

zpool add  zroot  mirror    <disk_1>  <disk_2>

Replacing HDDs

Configure zpool

# zpool should consume extra space on avail partition
sudo zpool set autoexpand=on zroot

Identify HDDs

# identify oldest HDDs to replace
# (try for all HDDs, in my case `ada${N}` )
smartctl -a /dev/ada1 | grep -i 'power_on_hours'
camcontrol identify /dev/ada1 | grep 'serial number'  # get serial num (on HDD label)

Partition HDD

See above for creation instructions.

Replace HDD

# find GUID ('12345...') of prev HDD
zpool status

# swap HDDs within mirror
zpool replace ${zpool} ${guid} ada4p1

# wait for resilver
zpool status

# once everything works, remove original disk (NOTE: guid, not /dev!)
sudo zpool detach ${zpool} ${guid}