Zfs configurations: Zfs on Root (FreeBSD)

Documentations

https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE		ZFS On Root Instructions (GPT)
https://wiki.freebsd.org/RootOnZFS/ZFSBootPartition		ZFS on Root Instructions (MBR)

Naming Disk Partitions

As your storage array grows in size, it is handy to have something more informative than randomly assigned da1, da2, da3, ... for your disks. Create a naming/numbering scheme based on the harddrive location in the box.

Manually change the SATA connections, and reboot machine until the Disk-Order is ascending (/dev/da0 at the top). You can then label your disks with:

camcontrol identify /dev/ada2 | grep -i serial          ## displays same serial num as printed on physical disk

glabel create   s0d0-2-4HAx0H    /dev/ada0p2            ## create label (shelf0, disk0, partition2, last 6x digits of hdd serial (printed on disk))
glabel destroy /dev/label/s0d0-2-4HAx0H                 ## destroy label

Create ZPool

Okay, so right off the bat there are a few differences here. ZFS is at once a filesystem, a partition tool, and a volume manager. End-To-End ZFS can control the entire stack. While it can on a raw disk, it highly recommended to use ZFS on top of a UFS partition with an additional 100MB or so at the end of the disk left free. UFS is readable by the kernel and it will prevent most boot issues, and the additional bytes are to account for variance in hard-disk sizes (not all 6TB disks have the same amount of MB.

NOTE As of FreeBSD 10, ZFS on Root is an option from the FreeBSD install media.

## Pop in the FreeBSD USB stick, Run the standard BSDinstall
## Continue through installation (selecting keyboard, etc) until it is time
## to partition your disks. At this point choose <SHELL>

## Create 2x UFS partitions, leaving empty space at
## the end of your HDD.
gpart destroy -F da1            ## delete MBG/gpt
gpart create  -s gpt da1    ## Create new GPT
#fdisk    -I /dev/da1            ## new slice covering entire disk
#bsdlabel -w /dev/da1s1        ## new partition on slice
#newfs    -U /dev/da1s1        ## new ufs filesystem

## Create new GPT
gpart destroy -F     ada0
gpart create  -s gpt ada0


## Create Partitions
gpart add -s 222 -a 4k -t freebsd-boot -l boot0 ada0
gpart add -s 8g  -a 4k -t freebsd-swap -l swap0 ada0
gpart add        -a 4k -t freebsd-zfs  -l disk0 ada0
gpart show -l


## Set Bootcode
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

## If using 'Advanced Format' HDD (yes) create virtual devices with 4k sectors
kldload /boot/kernel/zfs.ko
sysctl vfs.zfs.min_auto_ashift=12


## Create Zpool
kldload zfs                                                                # Load ZFS kernel module
##zpool create -o altroot=/mnt -O canmount=off -m none zroot /dev/ada0p3   # Create zpool called 'zroot'
zpool create -o altroot=/mnt -O canmount=off -m none zroot /dev/gpt/disk0
zfs list                                                                   # Confirm 'zroot's creation


## Global ZFS Settings
zfs set checksum    = fletcher4    zroot  # fletcher4 checksum algorithm is supposed to be good
zfs set atime       = off          zroot  # do not write metatdata every time file is accessed. Keeps performance up.

Create FS Mountpoints

In ZFS you have a zpool, and that zpool can be broken up into managed sections. Before continuing with bsdinstall, we need to create the filesystem mountpoints so that freebsd can be installed/run.

Create MountPoints

## /, /tmp
zfs create   -o mountpoint=none                                      zroot/ROOT
zfs create   -o mountpoint=/                                         zroot/ROOT/default
zfs create   -o mountpoint=/tmp -o compression=lz4   -o setuid=off   zroot/tmp
chmod 1777 /mnt/tmp

## /usr, /usr/local
zfs create   -o mountpoint=/usr                                      zroot/usr
zfs create                                                           zroot/usr/local

## /home
zfs create   -o mountpoint=/home                   -o setuid=off   zroot/home

## /usr/ports
zfs create   -o compression=lz4                    -o setuid=off   zroot/usr/ports
zfs create   -o compression=off      -o exec=off     -o setuid=off   zroot/usr/ports/distfiles
zfs create   -o compression=off      -o exec=off     -o setuid=off   zroot/usr/ports/packages

## /usr/src
zfs create   -o compression=lz4      -o exec=off     -o setuid=off   zroot/usr/src
zfs create                                                          zroot/usr/obj

## logs
zfs create   -o mountpoint=/var                                    zroot/var
zfs create   -o compression=lz4      -o exec=off     -o setuid=off   zroot/var/crash
zfs create                           -o exec=off     -o setuid=off   zroot/var/db
zfs create   -o compression=lz4      -o exec=on     -o setuid=off   zroot/var/db/pkg
zfs create                           -o exec=off     -o setuid=off   zroot/var/empty
zfs create   -o compression=lz4      -o exec=off     -o setuid=off   zroot/var/log
zfs create   -o compression=gzip     -o exec=off     -o setuid=off   zroot/var/mail
zfs create                           -o exec=off     -o setuid=off   zroot/var/run
zfs create   -o compression=lz4      -o exec=on     -o setuid=off   zroot/var/tmp
chmod 1777 /mnt/var/tmp


##
### Type 'exit' and continue with install
##

Post Install

After Install has completed, before rebooting, choose to drop into a shell on your new system.

zpool set bootfs=zroot/ROOT/default zroot  # Choose Pool to boot from

# /tmp/bsdinstall_etc/fstab
# (Add Each Swap partition for each disk )
# Device                       Mountpoint              FStype  Options         Dump    Pass
/dev/gpt/swap0                 none                    swap    sw              0       0

mount -t devfs devfs /dev

# /etc/rc.conf
zfs_enable="YES"

# /boot/loader.conf
zfs_load="YES"

zfs set readonly=on zroot/var/empty
zpool set cachefile=/boot/zfs/zpool.cache zroot
reboot