ZFS

zpool — configure ZFS storage pools

The zpool command configures ZFS storage pools. A storage pool is a collection of devices that provides physical storage and data replication for ZFS datasets. All datasets within a storage pool share the same space.

Virtual Devices (vdevs): A “virtual device” describes a single device or a collection of devices organized according to certain performance and fault characteristics.

zpool-status — Display detailed health status for the given ZFS storage pools

zpool status

(also shows if a scrub is in progress)

zpool-list — Lists ZFS storage pools along with a health status and space usage

zpool list
zpool list -v

zpool-create — Creates a new ZFS storage pool
zpool-destroy — Destroys the given ZFS storage pool, freeing up any devices for other use

zpool create SAN200 /dev/mapper/mpathl /dev/mapper/mpathk /dev/mapper/mpathm /dev/mapper/mpathn
zpool create -o ashift=12 storage raidz1 /dev/sda /dev/sdb
zpool create -o ashift=12 wdExternPassport /dev/mapper/luks-5295ac32-4ce8-4ce3-9d54-c60d28522031
zpool destroy SAN200

zpool-import — Lists ZFS storage pools available to import or import the specified pools

zpool-export — Exports the given ZFS storage pools from the system

Rename a zpool:

zpool export [poolname]
zpool import [poolname] [newpoolname]

zpool-get — Retrieves properties for the specified ZFS storage pool(s)

zpool-set— Sets the given property on the specified pool

https://openzfs.github.io/openzfs-docs/man/5/zpool-features.5.html

zpool get all tank-s-p-w
zpool get size tank
zpool get feature@large_dnode tank

zpool-scrub — begin or resume scrub of ZFS storage pools

zpool scrub [-s|-p] [-w] pool…
-sStop scrubbing.
-pPause scrubbing. Scrub pause state and progress are periodically synced to disk. If the system is restarted or pool is exported during a paused scrub, even after import, scrub will remain paused until it is resumed. Once resumed the scrub will pick up from the place where it was last checkpointed to disk. To resume a paused scrub issue zpool scrub again.
-wWait until scrub has completed before returning.

zpoolprops — properties of ZFS storage pools

ashift=ashiftPool sector size exponent, to the power of 2 (internally referred to as ashift). I/O operations will be aligned to the specified size boundaries. The typical case for setting this property is when performance is important and the underlying disks use 4KiB sectors but report 512B sectors to the OS (for compatibility reasons); in that case, set ashift=12 (which is 1<<12 = 4096).
autoexpand=on|offControls automatic pool expansion when the underlying LUN is grown.

https://openzfs.github.io/openzfs-docs/man/7/zpoolprops.7.html?highlight=zpoolprops

zfs — configures ZFS file systems

zfsprops — native and user-defined properties of ZFS datasets

https://openzfs.github.io/openzfs-docs/man/7/zfsprops.7.html

acltype=off|nfsv4|posixControls whether ACLs are enabled and if so what type of ACL to use.
atime=on|offControls whether the access time for files is updated when they are read.
compression=on|off|gzip|gzip-N|lz4|lzjb|zle|zstd|zstd-N|zstd-fast|zstd-fast-NControls the compression algorithm used for this dataset. When set to on (the default), indicates that the current default compression algorithm should be used.
dnodesize=legacy|auto|1k|2k|4k|8k|16kSpecifies a compatibility mode or literal value for the size of dnodes in the file system. The default value is legacy. Setting this property to a value other than legacy requires the large_dnode pool feature to be enabled.
recordsize=sizeSpecifies a suggested block size for files in the file system. This property is designed solely for use with database workloads that access files in fixed-size records. ZFS automatically tunes block sizes according to internal algorithms optimized for typical access patterns.
relatime=on|offControls the manner in which the access time is updated when atime=on is set.
sharesmb=on|off|optsControls whether the file system is shared by using Samba USERSHARES and what options are to be used.
sharenfs=on|off|optsControls whether the file system is shared via NFS, and what options are to be used. A file system with a sharenfs property of off is managed with the exportfs(8) command and entries in the /etc/exports file.
xattr=on|off|saControls whether extended attributes are enabled for this file system.
encryption=off|on|aes-128-ccm|aes-192-ccm|aes-256-ccm|aes-128-gcm|aes-192-gcm|aes-256-gcmControls the encryption cipher suite (block cipher, key length, and mode) used for this dataset.
keyformat=raw|hex|passphraseControls what format the user’s encryption key will be provided as. This property is only set when the dataset is encrypted.
keylocation=prompt|file://</absolute/file/path>|https://<address> |http://<address>Controls where the user’s encryption key will be loaded from by default for commands such as zfs load-key and zfs mount -l. This property is only set for encrypted datasets which are encryption roots. If unspecified, the default is prompt.
keylocation=prompt|file://</absolute/file/path>|https://<address> |http://<address>

zfs list

-HUsed for scripting mode. Do not print headers and separate fields by a single tab instead of arbitrary white space.
-o propertyA comma-separated list of properties to display.
-rRecursively display any children of the dataset on the command line.
-s propertyA property for sorting the output by column in ascending order based on the value of the property.
-t typeA comma-separated list of types to display, where type is one of filesystem, snapshot, volume, bookmark, or all.
zfs list
zfs list -t volume
zfs list -t snapshot
zfs list -t snapshot tank/dataset
zfs list -r -t snapshot tank/dataset
zfs list -H -r -t snapshot -o name

zfs clone

zfs clone [-p] [-o property=value]… snapshot filesystem|volume

Creates a clone of the given snapshot. See the Clones section for details. The target dataset can be located anywhere in the ZFS hierarchy, and is created as the same type as the original.

zfs clone tank/dataset@morning tank/dataset_morning

zfs create/destroy

zfs create -p tank/backup/test

-p Creates all the non-existing parent datasets.

zfs destroy -r SAN200/backup

-r Destroy (or mark for deferred deletion) all snapshots with this name in descendent file systems.

Create a ZVOL

-sCreates a sparse volume with no reservation. See volsize in the Native Properties section for more information about sparse volumes.
-Vcreates a ZVOL
zfs create -s -V 4GB tank/vol

zfs get/set

zfs get compression
zfs set compression=lz4 SAN200
zfs set compression=on pool/home/anne
zfs set compression=lz4 pool/home/anne
zfs get all pool/home/bob
zfs get xattr,compression,atime,recordsize,acltype,dnodesize, localZFS
zfs get compression,xattr,acltype,dnodesize,atime tank/smbtest
zfs set compression=lz4 xattr=sa acltype=posixacl dnodesize=auto atime=on tank/smbtest
zfs get xattr,compression,atime,relatime,recordsize,acltype,dnodesize, localZFS
zfs set xattr=sa compression=lz4 atime=off relatime=on recordsize=1M acltype=posixacl dnodesize=auto localZFS
zfs get mountpoint storage/music
zfs set mountpoint=/home/bob/Music storage/music

sharenfs

zfs get sharenfs backup/proxmox/vm
zfs set sharenfs="rw=@192.168.0.0/24,rw=@10.0.0.0/24" pool-name/dataset-name

https://blog.programster.org/sharing-zfs-datasets-via-nfs

sharesmb

zfs get sharesmb tank/smbtest

zfs rename

-fForce unmount any filesystems that need to be unmounted in the process.
-pCreates all the nonexistent parent datasets.
-r (snapshots)Recursively rename the snapshots of all descendant datasets.
zfs rename -p tank/projects tank/data/projects

zfs-snapshot — create snapshots of ZFS datasets

All previous modifications by successful system calls to the file system are part of the snapshots.

zfs snapshot [-r] [-o property=value]… dataset@snapname…
-o property=valueSet the specified property; see zfs create for details.
-rRecursively create snapshots of all descendent datasets

Create snapshot

(must be filesystem@snapname or volume@snapname)

zfs snapshot tank/backup/projects@version1
zfs destroy tank/backup@zfs-auto-snap_frequent-2021-04-23-1900
zfs list -t snapshot -o name | grep ^BACKUP/projects@zfs-auto-snap_frequent | xargs -n 1 zfs destroy
zfs list -r -t snapshot tank

Look into the snapshots

The info is available at the root of the dataset in .zfs/snapshots

zfs-rollback — roll ZFS dataset back to snapshot

-RDestroy any more recent snapshots and bookmarks, as well as any clones of those snapshots.
-fUsed with the -R option to force an unmount of any clone file systems that are to be destroyed.
-rDestroy any snapshots and bookmarks more recent than the one specified.

Remove all modifications since snapshot

zfs rollback tank/backup/test@zfs-auto-snap_removeme-2021-04-30-1637
zfs rollback -r tank/backup/test@zfs-auto-snap_removeme-2021-04-30-1637

zfs send/receive

zfs-send — generate backup stream of ZFS dataset

-I snapshotGenerate a stream package that sends all intermediary snapshots from the first snapshot to the second snapshot.
-R, --replicateGenerate a replication stream package, which will replicate the specified file system, and all descendent file systems, up to the named snapshot.
-v, --verbosePrint verbose information about the stream package generated. This information includes a per-second report of how much data has been sent.
-L, --large-blockGenerate a stream which may contain blocks larger than 128KB.

zfs-receive — create snapshot from backup stream

-FForce a rollback of the file system to the most recent snapshot before performing the receive operation. If receiving an incremental replication stream (for example, one generated by zfs send -R [-i|-I]), destroy snapshots and file systems that do not exist on the sending side.
-o property=valueSets the specified property as if the command zfs set property=value was invoked immediately before the receive.
-vPrint verbose information about the stream and the time required to perform the receive operation.

zfs send [-DLPRbcehnpvw] [[-I|-i] snapshot] snapshot
zfs send [-LPcenvw] [-i snapshot|bookmark] filesystem|volume|snapshot

zfs receive [-Fhnsuv] [-o origin=snapshot] [-o property=value] [-x property] filesystem|volume|snapshot
zfs receive [-Fhnsuv] [-d|-e] [-o origin=snapshot] [-o property=value] [-x property] filesystem

first time:

zfs send tank/backup/projects@zfs-auto-snap_daily-2021-04-24-0425 | pv -rtab | zfs recv SAN200/backup/projects

after that:

zfs send -R -I @zfs-auto-snap_daily-2021-04-24-0425 tank/backup/projects@zfs-auto-snap_daily-2021-04-25-0425 | pv -rtab | zfs receive -F SAN200/backup/projects
Send snapshot between machines
  • dest is a dataset and must exist
  • port 8000 is arbitrary but must be the same on both sides
nc -l 8000 | mbuffer -q -m 1G | pv -rtab | zfs receive -vF dest
  • @base snapshot must exist on both incoming and destination datasets
  • inc is a dataset and @transfer is the snapshot we copy to the dest
  • dest_comp_ip is the IP of the receiving computer waiting for transfer
zfs send -R -I @base inc@transfer | mbuffer -q -m 1G | pv -b | nc dest_comp_ip 8000

src: https://www.polyomica.com/improving-transfer-speeds-for-zfs-sendreceive-in-a-local-network/

https://ithero.eu/documentation/networking/netcat/

Encryption

encryption=off|on|aes-128-ccm|aes-192-ccm|aes-256-ccm|aes-128-gcm|aes-192-gcm|aes-256-gcmControls the encryption cipher suite (block cipher, key length, and mode) used for this dataset.
keyformat=raw|hex|passphraseControls what format the user’s encryption key will be provided as. This property is only set when the dataset is encrypted.
keylocation=prompt|file://</absolute/file/path>|https://<address> |http://<address>Controls where the user’s encryption key will be loaded from by default for commands such as zfs load-key and zfs mount -l. This property is only set for encrypted datasets which are encryption roots. If unspecified, the default is prompt.
zfs create -o encryption=on -o keyformat=passphrase mydataset
Enter new passphrase:
Re-enter new passphrase:
zfs get encryption,keyformat,keylocation mydataset
NAME       PROPERTY     VALUE        SOURCE
mydataset  encryption   aes-256-gcm  -
mydataset  keyformat    passphrase   -
mydataset  keylocation  prompt       local
/usr/sbin/syncoid --identifier=usbdrive --recvoptions="x encryption" mydataset usbdrive/mydataset
zfs load-key mydataset
zfs mount -a
zfs unmount mydataset
zfs unload-key mydataset
zpool export myzpool

zfs share/unshare

zfs share -a | filesystem
zfs unshare -a | filesystem|mountpoint

zfs hold/holds

cannot destroy snapshot [snapshot] : dataset is busy

z$ zfs holds [snapshot]
NAME        TAG              TIMESTAMP
[snapshot]  .send-2964719-1  Mon Mar 14 15:17 2022

zfs share -a | filesystem

zfs unshare -a | filesystem|mountpoint

zdb — display zpool debugging and consistency information

-l device : Read the vdev labels from the specified device (obtained with zpool list -v)

# zdb -l mpathp

-C : Display information about the configuration

# zdb -C tank

-d : Display information about datasets

# zdb -d tank/backup

zfs-auto-snapshot – take regular ZFS snapshots

/sbin/zfs-auto-snapshot -q -g --label=hourly --keep=24 tank/backup/projects

Helper script to run after zfs-auto-snapshot (note that the –keep switch takes affect after the –post-snapshot):

#!/bin/sh

#echo "dataset = $1"
#echo "snapname = $2"

latest_synced_snapname=`zfs list -H -o name -S creation -t snapshot SAN200/test  | head -1 | sed 's/.*\(@.*\)/\1/'`

zfs send -R -I "$latest_synced_snapname" "$1@$2" | zfs receive -F SAN200/test

Documentation

https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/index.html

https://openzfs.github.io/openzfs-docs/man/8/index.html

https://wiki.archlinux.org/title/ZFS

Installation

sudo apt install zfsutils-linux

Misc

zfs tuning: https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/

advanced format drives: https://wiki.lustre.org/Optimizing_Performance_of_SSDs_and_Advanced_Format_Drives

official FAQ: https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html

Record size: https://blog.programster.org/zfs-record-size

Memory used: https://superuser.com/questions/1137416/how-can-i-determine-the-current-size-of-the-arc-in-zfs-and-how-does-the-arc-rel

Root on ZFS: https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS.html

Cheat sheet: https://www.thegeekdiary.com/solaris-zfs-command-line-reference-cheat-sheet/

System administration: https://openzfs.org/wiki/System_Administration

Random

$ more /etc/cron.hourly/zfs-auto-snapshot
#!/bin/sh

# Only call zfs-auto-snapshot if it's available
which zfs-auto-snapshot > /dev/null || exit 0

exec zfs-auto-snapshot --quiet --syslog --label=hourly --keep=24 --recursive --post-snapshot=/home/technician/zfs/myscript SAN300/projects
#exec zfs-auto-snapshot --quiet --syslog --label=hourly --keep=24 --recursive SAN300/projects
$ more /home/technician/zfs/myscript
#!/bin/sh

#echo "dataset = $1"
#echo "snapname = $2"

zfs create -p BACKUP/$1

latest_synced_snapname=`zfs list -H -o name -S creation -t snapshot BACKUP/$1  | head -1 | sed 's/.*\(@.*\)/\1/'`

if [ -z "$latest_synced_snapname" ]
then
        zfs send -R "$1@$2" | zfs receive -F BACKUP/$1
else
        zfs send -R -I "$latest_synced_snapname" "$1@$2" | zfs receive -F BACKUP/$1
fi

#zfs list -t snapshot -o name | grep ^BACKUP/$1@zfs-auto-snap_frequent | xargs -n 1 zfs destroy

return 0