ZFS Metadata Special Device: Z - L1 Articles & Video-related - Level1Techs Forums
Advice on the special VDEV in my ZFS setup
How do I make a metadata special device vdev?
What does the ZFS Metadata Special Device do?
What metadata actually is in ZFS
Who needs one - workload patterns that benefit
I am planning out my ZFS setup as I'm moving from snapraid. I bought a few sticks of 118gb optane to play around with and am considering using them or some high end SSD's mirrored in a special vdev. I'm considering using some 2tb sn850x's instead of optane to be able to store small blocks on the special vdev. I store mostly video and photos in my server and plan on having a 5x20tb raidz2 and a 3x8tb mirror vdev in my pool. I have 64gb of non ecc ram and my server is on a 1gbit nic. The performance improvement I want to see is faster loading of my folders as it currently takes 10-20 seconds to load the file structure and thumbnails in the worse case. Would a special vdev suit my needs or would arc and l2arc be fine enough for my needs? I would appreciate any advice on my setup.
I have 100 TB raidz1 pool that I am about to create, and I want to store the metadata for it on mirrored SSDs.
How do I create the metadata storage for only the 100TB pool (not the OS)?
zpool create -f -o ashift=12 -m /media storage \
-o recordsize=1M \
-o primarycache=metadata -o secondarycache=none \
raidz \
ata-ST3000DM001-9YN166_HWID \
ata-ST3000DM001-9YN166_HWID \
ata-ST3000DM001-9YN166_HWID \
ata-ST3000DM001-9YN166_HWIDzpool add storage -o ashift=12 special mirror /dev/ssd0n1 /dev/ssd1n1
zfs set special_small_blocks=128K storage
In order to view the current value of a specific setting like ashift, you will need to use the zdb command instead of the zpool command.
Running zdb on its own with no arguments will give you a view of any pools found on the system, and their vdevs, and disks within the vdevs.
root@pve1:/home/tim# zdb
pm1:
version: 5000
name: 'pm1'
state: 0
txg: 801772
pool_guid: 13783858310243843123
errata: 0
hostid: 2831164162
hostname: 'pve1'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 13783858310243843123
children[0]:
type: 'raidz'
id: 0
guid: 13677153442601001142
nparity: 2
metaslab_array: 34
metaslab_shift: 33
ashift: 9
asize: 1600296845312
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 4356695485691064080
path: '/dev/disk/by-id/ata-DENRSTE251M45-0400.C_A181B011241000542-part1'
whole_disk: 1
not_present: 1
DTL: 64
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 14648277375932894482
path: '/dev/disk/by-id/ata-DENRSTE251M45-0400.C_A181B011241000521-part1'
whole_disk: 1
DTL: 82
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 11362800770521042303
path: '/dev/disk/by-id/ata-DENRSTE251M45-0400.C_A181B011241000080-part1'
whole_disk: 1
DTL: 59
create_txg: 4
children[3]:
type: 'disk'
id: 3
guid: 10494331395233532833
path: '/dev/disk/by-id/ata-DENRSTE251M45-0400.C_A181B011241000517-part1'
whole_disk: 1
DTL: 58
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
or, for just ashift with some context:
root@pve1:/home/tim# sudo zdb | egrep 'ashift|vdev|type' | grep -v disk
vdev_children: 1
vdev_tree:
type: 'root'
type: 'raidz'
ashift: 9
Here is an old blog post about zdb that is still very informative about the origins and intent, and the information that comes out of zdb. A quick google also reveals many posts that may be more specifically relevant to ZFS on Linux.
As of today (ZFS 2.2.4), it is possible to get and set vdev properties:
zpool get <property> <pool> <vdev>
The keyword all-vdevs can be used to specify all vdevs. For example, to get ashift for all vdevs:
zpool get ashift rpool all-vdevs
Racking my brain trying to figure out it's actual behavior. No documentation anywhere I can find actually comments on this.
The special vdev is used to store metadata from the pool, so operations like directory listing are at the speed of an NVME drive, which will also improve the performance of spinning rust by reducing the load of small IO required to lookup the pool metadata. This makes sense.
What confuses me is the behavior of the small block allocation class when the special vdev is at capacity. There seems to be a few scenarios that aren't talked about anywhere in the documentation that would be important for performance in various scenarios.
From how i've seen it talked about my understanding is that it acts as a write back cache for small IO based on the special_small_blocks value of the dataset, and when the special vdev is at capacity or needs more space for metadata, small block allocation is offloaded back to the data vdevs.
However i've never actually seen this mentioned anywhere. From the closest thing to a confirmation is in the TrueNas documentation on a fusion pool saying
If the special class becomes full, then allocations spill back into the normal class.
By spill, does that mean, removed from the special class so new incoming small block writes are added to the cache. Or does that mean when the special class is full small IO is sent directly back to the normal class bypassing the special vdev.