There's two types of "data" in the pool: the actual data of whatever it is you're storing, and the metadata, which is all the tables, properties, indexes and other "stuff" that defines the pool structure, the datasets, and the pointers that tell ZFS where on disk to find the actual data.
Normally this is all mixed in together on the regular pool vdevs (mirror, raidz, etc). If you add a special vdev to your pool, then ZFS will prefer to store the metadata there, and send the data proper to the regular vdevs. The main reason for doing this is if you have "slow" data vdevs; adding a special vdev of a SSD mirror can speed up access times, as ZFS can look to the SSDs to know where on the data vdevs to find its data and go directly there, rather than loading the metadata off the slow vdevs and then needing another access to get the real data.
There's another possible advantage: ZFS can store "small" files on the special vdev, leaving larger ones for the regular data vdevs.
One important thing to remember is that special vdevs are a proper part of the pool, not an add-on - just like the regular data vdevs, if the special vdev fails, the pool is lost. An SSD mirror is typical for this vdev.
This is only a rough explanation, see also:
zpoolconcepts(7)- Level1Techs writeup
GitHub
openzfs.github.io › openzfs-docs › man › master › 7 › zpoolconcepts.7.html
zpoolconcepts.7 — OpenZFS documentation
But, when an active device fails, it is automatically replaced by a hot spare. To create a pool with hot spares, specify a spare vdev with any number of devices. For example, ... Spares can be shared across multiple pools, and can be added with the zpool add command and removed with the zpool ...
OpenZFS
openzfs.github.io › openzfs-docs › Basic Concepts › Pool Structure › VDEVs.html
VDEVs — OpenZFS documentation
July 22, 2026 - Administrators can zpool-replace(8) a failed leaf vdev with a new one and ZFS will automatically resilver (rebuild) the data onto it to return the pool to a healthy state. See Scrub and Resilver. Vdevs are managed using the zpool(8) command-line utility.
ZFS Metadata Special Device | Proxmox Support Forum
Hello all, I would like to configure a metadata special device for my ZFS mirror pool (2x HDD's, 18TB each) but I am not entirely sure of the steps. I've found some documentation online but I am still hesitant as I don't want to mess things up & then re-build the mirror. I do have data backed... More on forum.proxmox.com
ZFS Special VDEV | Proxmox Support Forum
Hi, On the beginning of 2024, I set up a new storage server for my work using ZFS and Samba on Proxmox. I added to the ZFS pool a "special" vdev, which gives a really good performance when the 40TB (around 50 Million files) of data are backed up, as the file metadata can be analysed very quickly... More on forum.proxmox.com
PBS and ZFS Special Allocation Class VDEV ... aka Fusion Drive | Proxmox Support Forum
I've been chewing on this for months. Getting hardware into place to test it. Won't bore you with the story. I didn't waste much money if it isn't wonderful, but I'm hoping for wonderful. Special VDEV is ... you can add a couple SSDs to a ZFS pool to speed it up. This isn't old-style hybrid... More on forum.proxmox.com
Advice on the special VDEV in my ZFS setup
If you want to be certain that your thumbnails and whatnot render faster in Explorer windows, you'll want a special with special_small_blocks set large enough to capture the thumbnails (and you'll want to double-check to see how large those thumbnails are). L2ARC might or might not do that great a job at keeping those cached. Then again, L2ARC is safe whereas the special is an additional element of risk. I'd probably start out with L2ARC (which can be safely removed later if necessary) and see if that did what I wanted it to do well enough. More on reddit.com
38
7
March 6, 2023Author: openzfs
Cr0x
cr0x.net › tuning › zfs special vdev sizing: how big it should be (so you don’t regret it)
ZFS Special VDEV Sizing: How Big It Should Be (So You Don’t Regret It) – cr0x.net
February 3, 2026 - cr0x@server:~$ zfs list -o name,used,avail,refer,mountpoint tank NAME USED AVAIL REFER MOUNTPOINT tank 120T 48T 256K /tank · Output meaning: Pool space is not the same as special vdev space. This command doesn’t show special usage directly.
GitHub
openzfs.github.io › openzfs-docs › man › v0.8 › 8 › zpool.8.html
zpool.8 — OpenZFS documentation
Cause zpool subcommands to output full vdev path names by default. This behavior is identical to the zpool status -p command line option. ... Older ZFS on Linux implementations had issues when attempting to display pool config VDEV names if a devid NVP value is present in the pool's config.
GitHub
github.com › openzfs › zfs › discussions › 17798
Special vdev for metadata and small blocks on raidz not just mirror · openzfs/zfs · Discussion #17798
If I do that, will the special vdev simply inherit/adopt the pool’s settings (like encryption)? ... zpool add <pool_name> special raidz2 <ssd_device1> <ssd_device2> …<ssd_deviceN> zfs set special_small_blocks=16M <pool_name> zfs rewrite -rvx /mnt/pool_name
Author: openzfs
Pthree
pthree.org › 2012 › 12 › 04 › zfs-administration-part-i-vdevs
Aaron Toponce : ZFS Administration, Part I- VDEVs
December 4, 2012 - VDEVs can be nested. A perfect example is a standard RAID-1+0 (commonly referred to as "RAID-10"). This is a stripe of mirrors. In order to specify the nested VDEVs, I just put them on the command line in order (emphasis mine):
Proxmox
forum.proxmox.com › home › forums › proxmox virtual environment › proxmox ve: installation and configuration
ZFS Metadata Special Device | Proxmox Support Forum
June 16, 2023 - Thanks in advance! Click to expand... It is more or less not much more than: zpool add POOLNAME special mirror /dev/sdX /dev/sdY and for the blocksize: zfs set special_small_blocks=1M POOLNAME
Proxmox
forum.proxmox.com › home › forums › proxmox virtual environment › proxmox ve: installation and configuration
ZFS Special VDEV | Proxmox Support Forum
January 1, 2025 - Hi, On the beginning of 2024, I set up a new storage server for my work using ZFS and Samba on Proxmox. I added to the ZFS pool a "special" vdev, which gives a really good performance when the 40TB (around 50 Million files) of data are backed up, as the file metadata can be analysed very quickly thanks to the special vdev.
Proxmox
forum.proxmox.com › home › forums › proxmox backup server › proxmox backup: installation and configuration
PBS and ZFS Special Allocation Class VDEV ... aka Fusion Drive | Proxmox Support Forum
June 14, 2024 - ===================================================== Add a special vdev. zpool add rpool -f -o ashift=12 special mirror scsi-<>-part3 scsi-<>-part3 scsi-<>-part3 Configure it. zfs set recordsize=1M rpool zfs set special_small_blocks=512K rpool ===================================================== Test results
Reddit
reddit.com › r/zfs › advice on the special vdev in my zfs setup
r/zfs on Reddit: Advice on the special VDEV in my ZFS setup
March 6, 2023 -
I am planning out my ZFS setup as I'm moving from snapraid. I bought a few sticks of 118gb optane to play around with and am considering using them or some high end SSD's mirrored in a special vdev. I'm considering using some 2tb sn850x's instead of optane to be able to store small blocks on the special vdev. I store mostly video and photos in my server and plan on having a 5x20tb raidz2 and a 3x8tb mirror vdev in my pool. I have 64gb of non ecc ram and my server is on a 1gbit nic. The performance improvement I want to see is faster loading of my folders as it currently takes 10-20 seconds to load the file structure and thumbnails in the worse case. Would a special vdev suit my needs or would arc and l2arc be fine enough for my needs? I would appreciate any advice on my setup.
Top answer 1 of 5
6
If you want to be certain that your thumbnails and whatnot render faster in Explorer windows, you'll want a special with special_small_blocks set large enough to capture the thumbnails (and you'll want to double-check to see how large those thumbnails are). L2ARC might or might not do that great a job at keeping those cached. Then again, L2ARC is safe whereas the special is an additional element of risk. I'd probably start out with L2ARC (which can be safely removed later if necessary) and see if that did what I wanted it to do well enough.
2 of 5
4
metadata, ZFS Cache (L2ARC) and Log (ZIL/ SLOG device) Here are two video sources from pros that talk about this. https://www.youtube.com/watch?v=mD6i2toN7lE https://www.youtube.com/watch?v=rKXQQZTifzg Do no use Optane for VM's, docker or data. Regular NVME will be much better than Optane for that. All that is discussed on these videos.
Reddit
reddit.com › r/zfs › how do i make a metadata special device vdev?
r/zfs on Reddit: How do I make a metadata special device vdev?
January 31, 2023 -
I have 100 TB raidz1 pool that I am about to create, and I want to store the metadata for it on mirrored SSDs.
How do I create the metadata storage for only the 100TB pool (not the OS)?
zpool create -f -o ashift=12 -m /media storage \
-o recordsize=1M \
-o primarycache=metadata -o secondarycache=none \
raidz \
ata-ST3000DM001-9YN166_HWID \
ata-ST3000DM001-9YN166_HWID \
ata-ST3000DM001-9YN166_HWID \
ata-ST3000DM001-9YN166_HWIDzpool add storage -o ashift=12 special mirror /dev/ssd0n1 /dev/ssd1n1
zfs set special_small_blocks=128K storage
Top answer 1 of 3
2
I would highly recommend using something more robust than raidz1 on a 100TB pool. The likelihood of an error during a rebuild of a pool of that magnitude is not inconsequential. If this is just a backup pool, then so be it, but if this is to be primary storage then you're gambling with your data.
2 of 3
1
That looks reasonable to me, if all the devices and the pool are empty you can just try it and see how it goes? A special vdev is assigned to a specific pool, so if your / is zfs too it isn't going to end up sharing it. Are you sure you want to do this? You've got 4x 3T drives. Doesn't really seem worth wasting a couple of SSDs on a special vdev. I'd probably use them in their own fast pool instead personally. Also, why are you setting primarycache=metadata (and secondarycache=none)? I'd think that'd just cripple performance. I wonder if you can just tack on special mirror /dev/ssd0n1 /dev/ssd1n1 to the create, like you can do create mirror dev1 dev2 mirror dev3 dev4?
Reddit
reddit.com › r/zfs › guidance on how the special vdev performs.
r/zfs on Reddit: Guidance on how the special vdev performs.
January 31, 2022 -
Racking my brain trying to figure out it's actual behavior. No documentation anywhere I can find actually comments on this.
The special vdev is used to store metadata from the pool, so operations like directory listing are at the speed of an NVME drive, which will also improve the performance of spinning rust by reducing the load of small IO required to lookup the pool metadata. This makes sense.
What confuses me is the behavior of the small block allocation class when the special vdev is at capacity. There seems to be a few scenarios that aren't talked about anywhere in the documentation that would be important for performance in various scenarios.
From how i've seen it talked about my understanding is that it acts as a write back cache for small IO based on the special_small_blocks value of the dataset, and when the special vdev is at capacity or needs more space for metadata, small block allocation is offloaded back to the data vdevs.
However i've never actually seen this mentioned anywhere. From the closest thing to a confirmation is in the TrueNas documentation on a fusion pool saying
If the special class becomes full, then allocations spill back into the normal class.
By spill, does that mean, removed from the special class so new incoming small block writes are added to the cache. Or does that mean when the special class is full small IO is sent directly back to the normal class bypassing the special vdev.
Top answer 1 of 2
11
I have answered my own question eventually, partially from some old forum posts and practical testing. Once the special VDEV is full it stays full. The standard small block allocation is 75% of the drive space(default but adjustable), after that further blocks will be sent directly to the backing storage. It can be expanded after the fact, and also striped to another vdev if space becomes constrained, but there is no way to rebalance. Furthermore, the special vdev hits it's allocation limit and later expanded, the small IO will have to be re-written to move it back to it's allocation class. As long as you're using all striped mirrors in your pool with the same ashift(I am), you can "flush" the metadata and small IO back to your main pool by removing the special vdev. I.E zpool remove pool mirror- with x being the mirrored SSD's vdev name This will write all metadata and small blocks back to the main pool. But it also prevents the special vdev from being useful again if re-added without re-writing all your data as new metadata is only added on write.
2 of 2
2
What confuses me is the behavior of the small block allocation class when the special vdev is at capacity. There seems to be a few scenarios that aren't talked about anywhere in the documentation that would be important for performance in various scenarios. When it reaches 75% full, all further small blocks writes go to regular vdevs like normal. Current blocks stay where they are. The remaining space will be reserved for just metadata. Adjust the percentage here: /etc/modprobe.d/zfs.conf by adding zfs_special_class_metadata_reserve_pct=10% for example and rebooting. There might be a way to do it live (only lasts until reboot) I think, but I forget right now. You can search the source code here for "zfs_special_class_metadata_reserve_pct" and find what it touches. Sometimes there are comments you might find helpful. Searching the github issue and pull requests can also be helpful. You're probably thinking that all of this is horribly documented and scattered around like marbles. You're right. As long as you're using all striped mirrors in your pool with the same ashift(I am), you can "flush" the metadata and small IO back to your main pool by removing the special vdev Make absolutely certain you have things backed up and confirmed good (scrubbed) before you try this. This has rarely resulted in problems, mercenary_sysadmin saw corruption I believe when he did some testing when special vdevs first came out. The issue was never resolved and I don't know if he's revisited it. I personally won't trust vdev removal for a long time. Also I do highly recommend a triple mirror as the smallest you consider. Preferably on old enterprise ssds with real PLP (visible capacitors), which are cheap enough to find on ebay. Here's some other references for those curious about special vdevs. Various findings: https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954 Generate a histogram of block sizes Calculate data from ZDB output Clarifies behavior of what goes where when setting small block size: https://github.com/openzfs/zfs/issues/9131#issuecomment-528562601 Small blocks can now be set to anything you can set recordsize to (512B-1M) and even bigger if you enable larger recordsizes (up to 16M): https://github.com/openzfs/zfs/pull/9355
Klara Systems
klarasystems.com › home › openzfs – understanding zfs vdev types
OpenZFS - Understanding ZFS vdev Types - Klara Systems
May 11, 2026 - ZFS pools are built from virtual devices (vdevs), and the topology you choose directly affects performance, redundancy, rebuild times, and scalability. This guide explains ZFS vdev types including mirror, RAIDz1/2/3, dRAID, LOG, CACHE (L2ARC), SPECIAL, and SPARE vdevs, along with the tradeoffs of each configuration for different workloads and hardware environments.
Techno Tim
technotim.com › posts › special-vdev-truenas
Boost ZFS Performance with a Special VDEV in TrueNAS | Techno Tim
January 7, 2026 - Curious about how a special metadata VDEV can boost ZFS performance, especially with spinning disks? In this video, I walk through what it is, why it matters, and how to set it up in TrueNAS and some from the terminal. I’ll also share real-world benchmarks comparing pools with and without ...
Howto-Do
howto-do.it › home › understanding zfs: what is a vdev in zfs?
Understanding ZFS: What is a vdev in ZFS? - Howto-Do.IT
A zpool, which is the top-level structure in ZFS, consists of one or more storage vdevs and zero or more support vdevs. A storage vdev is a collection of block or character devices arranged in a specific topology, such as single, mirror, RAIDz1/2/3, or DRAID. Each vdev type has its own characteristics in terms of fault tolerance and performance. Additionally, there are support vdev types, including LOG, CACHE, SPECIAL, and SPARE, which provide specific functionality to the zpool.
Published: July 2, 2024
Cr0x
cr0x.net › health & recovery › zfs special vdev failure: how to survive the nightmare scenario
ZFS Special VDEV Failure: How to Survive the Nightmare Scenario – cr0x.net
February 3, 2026 - The way out is not clever commands. It’s redundancy, verification, and a refusal to treat tier-0 devices like accessories. Audit: run zpool status and confirm special vdevs are mirrored everywhere. Policy check: inventory special_small_blocks by dataset; decide where it’s justified and where it’s accidental. Monitoring: alert on ZFS checksum errors and NVMe media/data integrity errors, not just “online.”
Cr0x
cr0x.net › fundamentals & design › zfs: when to add a special vdev (and when it’s a bad idea)
ZFS: When to Add a Special VDEV (And When It’s a Bad Idea) – cr0x.net
February 9, 2026 - In ZFS, a special vdev is a dedicated vdev class where ZFS can store metadata (and optionally small data blocks) on faster devices—usually SSDs or NVMe—while the main pool data stays on slower, higher-capacity disks.
Level1Techs
forum.level1techs.com › l1 articles & video-related
ZFS Metadata Special Device: Z - L1 Articles & Video-related - Level1Techs Forums
May 2, 2024 - Introduction ZFS Allocation Classes: It isn’t storage tiers or caching, but gosh darn it, you can really REALLY speed up your zfs pool. From the manual: Special Allocation Class The allocations in the special class are dedicated to specific block types. By default this includes all metadata, ...
FreeBSD
forums.freebsd.org › base system › storage
ZFS - ZFS special device on shared drive | The FreeBSD Forums
September 7, 2022 - A "special device" is a VDEV just like a "log device(ZIL)" or a "cache device(L2ARC)". It just needs a partition/slice. Note a reddit link, ... It might be easier to comment if we know what you are trying to achieve.