RAID-Z2 works by taking the drives given to the vdev, and using two of those to store redundant data for recovery purposes in case something goes wrong.
Hence, the absolute minimum number of drives in a RAID-Z2 vdev is the two redundant ones plus one, which works out to three drives.
However, by doing it that way, you:
- are getting the effective storage capacity of only one of the drives
- can lose any two of the three drives involved without data loss (in the general case, you can survive the loss of any 2 drives in a N drive RAID-Z2 vdev)
- are forcing the system to calculate parity data for RAID-Z2
If you want a vdev consiting of three drives, or if you have three and want two drives' worth of redundancy, you are probably better off by setting them up as a mirror instead. That way, you:
- are getting the effective storage capacity of only one of the drives (regardless of the number of mirror devices)
- can survive the loss of any two of the three drives involved without data loss (in the general case, you can survive the loss of any N-1 drives in a N drive mirror vdev)
- are avoiding parity data calculations, because mirrors are bit-identical copies of each other; writing the same data to multiple disks is much cheaper than calculating separate parity data and writing that out to disk
The minimum number of drives where RAID-Z2 makes sense is four, which gives you the effective storage capacity of two of the drives and allows losing any two of the drives. This is advantageous over using the same four drives in a 2x2 mirror setup, because if you have two mirror vdevs of two drives each and lose some combinations of two drives, that vdev is dead and takes the pool with it in its fiery demise. In a 2x2 mirror vdev pool, at least one of the drives in each mirror pair must remain functional for the pool to remain available. However, the mirror may very well have better performance. As always, it's a trade-off between different choices, and one part of the system administrator's job description is to make those trade-offs appropriately for each specific situation.
Thus, while RAID-Z2 might technically work with the setup you describe, it offers no advantages whatsoever and even has some disadvantages compared to a simple three-way-mirror configuration.
It would be different if ZFS allowed us to dynamically change the redundancy level of or increase the number of devices in a RAID-Zn vdev, but it does not. The only way to actually grow a pool is by adding more vdevs or grow the size of the existing devices, not by adding devices to an existing vdev. (You can add and remove devices in a mirror vdev, but that only changes the amount of redundancy, not the usable storage capacity.)
You could also set up the three drives as a RAID-Z(1) vdev, which will give you the ability to survive the loss of any one of the drives before your data is at risk if anything whatsoever further goes wrong, reduce the CPU workload as compared to RAID-Z2 (because RAID-Z1 parity calculations are less computationally intensive) and give you the effective storage capacity of two of the drives combined.
Always keep in mind that if anything goes wrong, a full ZFS resilver is an arduous process for the remaining disks, and it isn't unheard of for another disk to develop problems or even die under the stress. For this reason, especially with rotational drives of the sizes common today, double redundancy should be the default choice with triple redundancy an option if you are really paranoid. Single redundancy should be considered only for less-important data where downtime can be tolerated. Single redundancy may be acceptable with a SSD-backed pool, however.
Hence my recommendation: If you want three drives ZFS, and want redundancy, set them up as a three-way mirror vdev. If you want RAID-Z2, use a minimum of four drives, but keep in mind that you lock in the number of drives in the vdev at the time of vdev creation. Currently, the only way to grow a ZFS pool is by adding additional vdevs, or increasing the size of the devices making up a vdev, or creating a new pool and transferring the data. You cannot increase the pool's storage capacity by adding devices to an existing vdev.
You also need to make sure you keep backups. This is doubly important with ZFS, because ZFS recovery after an actual failure that brings the pool below its redundancy threshold is very hard. ZFS has a very complex on-disk format that I am not aware of any common off-the-shelf data recovery software understanding, so even if you can read most of the data off the disks, if ZFS can't figure it out then you may very well be out of luck unless you are willing to spend tens of thousands of dollars on the problem. Backups are cheaper.
As an alternative to the three-way mirror, you could also run a two-way mirror with a spare, but most of the time, and certainly in the situation you describe, there is no advantage and some disadvantage to that.
As an aside, related to the issue of recovery, you should very strongly consider using ECC RAM in any system that runs a checksumming self-repairing file system including ZFS. Others agree.
Showing that RAID-Z2 with three devices is possible (this is ZFS On Linux 0.6.4). Notice that after hard removing two of the three backing files, "Sufficient replicas exist for the pool to continue functioning in a degraded state." and the vdev is DEGRADED, not FAULTED.
# truncate -s 1G /root/d1 /root/d2 /root/d3
# zpool create tank raidz2 /root/d1 /root/d2 /root/d3
# zpool status tank
pool: tank
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
/root/d1 ONLINE 0 0 0
/root/d2 ONLINE 0 0 0
/root/d3 ONLINE 0 0 0
errors: No known data errors
# zpool export tank
# rm /root/d1 /root/d2
# zpool import tank -d /root
# zpool scrub tank
# zpool status tank
pool: tank
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://zfsonlinux.org/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 0h0m with 0 errors on Tue Mar 29 11:00:23 2016
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
18130982121682915530 UNAVAIL 0 0 0 was /root/d1
18289483070703159278 UNAVAIL 0 0 0 was /root/d2
/root/d3 ONLINE 0 0 0
errors: No known data errors
#
Answer from user on Stack ExchangeWhat is a ZFS vdev?
A ZFS vdev (virtual device) is the fundamental storage building block of a zpool, defining how disks are grouped for performance and fault tolerance.
Is RAIDz2 better than RAIDz1?
RAIDz2 provides dual parity protection and can survive two drive failures, making it significantly safer for large modern drives.
Is RAIDZ1 safe for modern drives?
I recently am in the market for rebuilding my NAS, since I recently lost one of the drives in my Synology system. The Synology gave me enough warning to save all my files onto my main PC. Instead of replacing the drive, I wanted to learn and take a crack at building my own DIY NAS. I'm a bit of a noob when it comes to ZFS, and how many drives are needed for RaidZ1 and RaidZ2. I was looking at RaidZ2, but I seem to find conflicting information on how many drives are needed to enable it.
For Z2, this site says I only need 4 drives: https://calomel.org/zfs_raid_speed_capacity.html
While this site says I need 5 drives: https://raidcalculators.com/zfs-raidz-capacity.php
Hello,
I have been thinking about investing more into my storage solution recently.
Right now I have seagate 2tb compute drives in a fairly capable desktop. I was thinking about buying one more and doing a raidz1 with 3 drives. But while someone is saying it is cool, other says it is really dumb to do. I am really confused. I can get the third drive as soon as tomorrow to finally get some safety around my data however getting 4 with raidz2 might be a go for me as well for future upgrade. I am really all over the place since I have no experience building a nas. Should I go for 4 drives with raid-z2 or 3 drives with raidz1. And it is double the price for me right now with same storage space.
Here is another thing. A thread within this subreddit says "don't go more than 1tb for raidz1 not less than 3 more than 7". And then says "don't go less then 6 drives for a raidz2" which is needed in this case. How come it can be true? It means like no one can use 2tb disks for 4tb of pool. I want to buy 3 or 4 seagate reds around June, this statement makes this config obsolete as well. Everyone is saying something different.
PS: I am living in Turkey, Getting 4 drives of Seagate Ironwolfs is actually two times the minimum wage here. Please reply me gently :/
Hi - I cannot find a clear answer for this on the web. It seems there are no hard set rules, more like guidelines when it comes to raidz2 and the width of your vdev.
We have a ZFS system using raidz2 with 27 disks. This happened recently:
config:
NAME STATE READ WRITE CKSUM
zpool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
sda REMOVED 0 0 0 (resilvering)
sdb DEGRADED 0 0 0 too many errors
sdc DEGRADED 0 0 0 too many errors
sdd DEGRADED 0 0 0 too many errors
sde DEGRADED 0 0 0 too many errors
sdf DEGRADED 0 0 0 too many errors
sdg DEGRADED 0 0 0 too many errors
sdh DEGRADED 0 0 0 too many errors
sdi DEGRADED 0 0 0 too many errors
sdj DEGRADED 0 0 0 too many errors
sdk DEGRADED 0 0 0 too many errors
sdl DEGRADED 0 0 0 too many errors
sdm DEGRADED 0 0 0 too many errors
sdn DEGRADED 0 0 0 too many errors
sdo DEGRADED 0 0 0 too many errors
sdp DEGRADED 0 0 0 too many errors
sdq DEGRADED 0 0 0 too many errors
sdr DEGRADED 0 0 0 too many errors
sds DEGRADED 0 0 0 too many errors
sdt DEGRADED 0 0 0 too many errors
sdu DEGRADED 0 0 0 too many errors
sdv DEGRADED 0 0 0 too many errors
sdw DEGRADED 0 0 0 too many errors
sdx DEGRADED 0 0 0 too many errors
sdz DEGRADED 0 0 0 too many errors
sdaa DEGRADED 0 0 0 too many errors
sdab ONLINE 0 0 9 (resilvering)
spares
sdal FAULTED corrupted data
errors: 20595 data errors, use '-v' for a listWe had two disks fail recently and resilvering has seemed to have crashed the entire pool. We are going to destroy the pool and rebuild.
From what I've read, I am of the opinion that we should have multiple raidz2 vdevs instead of a single giant one like this. That way when one goes down, it doesn't take the entire pool offline and/or make the server inoperable/unbearably slow when it is resilvering.
Am I correct in this thinking?
Is this single 27 disk vdev the reason resilvering never finished?
Don't multiple vdevs afford better write performance due to the way ZFS works?
I've read your vdev shouldn't be "too wide"--certainly this is a case of being too wide?
Wouldn't it be better if we had these disks in 3 vdevs e.g.
.
raidz2-0 sda sdb sdc etc raidz2-1 sdf sdg sdh etc raidz2-3 sdx sdy sdz etc
My coworker wants to maximize storage space, but I don't want this to happen again. If storage space is a concern, then we need to buy more storage.
We used to have this setup in a mirror, which worked fantastically. But, obviously it halved our capacity. e.g.
mirror0 sda sdb mirror1 sdc sdd mirror2 sde sdf mirror3 sdg sdh mirror4 sdi sdj etc...
I've got 24 6TB drives to work with. Running the numbers on wintelguy's ZFS calculator resulted in an estimated 71.92TiB usable space (20% free space limit) with 3x 8-drive RAIDz2 VDEVs, or 67.17TiB with 4x 6-drive RAIDz2 VDEVs.
My question is what's the performance difference likely to look like? As I understand it, writes should be relatively evenly distributed to the vdevs just like multiple mirrored pair vdevs, and that should come with a performance increase with each additional vdev. The benefit of more drives in a given RAIDz2 vdev is theoretically higher sequential read/write, but I understand that benefit is only applicable for certain workloads whereas this will see a mixed workload backing VMs and containers as well as hosting colder data. I know with that many drives I could do separate pools optimized for the hot and cold data, but I'm throwing an Intel P3700 on for SLOG and a 1.6TB P4600 for L2ARC supporting the 128GB of RAM.
Has anyone done any real-world tests on this kind of thing? I know the obvious answer is: "Just try it both ways yourself" but I'm looking for the advice of the experts here, not some numbers on a synthetic benchmark that doesn't accurately represent my workload.
My use case is storage in a 8xbay back-up homelab server.
I turn it on when my local FS is full to empty it, so high speed is not necessary.
6 mixed SAS/SATA drives (HGST He8 used in a datacenter) have ~45k power-on hours (with 40 power-cycles) and the rest 2 (shucked, seem like WD Helium filled) are in the 10k region with 600 power cycles. No failed long SMART tests so far.
This is my list of pros/cons, am I missing anything?
I think it's prudent to not consider 2-way mirrors (4x 4 mirrors) because the drives are not new, right?
However, both my top two options are RaidZ which are more stressful to the disks, so RaidZ might cover me in a position to which I would never have been brought, had I used mirrors (!).
| 1 vdev of 8-drive RaidZ-3: | 2 vdevs of 4-drive RaidZ-2: |
|---|---|
| Any 3 drives can fail | Any 2 can fail and potentially more if they are in different vdevs |
| ~35.3TB usable storage | ~30TB usable storage |
| Expands with more money | Expands with less money (replacing drives in 1 vdev only) |
| Puts more stress to the drives | Puts a little bit less stress to the drives. |
| Worse performance, but I don't care | A bit better performance, but I don't care |
Thanks!
Whatever scheme you use, have a backup, use snapshots, and scrub regularly. RAIDz3 gives the best probability of self-healing during a scrub since there are 3 parity bits and the data, making a fault likely to be repaired correctly. RAIDz2 has 2 parity bits and the data. Still a strong choice. However, without a backup and you experience catastrophic failure, or have data corruption happen at the right time during certain events, then your risk of data loss increases.
With a good strategy catered to your needs and resources, you can enjoy data integrity without a single point of failure. There are risks to keeping a machine running and there are risks to turning off and turning on a machine regularly. Multiple backups can mitigate risks from either scenario. If you do not have a high IO use case then maybe you don't need RAIDz3.
This doesn't require so much thought.
Use a single 8-drive RAIDZ2.
Raid 10, RaidZ1, or RaidZ2?