benefits of zfs live dedup vs pbs offline dedup

The normal mode of operation for PBS is to run on the hypervisor host and carry out a block-level backup of the VM/LXC virtual disks. Since it's a block level backup the filesystem on the virtual disk is irrelevant at the time it is captured. However a key feature of PBS is that you can browse and extract individual files. That's only going to be possible if the PBS server can read the filesystem inside the backup. The backups created by PBS are de-duplicated. ZFS provides for de-duplication of the primary storage (i.e. where the live vmdk lives when the VM is running). Also, as Qemu is able to track the dirt blocks, when PBS creates a VM backup, it only needs to read/write the blocks changed since the last backup. This does not work for lxc's - where PBS creates an image of the whole device then compares this with the most recent backup to get the changed blocks. There's also proxmox-backup-client which reads the local devices and sends them to a PBS server. I've not used this myself, but I believe it is currently only available for Linux and in the absence of a mechanism for tracking dirty blocks, will behave similarly to an LXC backup. But since the host where the backup client is running cannot be quiesced you'd be looking at crash-recovery on restore.

1 of 3

2 of 3

As far as I know yes, ZFS dedup is a part from the ZFS FS so it makes sense, the entire server is build for you to use ZFS

reddit.com › r/proxmox › correlation of zfs dedup, compression and proxmox backup server

r/Proxmox on Reddit: Correlation of ZFS dedup, compression and Proxmox Backup Server

November 18, 2020 -

Today I played around with some settings in ZFS and Proxmox Backup Server. I use Proxmox Backup Server with a ZFS raidz2 as storage for the Backup space. The ZFS pool was created manually without the proxmox-backup-manager command.

I noticed that the WebUI shows a deduplication factor of (in my case) 94.02 but the zfs pool shows dedupratio 1.00x.

Does Proxmox Backup Server not use ZFS' deduplication functionality? Should I enable deduplication and compression in the ZFS pool? Which compression algorithm should I use?

EDIT: I have tested a bit with ZFS and Proxmox Backup Server for quite a while (both hardware and VMs) and ZFS' deduplication and compression have next to 0 gains. As pointed out by the comments deduplication does not make sense as Proxmox stores backups in binary chunks (mostly of 4MiB) and does the deduplication and most of the compression there. ZFS compression with zstd-12 ended up on a compressratio of 1.02x which is not very significant.

TL;DR It's not necessary to use ZFS compression or deduplication because it's both already handled by Proxmox's way of storing Archives and Block Devices.

reddit.com › r/zfs › deduplication - request for success stories and empirical best practices

1 of 2

I believe they do not use zfs's dedup functionality. It seems they have created their own file format, Proxmox File Archive (.pxar). So they handle deduplication and compression and data transport themselves. ZFS isn't a requirement for datastores. You can create datastores on ext4 for example. They store data at the file level, not block.

So, I would turn off compression because it's probably extraneous for achieves which are already compressed.

PBS only dedups on a per group (container/vm/host) basis. So you will have some data duplicated across groups. So enabling ZFS deduplication may help a little by deduping the whole datastore instead of each individual group. I don't know if it's worth it though, you'll have to test/decide for yourself.

ETA: a word

2 of 2

The PBS does not use ZFS deduplication. PBS dedupe works similar to Windows offline Dedupe. PBS does it on the fly instead of as a scan process like Windows. But with both, files and blocks are broken up into chunks and stored in a chunk store. The chunks are hashed and matching hashes are used to dedupe.

r/zfs on Reddit: Deduplication - Request for Success Stories and Empirical Best Practices

May 26, 2023 -

Good Morning everyone!

As the title states, would you be willing to share any useful and positive information you may have around deduplication? Really just looking for some success stories and what you have found to work well in reference to deduplication.

Typically all I ever hear and read about ZFS deduplication is "just don't. and if you think you might benefit, still don't." or "it benefits a very niche workload but not enough to be worth it... so just don't".

I am really curious if anyone has found a good use for it, how successful was it, what has the experience been like, and what configurations (hardware and software) are in use?

Thank you to anyone who is willing to share.

And last, I have a hot downvote for anyone showing up with a "just don't" post...

Well the problems seem to be RAM usage, write speed and how much someone can actually profit from deduplication. so if you need a success story: get 2TB RAM, a bunch of M2 SSDs, use only one dataset (as dedup only works inside one dataset) and always write many copies of the same files instead of linking them. then you will have success. that said (i was obv joking): i'd really like to see some kind of offline dedup in the future. personally i won't profit much but a cron job which checks daily wouldnt hurt me. and who doesnt one free space

I did have a successful dedup deployment, but I also knew exactly what I was getting into and the sort of data I was putting in there. Basically, I had a system with a pool of spinning disk, 128GB of RAM and dedup tables on SATA SSD... no, not NVMe. I had very little reason to pull data quickly from it since it was literally an archive store for my VM's and arbitrary user data (Nextcloud). It also was pretty good for SQL dumps and the like. I ran backups to it daily using ZFS SEND / RECV and it just happily ingested all the data I gave it, then I would snapshot the pool and do another dump the next day. Between compression and dedup I think I was averaging about 3:1 or which was pretty damned good in my opinion, and accessing the data actually ended up being a lot quicker than I had anticipated. Of course, accesses were single-threaded since I only needed to pull data for a restore and that didn't happen often and was just me doing it. Media data got backed up to a second pool in the same system, so the 128GB of RAM was doing "double-duty". I had 100GB of RAM dedicated to ARC that was split among the two pools, but media data doesn't dedup well and backing up the media data to the dedup pool in testing was noticeably slower than the non-dedup pool. In the end it worked well for my use case. It was retired about 6 months ago so sorry I can't pull recent statistics; I moved my backups to a new system (hosted on unRAID) that ingests the backups via rsync and does a "file level dedup" (the solution is called BackupPC)... this gives me less administrative headache since I don't have to manage two pools and the array can be expanded a lot more easily.

reddit.com › r/zfs › openzfs deduplication is good now and you shouldn't use it

r/zfs on Reddit: OpenZFS deduplication is good now and you shouldn't use it

October 28, 2024 - I used to be a dedup expert, and in general what you call "real time dedup" is a lot better than post process dedup for most tasks, but for it to be truly viable, you need to have variable block size dedup, and here ZFS falls short with its fixed block size. If you add something in the middle of a huge file, all the blocks basically change due to alignment, while a variable block size dedup will have some kind of secret sauce that detects that it's still mostly the same blocks. ... There is work being done in that area as well. What is still missing before block cloning can be used for offline dedup is support for the syscall `FIDUPERANGE` which is being worked on here: https://github.com/openzfs/zfs/pull/15393

reddit.com › r/zfs › deduplication (dedup) offline-only?

r/zfs on Reddit: Deduplication (dedup) offline-only?

December 7, 2021 -

Hey there,

Do I understand correctly, that zfs does dedup either online or not at all? I would like to dedup offline(out-of-band) maybe once a month but I did not find answer on how to do this in zfs. The system does not have a lot of RAM, so online-dedup is not an option.

You can't have deduplication without keeping the dedup table in memory. If you want to dedupe an existing dataset, you have to make a new dataset with it enabled, copy data over, destroy the old dataset and rename the new one. Also, in general, deduplication doesn't save a lot of space in most cases. You would probably rather enable compression, if you haven't already. The copy data thing still applies, if you want to compress existing data.

You can kinda achieve this with using something like jdupes to search for duplicate files but it doesn't do block level like zfs native dedupe.

reddit.com › r/proxmox › zfs deduplication of pve backups?

r/Proxmox on Reddit: ZFS deduplication of PVE backups?

November 11, 2024 -

I'm using Proxmox at home to host the usual things. One ZFS NVMe pool for all the containers/VMs and one ZFS HDD pool for nightly backups.

I just setup a second PC with TrueNAS for some "just in case" backups of the backups (from PVE and other sources around the house).

Each VM has 1-2 dozen nearly identical backups, so I decided to try ZFS' deduplication feature. I copied over backups for 4 VMs (500G total) and was shocked to see no gains/efficiency from the deduplication, not even 1.01!

Some of those backups are from a VM that was powered off, so the original disk was definitely unchanged. Shouldn't the resulting backups be bit-identical? Does anyone know if Proxmox includes some metadata that might spoil block-level deduplication?

Get rid of TrueNAS and use Proxmox Backup Server instead. I've never had decent results with ZFS deduplication. The PBS dedup feature works allot better. It's also allot faster because it only needs to backup changed data from the last backup. And I'll probably get booted out of this sub for saying this but Synology ABB for everything else. PBS only supports Proxmox and Debian clients. I've looked at other backup solutions both commercial and open source and I ended up getting a small Synology NAS instead. Dedup and compression works decent in ABB and no licensing costs per machine. (If you really want the free option, UrBackup also supports deduplication/incremental-forever and has clients for most OS.)

1 of 8

2 of 8

Not sure how regular backups are deduped (or if they are at all) and I have never seen much dedupe happening on ZFS (though I’ve never really looked into it and haven’t cared). But I can tell you, check out Proxmox Backup Server, which is Proxmox’s official backup solution that’s a huge step up and has absolutely crazy dedupe, incremental backups, etc. still free, just runs in a VM or LXC.

reddit.com › r/zfs › does it make sense to use both dedup and compression?

r/zfs on Reddit: Does it make sense to use both dedup and compression?

April 15, 2016 -

I use ZFS for backup. Performance isn't an issue, so I've enabled deduplication. Will enabling deduplication and compression reduce capacity used further than deduplication alone?

truenas.com › docs › references › zfsdeduplication

1 of 2

you should definitely enable compression and MAYBE deduplication if you think your scenario warrants it. not the other way around.

and yes, deduplication and compression can work together. but you need to enable them before you copy data onto that volume. enabling compression/deduplication on an already filled volume will have no effect on already stored data.

2 of 2

Assuming your data is already on ZFS, you can calculate your dedupe ratio with 'zdb -S'. I normally would not use dedupe on anything with less than a 16:1 aggregation ratio, mainly because that's achievable with compression. If the aggregation ratio reported by zdb -S is a lot higher than that, you have a good use case to suck up the RAM hit of holding the DDT in RAM.

The use of DDT is also generally a good use case for using L2ARC; the ideal use case is generally something like holding OS images.

TrueNAS

ZFS Deduplication | TrueNAS Documentation Hub

March 2, 2026 - Reduced ZFS Performance Deduplication adds extra lookups and hashing calculations into the ZFS data pathway, which slows ZFS down significantly. A deduplicated pool does not reach the same speeds as a non-deduplicated pool. When data is not sufficiently duplicated, deduplication wastes resources, slows the server down, and has no benefit.

Lobsters

lobste.rs › s › kuhlap › openzfs_deduplication_is_good_now_you

OpenZFS deduplication is good now and you shouldn't use it | Lobsters

October 28, 2024 - They have a dedup table that can scale dynamically with memory and will miss things when memory pressure grows but not degrade performance (beyond the extra writes required when dedup fails).

Find elsewhere

Google Bing Mojeek

despair labs

despairlabs.com › blog › posts › 2024-10-27-openzfs-dedup-is-good-dont-use-it

OpenZFS deduplication is good now and you shouldn't use it

October 26, 2024 - Fast dedup improves all three, making it far easier to balance these factors and rather less of a disaster if it doesn’t work out. However, it is still only of benefit if you have a truly enormous amount of data, that gets copied a lot, and aren’t able to take advantage of other “zero-copy” options within OpenZFS, like block cloning or snapshot clones.

reddit.com › r/zfs › zfs dedup performance

r/zfs on Reddit: ZFS dedup performance

August 5, 2022 -

I understand dedup can cause performance degradation. I also understand that it can offer performance improvements.

Are the hard demands, CPU & RAM usage, only a concern during write events? If a zvol is born with dedup=verify set but is later removed does the running state of ZFS always consume additional resources for all reads?

I also understand that it can offer performance improvements Not when you have to ask, it doesn't. The feature is for incredibly specific workloads and you won't ever see "performance improvements", your disks will still need to be read to be... read. But with specific datasets the blocks something is after may already reference other data and find it in the ARC more frequently. Again, nothing any of us are doing on the regular will fit in this workload. You will not see this with nearly anything you do at home or in most projects. This is an immensely specific feature designed for, mostly, very specific database configurations. But can be applied elsewhere when designed for it. Also, if you change your mind the feature doesn't completely go away without remaking your pool entirely, not worth the mistake of enabling it. You should enable compression, which has proven benefits for the majority of data and leave it at that.

github.com › jim-collier › x9dedupe

Don't use dedup.

GitHub

GitHub - jim-collier/x9dedupe: The first ZFS offline deduplication tool. Also deduplicates Btrfs and XFS. · GitHub

Stores extended file attributes to both incrementally compare, and incrementally dedup. Significantly speeds up successive runs and reduces disk wear.¹ · Xattrs can survive copying to different filesystems & across networks, with optional ...

Starred by 32 users

Forked by 2 users

Languages: Shell

Hacker News

news.ycombinator.com › item

OpenZFS deduplication is good now and you shouldn't use it | Hacker News

November 3, 2024 - To me, this is "obviously" the wrong approach in most cases. When I'm writing data, I want that write to complete as fast as possible, even at the cost of disk space. That's why I don't save files I'm actively working on in 7zip archives · But later on, when the system is quiet, I would love ...

Proxmox

forum.proxmox.com › home › tags

deduplication | Proxmox Support Forum

Therefore I can either: dedicate all of my SSDs and HDDs to one large PBS that is remote from my PVEs, or split my resources: locally (with the PVEs) place... ... There are a large amounts of improvements in ZFS 2.3.0 if you use NVME and deduplication: https://github.com/openzfs/zfs/releas...

reddit.com › r/zfs › anyone doing zfs deduplication?!

r/zfs on Reddit: Anyone doing ZFS deduplication?!

November 5, 2022 -

I know ZFS deduplication is discouraged, because apparently it takes a lot of RAM and CPU. But 1 GB RAM and 1 CPU core per 1 TB data is not that costly. These days, most PCs and laptops have +16 GB RAM and +8 cores, with 1–2 TB of NVME.

Any problem if I relocate 1 TB to a deduplicated data set on NVMe, with hardware mentioned above? The OS is not deduplicated, just data.

Anyone doing ZFS deduplication here? How much space do you save, what is your RAM, CPU and performance?

Apparently, the savings are not that much either.

Software such as Restic or Borg do deduplication easily, with great speeds and deduplication ratios. I could use these software, but I prefer not to use third party apps.

Update. From the comments, deduplication is continued to be discouraged. The savings are small, not worth the trouble.

Let me tell you a little story from my first foray into using dedup. This was back in... 2010? or so, when I was running a 4*500G raidz5 on OpenSolaris, in VMware on my Windows desktop. The pool was mostly storing ~1G files and I had dedup turned on because I figured any space saving would be helpful. Things were basically fine until the pool filled up, at which point I had to start seriously deleting quite a bit of stuff. Deleting files on that pool would cause ZFS to completely hang for about 20 minutes, maxing the disks out with random I/O. Deleting different files usually caused another 20 minute hang, and any attempt to unhang it by rebooting the VM made it start all over again on import. Eventually I worked out what it was doing: loading the relevant parts of the dedup tables from disk so it could mark the blocks as freed. (You, the reader, already know this problem was dedup-related, but I didn't and it took me a while to work it out.) Things did get better once a significant part of the tables were cached. Dedup wasn't really saving me any space either so using it was pointless, but if you want to remove dedup from existing data the only way to do it is to rewrite that data. Fortunately for me I didn't have any requirement for high availability, because it took a while to do that. It's worth pointing out that I didn't have a RAM problem -- the problem was that the dedup tables weren't in the RAM, they were highly fragmented all over the disks. More RAM would've helped only so long as I never rebooted the VM. Anyway, that was then. Today we have SSDs, L2ARC, dedicated metadata and dedup devices, large recordsize support, faster dedup-capable checksum algorithms and significantly better overall hardware. I suspect that a pool with dedup tables on NVMe will probably be tolerable, especially if you're careful to use large record sizes and limit dedup to just a small set of data that it's helpful for. But I haven't tested it in earnest, and any problems likely won't show up immediately or be obviously attributable to dedup.

Apparently, the savings are not that much either. Savings entirely depend on the type of data. If you have hundreds of VMs and all are running the same OS all that data is deduplicated down to basically a single instance of the VHD and a couple of blocks for the custome parts of the VHD of each VM (this is exaggerated). This would be where deduplication shines. A regular single instance like John Doe's Linux Computer with a couple of Steam games, some mp3s and the odd Office file will usually experience a deduplication rate of only a couple of percent ... usually less than 10%. One could argue that memory capacity is more than 10% more expensive than storage capacity it's not worth it. Restic or Borg do deduplication easily Restic and Borg don't do deduplication on your existing data - they are backup tools and their deduplication is done when writing to the backup destination. This is completely diofferent to what ZFS deduplication does. In the case of ZFS it's built into the block-level layer and it's completely transparent to the OS or user. The active data in the file system (including metadata) is deduplicated. You could compare it to software like fdupes or rdfind but they only work on-top of the file system either, they find files that are duplicates of others and replace them with links to the first find of that file. This is not transparent to the OS or the user, it can be noticed and it can cause trouble. Block-level deduplication as done by ZFS has no direct-compare candidate in the Linux world (no, not even BTRFS deduplication) - to my knowledge there is only one file system available that does have comparable (and indeed even better) functionality and performance (with regard to deduplication) and that is NetApps WAFL file system - which might have been the source of inspiration for some of ZFS's features.

reddit.com › r/proxmox › pve backup vs pbs. do i need pbs

r/Proxmox on Reddit: PVE backup vs PBS. Do I need PBS

May 29, 2024 -

Am I correct in that the only benefit of PBS is that it provides a file-level restore? Otherwise, for a single node Proxmox VE, is the built-in backup function sufficient for being able to restore a corrupted VM or CT?

I am planning to use a 2TB external HDD connected via USB and create a backup job within ProxMox VE 8.2.2

Thank you for the advice.

If you want to keep backup growth in check then yes, you want PBS. The de-dupe and incremental is reason enough to go through with PBS. Saying nothing of the offsite replication option. Here is the first guide i followed to get PBS running in our labs for testing, give it a go you might find it better then you think. https://www.virtualizationhowto.com/2023/12/proxmox-backup-server-ultimate-install-backup-and-restore-guide/