I just started a pbs server to check out how it works. I have 8 vm's at 50GB each, and each has 5 snapshots. Each snapshot appears to be the full 50GB and the dashboard is giving me a dedupe factor of 1. I have the backup retention set to last 14. How long before I should see the deduplication saving space?
How does deduplication reduce backup storage?
How much storage do I need for Proxmox backups?
What retention policy should I use?
Hi,
so im currently testing the proxmox backup server für my vms. I was wondering if the deduplication / dirtybit only works with zfs or if it also works on "normal" ext4 formated disks (mounted directory).
Best
Just got this up and running yesterday. Set it for daily backups. It did the first full backup and this morning did the first incremental. Shows only a value of 3.04. TechnoTim had a value like 64. Not sure what he was doing for that but I do think he was doing like hourly backups so that maybe is why. Anyhow just curious what things I could do to possibly increase this value?
On that note, does the garbage collection schedule make any difference? Right now I am doing on PBS pruning for 7 dailys and 2 weeklys (pve job retention is off) and garbage collection every 6hrs. Not sure if this impacts anything but wanted to mention.
I guess what is odd (more with the incremental aspect). I haven't changed a single thing on any of my 10 CTs yet it got an incremental backup and took an hour. Still better than the almost 3hrs it too for a full backup but confused why it was so much if it's just the difference. Since nothing changes shouldn't it be close to nothing?
I'm using Proxmox at home to host the usual things. One ZFS NVMe pool for all the containers/VMs and one ZFS HDD pool for nightly backups.
I just setup a second PC with TrueNAS for some "just in case" backups of the backups (from PVE and other sources around the house).
Each VM has 1-2 dozen nearly identical backups, so I decided to try ZFS' deduplication feature. I copied over backups for 4 VMs (500G total) and was shocked to see no gains/efficiency from the deduplication, not even 1.01!
Some of those backups are from a VM that was powered off, so the original disk was definitely unchanged. Shouldn't the resulting backups be bit-identical? Does anyone know if Proxmox includes some metadata that might spoil block-level deduplication?
Today I played around with some settings in ZFS and Proxmox Backup Server. I use Proxmox Backup Server with a ZFS raidz2 as storage for the Backup space. The ZFS pool was created manually without the proxmox-backup-manager command.
I noticed that the WebUI shows a deduplication factor of (in my case) 94.02 but the zfs pool shows dedupratio 1.00x.
Does Proxmox Backup Server not use ZFS' deduplication functionality? Should I enable deduplication and compression in the ZFS pool? Which compression algorithm should I use?
EDIT: I have tested a bit with ZFS and Proxmox Backup Server for quite a while (both hardware and VMs) and ZFS' deduplication and compression have next to 0 gains. As pointed out by the comments deduplication does not make sense as Proxmox stores backups in binary chunks (mostly of 4MiB) and does the deduplication and most of the compression there. ZFS compression with zstd-12 ended up on a compressratio of 1.02x which is not very significant.
TL;DR It's not necessary to use ZFS compression or deduplication because it's both already handled by Proxmox's way of storing Archives and Block Devices.
I believe they do not use zfs's dedup functionality. It seems they have created their own file format, Proxmox File Archive (.pxar). So they handle deduplication and compression and data transport themselves. ZFS isn't a requirement for datastores. You can create datastores on ext4 for example. They store data at the file level, not block.
So, I would turn off compression because it's probably extraneous for achieves which are already compressed.
PBS only dedups on a per group (container/vm/host) basis. So you will have some data duplicated across groups. So enabling ZFS deduplication may help a little by deduping the whole datastore instead of each individual group. I don't know if it's worth it though, you'll have to test/decide for yourself.
ETA: a word
The PBS does not use ZFS deduplication. PBS dedupe works similar to Windows offline Dedupe. PBS does it on the fly instead of as a scan process like Windows. But with both, files and blocks are broken up into chunks and stored in a chunk store. The chunks are hashed and matching hashes are used to dedupe.