When I originally set up my computer, I put an LVM on my primary SSD with a couple of partitions in it. As the system has evolved, it's now at the stage where /home takes up the whole of that LVM.
So I'm trying to decide whether or not it's actually worth the effort of swapping things round so /home is a direct filesystem rather than going via the LVM. Any ideas?
LVM is fairly lightweight for just normal volumes (without snapshots, for example). It's really just a table lookup in a fairly small table that block X is actually block Y on device Z. I've never done any benchmarking, but I've never noticed any performance differences between LVM and just using the raw device. It's some small extra CPU overhead on the disc I/O, so I really wouldn't expect much difference.
My gut reaction is that the reason there are no benchmarks is that there just isn't that much overhead in LVM.
The convenience of LVM, and being able to slice and dice and add more drives, IMHO, far outweighs what little (if any) performance difference there may be.
I am installing a 48T Dell MD-1200 and I was curious about this question. The MD1200 is connected to a hardware RAID card set up as RAID-6, so it looks to Linux like just a (big) drive. I tested an XFS filesystem on an LVM physical volume vs. an XFS filesystem on a straight disk partition. I used a Dell R630 machine with two E5-2699 CPUs in it. The system was set for Performance; whatever energy saving features I could find in the BIOS were turned off.
I installed CentOS 6.7 on it. Kernel is 2.6.32-573.el6.x86_64 (sorry for the oldie kernel but that's what I need for production). LVM is version 2.02.118.
I let CentOS create an XFS partition during the build. It is 1T in size. Then I created another 1T partition on the disk and created a logical volume:
vgcreate vol_grp1 /dev/sdb1
lvcreate -l 100%FREE -n lv_vol1 vol_grp1
mkfs.xfs /dev/vol_grp1/lv_vol1
My XFS-only filesystem was called /data_xfs. The LVM-backed XFS filesystem was called /data_lvm. I tested using bonnie++ v 1.03e.
The commands were: bonnie++ -u 0:0 -d /FILESYSTEM -s 400G -n 0 -m xfsspeedtest -f -b where FILESYSTEM was either /data_xfs or /data_lvm .
Results are summarized as follows:
Test XFS on Partition XFS on LVM
Sequential Output, Block 1467995 K/S, 94% CPU 1459880 K/s, 95% CPU
Sequential Output, Rewrite 457527 K/S, 33% CPU 443076 K/S, 33% CPU
Sequential Input, Block 899382 K/s, 35% CPU 922884 K/S, 32% CPU
Random Seeks 415.0 /sec. 411.9 /sec.
Results seemed comparable in my view. In the Sequential Input test, LVM actually seemed to perform a little better.
Videos
Note: For an answer from this decade, I refer you to a 2023 answer from Serverfault.
LVM is designed in a way that keeps it from really getting in the way very much. From the userspace point of view, it looks like another layer of "virtual stuff" on top of the disk, and it seems natural to imagine that all of the I/O has to now pass through this before it gets to or from the real hardware.
But it's not like that. The kernel already needs to have a mapping (or several layers of mapping actually) which connects high level operations like "write this to a file" to the device drivers which in turn connect to actual blocks on disk.
When LVM is in use, that lookup is changed, but that's all. (Since it has to happen anyway, doing it a bit differently is a negligible performance hit.) When it comes to actually writing the file, the bits take as direct a path to the physical media as they would otherwise.
There are cases where LVM can cause performance problems. You want to make sure the LVM blocks are aligned properly with the underlying system, which should happen automatically with modern distributions. And make sure you're not using old kernels subject to bugs like this one. Oh, and using LVM snapshots degrades performance (and increasingly so with each active snapshot). But mostly, the impact should be very small.
As for the last: how can you test? The standard disk benchmarking tool is bonnie++. Make a partition with LVM, test it with bonnie++, wipe that out and (in the same place, to keep other factors identical) create a plain filesystem and benchmark again. They should be close to identical.
LVM, like everything else, is a mixed blessing.
With respect to performance, LVM will hinder you a little bit because it is another layer of abstraction that has to be worked out before bits hit (or can be read from) the disk. In most situations, this performance hit will be practically unmeasurable.
The advantages of LVM include the fact that you can add more storage to existing filesystems without having to move data around. Most people like it for this advantage.
One disadvantage of LVM used in this manner is that if your additional storage spans disks (ie involves more than one disk) you increase the likelyhood that a disk failure will cost you data. If your filesystem spans two disks, and either of them fails, you are probably lost. For most people, this is an acceptable risk due to space-vs-cost reasons (ie if this is really important there will be a budget to do it correctly) -- and because, as they say, backups are good, right?
For me, the single reason to not use LVM is that disaster recovery is not (or at least, was not) well defined. A disk with LVM volumes that had a scrambled OS on it could not trivially be attached to another computer and the data recovered from it; many of the instructions for recovering LVM volumes seemed to include steps like go back in time and run vgcfgbackup, then copy the resulting /etc/lvmconf file to the system hosting your hosed volume. Hopefully things have changed in the three or four years since I last had to look at this, but personally I never use LVM for this reason.
That said.
In your case, I would presume that the VMs are going to be relatively small as compared to the host system. This means to me you are more likely to want to expand storage in a VM later; this is best done by adding another virtual disk to the VM and then growing the affected VM filesystems. You don't have the spanning-multiple-disks vulnerability because the virtual disks will quite likely be on the same physical device on the host system.
If the VMs are going to have any importance to you at all, you will be RAID'ing the host system somehow, which will reduce flexibility for growing storage later. So the flexibility of LVM is probably not going to be required.
So I would presume you would not use LVM on the host system, but would install VMs to use LVM.
I have 2 partitions on my laptop--one for the system and another for my media data. I want both encrypted and without LVM, I could add a key to my system partition to automatically decrypt the second partition after entering the passphrase for the first partition. However, it still takes twice the amount of time to decrypt than if I were to just decrypt 1 partition .
-
Maybe I could make the encryption for the second partition much weaker, but I don't know if that's a security risk in this setup. Or maybe I just combine the two partitions into 1 partition? I don't know what advantages I might be losing doing this way.
-
Another scheme is to use LVM on LUKS for the entire drive, where only one decryption is necessary. I already use Btrfs whose features tend to overlap LVM's. Of course an added bonus is being able to shrink/extend encrypted logical volumes if I ever want to do that.
I think the obvious answer is to just combine the two partitions into 1 partition, but I'm more interested in hearing the pros/cons of each method just so I can understand when they might be useful in other situations.
Thanks.
I don't see any advantage in having 2 partitions. I use a setup like this, one large btrfs filesystem with subvolumes on luks. really depends on why you have 2 partitions in the first place... seems to me much more inflexible, as you have to know beforehand how big the partitions need to be. one partition, it doesn't matter, both filesets just share all the space.
However, it still takes twice the amount of time to decrypt than if I were to just decrypt 1 partition
Is this 'twice the time' something you measured? Details please.
My experience is that with modern hardware, dm_crypt is almost too fast to measure.
Using LVM in general enables several features. Just a couple:
- Extending a volume is one step and online:
lvextend --resize. - Partitioning is not required. Without LVM, the common use case of resizing root fs requires downtime and probably editing partition tables from a second system.
- Snapshots are always available, even without such a feature on the storage system or hypervisor.
- Extreme use cases may require a volume backed by more one disk (LUN). Rare these days. But might as well standardize on the more flexible LVM.
(Some of the details are specific to Linux LVM, but LVM in general is implemented on many operating systems. On UNIX, AIX and HP-UX predate Linux and have similar LVMs.)
That particular allocation resembles defaults on distros like Red Hat reflecting some recommendations they have given for a long time.
To store user data separately from system data, create a dedicated partition within a volume group for the /home directory. This will enable you to upgrade or reinstall Red Hat Enterprise Linux without erasing user data files.
Paging space could be in a more convenient file, yes. I think this requires certain file systems. Directly on a block level device works regardless of file system, and it is obvious the space is not available for regular files. Further, you could move this paging space volume to fast less durable storage like a single local SSD.
Some administrators go further and isolate say /var on its own volume and /tmp on tmpfs. That way log files and such will not fill up space required for software in /usr.
Personally, absent size requirements, I would start /home off smaller say 20 GB. That leaves 70 GB free in the VG for whatever needs the next lvextend. This also has been a suggestion for a long time. I suspect it isn't the auto partition default because user data tends to grows unbounded, and they want to use the space in case capacity planning is never revisited again.
Leave Excess Capacity Unallocated, and only assign storage capacity to those partitions you require immediately. You may allocate free space at any time, to meet needs as they occur.
In many cases a single partition, or no partitioning is not detrimental, especially in a virtualized environment where you can easily increase the primary disk and root file system if and when you run out of space.
Until you can't...
Then you find out the hard way that hard partitioning is design choice that you're stuck with for the life time of your server.
(Not completely but most solutions will require downtime and/or service interruptions.)
If you have only cattle: that is not an issue, your servers are short-lived anyway. If you have more pets than you'd care to admit though...
The performance overhead / penalty you pay for using LVM is negligible and even you can't see an immediate need now LVM will potentially save your bacon in the future when LVM will allow you to perform online changes in your storage that would otherwise be either outright impossible or would at least require (significant) downtimes.
By default, LVM2 metadata is 1 MiB per physical volume. You use
pvs -o +pe_startto find out exactly where the payload starts.Volume groups and logical volumes have no metadata overhead; only physical volumes carry metadata.
Also by default, LVM uses physical extents of 4 MiB.
The end result is that if for aesthetic reasons you want pvdisplay to show "not usable 0" then you must:
Either make each physical volume a multiple of 4 MiB + 1 MiB,
Or you create your volume groups with
-s 1m, to have physical extents of 1 MiB.
Sources:
The default alignment of the data area (reflected in the start sector of the payload area
pvs -o +pe_start) is 1 MiB in accordance with this commit. This is actually a configurable parameter, see the manual page forlvm.conf; it can of course be controlled per physical volume at physical volume creation time.The default physical extent size is documented in the manual page for
vgcreate:-s, --physicalextentsize PhysicalExtentSize[
bBsSkKmMgGtTpPeE]
Sets the physical extent size on physical volumes of this volume group. A size suffix (kfor kilobytes up totfor terabytes) is optional, megabytes is the default if no suffix is present. The value must be at least 1 sector for LVM2 format (where the sector size is the largest sector size of the PVs currently used in the VG) or 8KiB for LVM1 format and it must be a power of 2. The default is 4 MiB.
The command dmsetup table shows the start position of a LV within its PV. For my PVs on older systems the lowest offset is 384:
start cmd: # dmsetup table | grep '\b384$'
storage-images: 0 62914560 linear 9:9 384
linux0-home: 0 2097152 linear 9:0 384
On newer systems the offset is larger, though:
linuxbackup-rootfs: 0 18874368 linear 9:1 2048
This probably means that depending on the LVM version (or the distro-specific configuration) the first 384 or 2048 sectors are reserved for the PV and VG metadata. There may be metadata at the end of the PV, too (if you enforce that).
If you want to know the relevant value for you then you can simply create a VG and check the used value. But for your purpose it might be enough to just calculate with the larger value.