How LVM stores data across multiple physical disks (PVs) in Linux

The data in Linux LVM (Logical Volume Manager) is stored on the underlying physical disks based on how the Logical Volume (LV) was created.

For example, if we add 4 data disks to the same Volume Group (VG) and create a Logical Volume (LV), LVM divides the space into small, fixed-size chunks called Physical Extents (PEs) on the Physical Volumes (PVs - your disks). The LV is then mapped to these PEs.

Data Distribution Types

The way the data is laid out depends on the segment type used when creating the Logical Volume (LV). By default, LVM uses a linear allocation policy.

Linear (Default):

• The Logical Volume consumes space from the Physical Volumes sequentially.
• LVM fills up the available Physical Extents on one PV (data disk) before moving to the next PV in the Volume Group.
• Data is contiguous logically, but may span multiple disks physically.
• The order in which the disks are used is typically the order in which they were added to the Volume Group (or as determined by LVM's internal allocation policy).

Striped (RAID 0):

• The Logical Volume divides data into stripes and writes them parallel across multiple Physical Volumes.
• This is done by using the --type striped or -i <number_of_stripes> option with lvcreate.
• This is primarily for performance improvement (increased I/O throughput) but means if one disk fails, all data on the LV is lost.

How to Identify Which Disk Stores the Data

To determine exactly which Physical Volume(s) a Logical Volume is using, you can use the lvdisplay command with the mapping option (-m), or the pvs command with a segments option.

1. Using lvdisplay -m (Detailed Mapping)

sudo lvdisplay -m <VolumeGroupName>/<LogicalVolumeName>

Which Logical Extents (LEs) of the LV map to which Physical Extents (PEs).

Which Physical Volume (PV) (e.g., /dev/sdb, /dev/sdc, etc.) contains those Physical Extents.

2. Using pvs --segments (Segments Report)

sudo pvs --segments -o+lv_name

The Start and SSize (Segment Size) of the allocated region in terms of Physical Extents.

LVM Data Storage Analysis

Calculation of Storage Allocation (in GB)

Storage Allocation from Each Disk = ( PE x SSize (Segment Size+1) )/ 1024

                                                         = (4 x 5120) / 1024

                                                         = 20480 /1024 = 20 GB

Let us take example for better understanding

I have one RHEL8 VM which is attached 4 data disk as shown below and configured with LVM and we will see how data is stored for /dev/vgdata/backup lv, currently the LV size is 70 GB and used only 30 GB

[root@RHEL8VM ~]# lvdisplay -m /dev/vgdata/backup

  --- Logical volume ---

  LV Path                /dev/vgdata/backup

  LV Size                70.00 GiB

  Current LE             17920

  Segments               3   ===> how many disks are segmented/data is distributed

--- Segments ---

  Logical extents 0 to 10238:

    Type                linear

    Physical volume     /dev/sdc

    Physical extents    0 to 10238 ==> (10238*4)/1024= 40 GB   ; PE Size  is 4.00 MiB

  Logical extents 10239 to 15357:

    Type                linear

    Physical volume     /dev/sde

    Physical extents    10240 to 15358   => ((15358-10240 )*4)/1024= 20 GB

  Logical extents 15358 to 17919:

    Type                linear

    Physical volume     /dev/sdb     

    Physical extents    5121 to 7682   ==> ((7682-5121 )*4)/1024= 10 GB

Which means , for the LV /dev/vgdata/backup , the data is stored in 3 DATA disks (sdc-40GB, sde-20GB and sdb-10GB).

Here, if you can see the type data distributed is linear which is by default LVM type which means , the data is first stored in /dev/sdc which is 40 GB 

Linear Allocation Flow

The linear allocation type means the data is written sequentially, filling up the first segment completely before moving to the next disk then follows.

LV Data Segment	Physical Volume (PV)	Logical Extents (LEs)	Physical Extents (PEs)	Segment Size
First Segment	/dev/sdc	0 to 10238	0 to 10238	40 GB (The first 40GB of LV)
Second Segment	/dev/sde	10239 to 15357	10240 to 15358	20 GB (The next 20GB of LV)
Third Segment	/dev/sdb	15358 to 17919	5121 to 7682	10 GB (The final 10GB of LV)

[root@RHEL8VM backup]# for i in $(seq 1 10000); do fallocate -l 4M "$i"; done

Filesystem                Type  Size  Used Avail Use% Mounted on

/dev/mapper/vgdata-backup xfs    70G   40G   30G  57% /backup

[root@RHEL8VM backup]# for i in $(seq 10001 15000); do fallocate -l 4M "$i"; done

Filesystem                Type  Size  Used Avail Use% Mounted on

/dev/mapper/vgdata-backup xfs    70G   60G   11G  85% /backup

[root@RHEL8VM backup]# for i in $(seq 15001 17000); do fallocate -l 4M "$i"; done

Filesystem                Type  Size  Used Avail Use% Mounted on

/dev/mapper/vgdata-backup xfs    70G   67G  3.1G  96% /backup

[root@RHEL8VM backup]# ls -lrt | wc -l

17001

[root@RHEL8VM backup]# file 1

1: data

[root@RHEL8VM backup]# file 10001

10001: data

[root@RHEL8VM backup]# file 15001

15001: data

I am removing 60GB data disk from backend