VDO – Virtual Data Optimizer - Compressing and Deduplicating Storage with VDO
Red Hat Enterprise Linux 8 includes the Virtual Data Optimizer (VDO) driver, which optimizes the data footprint on block devices.
VDO is a Linux device mapper driver that reduces disk space usage on block devices, and minimizes the replication of data, saving disk space and even increasing data throughput.
VDO consists of two kernel modules and two commands:
- kvdo – This module to transparently control data compression.
- uds – This module for deduplication.
- vdo- This command is used to create, remove, start, and stop VDO volumes, as well as performing other configuration changes.
- vdostats -This command is used to report on various aspects of VDO volumes, including effective reduction and physical volume utilization. Think of this as ‘df’ for VDO capacity.
The VDO layer is placed on top of an existing block storage device, such as a RAID device or a local disk. Those block devices can also be encrypted devices. The storage layers, such as LVM logical volumes and file systems are placed on top of a VDO device.
VDO config is written in /etc/vdoconf.yml
1:10 ratio for VDO logical size Virtual machines, containers
1:3 for object storage
The following diagram shows the placement of VDO in an infrastructure consisting of KVM virtual machines that are using optimized storage devices.
VDO applies three phases to data in the following order to reduce the footprint on storage devices:
1. Zero-Block Elimination filters out data blocks that contain only zeroes (0) and records the information of those blocks only in the metadata. The nonzero data blocks are then passed
to the next phase of processing. This phase enables the thin provisioning feature in the VDO devices.
2. Deduplication eliminates redundant data blocks. When you create multiple copies of the same data, VDO detects the duplicate data blocks and updates the metadata to use those
duplicate blocks as references to the original data block without creating redundant data blocks. The universal deduplication service (UDS) kernel module checks redundancy of the
data through the metadata it maintains. This kernel module ships as part of the VDO.
3. Compression is the last phase. The kvdo kernel module compresses the data blocks using LZ4 compression and groups them on 4 KB blocks.
1. Enabling VDO
Install the vdo and kmod-kvdo packages to enable VDO in the system.
If VDO services not started, then you need to start.
2. Creating a VDO Volume
To create a VDO volume, run the vdo create command.
Make sure that you have a spare disk – or at least a partition – available for use by VDO.[root@tower ~]# lsblk
Next, we create the empty VDO volume on top of /dev/nvme0n2:
Verify the availability of the vdo1 volume using the vdo list command.
Verify that the vdo1 volume has both the compression and deduplication features enabled.
Use grep to search for the lines containing the string Deduplication in the output of the vdo status --name=vdo1 command.
Use grep to search for the lines containing the string Compression in the output of the vdo status --name=vdo1 command.
Format the vdo1 volume with the XFS file-system type and mount it on /mnt/vdo1.
Format the vdo1 volume with the XFS file system using the mkfs command.
The -K option in the preceding mkfs.xfs command prevents the unused blocks in the file system from being discarded immediately which lets the command return faster.
Use the udevadm command to register the new device node.
Create the /mnt/vdo1 directory using the mkdir command.
Mount the vdo1 volume on /mnt/vdo1 using the mount command.
Permanent mount for VDO
View the initial statistics and status of the volume using the vdostats command.
To start & stop particular VDO volumes
To activate/De-activate VDO volumes
Check status of vdo
Enable / Disable De-duplication
To increase logical size of the VDO volume
To increase Physical size of the VD0 volume
Remove VDO volumes
umount -f /dev/mapper/vdo1
dmsetup remove vdo1
vdo remove --force --name=vdo1
run discard unused blocks manually