#6 Linux LVM, RAID, caching 2015

2015-12-04, by René Rebe

In the initial LVM/RAID notes a very long time ago (in a galaxy far far away) we used the classic Linux Multiple Device software RAID setup.

However, in the meantime it is possible to setup some RAID levels directly within the LVM management to simplify and streamline the experience:

Native LVM RAID

Nowadays, we can intead let LVM manage all this by simply adding all single devices as physical volumes and combine them to a volume-group:

pvcreate /dev/sd*9
vgcreate vg0 /dev/sd*9

What is new is, that we can now set up the RAID directly inside the LVM:

lvcreate --type raid1 -L 200G -n lv-mirror vg0

Of course various RAID levels are available, too (but may require additional parametners). This not only simplifies the setup, it also makes the whole setup easier to manage, and to easily expand in the future.

Additional you can combine this with all the latest and greatest thin-pools, snapshots, and whatnot that may be great options for virutalization.

Adding SSD caching

One of the greatest and latest options is adding a cache to the logical volume, usually to improve the performance of large spinning disks. Especially seek times and thus operations per second are much higher for solid state storage. So by adding small, high performance SSD you can now enable caching for your LVM setup:

vgextend vg0 /dev/sdc9
lvcreate --type cache -L 20G -n lv-mirror-cache vg0/lv-mirror /dev/sdc9

and you should get a confirmation like:

Logical volume vg0/lv-mirror is now cached.

Behind the scene this command will create two volumes for the cache- and meta-data. However, latest lvm tools automatically manage this complexicty for you.

You can also ask for some basic statics about the utilization and hits vs. misses, like:

lvs -o lv_name,cache_read_hits,cache_read_misses,cache_write_hits,cache_write_misses

Living a dangerous life on the fast lane

By default LVM2 defaults to a saver, but slower write-through cache mode, which considers a write complete only when it has been stored in both the cache pool and on the origin LV. If your goal is to maximize performance by also hiding write latency by utilizing SSDs, you might want to setup write-back cache mode via:

lvcreate --cachemode writeback ...

which considers a write complete as soon as it is stored in the cache pool. To improve resilience, you could create the cache on a multi device, mirrosed LV.

Added bonus: Crypt the whole thing!1!

There are many reasons why you may want to add some crpyto into the mix. May it be simply your accouning and tax backup, your trade secret, running on an external root server, or all your business and prive emails and documents. With Linux, device mapper and all it's infrastrucutre it is as easy as:

cryptsetup luksFormat /dev/sda9
cryptsetup luksOpen /dev/sda9 crypt0
pvcreate /dev/mapper/crypto
...

and then setting everything up on top of /dev/mapper/crypto.

More notes

Create RAID5 with specifc devices, e.g. not on a ssd only intended as cache:

lvcreate --type raid5 -L 1T --stripes 2 -n bigdata vg0 /dev/sda2 /dev/sdb2 /dev/sdd2

Extend just created RAID5 volume to 2TB:

lvextend vg0/bigdata -L 2T

Followed by resizing the underlying file-system by something like:

btrfs filesystem resize max /mnt/bigdata

Uncaching (removing) a cache

lvconvert --uncache vg0/lv
(or, before w/ older versions: lvremove vg0/lv-cache)

One more thing: hot-adding RAID5 devices

So you need more storage and want to add another storage device to your RAID(5) array? It appears to be as simple as:

pvcreate /dev/sd-new
vgextend vg0 /dev/sd-new
lvconvert --stripes 3 vg0/bigdata

On success you should see your RAID re-syncing with a message simillar to:
md: reshape of RAID array mdX

Last but not least: replacing drives

So you got a new drive and want to migrate data before an old device failes, or simply accidently or automatically allocated space on a wrong, e.g. cache NVMe SSD? You can simply migrate data among physical volumes like:

lvconvert --replace /dev/nvme0n1p3 vg0/root /dev/sdc2

External links

The Author

René Rebe studied computer science and digital media science at the University of Applied Sciences of Berlin, Germany. He is the founder of the T2 Linux SDE (System Development Environment), and contributer to various projects in the open source landscape for more than 10 years, now. He also founded the Berlin-based software company ExactCODE GmbH. A company dedicated to exact software solutions that just work, everyday.