Tuesday, November 15, 2011

LLS=LXC+LVM+Snapshots

In our company developers deal with massive datasets that needs to be easy to copy, modify a fraction of it and scrape. Snapshotting is an ideal solution here. Each developer has a personal vserver (LXC container). The missing piece here is to provide them with a way to manipulate partitions and snapshots from inside of their vservers.

To enable users manipulate partitions from inside of the virtual server, I wrote LLS (LXC+LVM+Snapshots) scripts.

LXC+LVM+Snapshots = LLS

LLS system is set of scripts that enable LVM partitions and snapshots to be managed from inside of LXC vservers. It is safer to allow developers to use the scripts, instead of giving them superuser access to the physical machine.

Architecture

The LLS scripts consist from two parts. The daemon script on the host and client scripts on the vservers. The clients communicate with the server over a named pipe in a shared (bind mount) directory /lls (lls - LXC+LVM+Snapshots). The /lls directory is actually a small partition that contains configuration file, the shell scripts and the named pipe used for communcation.
The daemon script does all the necessary low-level manipulations, both on the physical machine and on the LXC vservers.
Usage

Each LLS vserver has /lls partition mounted. To preserve mounts across reboots, the /etc/rc.local file runs /lls/tools/lls_mount_on_boot.sh script.
There are several client scripts the /lls/tools partition that do various operations:
script nameOperation
lls_create_partition.shCreate a partition
lls_create_snapshot.shCreate an LVM snapshot from an existing LVM partition
lls_delete_partition.shDelete an LVM partition or snapshot
lls_list_partitions.shList available LLS partitions and refresh /dev/mf directory
lls_mount_on_boot.shMount LLS partitions using configuration in the /etc/fstab file

The scripts show informative Usage information when ran without arguments.
Developers are expected to operate the scripts by themselves. They are also expected to maintain the /etc/fstab file for mounts they want to survive reboot of their vserver. Unmounted snapshot is assumed to be not necessary anymore and might be deleted at any time.

Further work

As the LLS system is used, more features are asked by developers and system administrators. Here are some of them:
  • Track unused (unmounted) LVM snapshots and delete them automatically
  • Track disk space used/required by LVM snapshot and grow it automatically. Send email to sysadmin each time this happen.
  • Have a way to enable/disable visibility of a LLS partition. This is useful while the LLS partition is under construction.
  • Have a way to mark an LLS partition as non-mountable or mountable in read-only mode.
Download
Version 20111124.

2 comments:

Arie Skliarouk said...

Hmm, looks like the LVM snapshots slow down writes by x6...

http://www.mysqlperformanceblog.com/2009/02/05/disaster-lvm-performance-in-snapshot-mode/

Anybody knows why?

Arie Skliarouk said...

Almost 3 years later, we still use the LLS and are happy with it. Minor nuisance is that LVM snapshots are buggy and our developers learned not to store important data on snapshots.
This SF details some problems we encountered in person:
http://serverfault.com/questions/279571/lvm-dangers-and-caveats