LLS=LXC+LVM+Snapshots

In our company developers deal with massive datasets that needs to be easy to copy, modify a fraction of it and scrape. Snapshotting is an ideal solution here. Each developer has a personal vserver (LXC container). The missing piece here is to provide them with a way to manipulate partitions and snapshots from inside of their vservers.

To enable users manipulate partitions from inside of the virtual server, I wrote LLS (LXC+LVM+Snapshots) scripts.

LXC+LVM+Snapshots = LLS

LLS system is set of scripts that enable LVM partitions and snapshots to be managed from inside of LXC vservers. It is safer to allow developers to use the scripts, instead of giving them superuser access to the physical machine.

Architecture

The LLS scripts consist from two parts. The daemon script on the host and client scripts on the vservers. The clients communicate with the server over a named pipe in a shared (bind mount) directory /lls (lls - LXC+LVM+Snapshots). The /lls directory is actually a small partition that contains configuration file, the shell scripts and the named pipe used for communcation.
The daemon script does all the necessary low-level manipulations, both on the physical machine and on the LXC vservers.

Usage

Each LLS vserver has /lls partition mounted. To preserve mounts across reboots, the /etc/rc.local file runs /lls/tools/lls_mount_on_boot.sh script.
There are several client scripts the /lls/tools partition that do various operations:

script name	Operation
lls_create_partition.sh	Create a partition
lls_create_snapshot.sh	Create an LVM snapshot from an existing LVM partition
lls_delete_partition.sh	Delete an LVM partition or snapshot
lls_list_partitions.sh	List available LLS partitions and refresh /dev/mf directory
lls_mount_on_boot.sh	Mount LLS partitions using configuration in the /etc/fstab file

The scripts show informative Usage information when ran without arguments.
Developers are expected to operate the scripts by themselves. They are also expected to maintain the /etc/fstab file for mounts they want to survive reboot of their vserver. Unmounted snapshot is assumed to be not necessary anymore and might be deleted at any time.

Further work

As the LLS system is used, more features are asked by developers and system administrators. Here are some of them:

• Track unused (unmounted) LVM snapshots and delete them automatically
• Track disk space used/required by LVM snapshot and grow it automatically. Send email to sysadmin each time this happen.
• Have a way to enable/disable visibility of a LLS partition. This is useful while the LLS partition is under construction.
• Have a way to mark an LLS partition as non-mountable or mountable in read-only mode.

Download

Version 20111124.

Thoughts on photos-management system

Nowadays everybody around me have digital cameras. Photos that I am interested to look at are everywhere - on facebook, flickr, picasa, people's harddisks, some photo management system, etc. Rarely someone is kind enough to send me the photos.

Let's analyze the situation.

The situation

There are mainly two distinct categories of users of a photos management system - producers and consumers. Each of them want different features from the system.

Producers

Producers have many requirements:

• Editability: Producers need to operate (rotate, sort, tag, etc) on the photos. Automatic or manual face recognition would be nice to have.
• Multi-user, networkable: There might be several producers (members of the family) operating on the same photos. They connect from different accounts on the same computer or from different computers (and OSes) in the same network.
  
• Search: photos organizations must be flexible enough to support both folders and "search folders" - based on search string.
  
• Publishing: enable consumers to view the photos over internet. The requires Sync or Upload to a web server.
• Comments: enable consumers to comment on the photos, rate them, tag photos and faces.
• Portability: no lock-in into some proprietary photos management system, OSS or proprietary. Practically this means that all meta data must be stored/duplicated into the photo file itself, using some meta-data format (JFIF, EXIF, IPTC, XMP, etc).
  

Consumers

Consumers value different features than producers:

• easy and fast way to view the pics, preferably with captions (if any).
• they might want to specify filter that sorts out photos with nobody they know.
• ability to become producers (if allowed by host)

Current intermediate solution

Use shotwell program (I am linux user) to manage the photos on a network share. The shotwell maintains a database for quicker lookup, which is located along the photos. All changes are duplicated in metadata area of the photos as well. Publishing is done to picasa web album (it properly renders UTF8-encoded captions and tags in the IPTC header of the photos). Picasa is used as publishing point only, disregarding comments, ratings or faces tagging (blocking these if possible).

Ideal solution

The ideal solution would be in bi-directional synchronization of the photo files with some sharing-enabled cloud-based service, whereas the service embeds all additional meta-data generated by "consumers" into the files in open format. Clouds, are you listening?

How to install fb2pdf

Recently I installed fb2pdf on ubuntu maverick amd64. As there is no installation instructions anywhere, I decided to write down my notes I took during the installation:

cd /var/www
svn checkout FB2PDF
mkdir /var/www/fb2pdf-read-only/logs
chown -R www-data: /var/www/fb2pdf-read-only/logs
# Edit etc/apache.conf, register with /etc/apache2/sites-enabled
apt-get install libapache2-mod-php5
a2enmod rewrite
apt-get install texlive-latex-extra texlive-humanities python-boto \
python-pytils python-imaging libapache2-mod-php5 python2.4
texlive-lang-cyrillic

apt-get install php-pear php5-dev
pear update-channels
pear install Crypt_HMAC
pear install HTTP_Request

# Mysql server
apt-get install mysql-server php5-mysql
cd /var/www/fb2pdf-read-only/src/sql
mysqladmin create fb2pdf -p
cat create_db.sql | mysql fb2pdf -p
cat create_tables.sql | mysql fb2pdf -p
cat bootstrap.sql | mysql fb2pdf -p
#mysql
GRANT ALL ON fb2pdf.* to fb2pdf@"localhost" IDENTIFIED BY "THE_PWD";

cd /var/www/fb2pdf-read-only/www
cp awscfg.php.template awscfg.php

cd /usr/share/doc/texlive-lang-cyrillic/generic/t2/etc/utf-8/
cp utf-8.def /etc/texmf/tex/latex/pict2e/
gzip -dc utfcyr.def.gz > /etc/texmf/tex/latex/pict2e/utfcyr.def
gzip -dc utflat.def.gz > /etc/texmf/tex/latex/pict2e/utflat.def