Linux zealot

xls2csv using python-uno

2012-02-20T02:29:00.000-08:00

While several xls2csv converters exist, on a recent assignment none of them were able to convert a multi-sheet, 300+MB Excel file into CSV format.

While searching for possible solutions, I found this (dated) blog entry on using python-uno. After assembling it into a single script and updating for changed LibreOffice arguments, I made it available here. You can check the script using the sample .XLS file with three sheets.

The power of the solution lies in the LibreOffice+UNO (Universal Network Objects) platform it uses. While xls2csv looks like a trivial task, the platform automatically supports reading all spreadsheet types LibreOffice supports. This means that the script might as well be called xlsx2csv or ods2csv.

The script can become a starting point for someone trying to implement complex documents management automation scripts. Leave a comments if you do :) .

xkb files for russian and hebrew keyboard switchers

2012-01-16T07:25:00.000-08:00

Long ago, my xkb-based keyboard switcher method stopped working. I then moved over to using gnome-keyboard-properties to set up the keyboard switching. This worked for another several years until recent ubuntu changes in oneiric that removed the stand-alone gnome-keyboard-properties. Worse yet, the gnome-control-center crashes on me if not running from gnome-session.

Being loyal user of FVWM2 window manager for 15 years (see my config over here), I dug out the xkb keyboard switching files and fixed them to work with the evdev keyboard type. This works on ubuntu oneiric (11.10) system.

Here are the download links to my xkb files:

To load them use the following command:

xkbcomp -R/usr/share/X11/xkb/ russian.xkb $DISPLAY

LLS=LXC+LVM+Snapshots

2011-11-15T07:27:00.000-08:00

In our company developers deal with massive datasets that needs to be easy to copy, modify a fraction of it and scrape. Snapshotting is an ideal solution here. Each developer has a personal vserver (LXC container). The missing piece here is to provide them with a way to manipulate partitions and snapshots from inside of their vservers.

To enable users manipulate partitions from inside of the virtual server, I wrote LLS (LXC+LVM+Snapshots) scripts.

LXC+LVM+Snapshots = LLS

LLS system is set of scripts that enable LVM partitions and snapshots to be managed from inside of LXC vservers. It is safer to allow developers to use the scripts, instead of giving them superuser access to the physical machine.

Architecture

The LLS scripts consist from two parts. The daemon script on the host and client scripts on the vservers. The clients communicate with the server over a named pipe in a shared (bind mount) directory /lls (lls - LXC+LVM+Snapshots). The /lls directory is actually a small partition that contains configuration file, the shell scripts and the named pipe used for communcation.

The daemon script does all the necessary low-level manipulations, both on the physical machine and on the LXC vservers.

Usage

Each LLS vserver has /lls partition mounted. To preserve mounts across reboots, the /etc/rc.local file runs /lls/tools/lls_mount_on_boot.sh script.

There are several client scripts the /lls/tools partition that do various operations:

script name	Operation
lls_create_partition.sh	Create a partition
lls_create_snapshot.sh	Create an LVM snapshot from an existing LVM partition
lls_delete_partition.sh	Delete an LVM partition or snapshot
lls_list_partitions.sh	List available LLS partitions and refresh /dev/mf directory
lls_mount_on_boot.sh	Mount LLS partitions using configuration in the /etc/fstab file

The scripts show informative Usage information when ran without arguments.

Developers are expected to operate the scripts by themselves. They are also expected to maintain the /etc/fstab file for mounts they want to survive reboot of their vserver. Unmounted snapshot is assumed to be not necessary anymore and might be deleted at any time.

Further work

As the LLS system is used, more features are asked by developers and system administrators. Here are some of them:

Track unused (unmounted) LVM snapshots and delete them automatically
Track disk space used/required by LVM snapshot and grow it automatically. Send email to sysadmin each time this happen.
Have a way to enable/disable visibility of a LLS partition. This is useful while the LLS partition is under construction.
Have a way to mark an LLS partition as non-mountable or mountable in read-only mode.

Download

Version 20111124.

Thoughts on photos-management system

2011-07-26T04:28:00.000-07:00

Nowadays everybody around me have digital cameras. Photos that I am interested to look at are everywhere - on facebook, flickr, picasa, people's harddisks, some photo management system, etc. Rarely someone is kind enough to send me the photos.

Let's analyze the situation.

The situation

There are mainly two distinct categories of users of a photos management system - producers and consumers. Each of them want different features from the system.

Producers

Producers have many requirements:

Editability: Producers need to operate (rotate, sort, tag, etc) on the photos. Automatic or manual face recognition would be nice to have.
Multi-user, networkable: There might be several producers (members of the family) operating on the same photos. They connect from different accounts on the same computer or from different computers (and OSes) in the same network.
Search: photos organizations must be flexible enough to support both folders and "search folders" - based on search string.
Publishing: enable consumers to view the photos over internet. The requires Sync or Upload to a web server.
Comments: enable consumers to comment on the photos, rate them, tag photos and faces.
Portability: no lock-in into some proprietary photos management system, OSS or proprietary. Practically this means that all meta data must be stored/duplicated into the photo file itself, using some meta-data format (JFIF, EXIF, IPTC, XMP, etc).

Consumers

Consumers value different features than producers:

easy and fast way to view the pics, preferably with captions (if any).
they might want to specify filter that sorts out photos with nobody they know.
ability to become producers (if allowed by host)

Current intermediate solution

Use shotwell program (I am linux user) to manage the photos on a network share. The shotwell maintains a database for quicker lookup, which is located along the photos. All changes are duplicated in metadata area of the photos as well. Publishing is done to picasa web album (it properly renders UTF8-encoded captions and tags in the IPTC header of the photos). Picasa is used as publishing point only, disregarding comments, ratings or faces tagging (blocking these if possible).

Ideal solution

The ideal solution would be in bi-directional synchronization of the photo files with some sharing-enabled cloud-based service, whereas the service embeds all additional meta-data generated by "consumers" into the files in open format. Clouds, are you listening?

How to install fb2pdf

2011-02-05T13:44:00.000-08:00

Recently I installed fb2pdf on ubuntu maverick amd64. As there is no installation instructions anywhere, I decided to write down my notes I took during the installation:

cd /var/www
svn checkout FB2PDF
mkdir /var/www/fb2pdf-read-only/logs
chown -R www-data: /var/www/fb2pdf-read-only/logs
# Edit etc/apache.conf, register with /etc/apache2/sites-enabled
apt-get install libapache2-mod-php5
a2enmod rewrite
apt-get install texlive-latex-extra texlive-humanities python-boto \
python-pytils python-imaging libapache2-mod-php5 python2.4
texlive-lang-cyrillic

apt-get install php-pear php5-dev
pear update-channels
pear install Crypt_HMAC
pear install HTTP_Request

# Mysql server
apt-get install mysql-server php5-mysql
cd /var/www/fb2pdf-read-only/src/sql
mysqladmin create fb2pdf -p
cat create_db.sql | mysql fb2pdf -p
cat create_tables.sql | mysql fb2pdf -p
cat bootstrap.sql | mysql fb2pdf -p
#mysql
GRANT ALL ON fb2pdf.* to fb2pdf@"localhost" IDENTIFIED BY "THE_PWD";

cd /var/www/fb2pdf-read-only/www
cp awscfg.php.template awscfg.php

cd /usr/share/doc/texlive-lang-cyrillic/generic/t2/etc/utf-8/
cp utf-8.def /etc/texmf/tex/latex/pict2e/
gzip -dc utfcyr.def.gz > /etc/texmf/tex/latex/pict2e/utfcyr.def
gzip -dc utflat.def.gz > /etc/texmf/tex/latex/pict2e/utflat.def

Buggy Intel RAID

2010-12-20T03:46:00.000-08:00

Recently I solved an very nasty problem in the Intel RAID controller. The problem took me and my colleagues many hours and hairs, so I decided to describe it here in hope it will help someone.

The Intel RAID controller was configured from the BIOS to have two disks in RAID1 configuration and all was good.

00:1f.2 RAID bus controller: Intel Corporation 82801 SATA RAID Controller

Some time ago we put in additional harddisk (hot-swappable SATA) that was part of some other RAID in the past (it is important!), the machine recognized it, we formatted it and started using the disk. All was good until the moment we had to reboot the machine.

It just would not boot. The screen looked like this:

The message was (transcribing for keywords):
Gave up waiting for root device.
Check rootdelay=
Check root=
ALERT does not exist. Dropping to a shell!

"dmraid -ay" would sometimes detect the root filesystem and sometimes (with different disk) not.

After a while an idea struck me - could be that the RAID controller in addition to CMOS-kept configuration tries to autodetect the RAID partitions that might exist on the new disks.

This was the answer. After I cleared both start and end of the new disks, the RAID signature on the disks would no longer confuse dmraid and prevent the kernel to find the real root file system.

dd if=/dev/zero bs=1000000 count=200 of=/dev/sdc
# For a 1TB disk. For different disk you need to calculate bs and seek accordingly.
dd if=/dev/zero bs=1000000000 seek=1000 of=/dev/sdc

If this story helped someone, write in the comments.

Fiddler on Linux

2010-12-07T07:03:00.000-08:00

A colleague of mine (web developer) recently undertook effort to move from Windows to Ubuntu. One last obstacle was lack of adequate analogs of program "Fiddler". Wireshark is too general to work with and does not allow http-specific operations (for example - change values in a request and replay it). It's cumbersome interface was also hard to work with.

Enter burp proxy suite. I know, I know, it is proprietary, closed source, cpu and memory hogging java application. But hey, if the alternative for the colleague to stay in Windows, anything counts.

As good as the program is, it remains to be proxy application. This posed a problem because the developer had to intercept http requests from a stand-alone application.

While intercepting network-originating requests is not trivial, on Linux there is simple way to redirect local traffic to the burp proxy. The trick is to run the program in "invisible proxy" mode using "root" account and redirect all traffic coming from certain user to port 80 through the proxy. This solves the catch22 problem of "intercepting all outgoing requests".

Here are the commands that accomplish that:

sudo iptables -t nat -I OUTPUT -p tcp --dport 80 -m owner \
--uid-owner evgeny -j REDIRECT --to-port 8080
sudo java -jar burpsuite_v1.3.03.jar
sudo iptables -t nat -D OUTPUT -p tcp --dport 80 -m owner \
--uid-owner evgeny -j REDIRECT --to-port 8080

An nice advantage of the burp proxy over fiddler is that developer can choose to modify the request on-the-fly. Now I have something to brag about to all fiddler-wielding windows losers :) .

mp10 DV003-XXL manual on russian

2010-08-31T02:57:00.000-07:00

Инструкция по эксплуатации "шпионского фотоаппарата/камеры" MP10 (DV-003) на русском:
http://t11.mine.nu/mp10_russian.odt

O GTalk team, where were thou?

2010-08-04T00:28:00.000-07:00

Instant messaging (IM) penetrates our lives deeper and deeper, whether we like it or not. Most people start using whatever their peers so to be able to network with them. For that reason, switching default IM is tremendously hard thing to do, especially if one already accumulated number of contacts in previous network. This is called "Network Effect".

There are several factors that can make one consider the switch - presence of their peers on the alternative network, number of functions the IM client and the network offer, easy to use IM client, availability of the client on their platform and (unfortunately often neglected) - how proprietary the network protocol is.

For past several years GTalk team was consistently failing on almost every point (at least on Linux). There is no native GTalk client for Linux. Interoperability with open-source clients is poor, especially on advanced features (voice and video conferences). No Video in standalone GTalk client. No built-in group chat support. No way to transfer bunch of contacts. File transfer is not reliable. The XMPP servers occasionally refuse to allow login (on ports 5222 and 5223). No way to organize contacts per groups (in the official GTalk client). These are problems that I have struggled with. I am sure there are many others.

Some of the problems are caused by protocol restrictions. But Google already showed that it can extend protocols by adding Jingle to XMPP. Some are caused by open-source clients. Google also showed that it can fund that as part of summer of code projects. Apparently that was not enough. Google missed great opportunity to have an viable IM network by underfunding the GTalk team and most popular open source clients. For Google's finances that would be pocket charge expenses.

With the Network Effect the Skype network has accumulated, it will be impossible to revert the situation, unless something dramatically occurs. That makes me sad as XMPP+Jingle had great potential to become the IM of choice for the world.

I speculate that Google Voice will fail for this very reason, but once Google understand that, it will probably be too late.

Hebrew thesaurus for stardict

2010-05-03T03:43:00.000-07:00

Someone asked me to find him stardict-compatible version of hebrew thesaurus. I have found torrent link that has english-hebrew, hebrew-english dictionaries and hebrew thesaurus. All files are in stardict format:
babylon_stardict.torrent

I am still looking for stardict-compatible version of russian-hebrew and hebrew-russian dictionaries. If someone has informative links, drop me an email please.

adaptec CLI management tool

2010-03-04T04:59:00.000-08:00

Recently I had to install adaptec CLI management tool on an ubuntu 8.10 amd64 server. Despite of the aacraid driver present in kernel, it took me a lot of time to find the management tools that allow to see RAID status and manipulate it from command line. There are several names for the tools: afacli, aaccli, afaapps, afa-apps-snmp, arcconf, hrconf... oh my!
After I found the necessary tools (64bit arch) I packaged them into a tiny deb package which can be downloaded from here: DELETED_AS_REDISTRIBUTION OF BINARIES_IS_PROHIBITED_BY_ADAPTEC

I wrote a simple monitoring script that checks state of the Adaptec RAID system and sends out email if any of the published indicators is wrong (battery, failed disks, etc). The script can be downloaded from here: http://t11.mine.nu/adaptec-utils/adaptec_check.sh

Note that the script requires proprietary binary files to be installed. See the beginning of the script for the details.

Audiobook as a podcast

2009-12-07T07:14:00.000-08:00

Audiobooks have several specific features that differentiate them from bunch of songs and make it hard to listen to them using regular audio player:
* They can have the same title/album/artist headers, with filename the only difference.
* They must be in certain order.
* Once an audio file was listened to, it can be automatically safely deleted, thus freeing up space.

This makes audiobooks fit nicely into podcasts model.

To convert audiobooks into a podcast I wrote a python script that scans given directory of mp3 files and creates RSS feed out of the files. The verion 0.1 of the script can be downloaded from abook2podcast.

bittorrent and VoIP in the same sentence

2009-02-16T07:57:00.000-08:00

At my home I have an modest ADSL line with a standard 128kbits upload. The upload is shared between several computers, an bittorrent process and asterisk (VoIP) server. I have looked around for an simple script to balance the upload according to set of very simple rules:

VoIP packets go out first, no matter what
ssh, www, smtp, etc go after
everything else (including bittorrent) go last

After many fruitless hours of trying I have found a sample script at http://www.knowplace.org/pages/howtos/traffic_shaping_with_linux/examples.php and then heavily modified it.
There are only two lines in the script to change. You should be aware of following assumptions in the script however:

The machine serves as the gateway for your network
Internet is established using autodialer on interface ppp0
asterisk runs on the same machine as the shaper

As a result, the VoIP is hitchless, even in presence of unlimited bittorrent upload.

You can download my script here.

my 45 workspaces

2008-11-06T00:44:00.000-08:00

Most windows managers out there are windows oriented, so that you have single desktop and many overlapping windows. This is true for MS Windows, Mac OS, Gnome and KDE. There are ways to set up several more workspaces, but most people would continue to use in windows-oriented style as all running applications would still be represented in task bar.

If you think about it, the windows-oriented workflow has great overhead, in resizing windows, looking for application in the task bar or list of applications and managing misplaced overlapping windows. Working with more than certain number of applications becomes unfeasible, as the overhead of managing so many windows becomes unbearable. There are many band-aids to minimize number of open windows, in form of tabs (especially in firefox or konsole), lame attempts to integrate many application in the same window (as mozilla thunderbird does) or hiding the application and it's interface into tiny icon in the status bar.

Ideally, one needs not to think how to switch to the application he needs to access. It would be even more desirable for the application to launched automatically if it is not running already. The desktop should be organized so, that it would not matter whether you have couple of windows opened or many hundreds (provided you have enough resources).

After experimenting with many approaches, I would like to share with you one that works good enough for me.

The venerable fvwm2 window manager, albeit being very old, remains highly configurable one. The configuration uses text configuration files, that allow user to customize look and feel of the window manager up to the low-level details, such as autofocus of windows or executing a shell script on magic keypress. The features are pretty important for the workflow described.

Easiest, immediate, switching to application of interest is by using dedicated button. Actually there are many keyboards that have additional buttons, just for that purpose. The approach has several issues, such as limited amount of buttons, predetermined functionality (psychologically aggravated by inscribed picture on the button) and dependence on the keyboard in use.

Software buttons, albeit requiring a bit of learning, look much better in this regard. If you use a combination of CTRL, ALT, SHIFT, CTRL+SHIFT, ALT+SHIFT modifier buttons together with the numeric buttons on grey part of the keyboard, you can easily get 45 "hot-keys".

Once you get 45 "hot-keys", you can create 45 workspaces, arranged in blocks of 9. Module of fvwm2 "FvwmButtons" shows them nicely in a corner of the screen:

The bright rectangles are already open windows. You can drag them from one workspace to another using mouse.

Now, as we reached the goal of instant switch to any workspace, each workspace can be dedicated to hold window (in rare cases couple of windows) of an application. For example, you can assign workspace ALT+Grey7 to a internet browser, and workspace Alt+Grey1 to a mail client. After several uses you would learn the hot-keys by heart, like you remember your password (which you can always type but not always remember).

The above scenario assumes that you launch the application in the dedicated workspace manually. Flexibility of the fvwm2 window manager allows to make each switching keypress to run a shell script, that would check whether the workspace already runs the application it was dedicated for, and, if not - automatically launch it.

This is exactly what I did as you can see in the configuration files over here.

As a added bonus, my auto-execution script checks hostname of the computer I am working on, and takes it into account when populating the workspace. For example, one of my workspaces is automatic ssh to the firewall, which is only accessible from inside of the corporate network. On my home computer, switch to the workspace would not launch anything, as it would be pointless.

Back to my linux

2008-09-02T13:20:00.000-07:00

Recently I had to set up linux for my niece, which is eight years old. Besides making the system obscenely simple (that would be subject for a separate post), I was challenged to find a way to get back to her computer, should the need arise.

As the computer will be behind a NATing router, simply opening the ports was not sufficient. There had to be a way for me to connect to her computer. Here is what I have done:

I placed on her desktop icon of rescue buoy that was launching following shell script:

#!/bin/sh
gconftool-2 -s -t bool /desktop/gnome/remote_access/enabled true &&
xterm -e ssh -R 2000:127.0.0.1:5900 feshuk-amalia@callback.mycompany.com
gconftool-2 -s -t bool /desktop/gnome/remote_access/enabled false

To make the gnome's builtin VNC server vino to accept incoming connections, I set up it's preferences and password using following commands:
vino-preferences
vino-passwd

I chose vino over x11vnc server, as it has intrinsic awareness about the windows and background (which it can effectively hide) thus being more traffc-effective, whereas x11vnc treats the whole screen as an image.

Update 20090704: Users with NVidia card might have problems with vino (#353126). One of the workarounds is to use x1vnc server using following command:
x11vnc -display :0.0

The script would make vino to accept VNC connections and execute ssh port forwarding feature, that essentially logins into the feshuk-amalia@callback.mycompany.com, starts listening on port 2000 and redirects the port to 127.0.0.1:5900 on her's local machine.

To allow password-less logon, ssh keypair was created, and the public key was copied to /home/feshuk-amalia/.ssh/authorized_keys on the computer callback.mycompany.com.

This way, should she need my help, all she needs to do is to call me by phone, double click on the "Help me" icon and wait for me. Then all I need is to connect to the port 2000 on the callback.mycompany.com machine using following command:

xvncviewer -via root@callback.mycompany.com 127.0.0.1::2000

Once I enter VNC password, I have full access to display, mouse and keyboard of her computer, to install new game or explain how to play in already installed ones.