tag:blogger.com,1999:blog-75116405973288941292024-03-12T16:56:23.559-07:00Linux zealotArie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.comBlogger26125tag:blogger.com,1999:blog-7511640597328894129.post-86974622062193441672022-06-13T22:08:00.001-07:002022-06-13T22:09:14.373-07:00O GTalk team, where were thou? (part IV: The END)<p>It is finally happened:</p><p><br /><a href="https://www.theverge.com/2022/6/13/23166074/google-talk-messaging-xmpp-shutdown-gchat-pidgin-june-16">Google Talk is surprisingly still operational, but that ends on Thursday</a><br /><br />Took 12 years to die:<br /></p><h3 class="post-title entry-title" itemprop="name"><a href="http://skliarie.blogspot.com/2017/12/o-gtalk-team-where-were-thou-part-iii.html">O GTalk team, where were thou? (part III) (AKA: The other shoe dropped). (2017)</a><br /></h3><h3 class="post-title entry-title" itemprop="name"><a href="http://skliarie.blogspot.com/2014/09/o-gtalk-team-where-were-thou-part-ii.html">O GTalk team, where were thou? (part II) (2014)</a> </h3><h3 class="post-title entry-title" itemprop="name"><a href="http://skliarie.blogspot.com/2010/08/o-gtalk-team-where-were-thou.html"> O GTalk team, where were thou? (2010)</a><br /></h3><p></p>Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-34517054415512429452019-10-24T08:02:00.003-07:002019-10-24T08:02:52.812-07:00MYSQL/BTRFS/NVME failureIt is a very bad idea to run database (especially production one with lots of I/O) on BTRFS because the filesystem at any random time might become readonly:<br />
<blockquote class="tr_bq">
Oct 24 12:30:22 db02 kernel: BTRFS: error (device nvme0n1) in btrfs_run_delayed_refs:2936: errno=-28 No space left<br />Oct 24 12:30:22 db02 kernel: BTRFS info (device nvme0n1): forced readonly</blockquote>
<div>
And then you find that you need to do rebalance. You try and find out that rebalance can not be done because - you guessed it - there is no space left. They suggest to delete couple of snapshots though. You delete them, start rebalance and now the whole filesystem is stuck completely.</div>
<div>
<br /></div>
<div>
If you need HA mysql db with snapshots, then you should go with mysq/LVM/DRBD path, see this link for insight: https://rarforge.com/w/index.php/2_Node_Cluster:_Dual_Primary_DRBD_%2B_CLVM_%2B_KVM_%2B_Live_Migrations</div>
Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-88430891315135500702017-12-14T00:41:00.001-08:002017-12-14T00:41:34.925-08:00O GTalk team, where were thou? (part III) (AKA: The other shoe dropped).Today it happened for the first time. GTalk team, silently, without telling anyone, stopped messages sent using XMPP to be delivered to Android Hangouts clients. This caused me to miss important alert message from my monitoring system.<br />
<br />
Good bye GTalk/Hangouts, it was nice to know you.<br />
<br />
Hello telegram, the only popular and opensource API system out there!<br />
See you on tg://resolve?domain=skliarie<br />
<br />Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-29897647327360446602015-11-08T07:45:00.001-08:002018-04-25T01:21:48.160-07:00GRUB-based multiple iso booting flash driveWith huge USB flash drives of today it is sad that one can't easily put several bootable ISO images on it and have nice on-boot selection menu.<br />
<br />
GRUB to the rescue!<br />
<br />
Historically GRUB is being used for hard disks and syslinux for floppies and flash drives. But nothing prevents using GRUB for flash drives as well. Here are instructions for creating bootable GRUB-based USB flash drive (disk on key):<br />
<ol>
<li>Create vfat partition. For big drives you must use fat32 format.</li>
<li>Unpack the http://skliarie.meshanet.com/skliarie_blog/boot.tar.gz onto it. It will create single directory boot on the drive.</li>
<li>Customize boot/grub/grub.cfg file, put iso images accordingly</li>
<li>On linux box, put bootable MBR onto the DOK (for example on /dev/sdf):</li>
<ol>
<li>mount /dev/sdf /mnt/dok</li>
<li>grub-install --force --no-floppy --root-directory=/mnt/dok /dev/sdf</li>
<li>umount /mnt/dok </li>
</ol>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-tGYSBnWhVCk/VkBiUogkrbI/AAAAAAAACT8/i8BSUky64Ow/s1600/grub_menu_many_iso.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="240" src="https://2.bp.blogspot.com/-tGYSBnWhVCk/VkBiUogkrbI/AAAAAAAACT8/i8BSUky64Ow/s320/grub_menu_many_iso.jpg" width="320" /></a></div>
<br />
<ol><ol>
</ol>
</ol>
Caveats:<br />
<br />
The ISO image must support GRUB-based booting. Specifically it must be smart enough to locate ISO image on the DOK using parameters specified in grub.cfg file.<br />
<br />
Latest Ubuntu and Debian based ISO images are known to work.<br />
<br />
Thanks to <a href="http://unix.stackexchange.com/questions/102529/booting-a-knoppix-iso-from-grub-device-uri-madness">Jonathan Vollebregt</a> there is a way to boot knoppix as well. You will need custom initrd, with your flash drive specific tuning. I built one for mine: <a href="http://t11.meshanet.com/skliarie_blog/knoppix_minirt.gz">knoppix_minirt.gz</a>, you are free to take and modify it according to your DOK and filesystem parameters. Important commands here are:<br />
<ol>
<li>Unpack the initrd<br /><i>gzip -dc /mnt/dok/boot/knoppix/minirt_ska.gz | cpio -i</i></li>
<li>Modify the init file (put sector numbers that are correct for your DOK)<i> </i></li>
<li>Compress back the initrd<br />find ./ | cpio -H newc -o | gzip -9 -c > /mnt/dok/boot/knoppix/minirt_ska.gz</li>
</ol>
Please send me GRUB stanzas for other ISO images and I will put them into the <a href="http://t11.meshanet.com/skliarie_blog/grub.cfg">grub.cfg</a> file.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-21865636430079573232014-09-04T03:59:00.000-07:002014-09-04T03:59:05.439-07:00O GTalk team, where were thou? (part II)Four full years passed since my last post on <a href="http://skliarie.blogspot.co.il/2010/08/o-gtalk-team-where-were-thou.html">GTalk</a>. Unfortunately I can't say much good about Google Instant Messenger efforts.
It looks they would like everybody to switch from XMPP-based GTalk to proprietary protocol of Hangouts.<br />
<br />
We saw <a href="https://www.eff.org/deeplinks/2013/05/google-abandons-open-standards-instant-messaging">stop of XMPP federation</a>, wholesale upgrade of GTalk clients to Hangouts, last GTalk for Windows have been released<a href="http://en.wikipedia.org/wiki/Google_Talk">15 months ago</a>. Next logical step is to stop supporting XMPP altogether. What a sad day that would be..<br />
<br />
Of the handful biggest IM providers, none supports open protocol (besides Google/XMPP yet). With <a href="http://edition.cnn.com/2014/02/21/opinion/wolf-facebook-whatsapp/">19 billions valuations</a> the market is huge, and one would expect the fierce competition to leave no rock unturned in attracting more users. Unfortunately commercial interests prevail here.<br />
<br />
There never was better time for an established company to embrace open-protocol platform. May be there already is? Please post in comments.<br />
<br />
PS. Whoever is still on XMPP/Jabber/GTalk platform, uses pidgin on ubuntu 14.04 amd64 and misses "<a href="https://www.assembla.com/code/pidgin-xmpp-receipts/git/nodes/master/README">message delivery confirmation</a>" plugin - here is a bit of solace for you: <a href="http://t11.meshanet.com/skliarie_blog/pidgin-xmpp-receipts_0.6-1_amd64.deb">pidgin-xmpp-receipts_0.6-1_amd64.deb</a>
Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com2tag:blogger.com,1999:blog-7511640597328894129.post-4176386354948216842014-01-24T06:47:00.001-08:002014-01-24T06:52:35.397-08:00Lenovo IdeaPad A10 (Android) hands-on reviewBeing the "family geek" I am often asked for laptop recommendations. More often than not, functionality requirements are so low that even chromebook will be sufficient. Actually I already recommended two chromebooks and so far both continue to please their owners.<br />
<br />
Sometimes the requirements list Skype or greater autonomy as mandatory items. For such users an Android tablet should be enough. Occasional keyboard users require something more substantial. Bluetooth keyboards only complicate the picture.<br />
<br />
Since the Lenovo IdeaPad A10 (Android) appeared on the market (end of October 2013), I have been looking on the internet for a hands-on review of the device. And even now, three months later, nothing useful have shown up. This is probably related to Lenovo's refusal to market the laptop in USA (most possibly due to the shaky patents ground there). Lack of cyanogenmod posts on the device and abundance of clearly bought reviews on various blogs did not help either..<br />
<br />
Finally couple of days ago, I decided to bite a bullet and buy localized (Hebrew) Lenovo IdeaPad A10 for $310. So here is my list of pros and cons I found so far:<br />
<br />
Pros:<br />
<br />
<ul>
<li>USB charging is done by standard 2A 5v microUSB jack.</li>
<li>Touching the touchpad yields visible mouse pointer that simulates screen touches. This gives expected look and feel in laptop mode.</li>
<li>Interface is easy to use, but suffers from touch/pointer dilemma. Long story short - your hand starts aching after some time pressing buttons on vertical screen.</li>
<li>Two full-size USB connectors make it easy to connect flash-disks or mouse - this also improves look and feel in laptop mode.</li>
</ul>
Cons:<br />
<br />
<br />
<ul>
<li>There is no root jail-breaking application (at the moment). This makes it impossible to install VNC server - a must for tech support of the newbie users.</li>
<li>Lack of root access makes it impossible to mount a network share. Lack of any modders activity leaves little hope for that to change though.</li>
<li>Built-in "explorer" crashes when trying to connect to a webdav sever.</li>
<li>SFTP support in the "exporer" does not allow specifying target path.</li>
<li>The touchpad is single-figner only and can not interpret gestures - nice to have with most modern laptops (for example: two fingers down = scroll down).</li>
<li>SFTP support in the "exporer" does not allow to specify path on the server, does not show video files as icons, insist on copying the file locally first, and even that it does on ridiculously slow speed of 250 KBytes/s..</li>
<li>It is expensive for what it offers. A $150 tablet with a BT keyboard could be bought for less than that. Heck, add a bit more and you can buy weak laptop..</li>
</ul>
Verdict so far: The tablet/wannabe laptop combination is acceptable for users that are aware of its limitations and feel comfortable about it. Fortunately that was the case for me.<br />
<br />Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com8tag:blogger.com,1999:blog-7511640597328894129.post-47959252802295360382013-07-02T09:36:00.001-07:002013-07-02T09:36:41.695-07:00Ensure emails are accounted forEnsure emails are accounted for<br />
<br />
<br />
Sysadmins often set up scripts that send back "OK" or "Error" emails, but there is no mechanism to send alert if <b>*no*</b> email was received for certain time period. Lack of such emails might indicate serious system failure somewhere, that becomes obvious when it is too late to do anything. There are many stories on internet about backups that were set up properly, but for some reason were disconnected.<br />
<br />
Thus I wrote the script <a href="http://t11.meshanet.com/skliarie_blog/ensure_accounted_for.py">ensure_accounted_for.py</a> that informs sysadmins about emails that were supposed to arrive on daily basis, but for some reason did not. Its sample configuration can be taken from <a href="http://t11.meshanet.com/skliarie_blog/ensure_af.cfg">ensure_af.cfg</a>.<br />
<br />Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com2tag:blogger.com,1999:blog-7511640597328894129.post-85073363464896069992012-10-09T05:50:00.002-07:002014-02-17T04:43:47.268-08:00Multiple mysql engines on the same serverSomeone asked me for a howto on setting up multiple mysql engines on the same linux server. There are many articles on the topic, but none provides an ready-to-use set of files which one can simply grab and go.<br />
<br />
Thus I created <a href="http://t11.meshanet.com/skliarie_blog/multiple_mysql.tar.gz">multiple_mysql.tar.gz</a> file that includes all files to run four mysql engines. The configuration assumes per-database directories /opt/db-330[4567] with bin_log, mysql and relay_log directories. The mysql directory must have mysql database with associated tables (should be copied from an existing database).<br />
<br />
Note that the files were created on Ubuntu Lucid 10.04 and will likely not work on other distro. Also, the password in debian-330?.cnf files will be different for each DB and must be updated before use.<br />
Use the following command to set the password:<br />
<i>GRANT ALL ON mysql.* to "debian-sys-maint"@"localhost" IDENTIFIED BY "dxaskDkdkSkdSDLd";</i><br />
<br />
The bashrc file contains useful CLI shortcuts to common mysql commands. You are encouraged to add these to your own .bashrc file.<br />
<br />Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com2tag:blogger.com,1999:blog-7511640597328894129.post-32730693907386572982012-06-25T11:03:00.001-07:002012-06-26T04:12:46.758-07:00Laggy ssh with clear networkRecently, after reboot, my ssh sessions to remote servers started behaving sluggishly, sometimes taking several seconds for typed characters to appear. The lag was especially noticeable with last character that got typed. Nature of ssh sessions it to type several characters and checking the output. You can imagine the frustration seeing most of keypresses delivered, but getting feedback after some delay.
Ping to the server was showing reliable, speedy responses with not a single packet loss. Network was in excellent shape, no torrents or other heavy uploads. I did not know where to look or how to google for the problem.
I then remembered that I waited several months for the reboot to check an X-related crash and found that I disabled all X acceleration in the /etc/X11/xorg.conf file:
<pre>
Option "DRI" "false"
Option "shadow" "true"
Option "NoAccel" "True"
</pre>
Desperate about the whole issue, I enabled back all X acceleration options, restarted X and voila! The ssh is fast again!
Now, I don't know whether it is a bug or which subsystem it is in. So I decided to describe the problem sprinkled with keywords I used while googling for the problem in hope it helps someone. Write in comments if it did. :)Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com1tag:blogger.com,1999:blog-7511640597328894129.post-41863608070072080932012-02-23T03:34:00.000-08:002012-02-23T04:41:20.743-08:00Playlist detection on pythonThere are several popular playlists used for internet radio streaming: asx, m3u, smil, asf, pls... The playlist extension is not always visible in the URL that an ipradio station returns. Furthermore, the MIME type is not always correct. Sometimes one has no choice but to guess the playlist type by looking for the presence of certain markers.<br /><br />After fruitless searches on the internet, I wrote an python library that detects playlist type. You can download it <a href="http://t11.mine.nu/skliarie_blog/playlists_analyser.tar">here</a>.<br /><br />I am sure that the detection algorithm can be improved. Please write suggestions in the comments.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-54453071729172240032012-02-23T02:06:00.000-08:002012-02-23T04:11:54.889-08:00Easy creating and destroying of lxc vserversOne big advantage of using amazon's <a href="http://aws.amazon.com/ec2/">EC2</a> servce is its deceptively easy way to create new servers - it is just API call or mouseclick away. The actual physical machine allocation, OS image copying, hostname, IP, dns, firewall and routing setup are done transparently in the background. You are only required to setup the payment ;) .<br /><br />Big enterprise virtualization technology providers already have a mechanism to create virtual machines with given parameters on demand. Such a system would likely be overkill for a handful of physical servers. I also doubt they support the highly effective <a href="http://en.wikipedia.org/wiki/Operating_system-level_virtualization">pseudo-virtualization</a> LXC technology.<br /><br />Thus I created a python "lmachine" script for our LXC servers, that copies an OS image into a selected "slot" with already pre-allocated IP number and server name. The script also modifies a couple of system files from the OS image accordingly.<br /><br />The script can be downloaded <a href="http://t11.mine.nu/skliarie_blog/lmachine">here</a>.<br /><br /><span style="font-weight: bold;font-size:130%;" >Preparing an LXC image</span><br /><br />LXC images are easily created using the <a href="http://www.emanuelis.eu/2011/06/02/how-to-create-ubuntu-11-04-natty-narwhal-lxc-template/">debootstrap utility</a>. To make them ready for the lmachine script, you need to set the actual IP number with the IPNUMBER string and the actual server name with the VSERVERNAME string. Here is the list of files that needs to be modified:<br /><ul><li> fstab</li><li>lxc.conf</li><li>root/etc/network/interfaces</li><li>root/etc/hosts</li><li>root/etc/hostname</li><li>root/etc/mailname</li></ul>Make sure you pre-allocate IP numbers and ltestXX DNS domain names in advance. Update the lmachine script with the list of pre-allocated IP numbers.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com6tag:blogger.com,1999:blog-7511640597328894129.post-38192443608029878452012-02-20T02:29:00.000-08:002013-06-20T01:04:52.645-07:00xls2csv using python-unoWhile several xls2csv converters exist, on a recent assignment none of them were able to convert a multi-sheet, 300+MB Excel file into CSV format.<br />
<br />
While searching for possible solutions, I found <a href="http://www.logilab.org/blogentry/6130">this</a> (dated) blog entry on using python-uno. After assembling it into a single script and updating for changed LibreOffice arguments, I made it available <a href="http://t11.dyndns.org/skliarie_blog/xls2csv/xls2csv.py">here</a>. You can check the script using the <a href="http://t11.dyndns.org/skliarie_blog/xls2csv/sample3sheets.xls">sample .XLS file with three sheets</a>.<br />
<br />
The power of the solution lies in the LibreOffice+UNO (Universal Network Objects) platform it uses. While xls2csv looks like a trivial task, the platform automatically supports reading all spreadsheet types LibreOffice supports. This means that the script might as well be called xlsx2csv or ods2csv.<br />
<br />
The script can become a starting point for someone trying to implement complex documents management automation scripts. Leave a comments if you do :) .Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com6tag:blogger.com,1999:blog-7511640597328894129.post-49641816180461841712012-01-16T07:25:00.000-08:002013-08-05T09:19:56.810-07:00xkb files for russian and hebrew keyboard switchersLong ago, my xkb-based keyboard switcher method stopped working. I then moved over to using gnome-keyboard-properties to set up the keyboard switching. This worked for another several years until recent ubuntu changes in oneiric that removed the stand-alone gnome-keyboard-properties. Worse yet, the gnome-control-center crashes on me if not running from gnome-session.<br />
<br />
Being loyal user of FVWM2 window manager for 15 years (see my config <a href="http://skliarie.blogspot.com/2008/11/my-45-workplaces.html">over here</a>), I dug out the xkb keyboard switching files and fixed them to work with the evdev keyboard type. This works on ubuntu oneiric (11.10) system.<br />
<br />
Here are the download links to my xkb files:<br />
<ul>
<li><a href="http://t11.meshanet.com/skliarie_blog/russian.xkb">russian</a></li>
<li><a href="http://t11.meshanet.com/skliarie_blog/hebrew.xkb">hebrew</a></li>
</ul>
To load them use the following command:<br />
<blockquote>
</blockquote>
<blockquote>
xkbcomp -R/usr/share/X11/xkb/ russian.xkb $DISPLAY</blockquote>
<br />Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com3tag:blogger.com,1999:blog-7511640597328894129.post-2377901841689661062011-11-15T07:27:00.000-08:002016-04-14T05:15:13.707-07:00LLS=LXC+LVM+SnapshotsIn our company developers deal with massive datasets that needs to be easy to copy, modify a fraction of it and scrape. Snapshotting is an ideal solution here. Each developer has a personal vserver (LXC container). The missing piece here is to provide them with a way to manipulate partitions and snapshots from inside of their vservers.<br />
<br />
To enable users manipulate partitions from inside of the virtual server, I wrote LLS (LXC+LVM+Snapshots) scripts.<br />
<h2 style="font-weight: bold;">
<span style="font-size: 130%;"><a href="https://www.blogger.com/null" id="lxc_lvm_snapshots_lls" name="lxc_lvm_snapshots_lls">LXC+LVM+Snapshots = LLS</a></span><br /> </h2>
<div class="level1">
LLS system is set of scripts that enable LVM partitions and snapshots to be managed from inside of LXC vservers. It is safer to allow developers to use the scripts, instead of giving them superuser access to the physical machine. </div>
<div class="secedit">
<form action="/doku.php" class="button btn_secedit" method="post">
</form>
</div>
<h2 style="font-weight: bold;">
<span style="font-size: 130%;"><a href="https://www.blogger.com/null" id="architecture" name="architecture">Architecture</a></span></h2>
<div class="level2">
The LLS scripts consist from two parts. The daemon script on the host and client scripts on the vservers. The clients communicate with the server over a named pipe in a shared (bind mount) directory /lls (lls - LXC+LVM+Snapshots). The /lls directory is actually a small partition that contains configuration file, the shell scripts and the named pipe used for communcation. <br />
The daemon script does all the necessary low-level manipulations, both on the physical machine and on the LXC vservers. </div>
<span style="font-size: 130%; font-weight: bold;"><a href="https://www.blogger.com/null" id="usage" name="usage">Usage</a></span><br />
<br />
Each LLS vserver has /lls partition mounted. To preserve mounts across reboots, the /etc/rc.local file runs /lls/tools/lls_mount_on_boot.sh script. <br />
There are several client scripts the /lls/tools partition that do various operations: <br />
<table border="1"><tbody>
<tr><th>script name</th><th>Operation</th></tr>
<tr><td>lls_create_partition.sh</td><td>Create a partition</td> </tr>
<tr> <td>lls_create_snapshot.sh</td><td>Create an LVM snapshot from an existing LVM partition</td> </tr>
<tr> <td>lls_delete_partition.sh</td><td>Delete an LVM partition or snapshot</td> </tr>
<tr> <td>lls_list_partitions.sh</td><td>List available LLS partitions and refresh /dev/mf directory</td> </tr>
<tr> <td>lls_mount_on_boot.sh</td><td>Mount LLS partitions using configuration in the /etc/fstab file</td></tr>
</tbody></table>
<br />
The scripts show informative Usage information when ran without arguments. <br />
Developers are expected to operate the scripts by themselves. They are also expected to maintain the /etc/fstab file for mounts they want to survive reboot of their vserver. Unmounted snapshot is assumed to be not necessary anymore and might be deleted at any time. <br />
<div class="secedit">
<form action="/doku.php" class="button btn_secedit" method="post">
</form>
</div>
<h2 style="font-weight: bold;">
<span style="font-size: 130%;"><a href="https://www.blogger.com/null" id="further_work" name="further_work">Further work</a></span></h2>
<div class="level2">
As the LLS system is used, more features are asked by developers and system administrators. Here are some of them: <br />
<ul>
<li class="level1"><div class="li">
Track unused (unmounted) LVM snapshots and delete them automatically</div>
</li>
<li class="level1"><div class="li">
Track disk space used/required by LVM snapshot and grow it automatically. Send email to sysadmin each time this happen.</div>
</li>
<li class="level1"><div class="li">
Have a way to enable/disable visibility of a LLS partition. This is useful while the LLS partition is under construction.</div>
</li>
<li class="level1"><div class="li">
Have a way to mark an LLS partition as non-mountable or mountable in read-only mode.</div>
</li>
</ul>
<div style="font-weight: bold;">
Download</div>
</div>
<div class="secedit">
<form action="/doku.php" class="button btn_secedit" method="post">
</form>
</div>
<div class="clearer">
Version <a href="http://arnest.catchmedia.com/lls_20111124.tar.gz">20111124</a>.<br />
<br /></div>
Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com2tag:blogger.com,1999:blog-7511640597328894129.post-72024324130860600092011-07-26T04:28:00.000-07:002011-11-15T05:57:15.447-08:00Thoughts on photos-management systemNowadays everybody around me have digital cameras. Photos that I am interested to look at are everywhere - on facebook, flickr, picasa, people's harddisks, some photo management system, etc. Rarely someone is kind enough to send me the photos.<br /><br />Let's analyze the situation.<br /><br /><span style="font-size:130%;"><span style="font-weight: bold;">The situation</span></span><br /><br />There are mainly two distinct categories of users of a photos management system - producers and consumers. Each of them want different features from the system.<br /><br /><span style="font-weight: bold;font-size:130%;" >Producers</span><br /><br />Producers have many requirements:<br /><ul><li>Editability: Producers need to operate (rotate, sort, tag, etc) on the photos. Automatic or manual face recognition would be nice to have.</li><li>Multi-user, networkable: There might be several producers (members of the family) operating on the same photos. They connect from different accounts on the same computer or from different computers (and OSes) in the same network.<br /></li><li>Search: photos organizations must be flexible enough to support both folders and "search folders" - based on search string.<br /></li><li>Publishing: enable consumers to view the photos over internet. The requires Sync or Upload to a web server.</li><li>Comments: enable consumers to comment on the photos, rate them, tag photos and faces.</li><li>Portability: no lock-in into some proprietary photos management system, OSS or proprietary. Practically this means that all meta data must be stored/duplicated into the photo file itself, using some meta-data format (<span class="st">JFIF, EXIF, <em>IPTC</em>, XMP</span>, etc).<br /></li></ul><span style="font-weight: bold;font-size:130%;" >Consumers</span><br /><br />Consumers value different features than producers:<br /><ul><li>easy and fast way to view the pics, preferably with captions (if any).</li><li>they might want to specify filter that sorts out photos with nobody they know.</li><li>ability to become producers (if allowed by host)</li></ul><span style="font-weight: bold;">Current intermediate solution</span><br /><br />Use shotwell program (I am linux user) to manage the photos on a network share. The shotwell maintains a database for quicker lookup, which is located along the photos. All changes are duplicated in metadata area of the photos as well. Publishing is done to picasa web album (it properly renders UTF8-encoded captions and tags in the IPTC header of the photos). Picasa is used as publishing point only, disregarding comments, ratings or faces tagging (blocking these if possible).<br /><br /><span style="font-weight: bold;">Ideal solution</span><br /><br />The ideal solution would be in bi-directional synchronization of the photo files with some sharing-enabled cloud-based service, whereas the service embeds all additional meta-data generated by "consumers" into the files in open format. Clouds, are you listening?Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-55821663404578378362011-02-05T13:44:00.000-08:002011-02-05T13:49:37.893-08:00How to install fb2pdfRecently I installed fb2pdf on ubuntu maverick amd64. As there is no installation instructions anywhere, I decided to write down my notes I took during the installation:<br /><blockquote>cd /var/www<br />svn checkout FB2PDF<br />mkdir /var/www/fb2pdf-read-only/logs<br />chown -R www-data: /var/www/fb2pdf-read-only/logs<br /># Edit etc/apache.conf, register with /etc/apache2/sites-enabled<br />apt-get install libapache2-mod-php5<br />a2enmod rewrite<br />apt-get install texlive-latex-extra texlive-humanities python-boto \<br /> python-pytils python-imaging libapache2-mod-php5 python2.4<br /> texlive-lang-cyrillic<br /><br />apt-get install php-pear php5-dev<br />pear update-channels<br />pear install Crypt_HMAC<br />pear install HTTP_Request<br /><br /># Mysql server<br />apt-get install mysql-server php5-mysql<br />cd /var/www/fb2pdf-read-only/src/sql<br />mysqladmin create fb2pdf -p<br />cat create_db.sql | mysql fb2pdf -p<br />cat create_tables.sql | mysql fb2pdf -p<br />cat bootstrap.sql | mysql fb2pdf -p<br />#mysql<br />GRANT ALL ON fb2pdf.* to fb2pdf@"localhost" IDENTIFIED BY "THE_PWD";<br /><br />cd /var/www/fb2pdf-read-only/www<br />cp awscfg.php.template awscfg.php<br /><br />cd /usr/share/doc/texlive-lang-cyrillic/generic/t2/etc/utf-8/<br />cp utf-8.def /etc/texmf/tex/latex/pict2e/<br />gzip -dc utfcyr.def.gz > /etc/texmf/tex/latex/pict2e/utfcyr.def<br />gzip -dc utflat.def.gz > /etc/texmf/tex/latex/pict2e/utflat.def<br /></blockquote>Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-34633938185180622322010-12-20T03:46:00.000-08:002010-12-20T04:05:35.011-08:00Buggy Intel RAIDRecently I solved an very nasty problem in the Intel RAID controller. The problem took me and my colleagues many hours and hairs, so I decided to describe it here in hope it will help someone.<br /><br />The Intel RAID controller was configured from the BIOS to have two disks in RAID1 configuration and all was good.<br /><blockquote>00:1f.2 RAID bus controller: Intel Corporation 82801 SATA RAID Controller</blockquote>Some time ago we put in additional harddisk (hot-swappable SATA) that was part of some other RAID in the past (it is important!), the machine recognized it, we formatted it and started using the disk. All was good until the moment we had to reboot the machine.<br /><br />It just would not boot. The screen looked like this:<br /><div style="text-align: center;"><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_VbzJ1FZE72w/TQ9ErhC609I/AAAAAAAAAng/_jMm16tOJM8/s1600/stuck_on_boot.png"><img style="float: right; margin: 0pt 0pt 10px 10px; cursor: pointer; width: 320px; height: 208px;" src="http://2.bp.blogspot.com/_VbzJ1FZE72w/TQ9ErhC609I/AAAAAAAAAng/_jMm16tOJM8/s320/stuck_on_boot.png" alt="" id="BLOGGER_PHOTO_ID_5552732379966788562" border="0" /></a></div><br />The message was (transcribing for keywords):<br />Gave up waiting for root device.<br />Check rootdelay=<br />Check root=<br />ALERT does not exist. Dropping to a shell!<br /><br />"dmraid -ay" would sometimes detect the root filesystem and sometimes (with different disk) not.<br /><br />After a while an idea struck me - could be that the RAID controller in addition to CMOS-kept configuration tries to autodetect the RAID partitions that might exist on the new disks.<br /><br />This was the answer. After I cleared both start and end of the new disks, the RAID signature on the disks would no longer confuse dmraid and prevent the kernel to find the real root file system.<br /><blockquote>dd if=/dev/zero bs=1000000 count=200 of=/dev/sdc<br /># For a 1TB disk. For different disk you need to calculate bs and seek accordingly.<br />dd if=/dev/zero bs=1000000000 seek=1000 of=/dev/sdc<br /></blockquote>If this story helped someone, write in the comments.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-10946954064895395392010-12-07T07:03:00.000-08:002010-12-07T08:34:22.802-08:00Fiddler on LinuxA colleague of mine (web developer) recently undertook effort to move from Windows to Ubuntu. One last obstacle was lack of adequate analogs of program "<a href="http://www.fiddler2.com/fiddler2/">Fiddler</a>". Wireshark is too general to work with and does not allow http-specific operations (for example - change values in a request and replay it). It's cumbersome interface was also hard to work with.<br /><br />Enter <a href="http://portswigger.net/burp/proxy.html">burp proxy</a> suite. I know, I know, it is proprietary, closed source, cpu and memory hogging java application. But hey, if the alternative for the colleague to stay in Windows, anything counts.<br /><br />As good as the program is, it remains to be proxy application. This posed a problem because the developer had to intercept http requests from a stand-alone application.<br /><br />While intercepting network-originating requests is not trivial, on Linux there is simple way to redirect local traffic to the burp proxy. The trick is to run the program in "invisible proxy" mode using "root" account and redirect all traffic coming from certain user to port 80 through the proxy. This solves the catch22 problem of "intercepting all outgoing requests".<br /><br />Here are the commands that accomplish that:<br /><pre><span style="font-style: italic;">sudo iptables -t nat -I OUTPUT -p tcp --dport 80 -m owner \<br />--uid-owner evgeny -j REDIRECT --to-port 8080</span><br /><span style="font-style: italic;">sudo java -jar burpsuite_v1.3.03.jar</span><br /><span style="font-style: italic;">sudo iptables -t nat -D OUTPUT -p tcp --dport 80 -m owner \<br />--uid-owner evgeny -j REDIRECT --to-port 8080</span><br /><br /></pre>An nice advantage of the burp proxy over fiddler is that developer can choose to modify the request on-the-fly. Now I have something to brag about to all fiddler-wielding windows losers :) .Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com2tag:blogger.com,1999:blog-7511640597328894129.post-54491931639699672462010-08-31T02:57:00.000-07:002018-04-25T01:18:39.246-07:00mp10 DV003-XXL manual on russianИнструкция по эксплуатации "шпионского фотоаппарата/камеры" MP10 (DV-003) на русском:<br />
http://skliarie.meshanet.com/skliarie_blog/mp10_russian.odt<br />
<br />Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-43543412394387733092010-08-04T00:28:00.000-07:002010-08-04T01:24:27.742-07:00O GTalk team, where were thou?Instant messaging (IM) penetrates our lives deeper and deeper, whether we like it or not. Most people start using whatever their peers so to be able to network with them. For that reason, switching default IM is tremendously hard thing to do, especially if one already accumulated number of contacts in previous network. This is called "<a href="http://en.wikipedia.org/wiki/Network_effect">Network Effect"</a>.<br /><br />There are several factors that can make one consider the switch - presence of their peers on the alternative network, number of functions the IM client and the network offer, easy to use IM client, availability of the client on their platform and (unfortunately often neglected) - how proprietary the network protocol is.<br /><br />For past several years GTalk team was consistently failing on almost every point (at least on Linux). There is no native GTalk client for Linux. Interoperability with open-source clients is poor, especially on advanced features (voice and video conferences). No Video in standalone GTalk client. No built-in group chat support. No way to transfer bunch of contacts. File transfer is not reliable. The XMPP servers occasionally refuse to allow login (on ports 5222 and 5223). No way to organize contacts per groups (in the official GTalk client). These are problems that I have struggled with. I am sure there are many others.<br /><br />Some of the problems are caused by protocol restrictions. But Google already showed that it can extend protocols by adding <a href="http://en.wikipedia.org/wiki/Jingle_%28protocol%29">Jingle</a> to XMPP. Some are caused by open-source clients. Google also showed that it can fund that as part of <a href="http://developer.pidgin.im/wiki/GSoC2008/VoiceAndVideo">summer of code</a> projects. Apparently that was not enough. Google missed great opportunity to have an viable IM network by underfunding the GTalk team and most popular open source clients. For Google's finances that would be pocket charge expenses.<br /><br />With the Network Effect the Skype network has accumulated, it will be impossible to revert the situation, unless something dramatically occurs. That makes me sad as XMPP+Jingle had great potential to become the IM of choice for the world.<br /><br />I speculate that Google Voice will fail for this very reason, but once Google understand that, it will probably be too late.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-87612929954657199952010-05-03T03:43:00.000-07:002010-05-04T01:36:42.798-07:00Hebrew thesaurus for stardictSomeone asked me to find him stardict-compatible version of hebrew thesaurus. I have found torrent link that has english-hebrew, hebrew-english dictionaries and hebrew thesaurus. All files are in stardict format:<br /><a href="http://t11.mine.nu/babylon_stardict.torrent">babylon_stardict.torrent</a><br /><br />I am still looking for stardict-compatible version of russian-hebrew and hebrew-russian dictionaries. If someone has informative links, drop me an email please.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-16306526556499371652010-03-04T04:59:00.000-08:002018-04-25T01:20:44.213-07:00adaptec CLI management toolRecently I had to install adaptec CLI management tool on an ubuntu 8.10 amd64 server. Despite the fact that aacraid driver is present in kernel, it took me a lot of time to find the management tools that allow to see the RAID status and manipulate it from command line. There are several names for the tools: afacli, aaccli, afaapps, afa-apps-snmp, arcconf, hrconf... oh my!<br />
After I found the necessary tools (64bit arch), I packaged them into a tiny deb package adaptec-utils_0.0.1_amd64.deb, which can be downloaded from here: DELETED_AS_REDISTRIBUTION OF BINARIES_IS_PROHIBITED_BY_ADAPTEC<br />
<br />
Also, I wrote a simple monitoring script (included in the deb package) that checks the state of the Adaptec RAID system and sends out an email if any of the published indicators is wrong (battery, failed disks, etc). The script can be downloaded from here:<br />
http://skliarie.meshanet.com/skliarie_blog/adaptec-utils/adaptec_check.sh<br />
<br />
Note that the script requires proprietary binary files to be installed. See the beginning of the script for the details.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com9tag:blogger.com,1999:blog-7511640597328894129.post-18635441570578613422009-12-07T07:14:00.000-08:002009-12-07T07:50:28.438-08:00Audiobook as a podcastAudiobooks have several specific features that differentiate them from bunch of songs and make it hard to listen to them using regular audio player:<br />* They can have the same title/album/artist headers, with filename the only difference.<br />* They must be in certain order.<br />* Once an audio file was listened to, it can be automatically safely deleted, thus freeing up space.<br /><br />This makes audiobooks fit nicely into podcasts model.<br /><br />To convert audiobooks into a podcast I wrote a python script that scans given directory of mp3 files and creates RSS feed out of the files. The verion 0.1 of the script can be downloaded from <span style="font-weight:bold;"><a href="http://t11.mine.nu/abook2podcast.py">abook2podcast</a></span>.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-52443939173549475792009-02-16T07:57:00.001-08:002012-03-14T03:27:23.300-07:00bittorrent and VoIP in the same sentenceAt my home I have an modest ADSL line with a standard 128kbits upload. The upload is shared between several computers, an bittorrent process and asterisk (VoIP) server. I have looked around for an simple script to balance the upload according to set of very simple rules:<br /><ul><li>VoIP packets go out first, no matter what</li><li>ssh, www, smtp, etc go after</li><li>everything else (including bittorrent) go last</li></ul>After many fruitless hours of trying I have found a sample script at http://www.knowplace.org/pages/howtos/traffic_shaping_with_linux/examples.php and then heavily modified it.<br />There are only two lines in the script to change. You should be aware of following assumptions in the script however:<br /><ul><li>The machine serves as the gateway for your network<br /></li><li>Internet is established using autodialer on interface ppp0</li><li>asterisk runs on the same machine as the shaper</li></ul><span style="font-weight: bold; color: rgb(0, 0, 0);font-size:100%;" >As a result, the VoIP is hitchless, even in presence of unlimited bittorrent upload.</span><br /><br />You can <span style="font-size:180%;">download</span> my script <span style="font-size:180%;"><a href="http://t11.mine.nu/skliarie_blog/shaper_0_2.tar.gz">here</a></span>.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com0tag:blogger.com,1999:blog-7511640597328894129.post-28980337966526613382008-11-06T00:44:00.001-08:002012-03-14T03:31:49.427-07:00my 45 workspacesMost windows managers out there are windows oriented, so that you have single desktop and many overlapping windows. This is true for MS Windows, Mac OS, Gnome and KDE. There are ways to set up several more workspaces, but most people would continue to use in windows-oriented style as all running applications would still be represented in task bar.<br /><br />If you think about it, the windows-oriented workflow has great overhead, in resizing windows, looking for application in the task bar or list of applications and managing misplaced overlapping windows. Working with more than certain number of applications becomes unfeasible, as the overhead of managing so many windows becomes unbearable. There are many band-aids to minimize number of open windows, in form of tabs (especially in firefox or konsole), lame attempts to integrate many application in the same window (as mozilla thunderbird does) or hiding the application and it's interface into tiny icon in the status bar.<br /><br />Ideally, one needs not to think how to switch to the application he needs to access. It would be even more desirable for the application to launched automatically if it is not running already. The desktop should be organized so, that it would not matter whether you have couple of windows opened or many hundreds (provided you have enough resources).<br /><br />After experimenting with many approaches, I would like to share with you one that works good enough for me.<br /><br />The venerable fvwm2 window manager, albeit being very old, remains highly configurable one. The configuration uses text configuration files, that allow user to customize look and feel of the window manager up to the low-level details, such as autofocus of windows or executing a shell script on magic keypress. The features are pretty important for the workflow described.<br /><br />Easiest, immediate, switching to application of interest is by using dedicated button. Actually there are many keyboards that have additional buttons, just for that purpose. The approach has several issues, such as limited amount of buttons, predetermined functionality (psychologically aggravated by inscribed picture on the button) and dependence on the keyboard in use.<br /><br />Software buttons, albeit requiring a bit of learning, look much better in this regard. If you use a combination of CTRL, ALT, SHIFT, CTRL+SHIFT, ALT+SHIFT modifier buttons together with the numeric buttons on grey part of the keyboard, you can easily get 45 "hot-keys".<br /><br />Once you get 45 "hot-keys", you can create 45 workspaces, arranged in blocks of 9. Module of fvwm2 "FvwmButtons" shows them nicely in a corner of the screen:<br /><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 285px; height: 44px;" src="http://3.bp.blogspot.com/_VbzJ1FZE72w/SRLDTcxlnZI/AAAAAAAAAFQ/97pHYARyCdc/s320/workplaces.png" alt="" id="BLOGGER_PHOTO_ID_5265485653258640786" border="0" /><br />The bright rectangles are already open windows. You can drag them from one workspace to another using mouse.<br /><br />Now, as we reached the goal of instant switch to any workspace, each workspace can be dedicated to hold window (in rare cases couple of windows) of an application. For example, you can assign workspace ALT+Grey7 to a internet browser, and workspace Alt+Grey1 to a mail client. After several uses you would learn the hot-keys by heart, like you remember your password (which you can always type but not always remember).<br /><br />The above scenario assumes that you launch the application in the dedicated workspace manually. Flexibility of the fvwm2 window manager allows to make each switching keypress to run a shell script, that would check whether the workspace already runs the application it was dedicated for, and, if not - automatically launch it.<br /><br />This is exactly what I did as you can see in the configuration files over <a href="http://t11.mine.nu/skliarie_blog/fvwm_45_workspaces_autostart.tar.gz">here</a>.<br /><br />As a added bonus, my auto-execution script checks hostname of the computer I am working on, and takes it into account when populating the workspace. For example, one of my workspaces is automatic ssh to the firewall, which is only accessible from inside of the corporate network. On my home computer, switch to the workspace would not launch anything, as it would be pointless.Arie Skliaroukhttp://www.blogger.com/profile/17337009271103751185noreply@blogger.com3