F1x - wiki for devops

By definition devops role requires 24/7 on-demand attention. With separate NOC team to execute SOPs and track incidents. Single devops is nonsense. A company that has single devops is a recipe for disaster as there is no redundancy for that fragile and pesky human resource.

With more than one devops, knowledge sharing becomes the issue. And without further ado, let me introduce F1x - how we handle it in our company:

F1x

Have you ever wondered about possibility to press F1 and have a way to scribble notes about the very thing you are working on? And make these notes available to other members of your team? Popping up into their face when they happen to start working onto the same thing? And they can edit them too!

Introducing F1x - "F1+eXtra help button everywhere":

F1x is "F1 help button everywhere" system that recognizes user activity (context) and maps contexts and corresponding wiki pages.

At this stage it is able to track contexts of users doing ssh to servers and changing current path into a directory. But it already gives powerful notes-leaving mechanism:

Imagine you are trying to understand some weird functionality and ssh into the server. A popup shows up with a note that there is F1x note. You press F1 and browser is opened to the F1x wiki page, that is exactly about the server you are working on. Even more, when you chdir into some directory, and there is a note about it, you will get popup as well.

Here is short video demo of the F1x in action:

More info and code is on the gitlab project over here:
https://gitlab.com/skliarie/f1x

O GTalk team, where were thou? (part IV: The END)

It is finally happened:

Google Talk is surprisingly still operational, but that ends on Thursday

Took 12 years to die:

O GTalk team, where were thou? (part III) (AKA: The other shoe dropped). (2017)

O GTalk team, where were thou? (part II) (2014)

O GTalk team, where were thou? (2010)

MYSQL/BTRFS/NVME failure

It is a very bad idea to run database (especially production one with lots of I/O) on BTRFS because the filesystem at any random time might become readonly:

Oct 24 12:30:22 db02 kernel: BTRFS: error (device nvme0n1) in btrfs_run_delayed_refs:2936: errno=-28 No space left
Oct 24 12:30:22 db02 kernel: BTRFS info (device nvme0n1): forced readonly

And then you find that you need to do rebalance. You try and find out that rebalance can not be done because - you guessed it - there is no space left. They suggest to delete couple of snapshots though. You delete them, start rebalance and now the whole filesystem is stuck completely.

If you need HA mysql db with snapshots, then you should go with mysq/LVM/DRBD path, see this link for insight: https://rarforge.com/w/index.php/2_Node_Cluster:_Dual_Primary_DRBD_%2B_CLVM_%2B_KVM_%2B_Live_Migrations

O GTalk team, where were thou? (part III) (AKA: The other shoe dropped).

Today it happened for the first time. GTalk team, silently, without telling anyone, stopped messages sent using XMPP to be delivered to Android Hangouts clients. This caused me to miss important alert message from my monitoring system.

Good bye GTalk/Hangouts, it was nice to know you.

Hello telegram, the only popular and opensource API system out there!
See you on tg://resolve?domain=skliarie

GRUB-based multiple iso booting flash drive

With huge USB flash drives of today it is sad that one can't easily put several bootable ISO images on it and have nice on-boot selection menu.

GRUB to the rescue!

Historically GRUB is being used for hard disks and syslinux for floppies and flash drives. But nothing prevents using GRUB for flash drives as well. Here are instructions for creating bootable GRUB-based USB flash drive (disk on key):

1. Create vfat partition. For big drives you must use fat32 format.
2. Unpack the http://skliarie.meshanet.com/skliarie_blog/boot.tar.gz onto it. It will create single directory boot on the drive.
3. Customize boot/grub/grub.cfg file, put iso images accordingly
4. On linux box, put bootable MBR onto the DOK (for example on /dev/sdf):
1. mount /dev/sdf /mnt/dok
2. grub-install --force --no-floppy --root-directory=/mnt/dok /dev/sdf
3. umount /mnt/dok 

Caveats:

The ISO image must support GRUB-based booting. Specifically it must be smart enough to locate ISO image on the DOK using parameters specified in grub.cfg file.

Latest Ubuntu and Debian based ISO images are known to work.

Thanks to Jonathan Vollebregt there is a way to boot knoppix as well. You will need custom initrd, with your flash drive specific tuning. I built one for mine: knoppix_minirt.gz, you are free to take and modify it according to your DOK and filesystem parameters. Important commands here are:

1. Unpack the initrd
   gzip -dc /mnt/dok/boot/knoppix/minirt_ska.gz | cpio -i
2. Modify the init file (put sector numbers that are correct for your DOK)
3. Compress back the initrd
   find ./ | cpio -H newc -o | gzip -9 -c > /mnt/dok/boot/knoppix/minirt_ska.gz

Please send me GRUB stanzas for other ISO images and I will put them into the grub.cfg file.

O GTalk team, where were thou? (part II)

Four full years passed since my last post on GTalk. Unfortunately I can't say much good about Google Instant Messenger efforts. It looks they would like everybody to switch from XMPP-based GTalk to proprietary protocol of Hangouts.

We saw stop of XMPP federation, wholesale upgrade of GTalk clients to Hangouts, last GTalk for Windows have been released15 months ago. Next logical step is to stop supporting XMPP altogether. What a sad day that would be..

Of the handful biggest IM providers, none supports open protocol (besides Google/XMPP yet). With 19 billions valuations the market is huge, and one would expect the fierce competition to leave no rock unturned in attracting more users. Unfortunately commercial interests prevail here.

There never was better time for an established company to embrace open-protocol platform. May be there already is? Please post in comments.

PS. Whoever is still on XMPP/Jabber/GTalk platform, uses pidgin on ubuntu 14.04 amd64 and misses "message delivery confirmation" plugin - here is a bit of solace for you: pidgin-xmpp-receipts_0.6-1_amd64.deb

Lenovo IdeaPad A10 (Android) hands-on review

Being the "family geek" I am often asked for laptop recommendations. More often than not, functionality requirements are so low that even chromebook will be sufficient. Actually I already recommended two chromebooks and so far both continue to please their owners.

Sometimes the requirements list Skype or greater autonomy as mandatory items. For such users an Android tablet should be enough. Occasional keyboard users require something more substantial. Bluetooth keyboards only complicate the picture.

Since the Lenovo IdeaPad A10 (Android) appeared on the market (end of October 2013), I have been looking on the internet for a hands-on review of the device. And even now, three months later, nothing useful have shown up. This is probably related to Lenovo's refusal to market the laptop in USA (most possibly due to the shaky patents ground there). Lack of cyanogenmod posts on the device and abundance of clearly bought reviews on various blogs did not help either..

Finally couple of days ago, I decided to bite a bullet and buy localized (Hebrew) Lenovo IdeaPad A10 for $310. So here is my list of pros and cons I found so far:

Pros:

• USB charging is done by standard 2A 5v microUSB jack.
• Touching the touchpad yields visible mouse pointer that simulates screen touches. This gives expected look and feel in laptop mode.
• Interface is easy to use, but suffers from touch/pointer dilemma. Long story short - your hand starts aching after some time pressing buttons on vertical screen.
• Two full-size USB connectors make it easy to connect flash-disks or mouse - this also improves look and feel in laptop mode.

Cons:

• There is no root jail-breaking application (at the moment). This makes it impossible to install VNC server - a must for tech support of the newbie users.
• Lack of root access makes it impossible to mount a network share. Lack of any modders activity leaves little hope for that to change though.
• Built-in "explorer" crashes when trying to connect to a webdav sever.
• SFTP support in the "exporer" does not allow specifying target path.
• The touchpad is single-figner only and can not interpret gestures - nice to have with most modern laptops (for example: two fingers down = scroll down).
• SFTP support in the "exporer" does not allow to specify path on the server, does not show video files as icons, insist on copying the file locally first, and even that it does on ridiculously slow speed of 250 KBytes/s..
• It is expensive for what it offers. A $150 tablet with a BT keyboard could be bought for less than that. Heck, add a bit more and you can buy weak laptop..

Verdict so far: The tablet/wannabe laptop combination is acceptable for users that are aware of its limitations and feel comfortable about it. Fortunately that was the case for me.