Thursday, October 24, 2019

MYSQL/BTRFS/NVME failure

It is a very bad idea to run database (especially production one with lots of I/O) on BTRFS because the filesystem at any random time might become readonly:
Oct 24 12:30:22 db02 kernel: BTRFS: error (device nvme0n1) in btrfs_run_delayed_refs:2936: errno=-28 No space left
Oct 24 12:30:22 db02 kernel: BTRFS info (device nvme0n1): forced readonly
And then you find that you need to do rebalance. You try and find out that rebalance can not be done because - you guessed it - there is no space left. They suggest to delete couple of snapshots though. You delete them, start rebalance and now the whole filesystem is stuck completely.

If you need HA mysql db with snapshots, then you should go with mysq/LVM/DRBD path, see this link for insight: https://rarforge.com/w/index.php/2_Node_Cluster:_Dual_Primary_DRBD_%2B_CLVM_%2B_KVM_%2B_Live_Migrations

Thursday, December 14, 2017

O GTalk team, where were thou? (part III) (AKA: The other shoe dropped).

Today it happened for the first time. GTalk team, silently, without telling anyone, stopped messages sent using XMPP to be delivered to Android Hangouts clients. This caused me to miss important alert message from my monitoring system.

Good bye GTalk/Hangouts, it was nice to know you.

Hello telegram, the only popular and opensource API system out there!
See you on tg://resolve?domain=skliarie

Sunday, November 8, 2015

GRUB-based multiple iso booting flash drive

With huge USB flash drives of today it is sad that one can't easily put several bootable ISO images on it and have nice on-boot selection menu.

GRUB to the rescue!

Historically GRUB is being used for hard disks and syslinux for floppies and flash drives. But nothing prevents using GRUB for flash drives as well. Here are instructions for creating bootable GRUB-based USB flash drive (disk on key):
  1. Create vfat partition. For big drives you must use fat32 format.
  2. Unpack the http://skliarie.meshanet.com/skliarie_blog/boot.tar.gz onto it. It will create single directory boot on the drive.
  3. Customize boot/grub/grub.cfg file, put iso images accordingly
  4. On linux box, put bootable MBR onto the DOK (for example on /dev/sdf):
    1. mount /dev/sdf /mnt/dok
    2. grub-install --force --no-floppy --root-directory=/mnt/dok /dev/sdf
    3. umount /mnt/dok 

Caveats:

The ISO image must support GRUB-based booting. Specifically it must be smart enough to locate ISO image on the DOK using parameters specified in grub.cfg file.

Latest Ubuntu and Debian based ISO images are known to work.

Thanks to Jonathan Vollebregt there is a way to boot knoppix as well. You will need custom initrd, with your flash drive specific tuning. I built one for mine: knoppix_minirt.gz, you are free to take and modify it according to your DOK and filesystem parameters. Important commands here are:
  1. Unpack the initrd
    gzip -dc /mnt/dok/boot/knoppix/minirt_ska.gz | cpio -i
  2. Modify the init file (put sector numbers that are correct for your DOK)
  3. Compress back the initrd
    find ./ | cpio -H newc -o | gzip -9 -c > /mnt/dok/boot/knoppix/minirt_ska.gz
Please send me GRUB stanzas for other ISO images and I will put them into the grub.cfg file.

Thursday, September 4, 2014

O GTalk team, where were thou? (part II)

Four full years passed since my last post on GTalk. Unfortunately I can't say much good about Google Instant Messenger efforts. It looks they would like everybody to switch from XMPP-based GTalk to proprietary protocol of Hangouts.

We saw stop of XMPP federation, wholesale upgrade of GTalk clients to Hangouts, last GTalk for Windows have been released15 months ago. Next logical step is to stop supporting XMPP altogether. What a sad day that would be..

Of the handful biggest IM providers, none supports open protocol (besides Google/XMPP yet). With 19 billions valuations the market is huge, and one would expect the fierce competition to leave no rock unturned in attracting more users. Unfortunately commercial interests prevail here.

There never was better time for an established company to embrace open-protocol platform. May be there already is? Please post in comments.

PS. Whoever is still on XMPP/Jabber/GTalk platform, uses pidgin on ubuntu 14.04 amd64 and misses "message delivery confirmation" plugin - here is a bit of solace for you: pidgin-xmpp-receipts_0.6-1_amd64.deb

Friday, January 24, 2014

Lenovo IdeaPad A10 (Android) hands-on review

Being the "family geek" I am often asked for laptop recommendations. More often than not, functionality requirements are so low that even chromebook will be sufficient. Actually I already recommended two chromebooks and so far both continue to please their owners.

Sometimes the requirements list Skype or greater autonomy as mandatory items. For such users an Android tablet should be enough. Occasional keyboard users require something more substantial. Bluetooth keyboards only complicate the picture.

Since the Lenovo IdeaPad A10 (Android) appeared on the market (end of October 2013), I have been looking on the internet for a hands-on review of the device. And even now, three months later, nothing useful have shown up. This is probably related to Lenovo's refusal to market the laptop in USA (most possibly due to the shaky patents ground there). Lack of cyanogenmod posts on the device and abundance of clearly bought reviews on various blogs did not help either..

Finally couple of days ago, I decided to bite a bullet and buy localized (Hebrew) Lenovo IdeaPad A10 for $310. So here is my list of pros and cons I found so far:

Pros:

  • USB charging is done by standard 2A 5v microUSB jack.
  • Touching the touchpad yields visible mouse pointer that simulates screen touches. This gives expected look and feel in laptop mode.
  • Interface is easy to use, but suffers from touch/pointer dilemma. Long story short - your hand starts aching after some time pressing buttons on vertical screen.
  • Two full-size USB connectors make it easy to connect flash-disks or mouse - this also improves look and feel in laptop mode.
Cons:


  • There is no root jail-breaking application (at the moment). This makes it impossible to install VNC server - a must for tech support of the newbie users.
  • Lack of root access makes it impossible to mount a network share. Lack of any modders activity leaves little hope for that to change though.
  • Built-in "explorer" crashes when trying to connect to a webdav sever.
  • SFTP support in the "exporer" does not allow specifying target path.
  • The touchpad is single-figner only and can not interpret gestures - nice to have with most modern laptops (for example: two fingers down = scroll down).
  • SFTP support in the "exporer" does not allow to specify path on the server, does not show video files as icons, insist on copying the file locally first, and even that it does on ridiculously slow speed of 250 KBytes/s..
  • It is expensive for what it offers. A $150 tablet with a BT keyboard could be bought for less than that. Heck, add a bit more and you can buy weak laptop..
Verdict so far: The tablet/wannabe laptop combination is acceptable for users that are aware of its limitations and feel comfortable about it. Fortunately that was the case for me.

Tuesday, July 2, 2013

Ensure emails are accounted for

Ensure emails are accounted for


Sysadmins often set up scripts that send back "OK" or "Error" emails, but there is no mechanism to send alert if *no* email was received for certain time period. Lack of such emails might indicate serious system failure somewhere, that becomes obvious when it is too late to do anything. There are many stories on internet about backups that were set up properly, but for some reason were disconnected.

Thus I wrote the script ensure_accounted_for.py that informs sysadmins about emails that were supposed to arrive on daily basis, but for some reason did not. Its sample configuration can be taken from ensure_af.cfg.