Migration of tens of users from office365 to mailcow

For a long time our company used hybrid mail setup, with regular developers having mailboxes on linux mail server (postfix) and manager types - on office365. As they share same domain, Office365 connectors are used. This setup enabled significant cost savings as only users requiring shared calendar or Microsoft Office license had to be put on office365.

Recently, the company decided to rebrand and change primary mail domain. All managers got another new mailbox and mailboxes on old domain no longer required office365 features.

To save costs, we decided to migrate the expensive mailboxes to Linux, but it turned out to be very hard task:

• Outlook has no way to "export" mailboxes. PST (and NST) files are useless in this regard.
• Paid options allowed backup and restore back to office365. Also, they prevented mailbox to be used during the backup.
• New outlook requires IMAP accounts to be authenticated using Oauth protocol (citing security issues, usual EEE tactics). It also seems O365 is throttling IMAP connections (and favoring their proprietary MAPI/EAS protocols)

So we did the migration in several stages:

1. Set up  Thunderbird (one profile per account), subscribe to all folders, switch to offline mode (that will cause it to download all subscribed folders). The Thunderbird keeps folders in mbox format, that might be converted into Maildir and served by dovecot
2. Set up keycloak SSO with LDAP backend and mailcow client, to provide Oauth protocol
3.  Imported the Thunderbird mailboxes into mailcow using custom scripts

 If there is interest in details of the solution above, or scripts used - post in comments.

How to multiply amount of EV charging points

Here is business idea on how to multiply amount of EV chargers

Many homeowners are in position to provide EV charging points (power connection, parking places), even if they are not interested in EV vehicle at the moment. Some of them are willing to monetize the possibility, but don't want to deal with clearinghouse headaches and the necessary initial investment in expensive charger. Lack of the solution prevents many to share their existing EV charging point even with their next door neighbors.

So we need to invent:

1. infrastructure that would incentivize (reimburse) home owners to expose their power to any EV consumer.
2. Adapter device that is as cheap as possible, for every EV owner to buy.

How?

1. Create portable adapter that transforms home 3 phase power to NACS or CCS. Car driver will purchase and register the adapter with clearinghouse (see below) and carry it in the car.

 

 

2. Create DIY kit for homeowner to wire a standard 3 phase power socket (IEC 60309) in a rain-proof way on their parking place (or accessible from street) and stick "provider" RFID tag (received from clearinghouse). Should be doable by any certified electrician.

3. Create software infrastructure to charge consumers and reimburse homeowners, with secure producer-consumer authentication. The adapter will have a chip that reads RFID tag embedded into the power socket and calculate reimbursement amounts.

General notes:

• It would be nice if the adapter  cooperates with the car's computer/SIM to communicate with the clearinghouse using cellular network
• Homeowners should be able to set rates, down to zero to certain adapters (EV car owners), see who is connected at any given time and contact details of EV drivers.
• Some amount expected to be taken by clearinghouse for infrastructure support
• The adapter variations could be with embedded 3G SIM, charge indicator, anti theft device

Daeb - Carpooling-first robotaxi driven highly efficient transportation solution

 ====== Daeb ======

Daeb - pronounced "deb" - nothing special, I just like the short name, similarly to "cab" :)

Motto: Carpooling-first robotaxi driven highly efficient transportation solution.

===== intro =====

Bus, metro and cars transport are obsolete. People want to have the best features of the three, like comfort and speed of car, lack of parking headache and cost of bus or metro,

Robotaxi might sound as the solution, yet it has its own issues. It is very complex system that tries to solve all problems for every possible road and weather conditions. No wonder we are in wait for over a hundred years and counting..

Another issue is carpooling, or rather, lack of it. The robotaxi car is by design point-to-point solution that does not bode well with picking up and dropping off passengers en-route, like bus or metro. As consequence, its efficiency goes down with rising traffic loads.

We need something simple, like elevator or escalator, that is "carpooling" passengers by design and works reliably in closed, predictable space. A network of strategically put travelator-inspired transportation would enable passengers to reach remote places easily.

===== Daeb =====

Daeb is a system of autonomous cars that are purpose built for quick pickup and dropoff and whose purpose is to transport passengers along predefined short route, like a street or around a block. The limited environment would simplify requirements from autonomous car. Neighborhoods might choose to use different transportation means, depending on environment, like self driving scooters or callboats.

It would stop at any place by pressing a button (to exit) or waiving a hand (to pickup). Phone app might "order" empty Daeb car to await passenger outside, just like taxi. It might circulate this way between center and remote parts of the neighborhood, passing predefined doPon places, like a bus route.

The Daeb car might have option to become car for rent on the go. Say someone needs to carry heavy load or citizen with disability to remote place. In this case he might "take over" the Daeb car control, sit at the driver place and drive the car to destination.

Similarly, there will be separate set of likely bigger Daebs traveling between suburbs or to downtown, with even bigger Daeb system participating vehicles (like trains) moving between cities.

The multilevel Daeb architecture enforces resources efficient carpooling. The separate Daeb loops enable to differentiate carriages and frequency, depending on demand.

doPon codes

doPon is short from "Drop Off People and ONboard"

symbol “↙️P↗️”

DoPon is system to enumerate places where people drop off and onboard, near socially popular places, such as wedding halls, malls, stadiums, cemeteries, etc.

It is designed to be used in robotaxi, carpooling or taxi settings, when user is on the go and needs to enter pickup or drop off location.

Scheme

doPon system is global, thus it has both full, short and tiny schemes. Same doPon place might have several doPon code (see shorting below).

Tiny - used in city: @Cid (e.g. @33231) - first digit is always control digit of all digits in doPon code. Second digit specifies amount of digits that follow it (it also serves code shortening - see below). Digits from second and until first non-digit char is the doPon code of comprise number of the place in the city.

Certain prefixes might be reserved for already existing system. For example all public bus stations in Israel have unique 5 digits station code. These might be mapped to doPon at, say, “prefix” 5. Thus, a bus station 12345 in Israel will have doPon code @X512345 (where X is control digit)

Short - used in region: @Cid-CityRegionCode - where CityRegionCode might be mapped after phone code of the City or Region

Full - international: @Cid-CityRegionCode-CountryCode - where CountryCode is phone code of the country

Slots A popular doPon place (like in airport or mall) might have several “slots”, where passengers might await for drivers to pick them up. The slots are dynamically directed to, based on demand and availability. To specify slot, one should add + sign after the short doPon code part. For example, full doPon code for slot 12 of doPon place 831 (BenGurion airport) might be @X3831+12-3-972 (where X is control digit). For consistency, slot number is not used in control digit calculation.

Codes shorting

It is expected that popular places will have shorter codes. These are easier on people to remember and type. Codes with “eternal” prefix (such as 1 to 3) could be printed on billboards and memorized by people easily.

In practice it is hard to estimate place popularity, thus a graduation approach should be used. Initially, any place submitted into the system is assigned longest code possible (say 7 digits). If the system sees many lookups for the code, it will get graduated to shorter code. After that, use of old code will cause warning about the code graduation.

doPon place

A physical doPon place should have several features:

• It is well suited for a car (especially autonomous cars) to stop safely and allow passengers pickup and drop off
• It has easy way for driver to turn around
• It is wheel-chair or scooter friendly, passengers have easy way to reach socially popular places nearby
• A popular doPon place might have several clearly marked “slots”, probably spread at different entrances. Slots on both sides of a road might be arranged in street-like buildings order for more human-friendly navigation.

F1x - wiki for devops

By definition devops role requires 24/7 on-demand attention. With separate NOC team to execute SOPs and track incidents. Single devops is nonsense. A company that has single devops is a recipe for disaster as there is no redundancy for that fragile and pesky human resource.

With more than one devops, knowledge sharing becomes the issue. And without further ado, let me introduce F1x - how we handle it in our company:

F1x

Have you ever wondered about possibility to press F1 and have a way to scribble notes about the very thing you are working on? And make these notes available to other members of your team? Popping up into their face when they happen to start working onto the same thing? And they can edit them too!

Introducing F1x - "F1+eXtra help button everywhere":

F1x is "F1 help button everywhere" system that recognizes user activity (context) and maps contexts and corresponding wiki pages.

At this stage it is able to track contexts of users doing ssh to servers and changing current path into a directory. But it already gives powerful notes-leaving mechanism:

Imagine you are trying to understand some weird functionality and ssh into the server. A popup shows up with a note that there is F1x note. You press F1 and browser is opened to the F1x wiki page, that is exactly about the server you are working on. Even more, when you chdir into some directory, and there is a note about it, you will get popup as well.

Here is short video demo of the F1x in action:

More info and code is on the gitlab project over here:
https://gitlab.com/skliarie/f1x

O GTalk team, where were thou? (part IV: The END)

It is finally happened:

Google Talk is surprisingly still operational, but that ends on Thursday

Took 12 years to die:

O GTalk team, where were thou? (part III) (AKA: The other shoe dropped). (2017)

O GTalk team, where were thou? (part II) (2014)

O GTalk team, where were thou? (2010)

MYSQL/BTRFS/NVME failure

It is a very bad idea to run database (especially production one with lots of I/O) on BTRFS because the filesystem at any random time might become readonly:

Oct 24 12:30:22 db02 kernel: BTRFS: error (device nvme0n1) in btrfs_run_delayed_refs:2936: errno=-28 No space left
Oct 24 12:30:22 db02 kernel: BTRFS info (device nvme0n1): forced readonly

And then you find that you need to do rebalance. You try and find out that rebalance can not be done because - you guessed it - there is no space left. They suggest to delete couple of snapshots though. You delete them, start rebalance and now the whole filesystem is stuck completely.

If you need HA mysql db with snapshots, then you should go with mysq/LVM/DRBD path, see this link for insight: https://rarforge.com/w/index.php/2_Node_Cluster:_Dual_Primary_DRBD_%2B_CLVM_%2B_KVM_%2B_Live_Migrations

O GTalk team, where were thou? (part III) (AKA: The other shoe dropped).

Today it happened for the first time. GTalk team, silently, without telling anyone, stopped messages sent using XMPP to be delivered to Android Hangouts clients. This caused me to miss important alert message from my monitoring system.

Good bye GTalk/Hangouts, it was nice to know you.

Hello telegram, the only popular and opensource API system out there!
See you on tg://resolve?domain=skliarie

GRUB-based multiple iso booting flash drive

With huge USB flash drives of today it is sad that one can't easily put several bootable ISO images on it and have nice on-boot selection menu.

GRUB to the rescue!

Historically GRUB is being used for hard disks and syslinux for floppies and flash drives. But nothing prevents using GRUB for flash drives as well. Here are instructions for creating bootable GRUB-based USB flash drive (disk on key):

1. Create vfat partition. For big drives you must use fat32 format.
2. Unpack the http://skliarie.meshanet.com/skliarie_blog/boot.tar.gz onto it. It will create single directory boot on the drive.
3. Customize boot/grub/grub.cfg file, put iso images accordingly
4. On linux box, put bootable MBR onto the DOK (for example on /dev/sdf):
1. mount /dev/sdf /mnt/dok
2. grub-install --force --no-floppy --root-directory=/mnt/dok /dev/sdf
3. umount /mnt/dok 

Caveats:

The ISO image must support GRUB-based booting. Specifically it must be smart enough to locate ISO image on the DOK using parameters specified in grub.cfg file.

Latest Ubuntu and Debian based ISO images are known to work.

Thanks to Jonathan Vollebregt there is a way to boot knoppix as well. You will need custom initrd, with your flash drive specific tuning. I built one for mine: knoppix_minirt.gz, you are free to take and modify it according to your DOK and filesystem parameters. Important commands here are:

1. Unpack the initrd
   gzip -dc /mnt/dok/boot/knoppix/minirt_ska.gz | cpio -i
2. Modify the init file (put sector numbers that are correct for your DOK)
3. Compress back the initrd
   find ./ | cpio -H newc -o | gzip -9 -c > /mnt/dok/boot/knoppix/minirt_ska.gz

Please send me GRUB stanzas for other ISO images and I will put them into the grub.cfg file.

O GTalk team, where were thou? (part II)

Four full years passed since my last post on GTalk. Unfortunately I can't say much good about Google Instant Messenger efforts. It looks they would like everybody to switch from XMPP-based GTalk to proprietary protocol of Hangouts.

We saw stop of XMPP federation, wholesale upgrade of GTalk clients to Hangouts, last GTalk for Windows have been released15 months ago. Next logical step is to stop supporting XMPP altogether. What a sad day that would be..

Of the handful biggest IM providers, none supports open protocol (besides Google/XMPP yet). With 19 billions valuations the market is huge, and one would expect the fierce competition to leave no rock unturned in attracting more users. Unfortunately commercial interests prevail here.

There never was better time for an established company to embrace open-protocol platform. May be there already is? Please post in comments.

PS. Whoever is still on XMPP/Jabber/GTalk platform, uses pidgin on ubuntu 14.04 amd64 and misses "message delivery confirmation" plugin - here is a bit of solace for you: pidgin-xmpp-receipts_0.6-1_amd64.deb

Lenovo IdeaPad A10 (Android) hands-on review

Being the "family geek" I am often asked for laptop recommendations. More often than not, functionality requirements are so low that even chromebook will be sufficient. Actually I already recommended two chromebooks and so far both continue to please their owners.

Sometimes the requirements list Skype or greater autonomy as mandatory items. For such users an Android tablet should be enough. Occasional keyboard users require something more substantial. Bluetooth keyboards only complicate the picture.

Since the Lenovo IdeaPad A10 (Android) appeared on the market (end of October 2013), I have been looking on the internet for a hands-on review of the device. And even now, three months later, nothing useful have shown up. This is probably related to Lenovo's refusal to market the laptop in USA (most possibly due to the shaky patents ground there). Lack of cyanogenmod posts on the device and abundance of clearly bought reviews on various blogs did not help either..

Finally couple of days ago, I decided to bite a bullet and buy localized (Hebrew) Lenovo IdeaPad A10 for $310. So here is my list of pros and cons I found so far:

Pros:

• USB charging is done by standard 2A 5v microUSB jack.
• Touching the touchpad yields visible mouse pointer that simulates screen touches. This gives expected look and feel in laptop mode.
• Interface is easy to use, but suffers from touch/pointer dilemma. Long story short - your hand starts aching after some time pressing buttons on vertical screen.
• Two full-size USB connectors make it easy to connect flash-disks or mouse - this also improves look and feel in laptop mode.

Cons:

• There is no root jail-breaking application (at the moment). This makes it impossible to install VNC server - a must for tech support of the newbie users.
• Lack of root access makes it impossible to mount a network share. Lack of any modders activity leaves little hope for that to change though.
• Built-in "explorer" crashes when trying to connect to a webdav sever.
• SFTP support in the "exporer" does not allow specifying target path.
• The touchpad is single-figner only and can not interpret gestures - nice to have with most modern laptops (for example: two fingers down = scroll down).
• SFTP support in the "exporer" does not allow to specify path on the server, does not show video files as icons, insist on copying the file locally first, and even that it does on ridiculously slow speed of 250 KBytes/s..
• It is expensive for what it offers. A $150 tablet with a BT keyboard could be bought for less than that. Heck, add a bit more and you can buy weak laptop..

Verdict so far: The tablet/wannabe laptop combination is acceptable for users that are aware of its limitations and feel comfortable about it. Fortunately that was the case for me.

Ensure emails are accounted for

Ensure emails are accounted for


Sysadmins often set up scripts that send back "OK" or "Error" emails, but there is no mechanism to send alert if *no* email was received for certain time period. Lack of such emails might indicate serious system failure somewhere, that becomes obvious when it is too late to do anything. There are many stories on internet about backups that were set up properly, but for some reason were disconnected.

Thus I wrote the script ensure_accounted_for.py that informs sysadmins about emails that were supposed to arrive on daily basis, but for some reason did not. Its sample configuration can be taken from ensure_af.cfg.

Multiple mysql engines on the same server

Someone asked me for a howto on setting up multiple mysql engines on the same linux server. There are many articles on the topic, but none provides an ready-to-use set of files which one can simply grab and go.

Thus I created multiple_mysql.tar.gz file that includes all files to run four mysql engines. The configuration assumes per-database directories /opt/db-330[4567] with bin_log, mysql  and relay_log directories. The mysql directory must have mysql database with associated tables (should be copied from an existing database).

Note that the files were created on Ubuntu Lucid 10.04 and will likely not work on other distro. Also, the password in debian-330?.cnf files will be different for each DB and must be updated before use.
Use the following command to set the password:
GRANT ALL ON mysql.* to "debian-sys-maint"@"localhost" IDENTIFIED BY "dxaskDkdkSkdSDLd";

The bashrc file contains useful CLI shortcuts to common mysql commands. You are encouraged to add these to your own .bashrc file.

Laggy ssh with clear network

Recently, after reboot, my ssh sessions to remote servers started behaving sluggishly, sometimes taking several seconds for typed characters to appear. The lag was especially noticeable with last character that got typed. Nature of ssh sessions it to type several characters and checking the output. You can imagine the frustration seeing most of keypresses delivered, but getting feedback after some delay. Ping to the server was showing reliable, speedy responses with not a single packet loss. Network was in excellent shape, no torrents or other heavy uploads. I did not know where to look or how to google for the problem. I then remembered that I waited several months for the reboot to check an X-related crash and found that I disabled all X acceleration in the /etc/X11/xorg.conf file:

    Option "DRI" "false"
    Option "shadow" "true"
    Option "NoAccel" "True"

Desperate about the whole issue, I enabled back all X acceleration options, restarted X and voila! The ssh is fast again! Now, I don't know whether it is a bug or which subsystem it is in. So I decided to describe the problem sprinkled with keywords I used while googling for the problem in hope it helps someone. Write in comments if it did. :)

Playlist detection on python

There are several popular playlists used for internet radio streaming: asx, m3u, smil, asf, pls... The playlist extension is not always visible in the URL that an ipradio station returns. Furthermore, the MIME type is not always correct. Sometimes one has no choice but to guess the playlist type by looking for the presence of certain markers.

After fruitless searches on the internet, I wrote an python library that detects playlist type. You can download it here.

I am sure that the detection algorithm can be improved. Please write suggestions in the comments.

Easy creating and destroying of lxc vservers

One big advantage of using amazon's EC2 servce is its deceptively easy way to create new servers - it is just API call or mouseclick away. The actual physical machine allocation, OS image copying, hostname, IP, dns, firewall and routing setup are done transparently in the background. You are only required to setup the payment ;) .

Big enterprise virtualization technology providers already have a mechanism to create virtual machines with given parameters on demand. Such a system would likely be overkill for a handful of physical servers. I also doubt they support the highly effective pseudo-virtualization LXC technology.

Thus I created a python "lmachine" script for our LXC servers, that copies an OS image into a selected "slot" with already pre-allocated IP number and server name. The script also modifies a couple of system files from the OS image accordingly.

The script can be downloaded here.

Preparing an LXC image

LXC images are easily created using the debootstrap utility. To make them ready for the lmachine script, you need to set the actual IP number with the IPNUMBER string and the actual server name with the VSERVERNAME string. Here is the list of files that needs to be modified:

• fstab
• lxc.conf
• root/etc/network/interfaces
• root/etc/hosts
• root/etc/hostname
• root/etc/mailname

Make sure you pre-allocate IP numbers and ltestXX DNS domain names in advance. Update the lmachine script with the list of pre-allocated IP numbers.

xls2csv using python-uno

While several xls2csv converters exist, on a recent assignment none of them were able to convert a multi-sheet, 300+MB Excel file into CSV format.

While searching for possible solutions, I found this (dated) blog entry on using python-uno. After assembling it into a single script and updating for changed LibreOffice arguments, I made it available here. You can check the script using the sample .XLS file with three sheets.

The power of the solution lies in the LibreOffice+UNO (Universal Network Objects) platform it uses. While xls2csv looks like a trivial task, the platform automatically supports reading all spreadsheet types LibreOffice supports. This means that the script might as well be called xlsx2csv or ods2csv.

The script can become a starting point for someone trying to implement complex documents management automation scripts. Leave a comments if you do :) .

xkb files for russian and hebrew keyboard switchers

Long ago, my xkb-based keyboard switcher method stopped working. I then moved over to using gnome-keyboard-properties to set up the keyboard switching. This worked for another several years until recent ubuntu changes in oneiric that removed the stand-alone gnome-keyboard-properties. Worse yet, the gnome-control-center crashes on me if not running from gnome-session.

Being loyal user of FVWM2 window manager for 15 years (see my config over here), I dug out the xkb keyboard switching files and fixed them to work with the evdev keyboard type. This works on ubuntu oneiric (11.10) system.

Here are the download links to my xkb files:

• russian
• hebrew

To load them use the following command:

xkbcomp -R/usr/share/X11/xkb/ russian.xkb $DISPLAY

LLS=LXC+LVM+Snapshots

In our company developers deal with massive datasets that needs to be easy to copy, modify a fraction of it and scrape. Snapshotting is an ideal solution here. Each developer has a personal vserver (LXC container). The missing piece here is to provide them with a way to manipulate partitions and snapshots from inside of their vservers.

To enable users manipulate partitions from inside of the virtual server, I wrote LLS (LXC+LVM+Snapshots) scripts.

LXC+LVM+Snapshots = LLS

LLS system is set of scripts that enable LVM partitions and snapshots to be managed from inside of LXC vservers. It is safer to allow developers to use the scripts, instead of giving them superuser access to the physical machine.

Architecture

The LLS scripts consist from two parts. The daemon script on the host and client scripts on the vservers. The clients communicate with the server over a named pipe in a shared (bind mount) directory /lls (lls - LXC+LVM+Snapshots). The /lls directory is actually a small partition that contains configuration file, the shell scripts and the named pipe used for communcation.
The daemon script does all the necessary low-level manipulations, both on the physical machine and on the LXC vservers.

Usage

Each LLS vserver has /lls partition mounted. To preserve mounts across reboots, the /etc/rc.local file runs /lls/tools/lls_mount_on_boot.sh script.
There are several client scripts the /lls/tools partition that do various operations:

script name	Operation
lls_create_partition.sh	Create a partition
lls_create_snapshot.sh	Create an LVM snapshot from an existing LVM partition
lls_delete_partition.sh	Delete an LVM partition or snapshot
lls_list_partitions.sh	List available LLS partitions and refresh /dev/mf directory
lls_mount_on_boot.sh	Mount LLS partitions using configuration in the /etc/fstab file

The scripts show informative Usage information when ran without arguments.
Developers are expected to operate the scripts by themselves. They are also expected to maintain the /etc/fstab file for mounts they want to survive reboot of their vserver. Unmounted snapshot is assumed to be not necessary anymore and might be deleted at any time.

Further work

As the LLS system is used, more features are asked by developers and system administrators. Here are some of them:

• Track unused (unmounted) LVM snapshots and delete them automatically
• Track disk space used/required by LVM snapshot and grow it automatically. Send email to sysadmin each time this happen.
• Have a way to enable/disable visibility of a LLS partition. This is useful while the LLS partition is under construction.
• Have a way to mark an LLS partition as non-mountable or mountable in read-only mode.

Download

Version 20111124.

Thoughts on photos-management system

Nowadays everybody around me have digital cameras. Photos that I am interested to look at are everywhere - on facebook, flickr, picasa, people's harddisks, some photo management system, etc. Rarely someone is kind enough to send me the photos.

Let's analyze the situation.

The situation

There are mainly two distinct categories of users of a photos management system - producers and consumers. Each of them want different features from the system.

Producers

Producers have many requirements:

• Editability: Producers need to operate (rotate, sort, tag, etc) on the photos. Automatic or manual face recognition would be nice to have.
• Multi-user, networkable: There might be several producers (members of the family) operating on the same photos. They connect from different accounts on the same computer or from different computers (and OSes) in the same network.
  
• Search: photos organizations must be flexible enough to support both folders and "search folders" - based on search string.
  
• Publishing: enable consumers to view the photos over internet. The requires Sync or Upload to a web server.
• Comments: enable consumers to comment on the photos, rate them, tag photos and faces.
• Portability: no lock-in into some proprietary photos management system, OSS or proprietary. Practically this means that all meta data must be stored/duplicated into the photo file itself, using some meta-data format (JFIF, EXIF, IPTC, XMP, etc).
  

Consumers

Consumers value different features than producers:

• easy and fast way to view the pics, preferably with captions (if any).
• they might want to specify filter that sorts out photos with nobody they know.
• ability to become producers (if allowed by host)

Current intermediate solution

Use shotwell program (I am linux user) to manage the photos on a network share. The shotwell maintains a database for quicker lookup, which is located along the photos. All changes are duplicated in metadata area of the photos as well. Publishing is done to picasa web album (it properly renders UTF8-encoded captions and tags in the IPTC header of the photos). Picasa is used as publishing point only, disregarding comments, ratings or faces tagging (blocking these if possible).

Ideal solution

The ideal solution would be in bi-directional synchronization of the photo files with some sharing-enabled cloud-based service, whereas the service embeds all additional meta-data generated by "consumers" into the files in open format. Clouds, are you listening?