The multitude of ROM for the Xiaomi phones

I recently got a Xiaomi Mi5 (android) phone. One of the reasons I picked this phone was that apparently Xiaomi follows "release early, release often" principle for the firmware (aka ROM) of their devices. Eventually, I didn't get disappointed. Not only do they release often updates for the phone, but you have choices on which ROM goes on your phone. It took me some time to find out all the different ROMs, so as a note to myself, and maybe to others, here is a summary.

Firstly, let's mention that Xiaomi's firmware is called MIUI, and although it's developed by them, MIUI is also available for other brands of android phones. On the MIUI website, you have the choice between two ROMs:

• China: that's the one dedicated to the Chinese market (which cannot access anything related to Google due to the government policy). As the developers and main target customers are Chinese, it's the ROM the most often updated, and which receives the most attention.
• Global: that's the one for the rest of the world. It's main advantage is that it comes with Google-ware installed instead of Chinese software. It's less often updated, but still you can expect a new version once every couple of months.

Actually, for each of these ROMs there are two channels:

• Stable: That's the one most people should use.
• Developer: That's were beta versions are available. On the Chinese ROM, you can expect new ROM at least every second week! (For the Global one, it's a bit more irregular).

Ok, so far, there are just 4 ROM versions available for download at any time. Now, where it gets more muddy, is that there are several ways to update you ROM (5 on my last count). To support all the ways, there are actually, three different formats for each ROM. So we bump to 12 ROMs. But wait, what if you want to try older versions for bug bisection or due to a regression? Of course, all the older versions are also present!

That's pretty much all for the official ROMs... but there are also some non-official but famous ROMs. As pretty much the whole MIUI ROM is open-source, people have been making derivative. In particular, there is the Xiaomi.eu Multilang ROM. They basically take the China ROM, and modify it themselves to make a ROM adapted for the European users. So it's pretty much the same as the Global ROM, excepted it might be slightly ahead in terms of version, and they claim they've added their own improvements. Note that because it's not an official ROM, the phone bootloader must be unlocked, and the update is done via some special bootloader extension (TWRP). In my experience, the difference is very tiny, and I feel it's not worth the trouble of having to unlock the bootloader and drop some security and ease of updates.

One very weird aspect of my phone is that when I got it, it still was running none of these ROMs. I got the phone from a small Dutch vendor doing imports from China, when the phone had just been released so that might explain a bit this, but I didn't feel comfortable running some ROM from absolutely unknown source. Basically, the ROM was claiming to be the unofficial Xiaomi.eu, but the bootloader was locked (so the ROM was signed a Xiaomi key ?!). Although it seemed to behave fine, one particulary troubling aspect was the standard updater would never find any update, and would crash when trying to update manually. Trying to update via the MI PCSuite would just show an error saying the device firmware is too old to be updated (how ironic...). After much looking here was how I could update the ROM to a standard Global version (without any data loss):

1. Ask to have unlocking privileges on my Xiaomi account
2. (Two weeks later) unlock the bootloader of my phone
3. Flash the latest Global ROM using the fastboot method
4. Relocked the bootloader (using these instructions)

Maybe I should have been more paranoid, and completely erase the data to make sure no code from the old ROM was left-over but I was happy not to restore all my data.

Some shortcuts for the Samsung Galaxy Ace

While my new external USB hard drive came with a 50 pages manual, of which none were useful, my android smatphone came without manual. HHopefully it's just because I bought it in a small shop in Taiwan... but still it seems most android apps are poorly documented.

After a difficult (early) morning, I discover that there was a screen capture folder with a screenshot in it. It turns out that Samsung has added to the galaxy ace a shortcut to screen capture: back + home (you actually have to press in this order). Surprisingly, on other phones, it's a different combination... as if it was still too easy to find ;-)

I also recently realised that long pressing a key can bring a special action. For instance in the mail client and browser, pressing long menu brings the search widget. Long pressing back in the browser brings the history.

There is also this horrible design which makes the camera application to lock/unlock the interface if power is pressed. Not stupid but completely unstandard. I rebooted my phone twice because I thought the camera had gone blocked before realizing this.

So far I haven't any action activated by the accelerometer, like undo on the iPhone. Still, the picture gallery react a bit to it by tilting the images,  so there is hope to find such a hidden unpractical feature...

Any one knows some other shortcuts? Maybe I should start reading the source code to replace the user manual?

Just finished my first town

I've just finished mapping my first town in OpenStreetMap, and coincidently it's also the first town where I lived: Crépy-en-Valois. Completed it using mainly the "cadastre" from the French authorities, and the Bing imagery, and the memories I have from spending 15 years there (turned out to be extremely useful).

Of course a place is never fully mapped, but here is can be found:
* all the streets
* all the streets names
* the shape of every building

I finished by naming the streets, and to ensure the quality, I compared the list of street names found on internet (from the phone directory) to the list in OpenStreetMap. I converted the directory data into a text file with one name per street alphabetically sorted.

To get the list of streets in OSM, the nominatim service seemed the most appropriate but I seemed not to answer this type of query. So eventually I settled with a very basic command:
curl -g "http://api.openstreetmap.org/api/0.6/map?bbox=2.8634295463562,49.2131958007812,2.93606853485107,49.2601203918457"
| grep =\"name | sort -u | sed -re "s/.*v=\"(.*)\".*/\1/;s/'/'/" > names-osm.txt

As long as there aren't many amenities, it works fine. Then using a diff tool (I used Meld), you can see which street is still missing, or where there are spelling mistakes. It turned out to be really useful. In addition to knowing there are 212 streets in such a small town, it even allowed me to notice small streets I had forgotten to map, and some streets whose name are not on the cadastre! 

What happens when you plug a keyboard into your computer

While reporting a bug, I learned a lot about the different programs which are involved when a keyboard is plugged in. As simple as such an action can sound, it is amazingly complex as it traverse many layers. So, what happens when you are using gnome in Linux and you insert a new keyboard in your USB port?

1. The Linux kernel usb system detects a new device
2. From the device properties it is determined as an HID device, and therefore a new input device is created by the evdev module (which is loaded if it was not before)
3. Event info about a new device is sent to udev
4. Udev parses its rules and creates a new file in /dev for this new device
5. Udev transmits the info of a new device to listeners such as DeviceKit and Hal
6. Hal parses its rules, one of which is the distro script which set the default layout to the device
7. Hal transmits the info of a new device to its listeners, such as Xorg
8. Xorg add the new device to the children of the Xkb core keyboard
9. Hal transmits the options for this new device to Xorg via Xkb
10. Xorg, via the Xkb extension, transmits the info that the Xkb config has be modified to listeners such as xklavier
11. xklavier transmits the info of a change in config to its listeners, such as libgnomekdb
12. libgnomekdb reloads the configuration of the keyboard and update the status icon in the system tray
13. Xorg, via the XINPUT extension, transmits the info that a new device is present to the listeners, like xklavier
14. xklavier transmits the info of a new device to its listeners, such as gnome-settings-daemon (g-s-d)
15. g-s-d changes this new device's layout to the user preferences via libgnomekdb.

Voilà!

Note that it depends a bit on your system, and in particular in very modern Xorg's (xserver >= 1.8) the Hal part is removed and udev and Xorg communicate together directly. It might also contain some mistakes, don't hesitate to mention it in the comments.

Keyboard layouts could be so much simpler

My brand new laptop has a french keyboard, but I've connected to it an full size dutch keyboard. Linux is great because it can actually deal with such weird setup so that for both keyboards when you press a key you get what you would expect. To do that, I configured the french layout as the default (because it's always connected to the laptop!) and whenever I plug the dutch keyboard I've got a little script which does this:
xinput --list # to find the id of the keyboard
setxkbmap -device $kb-id nl


Now, all this troubles could be avoided if not every country had a different keyboard layout (Switzerland is special, they have several layouts). 100 years ago, this could make sense, but now that there is so much exchange between countries, this is bringing more pain than advantages. One solution would be to simply change the spelling of every language to restrain them to the good old 26 basic letters, and be happy with a US layout forever after. This could happen, the Turkish people did manage, but for sure not any time soon, and anyway the culture should not be driven by the technology. It's up to the technology to adapt.

So what I'd really like to see is a generic keyboard layout, which allows to type any latin character-based language (or at least all the official ones present in Europe). The us-international layout is not so bad, but they are lots of small adaptations needed to not make it a pain to use (especially on windows where by default the diacritics are added to the next letter). Some European peoples are at it, and I hope it goes somewhere. While at changing the layout, we could move to dvorak, so the loss of direct access for the special keys of a language would be more or less counter-balance by the increase speed provided by the dovrak layout.

Even more useful would be to have the input method follow more closely the normal behaviour of putting diacritics: you first put and the letter and then you put the diacritics. It would in addition allow to add easily diacritics while reviewing a text (put the cursor just after the letter, and press the diacritic you want to add). Today, in Xorg, a first layout providing such a mechanism appeared: "USA - International (AltGr Unicode combining)". As its name implies it uses some special features of unicode. This has the disadvantage of not yet being correctly supported in every program, but that's the way to go!

Test this new layout, and report bugs on application not correctly responding to it, and propose better versions of the layout :-)

Getting integrated

As a sign that I'm integrating to the Netherlands, I have a patch in pidgin for... the Dutch translation. Ok, it's just one word, but that's a beginning!

Annoying bug tracking systems

Bugs are extremely annoying, but unfortunately bug tracking systems are also very annoying. They are several existing systems used with open source software. Unfortunately they more or less all sports the same problems.

One which is rather known is the connection of the bugs between upstream (the original project) and downstream (the distros), and in between downstream. Each project and each distro has its own bug tracking system. As a result of this, first, bugs might end up being duplicated: in general the users report their problem in the tracking system of their distro, excepted for some advanced users who directly follow upstream and report there as well. Second, the bugs get reported to places which are not read by the authors of the project, which reduces a lot the chances to have them fixed. So far, only lauchpad has tried to answer this problem, by automatically following upstream, but this does not resolve the problems to link the bug reports in-between downstream. Of course, it doesn't seem reasonable to have only one global bug tracking system for all the projects in the world. What would be useful is a mechanism which allows to transmit a bug from downstream to upstream, just by decision of the bug triaging team. It is then up to the triage team to differentiate bugs related to packaging problems to the bugs which have to be treated upstream.

However, an even bigger problem is that the bug tracking systems are actually bug report tracking system. The main concept is not a "bug", but the "report from someone about a bug". The main description about a bug is the first description ever written about it... which is one less likely to be precise: written before anyone ask for precisions, by someone who is a simple user. Generally, valuable information about the bug (eg: why and when it happens, how it can be fixed) is only available disseminated in the comments, or, even worse, in the comments of bug reports marked as duplicates. To correct this, the main concept should be shifted from "bug report" to "bug". Each bug could have assigned to it several bug reports, and bug reproducers (people would are affected by this bug and most likely able to confirm that the bug is fixed). The main description of a bug would be much closer to a wiki page, which can be updated little by little, as more things get known. Moreover, associated to the bug, there should be test cases, which allow to reproduce the bug. Each test case can be either manual, or automatic, in which case it could be possible to run the test case regularly and automatically against the latest version of the project to detect if the bug is still present. A bug would also be associated to fixes and workarounds. The fixes are patches, which can be commented, and which eventually become a commit in the code repository of the project.

A third problem is that when someone finds a bug in a project and reports it. The reporter should first check for previous reports of the bug. However, if the bug has been corrected in the mean time (typically, in the development version), it will not be easily findable because marked as closed. That is why the bug tracker should also be more aware of the versions of the projects, usually ordered as a tree. So that for each version of the project, it is possible to know which (known) bug is present, likely present, or fixed. When someone search for a bug, he or she will also indicate the version on which the bug was seen, and the search will show bugs even now closed. Not only this avoid to create a duplicate bug report, but also allow the reporter to use a workaround if it is available, without having to update to a new version.

Lots of perspectives to improve bug tracking systems. Maybe it is possible to modify bugzilla or launchpad in this directions. Maybe first, a proof of concept should be implemented. What other important features are missing in current bug trackers?