DoudouLinux dev's blog

To content | To menu | To search

Tag - localization

Entries feed

Wednesday 15 April 2015

Nightly builds for translation packages

One of the issues our translation team has been facing for a while, was the long delay between the moment at which translations of messages are recorded on Transifex and the moment at which they are available in a stable release of DoudouLinux. This issue was highlighted more than one year ago as a potential cause for motivation loss of our translators (now more than 200 people working on 50 languages!). One of the proposed solution was to automatically fetch translation updates from Transifex every night in order to build nightly updates of all the translations, using the standard Debian package system (“apt-get upgrade”). This solution is now active: new translations should be available to every one the day after they are recorded on Transifex :). Instructions can be found on our website.

To achieve this, we set up a dedicated repository for our nightly Debian packages, that we called “DoudouLinux volatile”. It provides 2 types of packages: doudoulinux-l10n for applications, doudoulinux-icons for icons on the desktop of the most advanced activities. Around 00:30 GMT every night, a script is launched on our build server. It will offer newer updates packages two hours later whenever a message has been detected as changed or new on Transifex. Of course this assumes we can get all the modified files from Transifex without being blocked by their anti-DOS attack system (which already happened during our tests). May this happen, we would have one day or more of additional delay before publishing translations updates.

How does it work? The server owns a local copy of several branches from our SVN code repository and is able to query Transifex servers using their Python client application (CLI client). The process of building package updates is the following:

  1. SVN files are re-synchronized with the HEAD version.
  2. If SVN updates cause translation files to be updated (PO/TS files), any previous local change in these files is first reverted.
  3. All the PO/TS files get updated from Transifex[1].
  4. A Python script determines if there are really changes in message translations[2]. Insignificant changes are reverted.
  5. If there were real changes in message translations, doudoulinux-l10n packages are rebuilt with a package name containing nightly and the date of the day.
  6. If translations of icons have been changed, doudoulinux-icons packages are rebuilt too.
  7. New packages are registered into a Debian repository structure using reprepro, with key signing.
  8. The repository is made available through an Apache web server.

The main build script is on our SVN server: server/build/l10n-updates. It relies on several other scripts that have been adapted or rewritten:


[1] 77 resources × 50 languages = 3850 files, at least!!!

[2] Transifex may just change metadata like author or date

Saturday 19 January 2013

DoudouLinux ISO relocalization

One issue we have with the DoudouLinux ISO images is that, since it is designed to start without any user interaction, localization parameters are preset inside the ISO and cannot be easily changed. While this can be acceptable for most language variations (eg. French from Canada and French from France do not differ that much), this can be a real issue for the keyboard layout and the timezone. Indeed, if you try to teach children the keyboard and that it outputs an Azerty layout while you only have Qwerty ones in your country, this is rather annoying… This is however the case between France (Azerty), Canada (Qwerty) and Switzerland (Qwertz) in which French is spoken. Currently the only official solution is to play with combinations of Left Alt+Shift to change between predefined CD keyboard layouts[1].

So, as we can't propose an ISO for each country and each timezone, we have created a ticket quite a long time ago about this issue[2]. The problem to solve is that localization parameters are computed at each CD boot. It is then not possible to make changes persistent without patching the Debian live tools we're using for DoudouLinux. The only solution right now is then to unpack an ISO, change whatever needed, then repack a new ISO. We are indeed lucky enough to not have to unpack the whole CD filesystem because the CD localization parameters need to be accessed during the early boot process and then are contained in the CD boot parameters. Our relocalization process is then simply the following:

  1. Unpack the ISO image
  2. Change the boot configuration file
  3. Repack the ISO image

To achieve this, we will use rsync and genisoimage from the Debian packages of the same name. So first let's “loop” mount the ISO then copy its content into an empty directory:

$ sudo mount -o loop doudoulinux-2012-10-fr.iso /mnt/
$ mkdir isofs
$ rsync -at --del /mnt/ isofs/

File copying just takes few seconds. Now we can directly edit the boot configuration file:

$ sudo nano isofs/isolinux/live.cfg

You will find inside two boot lines of the following form (it's a very long line):

append initrd=/live/initrd.img boot=live config locales=fr_FR.UTF-8 keyboard-layouts=fr,us,de keyboard-variant=oss,, keyboard-options=grp:alt_shift_toggle,lv3:ralt_switch,compose:menu,terminate:ctrl_alt_bksp notimezone nox11autologin quiet splash nomodeset video=uvesafb:mode_option=640x480-16,mtrr=3,scroll=ywrap live-media=removable  persistent persistent-subtext=doudoulinux username=tux hostname=doudoulinux  quiet

The first line is the default boot entry, the second one allows to disable any live persistence that would have been set formerly. You can change or add the following parameters:

  • locales, this is the interface language, eg. fr_FR.UTF-8
  • keyboard-layouts, self-explanatory, it can be a single layout, eg. fr, or a series, eg. fr,us,de
  • keyboard-variant if you need to use a particular variant, eg. the French variant in Canada fr-legacy
  • keyboard-options if you don't need all the stuff we added to DoudouLinux like Left Shift+Alt
  • notimezone can be replaced with a particular timezone, eg. timezone=America/Montreal
  • utc=yes sets the hardware clock to UTC, utc=no sets it to the local time

Note that if your machine is unluckily running Windows :p, you have to set the hardware clock to local time. If it runs Linux, you probably have to set it to UTC. On the other hand, if the hardware clock is the local time, preserving the setting no timezone should automatically get the correct date and time. A list of timezones can be found on Wikipedia. Concerning keyboard layouts, the answer is at the bottom of one of our README files. Finally language codes and country codes can be found on Wikipedia too.

So, for a Canadian/Québec setting, we might use the following parameters:


Now we have to recompute the MD5 checksum of the modified file since the ISO integrity can be checked using it:

$ MD5=$(md5sum isofs/isolinux/live.cfg | grep -o '^[^ ]*')
$ sudo sed -ri 's|^[^ ]+(\s+./isolinux/live.cfg)|'$MD5'\1|' isofs/md5sum.txt

And we now just have to repack the ISO:

$ TITLE=$(file doudoulinux-2012-10-fr.iso | grep -o "'[^']*'" | sed "s/'//g")
$ sudo genisoimage -o doudoulinux-2012-10-fr_CA.iso -b isolinux/isolinux.bin -c isolinux/ -no-emul-boot -boot-load-size 4 -boot-info-table -V "$TITLE" -cache-inodes -r -J -l isofs/

Of course, once the job is done, cleaning useless files is a good idea:

$ sudo rm -fr isofs/
$ sudo umount /mnt/

Here is a screenshot of the resulting environment that you can't currently get out of one of our CD's without modification:

DoudouLinux fr relocalized into fr_CA

That's it!

What we are now dreaming of is a simple set of web pages that would allow to queue such ISO customization online. Our web server would do the job of unpacking/setting/repacking using its own set of CD's and would then send an email to users for them to download the resulting ISO before it is removed to make room for other users :). Any volunteer?

PS: credits to the Linux Mint ISO relocalization tool for the source of the command lines.


[1] DoudouLinux CD's are often configured with Azerty, Qwerty and Qwertz keyboards

[2] Unfortunately our project management software is temporarily down…