Worst Ever Qt Bug (for me!)

I think Qt is one of the best software development frameworks. It has a nice GUI library alongside with many other tools, provided as a neat and clean API. KDE desktop environment is almost entirely based on Qt framework. I’ve been using it for a decade now, and I have successfully delivered dozens of projects – both embedded and desktop – most of them only using Qt. Though starting from version 5.3, a tiny bug has appeared in the core library, that makes life difficult for me: It is impossible to insert invisible characters in Qt’s text engine. That is you can’t type ZWNJ, ZWJ, LRE, RLE and PDF characters. (I have described what they in this post: Control Charachters and Bi-Directional Text. The effect of this bug is so huge that makes it impossible for me to use KDE or any other application that uses Qt. To see how it didn’t work for us, imagine being unable to type, for example, “space” character in any application on your operating system! In this post I am going to show a workaround and explain how to fix this bug. It is already fixed in recent versions of Qt, though it is not in official repositories yet. Fortunately, the workaround is very simple and does not break anything. Finally I’ll provide a patch for 5.9. Hope this helps someone.

The Bugfix That Entailed Another Bug

Everything was going fine in Qt, until December 2013. That day someone reported a bug regarding accent characters. Apparently he couldn’t get combinational character codes to work (for example SHIFT+Ü in German). This bug affected older versions back to 5.2. The fix was simple: Add a filter code right at the text entry level, to filter out all un-printable characters. That includes every single control character that we use to format Persian, Arabic and all other right-to-left texts, also two space chars (Zero Width Non-Joining Space, and Zero Width Joining Space) that we use often in Persian text. In fact, you can’t spell proper Persian without ZWNJ at all. The fix was released with 5.4 and still exists in 5.8. You can still insert those characters elsewhere (like LibreOffice) and copy/paste them into your Qt application. But it is not a pleasant experience at all!

The Bugfix

The bug was reported by five or six people around the world, as expected, mostly from middle-east. QTBUG-42074 was reported in October 2014 and is followd by QTBUG-55608, QTBUG-57302, QTBUG-57003, and also QTBUG-58364. But the fix has not released till January 17th 2017! It will take another couple of years till it makes it way through official repositories of the common distributions (except for Arch Linux . Arch is crazy!) Currently the Qt version in latest Ubuntu release is still 5.6.1. Next release will contain 5.7.1 apparently. Debian is even worse! RedHat is the worst regarding outdated packages! So a workaround is necessary for now.

Unofficial Workarounds

Telegram uses a patch for they own software which fixes part of the problem. They have resolved the problem with ZWNJ. We can use the same approach and add all characters we need. As far as I researched, no Linux distro has released any patches for this problem. So we need a patch on distro’s package (not Qt itself). I already done this and got some partially satisfying results. It works well with widgets but not with, for example Kate, which apparently has migrated to KF5. QtQuick seems to be still problematic. My patch is based on Ubuntu, though it is also applicable for Debian-based distros. Similar steps must be followed to provide patches for other distributions as well.

To fix this issue, we must patch Qt Core library in Ubuntu package repository. While patching, we must also comply to the Debian packaging rules so the final package can be replaced with the system’s one. So let’s first download source code of QtCore:

$ ‌‌apt source libqt5core5a

This command will download Qt Core library with all of the patches added by Ubuntu developers. That will be around 50~is megabytes. In the output of this command, the git repository location is also shown. We can use it later to submit our patch.

$ Reading package lists... Done
Picking 'qtbase-opensource-src' as source package instead of 'libqt5core5a'
NOTICE: 'qtbase-opensource-src' packaging is maintained in the 'Git' version 
control system at:
https://anonscm.debian.org/git/pkg-kde/qt/qtbase.git
Please use:
git clone https://anonscm.debian.org/git/pkg-kde/qt/qtbase.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 47.5 MB of source archives.
Get:1 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main/qtbase-opensource-src 5.5.1+dfsg-16ubuntu7.2 (dsc) [5,096 B]
Get:2 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main/qtbase-opensource-src 5.5.1+dfsg-16ubuntu7.2 (tar) [47.2 MB]
...

So now we have the source code in path. There is no patches applied yet. At this time there are about 50 patches by Ubuntu devs. I happen to know that the infected source file is qwidgettextcontrol.cpp located at src/widgets/widgets/. So I am going to change it and add my codes. It is of grave importance that you do this the right way, or the final package will not be recognised by the dpkg (debian package management backend). It should be properly versioned and it also needs a ChangeLog entry as well.

$ cd qtbase-opensource-src-5.5.1+dfsg/
$ export QUILT_PATCHES=debian/patches
$ quilt push -a
File series fully applied, ends at patch debian/patches/fix-duplicate-qnam-finished.patch
$ quilt new fix-non-printable-filters-for-persian-keyboard
Patch debian/patches/fix-non-printable-filters-for-persian-keyboard is now on top
$ quilt add src/widgets/widgets/qwidgettextcontrol.cpp 
File src/widgets/widgets/qwidgettextcontrol.cpp added to patch debian/patches/fix-non-printable-filters-for-persian-keyboard
$ quilt add src/widgets/widgets/qwidgetlinecontrol.cpp
File src/widgets/widgets/qwidgetlinecontrol.cpp added to patch debian/patches/fix-non-printable-filters-for-persian-keyboard

Running these commands, I mark my files to be tracked for changes. That means when I am done, I can ask quilt system what have I done exactly. It will provide me an standard patch file, containing exact changes I have made. Now I can start modifying aforementioned files. You can see what are those changes at the end of this post.

Now, to generate the actual patch, I run these two commands:

$ quilt refresh
$ quilt pop -a

I’ll also need a ChangeLog entry. ChangeLog has a mundane syntax which if not followed word-by-word can be a pain in the back. Fortunately, Debian provides a tool to add ChangeLog entries. Using dch -i any ChangeLog file can be modified properly with less effort. It will automatically add date and signature, and even will open the editor for you, so you can edit your entry.

Our final change looks like below lines. Please note that the version is exactly one patch-number above the official version.

qtbase-opensource-src (5.5.1+dfsg-16ubuntu7.3) UNRELEASED; urgency=medium

  * Add a temporary workaround for QTBUG-42074

 -- Soroush Rabiei <soroush@ametisco.ir>  Tue, 09 May 2017 20:23:08 +0430
...

Finally I am ready to compile and install the package. These commands will do so:

$ DEB_BUILD_OPTIONS=nocheck debuild -us -uc -b -j10

Note that I have disabled all of the automated tests by setting DEB_BUILD_OPTIONS environment variable. Not doing so, I would have to wait a long time for tests to finish. (It might be a good idea though!) I also set the parallel compile degree to 10. As a rule of thumb, I always use 2 + number of cores for parallel builds. After build is finished, I’ll have your Debian packages ready to be installed. Following commands will do so:

$ cd .. # go up one level
$ sudo debi # install everything

Reults

Now I can write proper Persian in my Qt applications:

Resolved Bug

PS: Patches

Well, the patch works like this: Provide a white-list of all characters that you need, and make exceptions for them while filtering. This way, the fix for 35724 will not be undone, while it’s negative side effect will be resolved. The patch is this one. Hope this will help someone.

Index: 
qtbase-opensource-src-5.5.1+dfsg/src/widgets/widgets/qwidgetlinecontrol.cpp
===================================================================
--- 
qtbase-opensource-src-5.5.1+dfsg.orig/src/widgets/widgets/qwidgetlinecontrol.cpp
+++ qtbase-opensource-src-5.5.1+dfsg/src/widgets/widgets/qwidgetlinecontrol.cpp 
 
                                                                            
@@ -1884,7 +1884,11 @@ void QWidgetLineControl::processKeyEvent                 
 
                                                                            
         && event->modifiers() != Qt::ControlModifier                           
 
                                                                            
         && event->modifiers() != (Qt::ControlModifier | Qt::ShiftModifier)) {
         QString t = event->text();
-        if (!t.isEmpty() && t.at(0).isPrint()) {
+        ushort code = 0;
+        if(!t.isEmpty())
+            code = t.at(0).unicode();
+        if (!t.isEmpty() &&
+            (t.at(0).isPrint() || (0x2000 <= code && code <= 0x200F) || (0x2028 <= code && code <= 0x202F))) {
             insert(t);
 #ifndef QT_NO_COMPLETER
             complete(event->key());
Index: 
qtbase-opensource-src-5.5.1+dfsg/src/widgets/widgets/qwidgettextcontrol.cpp
===================================================================
--- 
qtbase-opensource-src-5.5.1+dfsg.orig/src/widgets/widgets/qwidgettextcontrol.cpp
+++ qtbase-opensource-src-5.5.1+dfsg/src/widgets/widgets/qwidgettextcontrol.cpp
@@ -1342,13 +1342,19 @@ void QWidgetTextControlPrivate::keyPress
 process:
     {
         // QTBUG-35734: ignore Ctrl/Ctrl+Shift; accept only AltGr (Alt+Ctrl) on German keyboards
-        if (e->modifiers() == Qt::ControlModifier
+        /* if (e->modifiers() == Qt::ControlModifier
             || e->modifiers() == (Qt::ShiftModifier | Qt::ControlModifier)) {
             e->ignore();
             return;
         }
+        */
         QString text = e->text();
-        if (!text.isEmpty() && (text.at(0).isPrint() || text.at(0) == 
QLatin1Char('\t'))) {
+        ushort code = 0;
+        if(!text.isEmpty())
+            code = text.at(0).unicode();
+        if (!text.isEmpty()
+                && (text.at(0).isPrint() || text.at(0) == QLatin1Char('\t')
+                    || (0x2000 <= code && code <= 0x200F) || (0x2028 <= code && code <= 0x202F))) {
             if (overwriteMode // no need to call deleteChar() if we have a selection, insertText
                 // does it already

PS2: Future Works

Applying this patch, fixes the problem for all widgets. But there is a similar issue, most probably in QtQuick or KF5. I will find a fix for that too, and I might release both fixes in a PPA.

Comments

comments powered by Disqus