Wednesday, August 5, 2015

Porting Drawpile to QML - part 2


The previous post dealt with rewriting the playback dialog in QML. Now it's time for the real meat: the canvas.



Getting pixels on the screen

The very first was task to get a QML environment running in the main window for the new canvas to inhabit. I knew porting over all the features was going to be a huge task, so rather than throw out the old QGraphicsScene canvas and start from scratch, I decided to go for an evolutionary approach and create the new canvas in parallel to the old one, until it is ready to replace it.

The main work area has a draggable splitter element that separates the canvas from the chat widget. This presented an easy way to move forward: I added a QQuickWindow (in a widget container) as a third widget to the splitter. The old architecture is pretty messy, but it was still MVC enough to support two simultaneous views of the canvas. With the old and new canvases co-existing, I could gradually re-implement features in QML with minimal interim breakage.

Now with the QML environment running, the very first thing to do was to create an item that displays the layer stack pixel content.

This turned out to be quite easy. In the old graphics scene based canvas, the layer stack is drawn to the screen by a custom QGraphiscItem. The item holds a private cache pixmap and when the layer stack content changes, the changed region is updated in the cache. The cache is then used to quickly redraw the screen when needed.

The QQuickItem solutions works almost exactly the same way. Using the QQuickPaintedItem, I could reuse the old graphics item code almost verbatim. This is just a temporary solution though, since it's very inefficient. There are now two buffers: the pixmap cache and the framebuffer object QQuickPaintedItem renders to. Later, I'm going to replace it with a plain customized QQuickItem that renders directly to the texture.

One optimization is lost, though. The graphics scene item knew which part of it was within the viewport and could thus update only the visible parts of the cache. With QQuickItems this isn't possible without cooperation from an enclosing viewport item.

Zooming around

Now I had a layer stack item, but it was completely static and didn’t respond to mouse, touch or pen input. I could handle the input in the layerstack item, but that is not the QML way. In QML, a separate item (like MouseArea) is typically used to capture input, and that is the approach I chose here too. It makes the code a little cleaner by splitting the nasty input handling code from the drawing code and allows easier reuse of the individual items. For example, I can use the same LayerStack item for brush previews without having to add any logic to make the item non-interactive.

Before tackling pen input, I wanted to get zooming, panning and rotation working. QtQuick provides two items that work like QScrollArea from the widget side: ScrollView and Flickable. However, these support neither zooming nor rotation, not to mention the button combos used by Drawpile to switch between drawing, panning, zooming and rotation modes.

Instead, I created CanvasInputArea to handle all the input events relevant to the canvas, including the ones related to transforming the view. The item itself does not apply any transformations to its child items, but merely exposes offset, zoom and rotation properties.

The LayerStack is placed inside the CanvasInputArea and standard Transform objects, whose properties are bound to the input area's properties, are used to perform the actual view transformation. For convenience, I created a custom QQuickTransform (CanvasTransform) class that wraps the needed Translate, Scale and Rotate matrix operations into a single easy to use package.

Implementing gesture handling was very straightforward using QNativeGestureEvents.

Pen & mouse

Pen input presented a problem. QtQuick doesn't have a tablet event! There was an initial implementation, but it was never merged. Even on the widget side, the tablet events are a bit dodgy. On some tablets, on some platforms, they work fine and on others... not so well. A common problem is jittery cursor coordinates. The amount of jitter can range from annoying to completely unusable.

A somewhat ugly workaround I had discovered earlier was to grab the pressure info from the tablet event and then use the mouse event generated for said tablet event for the actual drawing. (For some reason, even if tablet event coordinates are jittery, the generated mouse events are OK.)

Rather than hack tablet event dispatching to QQuickWindow, I decided to just use an event filter to catch the pressure data from tablet events, and just handle mouse events in the CanvasInputArea. This seems to work well enough, but I hope Qt's tablet support will be improved in the future so I won't need to rely on ugly workarounds like these anymore.

Finally, what to do with the pen coordinates? The old view class was fairly tightly coupled to the tool controllers and would call their input handlers directly. The new input item is much simpler: it merely emits penDown, penMove and penUp signals, which are connected to the tool controller’s corresponding slots.

Models, views, controllers, oh my!

Drawpile's canvas displays more than just the layer stack:
  • Tool previews (line, rectangle and ellipse tools)
  • Annotations (text layers that float over the layers)
  • User cursors (show user cursor positions with their names attached)
  • Laser trails (the laser pointer tool)
  • Selection (may contain a pasted image)
These were all implemented as graphics items that were poked and prodded by the tool controllers. Exposing QtQuick items to C++ in the same way is possible, but not as easy, but this is not really a good architecture in the first place. Instead, the model classes should be UI agnostic, so they can be visualized as easily by QML or a graphics scene.

By using QAbstractItemModel based classes, exposing all this data to the QML view is quite simple.

Tool previews

The shape drawing tools (line, rectangle and ellipse) use a special overlay item to visualize the end result until the mouse button/pen is released and the actual drawing commands sent.

The problem is, QtQuick only comes with a single geometric primitive: a rectangle. This meant I would have to write the other shape items myself, and drawing lines in OpenGL is surprisingly hard.

This got me thinking. Why are the previews drawn with overlay items in the first place? Turns out, this design choice dates back to version 0.4.0 (when the Line and Rectangle tools were first implemented) and has never been reevaluated since.

A lot has changed since then, most importantly the addition of layer support in version 0.7.0 and the “realistic” stroke preview mode in version 0.8.5. This meant all the tools needed to replace the overlay previews with realistic canvas based previews already existed in the code.

So, this one was taken care of by simply throwing out the old preview items and drawing the previews onto a temporary layer using real brushes. This not only looks better, but actually simplified the code too!

Annotations

I started by separating the annotation model from LayerStack into its own QAbstractItemModel based AnnotationModel class. Internally, it is just a list of annotation objects, but by making it into a standard model class, I can use the QML item Repeater to manage the view delegates. For compatibility with the old canvas during the transition, I also implemented the old style change notification signals in the new model class.

Rather than writing a QQuickWidget for displaying an annotation, I wrote the view delegate in pure QML. This is much less code than writing the equivalent in C++ and also makes it easier to change its appearance if needed. The interaction is still handled by the tool controller in C++, like before.

One problem I ran into was how to draw the preview for the annotation object, since there are no longer any tool preview items. The solution I decided on was to create a temporary annotation object which gets replaced by the real one when the user finishes drawing it. This way, the annotation preview looks just like the real thing and by using an ID outside the protocol range, there is no chance of mixing it with the real annotations.

User cursors & lasers

User cursors are the little popover bubbles that follow remote user's pointers and lets you know who is drawing. Reimplementing these in QML was fairly straightforward. As with annotations, I created a model class to hold the list of cursors and a QML item delegate to visualize them. The item is drawn using a QML canvas item (which is like the HTML5 canvas element.)

Closely related to user cursors are laser trails. These are lines drawn over the canvas that automatically fade out after a while that can be used for a variety of things, such as to point out things like with laser pointer, or as quick red-lines to give drawing tips.

They are implemented almost exactly the same way as user cursors. The main difference is the rendering: that lack of native geometric primitives again! Luckily OpenGL’s standard line drawing (GL_LINESTRIP) is good enough for this use case, so implementing a custom PolyLine item was fairly straightforward, especially since Qt’s examples had demo code for exactly this sort of line drawing.

For the sake of simplicity and efficiency, I changed the way laser fadeout works. Previously, each line segment was a separate object and faded out on its own, creating a nice progressive fade effect. Now the laser is rendered as a single primitive, so the entire trail now fades out at once. (The old behaviour could be reimplemented with per vertex colors, though.)

The laser flicker effect I implemented with a simple fragment shader.

Selection

The selection model is a bit simpler than the others, since there can be only one at the time. The selection object is therefore implemented as a simple QObject that belongs to the canvas model.

Setting a selection deletes the old one (if any) and triggers a selectionChanged signal, which the view layer detects and updates itself. Changing the selection itself will trigger another set of signals, which the cause the view item to refresh.

Bugs!

Qt Quick, being still quite new, naturally has bugs. What I did not expect was the number of them, since I had got used to the general high quality of Qt widgets. Although since the widget side has been around for 20 years and QtQuick2 only for about four, perhaps I’m being unfair.

First, A QQuickWindow inside a widget container (QWidget::createWindowContainer) has a strange bug (that apparently affects only some platforms): If focus is passed to a widget, the quick window cannot regain it! (QTBUG-39362)

This is seriously bad. It means keyboard commands (like hold Alt to pan) won't work, and moving the chat box to QML is a non-starter.

QQuickWidget does not have this bug, so I tried it instead, even though it adds an additional level of indirection and disables threaded rendering. But, it fixed the focus problem.

However, QQuickWidget has a set of bugs of its own. After losing focus, its cursor changes stop working. (Possibly related to QTBUG-45557). Custom cursors inside the canvas are pretty important, so this is just as much of a deal breaker as QQuickWindow's focus bug.

After this, I decided to go back to QQuickWindow, since at least the cursors work and the focus bug manifests only on some platforms (mine, unfortunately.) Hopefully, it will be fixed before Drawpile 2.0 is ready for release.

On OSX, my custom shaders didn’t work because Qt defaults to an ancient OpenGL 2 profile, and selecting a newer one (like 3.2 core) doesn’t work because of QTBUG-42107.

On Linux, opening the host dialog causes the canvas to disappear. I isolated to problem to a single line that is entirely unrelated, so I’m guessing memory corruption is the culprit. Running Drawpile in valgrind doesn’t help much, since it immediately segfaults somewhere in the scenegraph code, even with all my custom items disabled. I haven’t figured out why yet, or even if the bug is in Qt or my own code.

Some of these bugs could be bypassed by not using widgets and going pure QML, but that is just not feasible yet. (QtQuick still lacks replacements for QDockWidget and QTreeView, for example.) So, I’m putting the QML port on hold until Qt Quick has matured a bit more. The code is now in the master branch and the QML canvas can be activated by changing a single #define, so I’ll keep trying every time a new Qt version is released.

If it seems like these bugs will go unfixed for a long time, one option would be to do the pure QML version (a mobile oriented Drawpile-lite) in parallel with the QGraphicsScene based normal version. This would involve further refactoring the Drawpile client into three pieces:
  • Shared UI agnostic parts (models, paint engine, networking, etc.)
  • QML based UI
  • Widget based UI
Ideally, both UIs would be thin wrappers around the shared code. The long-term goal would be a “convergent” client that can switch between desktop and tablet (pen&touch) style interface on the fly.

For now, the QML based canvas implementation is there, but disabled by default. I’ve modified the old graphics scene canvas to use the same model classes, so development can continue with minimal duplication of effort.

Summary

I refactored Drawpile’s main canvas code to better separate the model from the controllers and the controllers from the view. Then, I reimplemented the primary canvas features with custom QQuickItems and QML. However, I ran into enough bugs in Qt Quick itself that I had to put the migration project on hold.

In the meantime, Drawpile’s architecture did benefit from the refactoring, so this wasn’t a total loss. There are other things on the roadmap (namely, the new protocol) I’m going to focus on next while waiting for new Qt versions. The worst case scenario is that Drawpile 2.0 will be released with the old QGraphicsScene based canvas.

Many of the bugs encountered are related to integrating QML with traditional widgets, so building a separate pure QML GUI may be a feasible approach.

This blog series will continue in part 3, once I resume work on the QML migration.

4 comments:

  1. I don’t know what I would have done if I had not encountered such a step like this Your good knowledge and kindness in playing with all the pieces was very useful. MOM Canada

    ReplyDelete
  2. I would like to thank you for the efforts you have made while writing this post. I am hoping for the best work of the same from you in future secure self storage

    ReplyDelete
  3. I just now couldn’t depart your website before letting you know that any of us honestly relished the highest high quality data an individual indicate your visitors? Let me another frequently to look at brand-new information. American Samoa European License Plate embossed with custom number or text

    ReplyDelete
  4. Good article, but it would be better if in future you can share more about this subject. Keep posting. hair growth supplements for men

    ReplyDelete