Hardening consulting

Let's play with Wayland

A post written after an error I have made while coding UWAC (Using Wayland As Client), my library to use wayland on the client side with the spirit of the Xlib. The symptoms were that an application using UWAC was working with the X11 compositor and was freezing with the RDP compositor. After some testing, it appeared that the pixman renderer was the guilty component (using the X11 compositor and forcing the pixman renderer, the behaviour was the same).

Pools, buffers, frames et friends...

Before analysing what was going on, let's explain how it works under wayland. If you're a complete noob on the subject this post (only un french for now sorry, but google translate is your friend) could help.

Pools and buffers

In my case the rendering was done using wl_shm, ie the communication between the compositor and the wayland client is done through a shared memory.

So the client have to:

  • create a shared memory segment, for example by mmaping an "anonymous" file;
  • create a wl_shm_pool by calling wl_shm_create_pool(), giving the file descriptor of the mmaped file and the size to use in this file;
  • once the pool exists, you can allocate wl_buffers in this pool by calling wl_shm_pool_create_buffer.

Surfaces, frames

So now we have buffers that we can use to draw the content to show, the next steps are to attach a buffer to a surface (wl_surface_attach), notify damaged areas (wl_surface_damage), and finally the compositor is notified that it can use the buffer to create a frame (wl_surface_commit).

When you've attached a buffer to a surface and that you have asked the compositor for a frame rendering, you're not supposed to touch to the buffer anymore. When the compositor will have no more usage of the buffer (typically when the frame has been created), it will send a release event which allows the client to reuse it.

By calling wl_surface_frame, you can be notified when the frame is rendered, and we will receive a done event.


The bug

In UWAC, I had implemented a triple buffering:

  • a currrent buffer where to draw;
  • a buffer that would have been sent to the compositor;
  • and finally a buffer that would be a frame waiting to be sent to the compositor (waiting for the done event of the frame).

So when initializing a window, I was creating 3 wl_buffer, and then I was swapping them.

I was using the event of frame rendered (wl_surface.done) to notify the user of the library that it could draw a new frame. Anyway my buffers management was controlled by the wl_buffer.release event: I was waiting for the compositor to notify, to swap the buffer and display the next computed frame.

My bug was there, in some situations the compositor doesn't send the wl_buffer.release event instantly. With the pixman renderer, the buffer is released when a new one is attached to the surface.

Conclusion

I have changed my implementation, now the wl_buffers are dynamcally allocated and the frames are controlled by the reception of the frame.done event. If the compositor is not fast enough to release the buffer, we allocate a new one. I was expecting a lot more buffers to be allocated and in fact we only have a total of 3.

Thanks to Derek Foreman that found the bug and suggested the solution.